Model performance monitoring method, and first device, second device and storage medium

Abstract

Provided in the present disclosure are a model performance monitoring method, and a first device, a second device and a storage medium. The method comprises: receiving second data sent by a second device, wherein the second data is data related to first data, and the first data is reference data used for monitoring the performance of a first AI model; inputting the second data into the first AI model, so as to obtain third data that is output by the first AI model; and on the basis of the difference between the third data and the first data, determining a performance monitoring result for the first AI model. In the present disclosure, the performance of an AI model can be monitored, such that the availability and reliability of the AI model can be improved in various communication scenarios, thereby facilitating the development of integration of AI and communications.

Description

Model performance monitoring method, first device, second device, and storage medium TECHNICAL FIELD

[0001] The present disclosure relates to the field of artificial intelligence, and in particular, to a model performance monitoring method, a first device, a second device, and a storage medium. BACKGROUND

[0002] Currently, an artificial intelligence (AI) model can be applied to multiple scenarios of a mobile communication system, thereby meeting service requirements such as reliable communication in each scenario and realizing future wireless communication visions such as smart cities and smart transportation.

[0003] SUMMARY

[0004] To improve the reliability of an AI model, embodiments of the present disclosure provide a model performance monitoring method, a first device, a second device, and a storage medium.

[0005] According to a first aspect of embodiments of the present disclosure, a model performance monitoring method is provided, the method being performed by a first device, and the method comprising:

[0006] receiving second data sent by a second device; wherein the second data is data related to first data; wherein the first data is reference data used to monitor the performance of a first artificial intelligence (AI) model;

[0007] inputting the second data into the first AI model to obtain third data output by the first AI model;

[0008] determining a performance monitoring result of the first AI model based on the difference between the third data and the first data.

[0009] According to a second aspect of embodiments of the present disclosure, a model performance monitoring method is provided, the method being performed by a second device, and the method comprising:

[0010] determining second data; wherein the second data is data related to first data; wherein the first data is reference data used to monitor the performance of a first artificial intelligence (AI) model;

[0011] sending the second data to a first device.

[0012] According to a third aspect of embodiments of the present disclosure, a first device is provided, comprising:

[0013] The transceiving module is configured to receive second data sent by a second device; wherein the second data is data related to the first data; wherein the first data is reference data used for monitoring performance of a first artificial intelligence (AI) model;

[0014] The processing module is configured to input the second data into the first AI model, and obtain third data output by the first AI model;

[0015] The processing module is further configured to determine a performance monitoring result of the first AI model based on a difference between the third data and the first data.

[0016] According to a fourth aspect of embodiments of the present disclosure, a second device is provided, comprising:

[0017] The processing module is configured to determine second data; wherein the second data is data related to the first data; wherein the first data is reference data used for monitoring performance of a first artificial intelligence (AI) model;

[0018] The transceiving module is configured to send the second data to a first device.

[0019] According to a fifth aspect of embodiments of the present disclosure, a first device is provided, comprising:

[0020] One or more processors;

[0021] The processor is configured to perform the model performance monitoring method in any of the first aspect.

[0022] According to a sixth aspect of embodiments of the present disclosure, a second device is provided, comprising:

[0023] One or more processors;

[0024] The processor is configured to perform the model performance monitoring method in any of the second aspect.

[0025] According to a seventh aspect of embodiments of the present disclosure, a communication system is provided, comprising:

[0026] The first device is configured to implement the model performance monitoring method in any of the first aspect;

[0027] The second device is configured to implement the model performance monitoring method in any of the second aspect.

[0028] According to an eighth aspect of the embodiments of the present disclosure, a storage medium is provided, and the storage medium stores instructions, which, when executed on a communication device, cause the communication device to perform the model performance monitoring method according to any one of the first aspect or the second aspect.

[0029] According to a ninth aspect of the embodiments of the present disclosure, a computer program product is provided, and the computer program product comprises a computer program, which, when executed by a processor, is configured to implement the model performance monitoring method according to any one of the first aspect or the second aspect.

[0030] In the embodiments of the present disclosure, the performance of the AI model can be monitored, and the usability and reliability of the AI model in various communication scenarios can be improved, which helps to promote the integration of AI and communication.

[0031] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and are not limiting to the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0032] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present disclosure and serve to explain the principles of the present disclosure together with the specification.

[0033] FIG. 1A is one exemplary schematic diagram of an architecture of a communication system according to an embodiment of the present disclosure.

[0034] FIG. 1B is one exemplary schematic diagram of an AI interface scheme according to an embodiment of the present disclosure.

[0035] FIG. 1C is another exemplary schematic diagram of an AI interface scheme according to an embodiment of the present disclosure.

[0036] FIG. 2A is one exemplary interaction schematic diagram of a model performance monitoring method according to an embodiment of the present disclosure.

[0037] FIG. 2B is one exemplary interaction schematic diagram of a model performance monitoring method according to an embodiment of the present disclosure.

[0038] FIG. 3A is one exemplary flow schematic diagram of a model performance monitoring method according to an embodiment of the present disclosure.

[0039] FIG. 3B is one exemplary flow schematic diagram of a model performance monitoring method according to an embodiment of the present disclosure.

[0040] FIG. 4 is one exemplary process schematic diagram of a model performance monitoring method according to an embodiment of the present disclosure.

[0041] FIG. 5A is one exemplary block diagram of a first device according to an embodiment of the present disclosure.

[0042] FIG. 5B is an example block diagram of a second device, according to embodiments of the present disclosure.

[0043] FIG. 6A is an example block diagram of a communication device, according to embodiments of the present disclosure.

[0044] FIG. 6B is an example block diagram of a chip, according to embodiments of the present disclosure. DETAILED DESCRIPTION

[0045] The exemplary embodiments will be described in detail herein with reference to the attached drawings. The following description is only exemplary and is not intended to limit the scope, applicability or configuration of the present disclosure. Rather, the description is provided as an example with regard to various aspects of the present disclosure.

[0046] Embodiments of the present disclosure provide a model performance monitoring method, a first device, a second device, and a storage medium.

[0047] In a first aspect, embodiments of the present disclosure provide a model performance monitoring method. The method is performed by a first device, and includes: receiving second data sent by a second device; wherein the second data is data related to first data; wherein the first data is reference data for monitoring performance of a first artificial intelligence (AI) model; inputting the second data into the first AI model to obtain third data output by the first AI model; and determining a performance monitoring result of the first AI model based on a difference between the third data and the first data.

[0048] In the above embodiments, the first device can determine the performance monitoring result of the first AI model based on the difference between the third data output by the first AI model and the first data. This achieves the purpose of monitoring the performance of the AI model, and effectively improves the usability and reliability of the AI model in various communication scenarios, thereby helping to promote the integrated development of AI and communication.

[0049] In some embodiments of the first aspect, in some embodiments, the first AI model is used for demodulation, the first data is original sending data without modulation, and the second data is data obtained after the first data is modulated; or the first AI model is used for device perception, the first data includes actual perception information of a third device, and the second data is first channel measurement data used for device perception; wherein the third device is a device that needs to be perceived; or the first AI model is used for device positioning, the first data includes actual position information of a fourth device, and the second data is second channel measurement data used for device positioning; wherein the fourth device is a device that needs to be positioned.

[0050] In the above embodiments, the specific content of the first data and the second data is limited respectively when the first AI model is used to implement different functions, which is beneficial to improve the reliability of AI model performance monitoring.

[0051] In some embodiments of the first aspect, in some embodiments, the method further includes: determining the first data; and sending the first data to the second device.

[0052] In the above embodiments, the first data can be determined by the first device and then sent to the second device, so that the first device and the second device have consistent understanding of the first data, and the reliability of the performance monitoring result of the AI model is improved.

[0053] In some embodiments of the first aspect, in some embodiments, the first AI model is used for demodulation, and different modulation modes correspond to different first data; and / or different modulation orders correspond to different first data.

[0054] In the above embodiments, when the first AI model is used for demodulation, different modulation modes and / or different modulation orders can correspond to different first data, which improves the accuracy of the selected first data.

[0055] In some embodiments of the first aspect, in some embodiments, the method further includes: receiving the first data sent by the second device.

[0056] In the above embodiments, the first data can be determined by the second device and then sent to the first device, so that the first device and the second device have consistent understanding of the first data, and the reliability of the performance monitoring result of the AI model is improved.

[0057] In some embodiments of the first aspect, in some embodiments, the determining the performance monitoring result of the first AI model based on the difference between the third data and the first data comprises: determining a first parameter value, wherein the first parameter value is used to measure the difference between the third data and the first data; and when the first parameter value reaches a first value, determining that the performance monitoring result is that the performance of the first AI model does not meet the accuracy requirement; or when the first parameter value does not reach the first value, determining that the performance monitoring result is that the performance of the first AI model meets the accuracy requirement.

[0058] In the above embodiments, the first device can determine the performance monitoring result of the first AI model in the above manner, which is simple and convenient to use.

[0059] In some embodiments of the first aspect, in some embodiments, the method further comprises any one of the following: when the performance monitoring result is that the performance of the first AI model does not meet the accuracy requirement, obtaining fourth data based on the second data in a non-AI manner; updating the first AI model; and switching the first AI model to a second AI model.

[0060] In the above embodiments, when the performance monitoring result of the first AI model is that the performance of the first AI model does not meet the accuracy requirement, the first device can process in the above manner, which improves the usability and reliability of the AI model and helps promote the integrated development of AI and communication.

[0061] In the second aspect, the embodiments of the present disclosure provide a model performance monitoring method, which is performed by a second device and comprises: determining second data; wherein the second data is data related to first data; wherein the first data is reference data used to monitor the performance of a first artificial intelligence (AI) model; and sending the second data to a first device.

[0062] In the above embodiments, the second device can send the second data to the first device, and the first device can monitor the performance of the first AI model, which achieves the purpose of monitoring the performance of the AI model, effectively improves the usability and reliability of the AI model in various communication scenarios, and helps promote the integrated development of AI and communication.

[0063] In some embodiments of the second aspect, in some embodiments, the first AI model is used for demodulation, the first data is original transmission data without modulation, and the second data is data obtained after modulation of the first data; or the first AI model is used for device perception, the first data includes actual perception information of a third device, and the second data is first channel measurement data used for device perception; wherein the third device is a device that needs to be perceived; or the first AI model is used for device positioning, the first data includes actual position information of a fourth device, and the second data is second channel measurement data used for device positioning; wherein the fourth device is a device that needs to be positioned.

[0064] In some embodiments of the second aspect, in some embodiments, the method further includes: receiving the first data sent by the first device.

[0065] In some embodiments of the second aspect, in some embodiments, the method further includes: determining the first data; and sending the first data to the first device.

[0066] In some embodiments of the second aspect, in some embodiments, the first AI model is used for demodulation, and different modulation modes correspond to different first data; and / or different modulation orders correspond to different first data.

[0067] In a third aspect, the embodiments of the present disclosure provide a first device, including: a transceiver module configured to receive second data sent by a second device; wherein the second data is data related to first data; wherein the first data is reference data used for monitoring performance of a first artificial intelligence (AI) model; and a processing module configured to input the second data into the first AI model to obtain third data output by the first AI model; and the processing module is further configured to determine a performance monitoring result of the first AI model based on a difference between the third data and the first data.

[0068] In a fourth aspect, the embodiments of the present disclosure provide a second device, including: a processing module configured to determine second data; wherein the second data is data related to first data; wherein the first data is reference data used for monitoring performance of a first artificial intelligence (AI) model; and a transceiver module configured to send the second data to a first device.

[0069] In a fifth aspect, the embodiments of the present disclosure provide a first device, including: one or more processors; wherein the processor is configured to execute the model performance monitoring method of any one of the first aspect.

[0070] In a sixth aspect, an embodiment of the present disclosure provides a second device, comprising: one or more processors; wherein the processor is configured to execute the model performance monitoring method in any of the second aspect.

[0071] In a seventh aspect, an embodiment of the present disclosure provides a communication system, comprising: a first device configured to implement the model performance monitoring method in any of the first aspect; and a second device configured to implement the model performance monitoring method in any of the second aspect.

[0072] In an eighth aspect, an embodiment of the present disclosure provides a storage medium, which stores instructions, when the instructions are executed on a communication device, causing the communication device to execute the model performance monitoring method in any of the first aspect or the second aspect.

[0073] In a ninth aspect, an embodiment of the present disclosure provides a computer program product, comprising a computer program, which is executed by a processor to implement the model performance monitoring method in any of the first aspect or the second aspect.

[0074] In a tenth aspect, an embodiment of the present disclosure provides a program product, which is executed by a communication device to cause the communication device to execute the method described in the optional implementation of the first aspect or the second aspect.

[0075] In an eleventh aspect, an embodiment of the present disclosure provides a chip or chip system. The chip or chip system comprises a processing circuit configured to execute the method described in the optional implementation of the first aspect or the second aspect.

[0076] It can be understood that the first device, the second device, the communication system, the storage medium, the program product, the computer program, the chip or the chip system are all used to execute the method provided by the embodiments of the present disclosure. Therefore, the beneficial effects achieved thereby can refer to the beneficial effects in the corresponding method, which will not be described here.

[0077] The embodiments of the present disclosure provide a model performance monitoring method, a first device, a second device and a storage medium. In some embodiments, the terms of model performance monitoring method and performance monitoring method, processing method, etc. can be replaced with each other, the terms of model performance monitoring device and performance monitoring device, processing device, etc. can be replaced with each other, and the terms of model performance monitoring system and communication system, etc. can be replaced with each other.

[0078] The embodiments of the present disclosure are not exhaustive, but only illustrate some embodiments, and are not specific limitations on the protection scope of the present disclosure. In the case of no contradiction, each step in an embodiment can be implemented as an independent embodiment, and the steps can be combined arbitrarily, for example, the scheme after removing part of the steps in an embodiment can also be implemented as an independent embodiment, and the order of the steps in an embodiment can be exchanged arbitrarily, in addition, the optional implementation manners in an embodiment can be combined arbitrarily; in addition, the embodiments can be combined arbitrarily, for example, part or all steps of different embodiments can be combined arbitrarily, an embodiment can be combined with optional implementation manners of other embodiments arbitrarily.

[0079] In each embodiment of the present disclosure, the terms and / or descriptions between the embodiments are consistent if there is no special description and logical conflict, and can be referred to each other, and the technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationship.

[0080] The terms used in the embodiments of the present disclosure are only for the purpose of describing the specific embodiments, and not as a limitation on the present disclosure.

[0081] In the embodiments of the present disclosure, unless otherwise specified, the elements expressed in singular form, such as "one", "a", "the", "above", "said", "preceding", "this" and the like, can represent "one and only one", and can also represent "one or more", "at least one" and the like. For example, in the case of using articles such as "a", "an", "the" and the like in English, the noun after the article can be understood as singular expression, and can also be understood as plural expression.

[0082] In the embodiments of the present disclosure, "a plurality of" means two or more.

[0083] In some embodiments, the terms "at least one of", "one or more", "a plurality of", "multiple" and the like can be replaced with each other.

[0084] In some embodiments, the description of "at least one of A, B", "A and / or B", "in a case A, in another case B", "in response to a case A, in response to a case B", and the like, can include the following technical solutions according to the case: in some embodiments, A (A is executed independently of B); in some embodiments, B (B is executed independently of A); in some embodiments, A and B are selectively executed (A and B are selected from A and B); in some embodiments, A and B (A and B are executed). When there are more branches of A, B, C, and the like, the above is similar.

[0085] In some embodiments, the description of "A or B" and the like can include the following technical solutions according to the case: in some embodiments, A (A is executed independently of B); in some embodiments, B (B is executed independently of A); in some embodiments, A and B are selectively executed (A and B are selected from A and B). When there are more branches of A, B, C, and the like, the above is similar.

[0086] In some embodiments, the prefix words "first", "second", and the like in the embodiments of the present disclosure are only used to distinguish different description objects, and do not constitute a limitation on the position, order, priority, quantity, or content of the description objects. The description of the description objects should refer to the description in the context of the claims or embodiments, and should not constitute an unnecessary limitation because of the use of the prefix words. For example, the description objects are "fields", and the ordinal words before "fields" in "first field" and "second field" do not limit the position or order between "fields". "First" and "second" do not limit whether the "fields" they modify are in the same message, nor do they limit the order of "first field" and "second field". For another example, the description objects are "levels", and the ordinal words before "levels" in "first level" and "second level" do not limit the priority between "levels". For another example, the quantity of the description objects is not limited by the ordinal words, and can be one or more. For example, "first device", where the quantity of "devices" can be one or more. In addition, the objects modified by different prefix words can be the same or different, for example, the description objects are "devices", and "first device" and "second device" can be the same device or different devices, and their types can be the same or different. For another example, the description objects are "information", and "first information" and "second information" can be the same information or different information, and their contents can be the same or different.

[0087] In some embodiments, "including A", "containing A", "for indicating A", "carrying A", can be interpreted as directly carrying A, or indirectly indicating A.

[0088] In some embodiments, an apparatus or the like can be interpreted as an entity, and can also be interpreted as virtual, and the name thereof is not limited to the name described in the embodiments. The terms "apparatus", "equipment", "device", "circuit", "network element", "node", "function", "unit", "section", "system", "network", "chip", "chip system", "entity", "subject" and the like can be replaced with each other.

[0089] In some embodiments, "device" can be interpreted as "network", or can be interpreted as "terminal".

[0090] In some embodiments, "network" can be interpreted as an apparatus (for example, an access network device, a core network device, and the like) included in the network.

[0091] In some embodiments, the terms "access network device (AN device)", "radio access network device (RAN device)", "base station (BS)", "radio base station", "fixed station", "node", "access point", "transmission point (TP)", "reception point (RP)", "transmission / reception point (TRP)", "panel", "antenna panel", "antenna array", "cell", "macro cell", "small cell", "femto cell", "pico cell", "sector", "cell group", "serving cell", "carrier", "component carrier", "bandwidth part (BWP)" and the like can be replaced with each other.

[0092] In some embodiments, the terms "terminal," "terminal device," "user equipment (UE)," "user terminal," "mobile station (MS)," "mobile terminal (MT)," "subscriber station," "mobile unit," "subscriber unit," "wireless unit," "remote unit," "mobile device," "wireless device," "wireless communication device," "remote device," "mobile subscriber station," "access terminal," "mobile terminal," "wireless terminal," "remote terminal," "handset," "user agent," "mobile client," "client," and so on can be replaced with each other.

[0093] In some embodiments, the access network device, the core network device, or the network device can be replaced with a terminal. For example, the embodiments of the present disclosure can also be applied to a structure in which communication between the access network device, the core network device, or the network device and the terminal is replaced with communication between a plurality of terminals (e.g., device-to-device (D2D), vehicle-to-everything (V2X), etc.). In this case, the terminal can also be configured to have all or part of the functions of the access network device. In addition, the terms "uplink," "downlink," and the like can also be replaced with terms corresponding to the inter-terminal communication (e.g., "side"). For example, the uplink channel, the downlink channel, and the like can be replaced with the side channel, and the uplink, the downlink, and the like can be replaced with the sidelink.

[0094] In some embodiments, the terminal can be replaced with the access network device, the core network device, or the network device. In this case, the access network device, the core network device, or the network device can also be configured to have all or part of the functions of the terminal.

[0095] In some embodiments, the data, information, etc. can be obtained in compliance with the laws and regulations of the country where the location is situated.

[0096] In some embodiments, the data, information, etc. can be obtained after obtaining the consent of the user.

[0097] In addition, each element, each row, or each column in the table of the embodiments of the present disclosure can be implemented as an independent embodiment, and any combination of any element, any row, or any column can also be implemented as an independent embodiment.

[0098] FIG. 1A is a schematic diagram of an architecture of a communication system according to an embodiment of the present disclosure.

[0099] As shown in FIG. 1A, the communication system 100 includes a first device 101 and a second device 102.

[0100] In some embodiments, the first device 101 can be a device for receiving second data.

[0101] In some embodiments, the second data can be data associated with the first data. The first data can be reference data for monitoring the performance of a first artificial intelligence (AI) model.

[0102] In some embodiments, the first AI model can be deployed on the first device 101, and the first AI model can be used for demodulation, device perception, device positioning, etc., which are not limited by the present disclosure.

[0103] In some embodiments, the first AI model is used for demodulation, and the first data can be original transmission data that has not been modulated. The second data can be data obtained after the first data is modulated.

[0104] In some embodiments, the first AI model is used for device perception, and the first data can include actual perception information of a third device, which is a device to be perceived, also referred to as a “perception target”. For example, the first data can include the distance of the “perception target” (the distance of the “perception target” relative to a specified device), the speed of the “perception target” (the moving speed of the “perception target”), the angle of the “perception target” (the horizontal angle and / or the zenith angle of the “perception target” relative to the specified device), etc. The second data can be first channel measurement data for device perception, and the second data includes but is not limited to a Channel State Information (CSI) matrix H.

[0105] By way of example, the horizontal angle refers to the angle obtained after the projection of the line connecting the reference point, such as the center of the earth, or other specified location point, to the “perception target” and the specified device on the horizontal plane. The zenith angle refers to the angle of the line connecting the “perception target” and the specified device relative to the normal of the ground.

[0106] In some embodiments, the first AI model can also be used for positioning, the first data can include actual position information of the fourth device, for example, the first data includes actual position coordinates of the fourth device, and the second data can be second channel measurement data used for device positioning, for example, the second data can be a CSI matrix and the like.

[0107] In some embodiments, the first AI model can also be used to perform other functions, which are not limited in the present disclosure.

[0108] In some embodiments, the specific content of the first data and the second data is not limited.

[0109] In some embodiments, the first device 101 can be a terminal or a network device, for example, an access network device or a core network device.

[0110] In some embodiments, the terminal includes at least one of a mobile phone, a wearable device, an Internet of Things device, a communication-enabled automobile, a smart automobile, a tablet computer (Pad), a computer with wireless transceiver function, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal device in industrial control, a wireless terminal device in self-driving, a wireless terminal device in remote medical surgery, a wireless terminal device in smart grid, a wireless terminal device in transportation safety, a wireless terminal device in smart city, and a wireless terminal device in smart home, but is not limited thereto.

[0111] In some embodiments, the access network device is, for example, a node or device that accesses a terminal to a wireless network, and the access network device can include at least one of an evolved NodeB (eNB) in a 5G communication system, a next generation eNB (ng-eNB), a next generation NodeB (gNB), a node B (NB), a home node B (HNB), a home evolved node B (HeNB), a wireless backhaul device, a radio network controller (RNC), a base station controller (BSC), a base transceiver station (BTS), a base band unit (BBU), a mobile switching center, a base station in a 6G communication system, an Open RAN, a Cloud RAN, a base station in other communication systems, an access node in a Wi-Fi system, but is not limited thereto.

[0112] In some embodiments, the first device 101 can be an unmanned device, an Internet of Things device, an environmental Internet of Things device, etc. The unmanned device includes, but is not limited to, an unmanned vehicle, a drone, etc. The Internet of Things device includes, but is not limited to, a sensor using radio frequency identification, a scanner, a smart home device, a robot, etc. The environmental Internet of Things device includes, but is not limited to, an Internet of Things device that can obtain energy from the outside world and charge.

[0113] In some embodiments, the technical solutions of the present disclosure can be applied to an Open RAN architecture, at this time, the interfaces between or within the access network devices involved in the embodiments of the present disclosure can become internal interfaces of the Open RAN, and the processes and information interactions between these internal interfaces can be realized through software or programs.

[0114] In some embodiments, the access network device can be composed of a central unit (CU) and a distributed unit (DU), wherein the CU can also be referred to as a control unit. The CU-DU structure can split the protocol layers of the access network device, and the functions of part of the protocol layers are controlled by the CU, and the functions of the remaining part or all of the protocol layers are distributed in the DU and controlled by the CU, but are not limited thereto.

[0115] In some embodiments, the core network device can be one device including the first network element, the second network element, etc., or can be multiple devices or device groups including all or part of the first network element, the second network element, etc., respectively. The network element can be virtual or physical. The core network includes at least one of an evolved packet core (EPC), a 5G core network (5GCN), and a next generation core (NGC), for example.

[0116] In some embodiments, the core network device is a location management function (LMF), for example.

[0117] In some embodiments, the core network device is used for device perception, and the name is not limited to a device perception network element or function.

[0118] In some embodiments, the second device 102 can be a device for sending second data.

[0119] In some embodiments, the second device 102 can be a terminal, or an unmanned device, an Internet of Things device, an environmental Internet of Things device, etc.

[0120] It can be understood that the communication system described in the embodiments of the present disclosure is for more clearly illustrating the technical solutions of the embodiments of the present disclosure, and does not constitute a limitation on the technical solutions proposed in the embodiments of the present disclosure. It can be known by those skilled in the art that, with the evolution of system architecture and the appearance of new business scenarios, the technical solutions proposed in the embodiments of the present disclosure are also applicable to similar technical problems.

[0121] The following embodiments of the present disclosure can be applied to the communication system 100 shown in FIG. 1A or part of the subject, but are not limited thereto. The subjects shown in FIG. 1A are examples, and the communication system can include all or part of the subjects in FIG. 1A, or include other subjects other than those in FIG. 1A. The number and form of each subject is arbitrary, each subject can be physical or virtual, the connection relationship between each subject is an example, each subject can not be connected or can be connected, and the connection can be in any way, can be direct connection or indirect connection, can be wired connection or wireless connection.

[0122] Embodiments of the present disclosure can be applied to Long Term Evolution (LTE), LTE-Advanced (LTE-A), LTE-Beyond (LTE-B), SUPER 3G, IMT-Advanced, 4th generation mobile communication system (4G), 5th generation mobile communication system (5G), 5G new radio (NR), Future Radio Access (FRA), New-Radio Access Technology (RAT), New Radio (NR), New radio access (NX), Future generation radio access (FX), Global System for Mobile communications (GSM (registered trademark)), CDMA2000, Ultra Mobile Broadband (UMB), IEEE 802.11 (Wi-Fi (registered trademark)), IEEE 802.16 (WiMAX (registered trademark)), IEEE 802.20, Ultra-WideBand (UWB), Bluetooth (registered trademark), Public Land Mobile Network (PLMN) network, Device-to-Device (D2D) system, Machine to Machine (M2M) system, Internet of Things (IoT) system, Vehicle-to-Everything (V2X), system using other communication methods, next-generation system expanded based thereon, and the like. In addition, a plurality of systems can be combined (for example, combination of LTE or LTE-A and 5G, and the like).

[0123] At present, taking the application of an AI model to signal demodulation as an example, accurate signal demodulation is one of the basic conditions for realizing a wireless communication system with low bit error rate and high-speed transmission, and is related to the overall performance of the communication system.

[0124] The optimal demodulator in traditional wireless communication systems is generally implemented by classical methods derived from the Neyman-Pearson theorem and the Bayes theorem. These demodulators usually require accurate channel state information and channel noise distribution, and their performance depends on the parameter settings of various modules, including filters, phase-locked loops, product modulators, analog-to-digital converters, etc.

[0125] The main limitations of traditional demodulators include:

[0126] 1. Large time delay and complex implementation: The use of traditional demodulators often results in significant time delay and relatively complex implementation.

[0127] 2. Unstable module performance: Due to factors such as module vibration, acceleration, temperature fluctuations, aging, and unstable discrete components, module performance can change, leading to a decline in overall receiver performance.

[0128] 3. Poor environmental adaptability: Actual wireless communication channels may suffer from multipath fading, impulse noise, clutter, or discrete interference, which can significantly degrade demodulation performance. For example, some coherent demodulators require carrier synchronization, and when there are phase errors and frequency offsets during synchronization, demodulation errors can occur.

[0129] 4. High dependence on prior knowledge such as CSI: Traditional demodulation methods usually have a high dependence on prior knowledge, while in actual communication, especially in fast fading scenarios, it is difficult to accurately estimate CSI, so the channel model may be unknown at the receiving end.

[0130] 5. Traditional demodulation methods do not fully utilize prior waveform information and time series information.

[0131] Traditional wireless communication systems are generally designed based on strict mathematical theories and precise system models. However, with the increasing demand for wireless services such as smartphones, virtual reality, and the Internet of Things, these systems need to handle more complex and diverse communication requirements, and traditional mathematical models may not be able to cope with these challenges well. In this case, deep learning is introduced as a powerful solution to wireless communication systems.

[0132] Since the information of the modulated signal is represented by amplitude and phase, feature extraction is crucial for signal demodulation. Deep learning-based demodulators (or learning-based demodulators) can automatically learn and extract important features in the signal using techniques such as neural networks, thereby achieving more accurate and robust demodulation. The main advantages of deep learning-based demodulators are:

[0133] 1. Greater flexibility and adaptability through learning from large datasets. Compared to traditional methods, they require less or even eliminate the need for prior knowledge.

[0134] 2. Because the model learns features under various undesirable conditions during training, the deep learning-based demodulator has stronger noise resistance.

[0135] 3. The requirements for prior knowledge (such as CSI and channel noise) can be relaxed or even eliminated.

[0136] Therefore, deep learning-based demodulators have great potential in dealing with complex wireless communication environments and improving communication performance.

[0137] Under different modulation schemes such as Quadrature Phase Shift Keying (QPSK), 16 Quadrature Amplitude Modulation (16QAM), and 64 Quadrature Amplitude Modulation (64QAM), the mapping between constellation symbols and data differs, and one data symbol to be demodulated can correspond to data bits of different lengths. Therefore, for symbols using different modulation schemes, a matching demodulation method is required for data demodulation. For demodulation schemes based on AI models, the different modulation schemes also need to be considered.

[0138] Training an AI model requires input and output data of a defined dimension. The length of the demodulated data corresponding to a symbol to be demodulated varies depending on the modulation scheme. For example, in 16QAM modulation, a constellation point corresponds to 4 bits of data, while in 64QAM modulation, a constellation point corresponds to 6 bits of data. Therefore, the AI ​​demodulation models used for 16QAM and 64QAM modes output data with lengths of 4 bits and 6 bits respectively, and it is necessary to learn the mapping relationship between constellation points and data under the two modulation schemes respectively.

[0139] Therefore, current AI-based demodulation schemes can demodulate data under different modulation methods, but require known modulation method information. This information can be indicated by the transmitter to the receiver during data transmission as auxiliary information for selecting the demodulation model, as shown in Figure 1B.

[0140] In addition, in the case where the modulation mode information is not indicated at the sending end, existing research also explores modulation mode identification based on AI models, that is, using the to-be-demodulated data and the corresponding modulation mode information as training data, an AI modulation mode identification model obtained by training can determine the corresponding sending end signal modulation mode based on the features of the to-be-demodulated data, and then select an AI demodulation model under the corresponding modulation mode to demodulate the data, so as to realize an AI demodulation scheme without indicating the modulation information, as shown in FIG. 1C.

[0141] The main application process of the data demodulation scheme based on deep learning is as follows: after the AI model for data demodulation is trained using the training data, the trained AI demodulation model is deployed on the channel receiving end device, and then the receiving end data demodulation task in the actual system is completed, that is, model inference is performed.

[0142] The mapping relationship between the to-be-demodulated data symbols and the demodulated data is different under different modulation modes, and different AI models are usually used. In a relatively fixed scene and without obvious environmental characteristics and channel changes, usually one model can achieve high-precision data demodulation.

[0143] However, in an actual communication system, due to device movement, channel scene changes, and instability of the wireless environment, the distribution of the to-be-demodulated data input in the AI demodulation model inference application process may change greatly, that is, there is a certain difference between the input data distribution in the model application process and the input data distribution in the model training, and then the input-output mapping relationship learned by the AI demodulation model from the training data set is no longer completely suitable for the test data, causing the performance of the AI demodulation model to decline.

[0144] In order to improve the reliability and usability of the AI model, the present disclosure provides a model performance monitoring method, a first device, a second device, and a storage medium.

[0145] FIG. 2A is an interaction diagram of a model performance monitoring method according to an embodiment of the present disclosure. As shown in FIG. 2A, the present embodiment relates to a model performance monitoring method, and the method comprises:

[0146] In step S2101, the second device 102 determines the first data.

[0147] In some embodiments, the second device 102 can be a terminal, a network device, an environmental Internet of Things device, an Internet of Things device, an unmanned device, etc., and the present disclosure does not limit the same.

[0148] In some embodiments, the second device 102 can be a device that transmits second data. The second data is data related to the first data, and the specific content will be described in subsequent embodiments, which will not be described here.

[0149] In some embodiments, the first data is reference data for monitoring the performance of a first artificial intelligence (AI) model.

[0150] In some embodiments, the first data can be a bit sequence composed of "0" and "1".

[0151] In some embodiments, the first AI model can be used for demodulation, device perception, device positioning, etc., and the present disclosure does not limit the same.

[0152] In some embodiments, when the first AI model is used for demodulation, the first data can be original transmission data that has not been modulated.

[0153] In some embodiments, when the first AI model is used for device perception, the first data can be actual perception information of a third device, such as actual values of parameters such as distance, angle, and speed of the third device. The third device is a device that needs to be perceived, which can also be referred to as a "perception target".

[0154] In some embodiments, when the first AI model is used for device positioning, the first data can be actual position information of a fourth device, such as actual position coordinates of the fourth device. The fourth device is a device that needs to be positioned, which can also be referred to as a "positioning target".

[0155] In some embodiments, the specific content of the first data is not limited.

[0156] In some embodiments, the name of the first data is not limited and can be interchangeable with reference data, model performance monitoring data, etc.

[0157] In some embodiments, the second device 102 can select one or more groups of first data.

[0158] In some embodiments, when the first AI model is used for demodulation, the second device 102 can select different first data for different modulation methods.

[0159] In one example, the modulation method can include, but is not limited to, QPSK, 16QAM, 64QAM, etc.

[0160] For example, if the modulation scheme is 16QAM, the second device 102 can select the first data #11; if the modulation scheme is 64QAM, the second device 102 can select the first data #12.

[0161] In some embodiments, when the first AI model is used for demodulation, the second device 102 can select different first data for different modulation orders.

[0162] In one example, the modulation order can refer to the number of bits corresponding to a constellation diagram.

[0163] For example, if the modulation order is 4, the second device 102 can select first data #21, where first data #21 includes 4 × M1 bits, and M1 is a positive integer. As another example, if the modulation order is 6, the second device 102 can select first data #22, where first data #22 includes 6 × N1 bits, and N1 is a positive integer.

[0164] In some embodiments, when the first AI model is used for demodulation, the second device 102 can select different first data for different modulation orders and different modulation orders.

[0165] For example, if the modulation scheme is 16QAM and the modulation order is 4, the second device 102 can select the first data #31, which corresponds to 16QAM and includes 4 × M2 bits, where M2 is a positive integer. As another example, if the modulation scheme is 64QAM and the modulation order is 6, the second device 102 can select the first data #32, which corresponds to 64QAM and includes 6 × N2 bits, where N2 is a positive integer.

[0166] In some embodiments, when the first AI model is used for demodulation, if the scene is the same and there are no significant environmental features or channel changes, the first data selected by the second device 102 can be the same. Of course, the second device 102 can also select different first data, and this disclosure does not limit this.

[0167] In some embodiments, when the first AI model is used for device perception or device positioning, the second device 102 may select one or more sets of first data, which is not limited in this disclosure.

[0168] In step S2102, the second device 102 sends the first data to the first device 101.

[0169] In some embodiments, the second device 102 may send the selected first data to the first device 101.

[0170] In some embodiments, the first device 101 receives first data.

[0171] In some embodiments, the second device 102 can send indication information to the first device 101, which can be used to indicate that the first data is reference data for monitoring the performance of the first artificial intelligence AI model.

[0172] In some embodiments, the first device 101 receives the indication information.

[0173] In some embodiments, the second device 102 can send the first data and the indication information to the first device 101 through the same signaling, or can separately send the first data, the indication information to the first device 101 through different signaling, which is not limited in the present disclosure.

[0174] In some embodiments, the second device 102 can not send the indication information, and the first device 101 defaults that the received first data is reference data for monitoring the performance of the first artificial intelligence AI model in the case of reaching a specified period or meeting a certain condition.

[0175] In one example, the specified period can refer to the period of monitoring the first AI model.

[0176] In one example, the condition that is met can refer to a condition for monitoring the first AI model, such as the usage time of the first AI model reaching a preset time length, etc.

[0177] Step S2103, the second device 102 determines the second data.

[0178] In some embodiments, the second data is data associated with the first data.

[0179] In some embodiments, in the case that the first AI model is used for demodulation, the second data is data obtained after the first data is modulated.

[0180] In some embodiments, the second device 102 modulates the first data in any modulation manner to obtain the second data. The modulation manner includes but is not limited to any one of QPSK, 16QAM, 64QAM.

[0181] In some embodiments, in the case that the first AI model is used for device sensing, the second data is first channel measurement data for device sensing, including but not limited to CSI matrix.

[0182] In some embodiments, the second device 102 as a "sensing target" can send a sensing signal, which is received by other devices such as base stations, Internet of Things devices, environmental Internet of Things devices after reflection, refraction and / or scattering, and based on the received signal, the second device 102 performs channel measurement to obtain the second data.

[0183] In some embodiments, the “perception target” is a device different from the second device 102, the perception signal can be sent by the device, the second device 102 can receive the signal reflected, refracted and / or scattered by the “perception target”, and perform channel measurement based on the received signal to obtain the second data.

[0184] In some embodiments, when the first AI model is used for device positioning, the second data is second channel measurement data used for device positioning, including but not limited to CSI matrix, and can also include incident angle value, signal transmission delay, etc.

[0185] In the above embodiments, the perception signal can refer to a signal used for device perception. For example, the perception signal can be a sounding reference signal, a random sequence, etc.

[0186] In some embodiments, the specific content of the second data is not limited.

[0187] In some embodiments, the name of the second data is not limited, and can be interchangeable with modulation signal, modulation data, channel measurement data, channel measurement information, etc.

[0188] Step S2104, the second device 102 sends the second data to the first device 101.

[0189] In some embodiments, the first device 101 receives the second data.

[0190] Step S2105, the first device 101 inputs the second data into the first AI model to obtain third data output by the first AI model.

[0191] In some embodiments, the first device 101 takes the received second data as the input value of the first AI model, and performs demodulation, device perception or device positioning, etc. through the first AI model to obtain the third data output by the first AI model.

[0192] Step S2106, the first device 101 determines the performance monitoring result of the first AI model based on the difference between the third data and the first data.

[0193] In some embodiments, the first device 101 can determine a first parameter value, which is used to measure the difference between the third data and the first data.

[0194] In some embodiments, the first parameter value can include but is not limited to bit error rate, bit error rate, etc.

[0195] In some embodiments, if the first parameter value is lower than the first value, e.g., the error code rate is lower than the first value, it indicates that the demodulation performance of the first AI model is better, and the first device 101 can determine that the performance monitoring result of the first AI model is that the performance of the first AI model meets the accuracy requirement.

[0196] In some embodiments, if the first parameter value reaches or is higher than the first value, e.g., the error code rate reaches or is higher than the first value, it indicates that the demodulation performance of the first AI model is poorer, and the first device 101 can determine that the performance monitoring result of the first AI model is that the performance of the first AI model does not meet the accuracy requirement.

[0197] In some embodiments, the specific content of the accuracy requirement can be defined by a protocol or determined by negotiation between the first device 101 and the second device 102. For example, the accuracy requirement can be agreed by the protocol as the first parameter value being lower than the first value.

[0198] Step S2107, the first device 101 performs the first operation.

[0199] In some embodiments, step S2107 is an optional step.

[0200] In some embodiments, the first device 101 performs the first operation in the case that the performance monitoring result is that the performance of the first AI model does not meet the accuracy requirement.

[0201] In one example, the first operation includes but is not limited to at least one of the following:

[0202] updating the first AI model;

[0203] performing model switching;

[0204] falling back to a non-AI mode.

[0205] Illustratively, the first device 101 can collect new sample data to update the network parameters and / or network architecture of the first AI model in the case that the performance of the first AI model is poorer.

[0206] Illustratively, the first device 101 can switch the first AI model to a second AI model in the case that the performance of the first AI model is poorer. The second AI model has the same function as the first AI model but is applicable to different channel scenarios. For example, the first AI model is used for demodulation, and the AI model used can be switched to the second AI model, e.g., the second data is input to the second AI model to obtain the fifth data output by the second AI model.

[0207] Exemplarily, the first device 101 can fall back to a non-AI manner in a case where the first AI model performs poorly, and obtain fourth data based on the second data. For example, the second data is demodulated by using a traditional non-AI manner such as a traditional demodulator to obtain demodulated fourth data. For another example, device sensing or device positioning can be performed by using a traditional non-AI manner to obtain device sensing results or device positioning results.

[0208] In some embodiments, the names of signals and the like are not limited to the names described in the embodiments, and terms such as "information", "message", "signal", "signaling", "report", "configuration", "indication", "instruction", "command", "channel", "parameter", "domain", "field", "symbol", "symbol", "codebook", "codeword", "codepoint", "bit", "data", "program", "chip", and the like can be replaced with each other.

[0209] In some embodiments, "acquire", "obtain", "get", "receive", "transmit", "bidirectional transmission", "send and / or receive", and the like can be replaced with each other, and can be interpreted as various meanings such as receiving from other subjects, acquiring from a protocol, acquiring from a higher layer, processing to obtain by oneself, autonomously implementing, and the like.

[0210] In some embodiments, terms such as "send", "transmit", "report", "issue", "transmit", "bidirectional transmission", "send and / or receive", and the like can be replaced with each other.

[0211] In some embodiments, terms such as "certain", "preset", "pre-set", "set", "indicated", "a certain", "arbitrary", "first", and the like can be replaced with each other, and "certain A", "preset A", "pre-set A", "set A", "indicated A", "a certain A", "arbitrary A", "first A" can be interpreted as A specified in advance in a protocol and the like, can be interpreted as A obtained by setting, configuring, or indicating, and the like, and can be interpreted as certain A, a certain A, arbitrary A, or first A, but are not limited thereto.

[0212] The data processing method related to the embodiments of the present disclosure can include at least one of steps S2101-S2107. For example, step S2101 can be implemented as an independent embodiment, step S2102 can be implemented as an independent embodiment, steps S2101+S2102 can be implemented as an independent embodiment, step S2103 can be implemented as an independent embodiment, step S2104 can be implemented as an independent embodiment, steps S2103+S2104 can be implemented as an independent embodiment, step S2105 can be implemented as an independent embodiment, step S2106 can be implemented as an independent embodiment, step S2107 can be implemented as an independent embodiment, steps S2106+S2107 can be implemented as an independent embodiment, steps S2101-S2107 can be implemented as an independent embodiment, but the present disclosure is not limited thereto.

[0213] In some embodiments, step S2107 is optional, and one or more of the steps can be omitted or replaced in different embodiments.

[0214] In some embodiments, steps S2101-S2102 are optional, and one or more of the steps can be omitted or replaced in different embodiments.

[0215] In some embodiments, the execution order of steps S2101-S2107 is not limited.

[0216] In some embodiments, steps S2101-S2107 are optional, and one or more of the steps can be omitted or replaced in different embodiments.

[0217] In the above embodiments, the first data can be determined by the second device and sent to the first device. The first device can take the second data provided by the second device as an input value of the first AI model, and determine the performance monitoring result of the first AI model based on the difference between the third data output by the first AI model and the first data. The purpose of monitoring the performance of the AI model is achieved, and the usability and reliability of the AI model can be effectively improved in various communication scenarios, which helps to promote the integration of AI and communication.

[0218] FIG. 2B is an interaction schematic diagram of a model performance monitoring method according to an embodiment of the present disclosure. As shown in FIG. 2B, the embodiments of the present disclosure relate to a model performance monitoring method, and the above method includes:

[0219] Step S2201, the first device 101 determines the first data.

[0220] In some embodiments, the first device 101 determines the first data in a manner similar to the aforementioned step S2101, which will not be described here.

[0221] At step S2202, the first device 101 sends the first data to the second device 102.

[0222] In some embodiments, the second device 102 receives the first data.

[0223] In some embodiments, the first device 101 sends indication information to the second device 102, which can be used to indicate that the first data is reference data for monitoring the performance of the first artificial intelligence AI model.

[0224] In some embodiments, the first device 101 receives the indication information.

[0225] In some embodiments, the first device 101 can also not send the indication information,

[0226] At step S2203, the second device 102 determines the second data.

[0227] In some embodiments, the implementation of step S2203 is similar to the aforementioned step S2103, which will not be described here.

[0228] At step S2204, the second device 102 sends the second data to the first device 101.

[0229] In some embodiments, the first device 101 receives the second data.

[0230] At step S2205, the first device 101 inputs the second data into the first AI model, and obtains third data output by the first AI model.

[0231] In some embodiments, the implementation of step S2205 is similar to the aforementioned step S2105, which will not be described here.

[0232] At step S2206, the first device 101 determines the performance monitoring result of the first AI model based on the difference between the third data and the first data.

[0233] In some embodiments, the implementation of step S2206 is similar to the aforementioned step S2106, which will not be described here.

[0234] At step S2207, the first device 101 performs the first operation.

[0235] In some embodiments, the implementation of step S2207 is similar to the aforementioned step S2107, which will not be described here.

[0236] In some embodiments, the execution order of steps S2201-S2207 is not limited.

[0237] In some embodiments, steps S2201-S2207 are optional, and one or more of these steps can be omitted or replaced in different embodiments.

[0238] In the above embodiments, the first device can determine the first data and then send the first data to the second device. The first device can take the second data provided by the second device as an input value of the first AI model, and determine the performance monitoring result of the first AI model based on the difference between the third data output by the first AI model and the first data. The purpose of monitoring the performance of the AI model is achieved, and the usability and reliability of the AI model can be effectively improved in various communication scenarios, which helps to promote the integration of AI and communication.

[0239] FIG. 3A is a flow diagram of a model performance monitoring method according to an embodiment of the present disclosure. As shown in FIG. 3A, the present embodiment relates to a model performance monitoring method, which can be performed by the first device 101. The method includes the following steps:

[0240] In step S3101, first data is obtained.

[0241] In some embodiments, the first data is reference data for monitoring the performance of a first artificial intelligence (AI) model.

[0242] In some embodiments, the optional implementation of step S3101 can refer to the optional implementation of step S2102 in FIG. 2A and other associated parts in the embodiments related to FIG. 2A, which will not be described here again.

[0243] In some embodiments, the first device 101 receives the first data sent by the second device 102, but is not limited thereto, and can also receive the first data sent by other subjects.

[0244] In some embodiments, the first device 101 obtains the first data specified by a protocol.

[0245] In some embodiments, the first device 101 obtains the first data from an upper layer.

[0246] In some embodiments, the first device 101 processes to obtain the first data.

[0247] In some embodiments, step S3101 is omitted, and the first device 101 autonomously implements the function indicated by the first data, or the above function is default or default.

[0248] In some embodiments, the optional implementation of step S3101 can refer to the optional implementation of step S2201 in FIG. 2B and other associated parts in the embodiments related to FIG. 2B, which will not be described here again.

[0249] Step S3102: sending first data.

[0250] In some embodiments, the first device 101 can send the first data to the second device 102 after determining the first data.

[0251] In some embodiments, the second device 102 receives the first data.

[0252] In some embodiments, step S3102 is an optional execution step. For example, when the second device 102 obtains the first data from other execution subject or the second device 102 determines the first data by itself, step S3102 can not be executed.

[0253] Step S3103: obtaining second data.

[0254] In some embodiments, the second data is data related to the first data.

[0255] In some embodiments, optional implementation of step S3103 can refer to optional implementation of step S2104 in FIG. 2A and other associated parts in the embodiments involved in FIG. 2A, which will not be repeated here.

[0256] In some embodiments, the first device 101 receives the second data sent by the second device 102, but is not limited thereto, and can also receive the second data sent by other subject.

[0257] In some embodiments, the first device 101 obtains the second data specified by a protocol.

[0258] In some embodiments, the first device 101 obtains the second data from upper layer(s).

[0259] In some embodiments, the first device 101 processes to obtain the second data.

[0260] In some embodiments, step S3102 is omitted, and the first device 101 autonomously implements the function indicated by the second data, or the above function is default or default.

[0261] Step S3104: determining third data.

[0262] In some embodiments, the first device 101 inputs the second data into the first AI model to obtain third data output by the first AI model.

[0263] In some embodiments, optional implementation of step S3104 can refer to optional implementation of step S2105 in FIG. 2A and other associated parts in the embodiments involved in FIG. 2A, which will not be repeated here.

[0264] Step S3105, determining the performance monitoring result of the first AI model.

[0265] In some embodiments, the optional implementation of step S3105 can refer to the optional implementation of step S2106 in FIG. 2A and other associated parts in the embodiments involved in FIG. 2A, which will not be repeated here.

[0266] Step S3106, performing a first operation.

[0267] In some embodiments, the optional implementation of step S3106 can refer to the optional implementation of step S2107 in FIG. 2A and other associated parts in the embodiments involved in FIG. 2A, which will not be repeated here.

[0268] In some embodiments, step S3106 is an optional execution step. For example, when the performance monitoring result is that the performance of the first AI model meets the accuracy requirement, step S3106 can not be executed.

[0269] In some embodiments, the execution order of steps S3101-S3106 is not limited.

[0270] In some embodiments, steps S3101-S3106 are optional, and one or more of these steps can be omitted or replaced in different embodiments.

[0271] In the above embodiments, the first device can determine the performance monitoring result of the first AI model based on the difference between the third data output by the first AI model and the first data. The purpose of monitoring the performance of the AI model is achieved, which can effectively improve the usability and reliability of the AI model in various communication scenarios, and helps to promote the integration of AI and communication.

[0272] FIG. 3B is a flow diagram of a model performance monitoring method according to an embodiment of the present disclosure. As shown in FIG. 3A, the embodiments of the present disclosure relate to a model performance monitoring method, and the above method can be executed by the first device 101, which includes:

[0273] Step S3201, obtaining first data.

[0274] In some embodiments, the first data is reference data for monitoring the performance of a first artificial intelligence (AI) model.

[0275] In some embodiments, the optional implementation of step S3201 can refer to the optional implementation of step S2101 in FIG. 2A and other associated parts in the embodiments involved in FIG. 2A, which will not be repeated here.

[0276] In some embodiments, the optional implementation of step S3201 can refer to the optional implementation of step S2202 in FIG. 2B and other associated parts in the embodiments involved by FIG. 2B, which will not be repeated here.

[0277] In some embodiments, the second device 102 receives the first data sent by the first device 101, but is not limited thereto, and can also receive the first data sent by other subjects.

[0278] In some embodiments, the second device 102 acquires the first data specified by a protocol.

[0279] In some embodiments, the second device 102 acquires the first data from upper layer(s).

[0280] In some embodiments, the second device 102 processes to obtain the first data.

[0281] In some embodiments, step S3201 is omitted, and the second device 102 autonomously implements the function indicated by the first data, or the above function is default or default.

[0282] Step S3202, sending the first data.

[0283] In some embodiments, the second device 102 can send the first data to the first device 101.

[0284] In some embodiments, the first device 101 receives the first data.

[0285] In some embodiments, step S3202 is an optional execution step. For example, when the first device 101 acquires the first data from other execution subjects or the first device 101 itself determines the first data, step S3202 can not be executed.

[0286] Step S3203, determining the second data.

[0287] In some embodiments, the second data is data related to the first data.

[0288] In some embodiments, the optional implementation of step S3203 can refer to the optional implementation of step S2103 in FIG. 2A and other associated parts in the embodiments involved by FIG. 2A, which will not be repeated here.

[0289] Step S3204, sending the second data.

[0290] In some embodiments, the second device 102 sends the second data to the first device 101.

[0291] In some embodiments, the first device 101 receives the second data.

[0292] In some embodiments, the optional implementation of step S3204 can refer to the optional implementation of step S2104 in FIG. 2A and other associated parts in the embodiments related to FIG. 2A, which are not described herein again.

[0293] In some embodiments, the execution order of steps S3201-S3204 is not limited.

[0294] In some embodiments, steps S3201-S3204 are optional, and one or more of these steps can be omitted or replaced in different embodiments.

[0295] In the above embodiments, the second device can provide the second data to the first device, and the first device determines the performance monitoring result of the first AI model based on the difference between the third data output by the first AI model and the first data. The purpose of monitoring the performance of the AI model is achieved, and the usability and reliability of the AI model can be effectively improved in various communication scenarios, which helps to promote the integration of AI and communication.

[0296] The above process is further illustrated as follows.

[0297] In the embodiments of the present disclosure, for model performance monitoring in the application process of the AI demodulation technology scheme, a data collection method for AI demodulation model performance monitoring is proposed, which effectively realizes data collection for model performance monitoring in the application process of the AI demodulation model, and further realizes relatively accurate demodulation model performance monitoring.

[0298] For data transmission in an actual communication system, the original data of the sending end is transmitted to the receiving end through a wireless channel after modulation and other steps. Due to the influence of factors such as channel noise and non-ideal characteristics of devices, there is a certain error between the data obtained by the receiving end after demodulation and the original data sent by the sending end. AI demodulation model performance monitoring requires label data, including to-be-demodulated data and corresponding ideal demodulation data, i.e., to-be-modulated data after encoding by the sending end. However, in the conventional signal sending and receiving process, it is difficult to obtain ideal demodulation data. Therefore, a data collection method for AI demodulation model performance monitoring is proposed, so that the receiving end can obtain ideal demodulation data corresponding to the to-be-demodulated data and the same as the original sending data of the sending end, to accurately and efficiently monitor the performance of the AI demodulation model.

[0299] The application process of the AI demodulation model performance monitoring method proposed in the present disclosure is shown in FIG. 4, including the following steps:

[0300] Step S4101, the sending end device (i.e., the second device) and the receiving end device (i.e., the first device) interact data for model performance monitoring.

[0301] The sending end device and the receiving end device first interact data information for AI demodulation model performance monitoring, that is, the sending end or the receiving end selects one or several sets of data for AI demodulation model performance monitoring and sends an indication to the other party, indicating that the two will use the data as data for AI demodulation model performance monitoring in the subsequent model performance monitoring process.

[0302] The main purpose of this step is for the sending end and the receiving end to determine the data dedicated to model performance monitoring in the subsequent communication process, which is similar to the pilot data in the channel sending and receiving process. In this way, in the model performance monitoring process, the sending end uses the data as the original sending data, and the receiving end uses the data as the label, which can achieve relatively accurate model performance monitoring.

[0303] Step S4102, the sending end device completes data modulation and other processes.

[0304] The sending end device uses the data determined in step S4101 for model performance monitoring as the original sending data to complete data modulation and other processes to generate a to-be-sent data signal.

[0305] Step S4103, data sending.

[0306] The sending end device sends the to-be-sent data signal to the receiving end device.

[0307] Step S4104, the receiving end device completes data demodulation and other processes.

[0308] The receiving end device acquires a received signal, completes demodulation and other processes, and acquires demodulation data.

[0309] Step S4105, AI demodulation model performance monitoring.

[0310] The receiving end device compares the correct demodulation data acquired with ideal demodulation data. If the error is large, it indicates that the AI demodulation model performance is poor, and countermeasures such as model updating, model switching, and falling back to a non-AI method need to be taken to ensure the relative stability of the demodulation accuracy.

[0311] In the above embodiment, the technical scheme for realizing data demodulation based on an AI model addresses the problem of model performance monitoring data collection in the application process of an AI demodulation model, proposes an AI demodulation model performance monitoring data collection method, effectively realizes data collection for model performance monitoring in the application process of an AI demodulation model, and thus helps to effectively improve the practicality of an AI-based demodulation scheme.

[0312] In the embodiments of the present disclosure, part or all of the steps, and optional implementation manners thereof, can be combined with part or all of the steps in other embodiments, or combined with optional implementation manners of other embodiments.

[0313] The embodiments of the present disclosure also propose a device for implementing any of the above methods, for example, a device comprising units or modules for implementing the steps performed by the first device in any of the above methods. For another example, another device is proposed, which comprises units or modules for implementing the steps performed by the second device in any of the above methods.

[0314] It should be understood that the division of each unit or module in the above device is only a logical function division, and all or part of the units or modules can be integrated into one physical entity, or can be physically separated. In addition, the units or modules in the device can be implemented in the form of processor calling software: for example, the device comprises a processor connected with a memory, the memory stores instructions, and the processor calls the instructions stored in the memory to implement any of the above methods or to implement the functions of each unit or module of the device, wherein the processor is, for example, a general processor such as a central processing unit (CPU) or a microprocessor, and the memory is a memory in the device or a memory outside the device. Alternatively, the units or modules in the device can be implemented in the form of hardware circuit, and the functions of part or all of the units or modules can be implemented by designing the hardware circuit, and the hardware circuit can be understood as one or more processors; for example, in one implementation, the hardware circuit is an application-specific integrated circuit (ASIC), and the functions of part or all of the units or modules are implemented by designing the logical relationship of elements in the circuit; for another example, in another implementation, the hardware circuit is a programmable logic device (PLD), and taking a field programmable gate array (FPGA) as an example, it can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by a configuration file, so as to implement the functions of part or all of the units or modules. All units or modules of the above device can be implemented in the form of processor calling software, or all units or modules can be implemented in the form of hardware circuit, or part of the units or modules are implemented in the form of processor calling software, and the remaining part is implemented in the form of hardware circuit.

[0315] In the embodiments of the present disclosure, the processor is a circuit with signal processing capability. In one implementation, the processor can be a circuit with instruction reading and running capability, such as a central processing unit (CPU), a microprocessor, a graphics processing unit (GPU) (which can be understood as a microprocessor), a digital signal processor (DSP), and the like. In another implementation, the processor can implement certain functions through a logical relationship of hardware circuits, and the logical relationship of the hardware circuits is fixed or reconfigurable. For example, the processor is a hardware circuit implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), such as an FPGA. In the reconfigurable hardware circuit, the processor loads a configuration document to implement the configuration of the hardware circuit. It can be understood that the processor loads instructions to implement the functions of the above part or all units or modules. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as a neural network processing unit (NPU), a tensor processing unit (TPU), a deep learning processing unit (DPU), and the like.

[0316] FIG. 5A is a structural schematic diagram of a first device according to an embodiment of the present disclosure. As shown in FIG. 5A, the first device 5100 can include a transceiver module 5101 and a processing module 5102.

[0317] In some embodiments, the transceiver module 5101 is configured to receive second data sent by a second device, wherein the second data is data related to first data, and the first data is reference data used for monitoring performance of a first artificial intelligence (AI) model.

[0318] In some embodiments, the processing module 5102 is configured to input the second data into the first AI model to obtain third data output by the first AI model, and determine a performance monitoring result of the first AI model based on a difference between the third data and the first data.

[0319] Optionally, the transceiver module 5101 is configured to perform at least one of the communication steps (for example, steps S2102, S2104, S2202, S2204, but not limited to) of transmitting and / or receiving performed by the first device 5100 in any of the above methods. Details are not repeated here.

[0320] Optionally, the processing module 5102 is configured to perform at least one of the other steps (for example, steps S2105, S2106, S2107, S2201, S2205, S2206, S2207, but not limited to) performed by the first device 5100 in any of the above methods. Details are not repeated here.

[0321] FIG. 5B is a structural schematic diagram of a first device according to an embodiment of the present disclosure. As shown in FIG. 5B, the first device 5200 can include a processing module 5201 and a transceiver module 5202.

[0322] In some embodiments, the processing module 5201 is configured to determine second data, wherein the second data is data related to the first data, and wherein the first data is reference data used to monitor performance of a first artificial intelligence (AI) model.

[0323] In some embodiments, the transceiver module 5202 is configured to transmit the second data to the first device.

[0324] Optionally, the processing module 5201 is configured to perform at least one of the other steps (for example, steps S2101, S2103, S2203, but not limited to) performed by the second device 5200 in any of the above methods. Details are not repeated here.

[0325] Optionally, the transceiver module 5202 is configured to perform at least one of the communication steps (for example, steps S2102, S2104, S2202, S2204, but not limited to) of transmitting and / or receiving performed by the second device 5200 in any of the above methods. Details are not repeated here.

[0326] In some embodiments, the processing module can be one module or can include multiple sub-modules. Optionally, the multiple sub-modules perform all or part of the steps required to be performed by the processing module respectively. Optionally, the processing module can be replaced by a processor.

[0327] FIG. 6A is a structural schematic diagram of a communication device 6100 according to an embodiment of the present disclosure. The communication device 6100 can be a first device or a second device, which can be a network device (for example, an access network device, a core network device, etc.), a terminal (for example, a user equipment, a vehicle, etc.), a chip, a chip system, or a processor supporting the network device to implement any of the above methods, or a chip, a chip system, or a processor supporting the terminal to implement any of the above methods. The communication device 6100 can be used to implement the methods described in the above method embodiments, and details can be referred to the descriptions in the above method embodiments.

[0328] As shown in FIG. 6A, the communication device 6100 includes one or more processors 6101. The processor 6101 can be a general purpose processor or a special purpose processor, for example, a baseband processor or a central processing unit. The baseband processor can be used to process communication protocols and communication data, and the central processing unit can be used to control the communication device (for example, a base station, a baseband chip, a terminal device, a terminal device chip, a DU or a CU, etc.), execute programs, and process data of the programs. Optionally, the communication device 6100 is configured to perform any of the above methods. Optionally, the one or more processors 6101 are configured to invoke instructions to cause the communication device 6100 to perform any of the above methods.

[0329] In some embodiments, the communication device 6100 further includes one or more transceivers 6102. When the communication device 6100 includes the one or more transceivers 6102, the transceiver 6102 performs at least one of the communication steps (for example, steps S2102, S2104, S2202, S2204, but not limited to) in the above methods, and the processor 6101 performs at least one of the other steps (for example, steps S2101, S2103, S2105, S2106, S2107, S2201, S2203, S2205, S2206, S2207, but not limited to) in the above methods. In optional embodiments, the transceiver can include a receiver and / or a transmitter, which can be separate or integrated together. Optionally, the terms of transceiver, transceiving unit, transceiver, transceiving circuit, interface circuit, interface, etc. can be replaced with each other, and the terms of transmitter, transmitting unit, transmitter, transmitting circuit, etc. can be replaced with each other, and the terms of receiver, receiving unit, receiver, receiving circuit, etc. can be replaced with each other.

[0330] In some embodiments, the communication device 6100 further includes one or more memories 6103 for storing data. Alternatively, all or part of the memories 6103 can be external to the communication device 6100. In optional embodiments, the communication device 6100 can include one or more interface circuits 6104. Optionally, the interface circuit 6104 is connected to the memory 6102, and the interface circuit 6104 can be used to receive data from the memory 6102 or other devices, and can be used to send data to the memory 6102 or other devices. For example, the interface circuit 6104 can read data stored in the memory 6102 and send the data to the processor 6101.

[0331] The communication device 6100 described in the above embodiments can be a network device or a terminal, but the scope of the communication device 6100 described in the present disclosure is not limited thereto, and the structure of the communication device 6100 can not be limited by FIG. 6A. The communication device can be a standalone device or can be part of a larger device. For example, the communication device can be: 1) a standalone integrated circuit (IC), or a chip, or a chip system or subsystem; (2) a set of one or more ICs, which can optionally include a storage component for storing data, programs; (3) an ASIC, such as a modem; (4) a module that can be embedded in other devices; (5) a receiver, a terminal device, a smart terminal device, a cellular phone, a wireless device, a handset, a mobile unit, a vehicle-mounted device, a network device, a cloud device, an artificial intelligence device, etc.; (6) other devices, etc.

[0332] FIG. 6B is a structural schematic diagram of a chip 6200 according to an embodiment of the present disclosure. For the case where the communication device 6100 is a chip or a chip system, the structural schematic diagram of the chip 6200 shown in FIG. 6B can be referred to, but is not limited thereto.

[0333] The chip 6200 includes one or more processors 6201. The chip 6200 is configured to perform any of the above methods.

[0334] In some embodiments, the chip 6200 further includes one or more interface circuits 6202. Optionally, the terms interface circuit, interface, transceiver pin, etc. can be replaced with each other. In some embodiments, the chip 6200 further includes one or more memories 6203 for storing data. Optionally, all or part of the memories 6203 can be external to the chip 6200. Optionally, the interface circuit 6202 is connected to the memory 6203, and the interface circuit 6202 can be used to receive data from the memory 6203 or other devices, and the interface circuit 6202 can be used to send data to the memory 6203 or other devices. For example, the interface circuit 6202 can read data stored in the memory 6203 and send the data to the processor 6201.

[0335] In some embodiments, the interface circuit 6202 performs at least one of the communication steps (for example, step S2102, step S2104, step S2202, step S2204, but not limited to) of transmitting and / or receiving in the above method. The interface circuit 6202 performing the communication steps such as transmitting and / or receiving in the above method means that the interface circuit 6202 performs data interaction between the processor 6201, the chip 6200, the memory 6203, or the transceiver device. In some embodiments, the processor 6201 performs at least one of the other steps (for example, step S2101, step S2103, step S2105, step S2106, step S2107, step S2201, step S2203, step S2205, step S2206, step S2207, but not limited to).

[0336] The modules and / or devices described in each of the embodiments of the virtual device, the physical device, the chip, etc. can be combined or separated according to the situation. Optionally, part or all of the steps can also be performed by a plurality of modules and / or devices in cooperation, which is not limited here.

[0337] The disclosure also proposes a storage medium, and the above storage medium stores instructions, which, when executed on the communication device 6100, causes the communication device 6100 to perform any of the above methods. Optionally, the above storage medium is an electronic storage medium. Optionally, the above storage medium is a computer readable storage medium, but is not limited to this, and it can also be a storage medium readable by other devices. Optionally, the above storage medium can be a non-transitory storage medium, but is not limited to this, and it can also be a transitory storage medium.

[0338] The disclosure also proposes a program product, which, when executed by the communication device 6100, causes the communication device 6100 to perform any of the above methods. Optionally, the above program product is a computer program product.

[0339] The disclosure also proposes a computer program, which, when executed on a computer, causes the computer to perform any of the above methods.

[0340] It should be understood that the disclosure is not limited to the precise construction that has been described above and shown in the accompanying drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the disclosure is only limited by the appended claims.

Claims

1. A model performance monitoring method characterized by, The method is performed by a first device, and the method comprises: receiving second data sent by a second device; wherein the second data is data related to first data; wherein the first data is reference data used for monitoring performance of a first artificial intelligence (AI) model; inputting the second data into the first AI model to obtain third data output by the first AI model; determining a performance monitoring result of the first AI model based on a difference between the third data and the first data.

2. The method of claim 1, wherein, The first AI model is used for demodulation, the first data is original transmission data that has not been modulated, and the second data is data obtained after the first data is modulated; or The first AI model is used for device perception, the first data comprises actual perception information of a third device, and the second data is first channel measurement data used for device perception; wherein the third device is a device that needs to be perceived; or The first AI model is used for device positioning, the first data comprises actual position information of a fourth device, and the second data is second channel measurement data used for device positioning; wherein the fourth device is a device that needs to be positioned.

3. The method according to claim 1 or 2, characterized in that, The method further comprises: determining the first data; sending the first data to the second device.

4. The method of claim 3, wherein, The first AI model is used for demodulation, and different modulation modes correspond to different first data; and / or Different modulation orders correspond to different first data.

5. The method according to claim 1 or 2, characterized in that, The method further comprises: receiving the first data sent by the second device.

6. The method according to any one of claims 1 to 5, characterized in that, The method further comprises: determining a first parameter value; wherein the first parameter value is used to measure the difference between the third data and the first data; when the first parameter value reaches a first value, determining that the performance monitoring result is that the performance of the first AI model does not meet an accuracy requirement; or when the first parameter value does not reach the first value, determining that the performance monitoring result is that the performance of the first AI model meets the accuracy requirement.

7. The method of claim 6, wherein, The method further comprises any one of the following: when the performance monitoring result is that the performance of the first AI model does not meet the accuracy requirement, obtaining fourth data based on the second data in a non-AI manner; updating the first AI model; switching the first AI model to a second AI model.

8. A model performance monitoring method characterized by, The method is performed by a second device, and the method comprises: determining second data; wherein the second data is data related to first data; wherein the first data is reference data used for monitoring performance of a first artificial intelligence (AI) model; sending the second data to a first device.

9. The method of claim 8, wherein, The first AI model is used for demodulation, the first data is original transmission data that has not been modulated, and the second data is data obtained after the first data is modulated; or The first AI model is used for device perception, the first data includes actual perception information of a third device, and the second data is first channel measurement data used for device perception; wherein the third device is a device that needs to be perceived. The first AI model is used for device positioning, the first data includes actual position information of a fourth device, and the second data is second channel measurement data used for device positioning; wherein the fourth device is a device that needs to be positioned.

10. The method according to claim 8 or 9, characterized in that, The method further includes: receiving the first data sent by the first device.

11. The method of claim 10, wherein, The method further includes: determining the first data; sending the first data to the first device.

12. The method of claim 11, wherein, The first AI model is used for demodulation, and different modulation modes correspond to different first data; and / or different modulation orders correspond to different first data.

13. A first device, comprising: comprising: a transceiver module configured to receive second data sent by a second device; wherein the second data is data related to first data; wherein the first data is reference data used for monitoring performance of a first artificial intelligence (AI) model; a processing module configured to input the second data into the first AI model to obtain third data output by the first AI model; the processing module is further configured to determine a performance monitoring result of the first AI model based on a difference between the third data and the first data.

14. A second device, comprising: comprising: a processing module configured to determine second data; wherein the second data is data related to first data; wherein the first data is reference data used for monitoring performance of a first artificial intelligence (AI) model; a transceiver module configured to send the second data to a first device.

15. A first device, comprising: comprising: one or more processors; wherein the processor is configured to execute the model performance monitoring method of any one of claims 1-7.

16. A second device, comprising: comprising: one or more processors; wherein the processor is configured to execute the model performance monitoring method of any one of claims 8-12.

17. A communication system, characterized by comprising: a first device configured to implement the model performance monitoring method of any one of claims 1-7; a second device configured to implement the model performance monitoring method of any one of claims 8-12.

18. A storage medium, the storage medium storing instructions, wherein, when the instructions run on a communication device, causing the communication device to execute the model performance monitoring method of any one of claims 1-7 or 8-12.

19. A computer program product comprising a computer program, characterized in that, The computer program is executed by a processor to implement the model performance monitoring method of any one of claims 1-7 or 8-12.

Images