Model monitoring method, device and system, storage medium and program product

Abstract

The invention discloses a model monitoring method, device and system, a storage medium and a program product. The method comprises the following steps: the terminal equipment acquires first information, wherein the first information comprises one or more first indexes of one or more pieces of truth value information; obtaining one or more pieces of truth value information corresponding to the one or more first indexes; wherein the one or more pieces of truth value information are used for obtaining the monitoring result of the performance of the AI model, and the output of the AI model is prediction information. By adopting the scheme, the terminal equipment can acquire the truth value information corresponding to the specific time unit by acquiring one or more first indexes of one or more pieces of truth value information, so that the performance of the prediction model can be flexibly monitored.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to a model monitoring method, device, system, storage medium and program product. Background Art

[0002] Model monitoring refers to monitoring the performance of artificial intelligence (AI) models. Determine whether the AI ​​model is working properly. If the AI ​​model performance is poor, it is necessary to switch to non-AI mode, or replace the AI ​​model, or update the AI ​​model, etc. Model monitoring can be monitoring the accuracy of the AI ​​model output (also known as an intermediate key performance indicator (KPI)). Monitoring the accuracy of the AI ​​model output is to compare the difference between the AI ​​model output and the corresponding ground-truth to determine whether the performance of the AI ​​model meets the requirements.

[0003] AI models can be used to predict information. For example, if the output of the AI ​​model is the predicted information at time t+5, the true value information at time t+5 is usually compared with the predicted information at time t+5. If the intermediate KPI meets the preset requirements, it means that the AI ​​model has good performance; otherwise, it means that the AI ​​model has poor performance. Considering that the AI ​​model is used in actual systems, the predicted information is used for base station scheduling downlink transmission, but the downlink transmission scheduled by the base station is not necessarily at the moment corresponding to the predicted information. For example, the information predicted by the AI ​​model is the information at time t+5, that is, if the period of the reference signal (RS) is 5ms, the AI ​​model can only predict the information on the 5ms grid, but the base station may schedule downlink transmission at any time, such as scheduling downlink transmission at time t+3. Then, only monitoring the performance of the predicted information at its corresponding moment cannot reflect the actual performance when the predicted information is used for downlink transmission.

[0004] There is currently no corresponding solution for how to flexibly monitor the performance of the prediction model so that the prediction performance can reflect the actual system performance. Summary of the invention

[0005] The present application provides a model monitoring method, device, system, storage medium and program product, in order to support flexible monitoring of the performance of the prediction model so that the predicted performance can reflect the actual system performance.

[0006] In a first aspect, a model monitoring method is provided, the method comprising: obtaining first information, the first information comprising one or more first indexes corresponding to one or more true value information respectively; and obtaining one or more truth value information corresponding to the one or more first indexes respectively; wherein the one or more truth value information are used to obtain monitoring results of the performance of an artificial intelligence (AI) model, and the output of the AI ​​model is prediction information.

[0007] In this aspect, the terminal device can obtain the truth information corresponding to a specific time unit by obtaining one or more first indexes of one or more truth information, so as to flexibly monitor the performance of the prediction model.

[0008] In a second aspect, a model monitoring method is provided, the method comprising: obtaining first information, the first information comprising the time domain resource location of one or more reference signals for measuring true value information; and determining one or more true value information based on the first information; wherein the one or more true value information are used to obtain monitoring results of the performance of the AI ​​model, the output of the AI ​​model is prediction information, and the time domain resource location of at least one of the one or more true value information is different from the time domain resource location of the prediction information.

[0009] In this aspect, the terminal device can obtain the true value information corresponding to any time domain resource position by obtaining the time domain resource position of one or more reference signals used to measure the true value information, so that the performance of the prediction model can be flexibly monitored.

[0010] In a possible implementation, the method further includes: sending the one or more truth value information.

[0011] In this implementation, by sending one or more truth value information to the access network device, the access network device can obtain the model monitoring results of the actual system based on the received one or more truth value information, so that the performance of the prediction model can be flexibly monitored, and the prediction performance can reflect the actual system performance.

[0012] In another possible implementation, the method further includes: obtaining one or more prediction information, wherein the one or more prediction information is the output of an artificial intelligence AI model; and sending the one or more prediction information.

[0013] In this implementation, by sending one or more true value information to the access network device, the access network device can obtain the model monitoring results of the actual system based on the received one or more true value information and one or more prediction information, so that the performance of the prediction model can be flexibly monitored, and the prediction performance can reflect the actual system performance.

[0014] In another possible implementation, the method further includes: obtaining one or more prediction information, wherein the one or more prediction information is the output of an artificial intelligence (AI) model; and obtaining one or more monitoring results of the performance of the AI ​​model based on the one or more true value information and the one or more prediction information.

[0015] In this implementation, the terminal device stores an AI model that can predict information, and the terminal device can also obtain monitoring results, so that the access network device can obtain the model monitoring results of the actual system, so that the performance of the prediction model can be flexibly monitored, and the prediction performance can reflect the actual system performance.

[0016] In another possible implementation, obtaining one or more prediction information includes: receiving second information, the second information including one or more second indexes corresponding to the one or more prediction information, or the second information including a relative time unit value of each of the one or more prediction information relative to a time unit corresponding to a reference resource; and obtaining one or more prediction information based on the second information.

[0017] In this implementation, the terminal device can obtain corresponding prediction information according to the instruction of the access network device.

[0018] In yet another possible implementation, the one or more first indexes and the one or more second indexes are based on a time unit corresponding to a reference resource.

[0019] In another possible implementation, the time unit corresponding to the reference resource is any one of the following: the first time unit where the resource for reporting the monitoring result report is located; or the second time unit where the downlink control signaling for scheduling the model monitoring is located; or a third time unit determined based on the first time unit or the second time unit; or a predefined / preconfigured fourth time unit.

[0020] In another possible implementation, a monitoring result report is sent; wherein the monitoring result report includes the one or more monitoring results; or the monitoring result report includes at least one monitoring result among the one or more monitoring results that is greater than or equal to a first threshold; or the monitoring result report includes at least one monitoring result among the one or more monitoring results that is less than or equal to a second threshold; or the monitoring result report includes an index of prediction information corresponding to the best monitoring result among the one or more monitoring results and an index of true value information; or the monitoring result report includes an index of prediction information corresponding to the worst monitoring result among the one or more monitoring results and an index of true value information; or the monitoring result report includes the absolute value of a first monitoring result among multiple monitoring results, and the relative values ​​of other monitoring results among the multiple monitoring results except the first monitoring result relative to the first monitoring result.

[0021] In the above implementation, the terminal device can report all acquired monitoring results to the access network device, so that the access network device can acquire complete monitoring results;

[0022] In some scenarios, the access network device may only know whether there is a monitoring result indicating that the model performance is good, without obtaining all monitoring results. Therefore, the monitoring result report includes at least one monitoring result greater than or equal to the first threshold value among the one or more monitoring results. The first threshold value may be configured or predefined by the access network device. This implementation method can save signaling overhead.

[0023] In some scenarios, the access network device may only know whether there is a monitoring result indicating that the model performance is poor, without obtaining all monitoring results. Therefore, the monitoring result report may include at least one monitoring result that is less than or equal to the second threshold value among the one or more monitoring results. The second threshold value may be configured or predefined by the access network device. The second threshold value is less than the first threshold value. This implementation method can save signaling overhead.

[0024] In some scenarios, the access network device may only know whether there is a monitoring result indicating that the model performance is good, without obtaining all monitoring results. Therefore, the monitoring result report may include the index of the prediction information corresponding to the optimal monitoring result among the above one or more monitoring results and the index of the true value information. The access network device may have received one or more prediction information and one or more true value information in advance, and can obtain the optimal monitoring result itself based on the index of the prediction information corresponding to the optimal monitoring result and the index of the true value information. This implementation method can save signaling overhead and has low computing power requirements for the terminal device.

[0025] In some scenarios, the access network device may only know whether there is a monitoring result indicating that the model performance is poor, without obtaining all monitoring results. Therefore, the monitoring result report may include the index of the prediction information corresponding to the worst monitoring result among the one or more monitoring results and the index of the true value information. The access network device may have received one or more prediction information and one or more true value information in advance, and can obtain the worst monitoring result itself based on the index of the prediction information corresponding to the worst monitoring result and the index of the true value information. This implementation method can save signaling overhead and has low computing power requirements for the terminal device.

[0026] In some scenarios, the terminal device can adopt a differential reporting method, and the monitoring result report includes the absolute value of the first monitoring result among multiple monitoring results, and the relative values ​​of other monitoring results other than the first monitoring result among the multiple monitoring results relative to the first monitoring result. This implementation method can save signaling overhead.

[0027] In another possible implementation, the first monitoring result is determined based on an order of the true value information used and / or an order of the prediction information used when acquiring the multiple monitoring results.

[0028] In this implementation, the generation order of one or more monitoring results can be determined based on the order in which the terminal device uses true value information and / or prediction information when generating the monitoring results, so that the first monitoring result can be determined based on the generation order.

[0029] In another possible implementation, the terminal device may report part of the monitoring results according to the instruction of the access network device. For example, the terminal device may be instructed to report part of the monitoring results among all the above monitoring results in a bitmap manner.

[0030] In another possible implementation, the monitoring result report and the information feedback for sending the reference signal measurement result are reported in the same information report, or the monitoring result report and the information feedback for sending the reference signal measurement result are reported in different information reports.

[0031] According to a third aspect, a model monitoring method is provided, the method comprising: sending first information, the first information comprising one or more first indexes corresponding to one or more true value information respectively; and receiving one or more truth value information corresponding to the one or more first indexes respectively; wherein the one or more truth value information are used to obtain monitoring results of the performance of an artificial intelligence (AI) model, and the output of the AI ​​model is prediction information.

[0032] In this aspect, the access network device indicates one or more first indexes of one or more truth information to the terminal device, so that the terminal device can obtain the truth information corresponding to a specific time unit, thereby flexibly monitoring the performance of the prediction model.

[0033] In a fourth aspect, a model monitoring method is provided, the method comprising: sending first information, the first information comprising the time domain resource location of one or more reference signals for measuring true value information; and receiving one or more true value information based on the first information; wherein the one or more true value information are used to obtain monitoring results of the performance of the AI ​​model, the output of the AI ​​model is prediction information, and the time domain resource location of at least one of the one or more true value information is different from the time domain resource location of the prediction information.

[0034] In this aspect, the access network device indicates to the terminal device the time domain resource location of one or more reference signals used to measure true value information, so that the terminal device can obtain the true value information corresponding to any time domain resource location, thereby flexibly monitoring the performance of the prediction model.

[0035] In one possible implementation, the method further includes: obtaining one or more prediction information; and obtaining one or more monitoring results of the performance of the AI ​​model based on the one or more true value information and the one or more prediction information.

[0036] In another possible implementation, the method further includes: receiving a monitoring result report, wherein the monitoring result report includes one or more monitoring results, and the one or more monitoring results are obtained based on the one or more true value information and the one or more prediction information.

[0037] In another possible implementation, the method includes: sending second information, the second information including one or more second indexes corresponding to the one or more prediction information respectively, or the second information including a relative time unit value of each of the one or more prediction information relative to the time unit corresponding to the reference resource; and receiving the one or more prediction information.

[0038] In yet another possible implementation, the one or more first indexes and the one or more second indexes are based on a time unit corresponding to a reference resource.

[0039] In another possible implementation, the time unit corresponding to the reference resource is any one of the following: the first time unit where the resource for reporting the monitoring result report is located; or the second time unit where the downlink control signaling for scheduling the model monitoring is located; or a third time unit determined based on the first time unit or the second time unit; or a predefined / preconfigured fourth time unit.

[0040] In another possible implementation, the monitoring result report includes the one or more monitoring results; or the monitoring result report includes at least one monitoring result among the one or more monitoring results that is greater than or equal to a first threshold; or the monitoring result report includes at least one monitoring result among the one or more monitoring results that is less than or equal to a second threshold; or the monitoring result report includes an index of prediction information corresponding to the best monitoring result among the one or more monitoring results and an index of true value information; or the monitoring result report includes an index of prediction information corresponding to the worst monitoring result among the one or more monitoring results and an index of true value information; or the monitoring result report includes the absolute value of a first monitoring result among multiple monitoring results, and the relative values ​​of other monitoring results among the multiple monitoring results except the first monitoring result relative to the first monitoring result.

[0041] In another possible implementation, the first monitoring result is determined based on an order of the true value information used and / or an order of the prediction information used when acquiring the multiple monitoring results.

[0042] In another possible implementation, the access network device may also instruct the terminal device to report part of the monitoring results. For example, the terminal device may be instructed to report part of the monitoring results among all the above monitoring results in a bitmap manner.

[0043] In another possible implementation, the monitoring result report and the information feedback for sending the reference signal measurement result are reported in the same information report, or the monitoring result report and the information feedback for sending the reference signal measurement result are reported in different information reports.

[0044] In any one of the above-mentioned first to fourth aspects, the predicted information is predicted channel state information (CSI), and the true information is true CSI; or the predicted information is predicted beam information, and the true information is true beam information.

[0045] The information of the predicted beam may be one or more of the predicted optimal beam or the reference signal receiving power (RSRP) of the predicted optimal beam. The information of the true beam may be one or more of the actual optimal beam or the RSRP of the actual optimal beam. In other words, the true beam information includes at least one of the following: the true optimal beam, the true reference signal receiving power.

[0046] The methods of the first and second aspects mentioned above can be executed by a terminal device, or by a module (such as a processor, chip, or chip system, etc.) applied to the terminal device, or by a logical node, logical module or software that can realize all or part of the functions of the terminal device.

[0047] The methods of the third and fourth aspects mentioned above can be executed by an access network device, or by a module (such as a processor, chip, or chip system, etc.) applied to the access network device, or by a logical node, logical module or software that can realize all or part of the functions of the access network device.

[0048] In a fifth aspect, a model monitoring device is provided, which is used to implement the model monitoring method in the above-mentioned first aspect or any one of the implementations of the first aspect; or, it is used to implement the model monitoring method in the above-mentioned second aspect or any one of the implementations of the second aspect. The device can be a terminal device, or a module applied to the terminal device (such as a processor, a chip, or a chip system, etc.), or a logical node, a logical module or software that can implement all or part of the functions of the terminal device. In one implementation, the model monitoring device may include a sending unit, a receiving unit, and may also include a processing unit. The sending unit and the receiving unit may be independent or combined together (which may be referred to as a "transceiver unit").

[0049] In a sixth aspect, a model monitoring device is provided, which is used to implement the model monitoring method in the third aspect or any one of the implementations of the third aspect; or, it is used to implement the model monitoring method in the fourth aspect or any one of the implementations of the fourth aspect. The device can be an access network device, or a module (such as a processor, a chip, or a chip system, etc.) applied to the access network device, or a logical node, a logical module or software that can implement all or part of the functions of the access network device. In one implementation, the model monitoring device may include a sending unit, a receiving unit, and may also include a processing unit. The sending unit and the receiving unit may be independent or combined together (which may be referred to as a "transceiver unit").

[0050] In a possible implementation, the model monitoring device in the fifth to sixth aspects includes a module for respectively executing the method in any one of the first to fourth aspects or any one of their implementations.

[0051] In another possible implementation, the model monitoring device in the fifth to sixth aspects above includes a processor coupled to a memory; the processor is configured to enable the device to perform the corresponding functions in the above model monitoring method. The memory is used to couple with the processor, which stores the necessary programs (instructions) and / or data for the device. Optionally, the model monitoring device may also include a communication interface for realizing communication between the device and other network elements. Optionally, the memory may be located inside the model monitoring device or outside the model monitoring device.

[0052] In another possible implementation, the model monitoring device in the fifth to sixth aspects includes a processor and a transceiver, the processor is coupled to the transceiver, and the processor is used to execute a computer program or instruction to control the transceiver to receive and send information; when the processor executes the computer program or instruction, the processor is also used to implement the above method through a logic circuit or execute code instructions. The transceiver can be a transceiver, a transceiver circuit or an input-output interface, which is used to receive signals from other model monitoring devices other than the model monitoring device and transmit them to the processor or send signals from the processor to other model monitoring devices other than the model monitoring device. When the model monitoring device is a chip, the transceiver is a transceiver circuit or an input-output interface.

[0053] When the model monitoring device in the fifth to sixth aspects is a chip, the sending unit may be an output unit, such as an output circuit or a communication interface; the receiving unit may be an input unit, such as an input circuit or a communication interface. When the model monitoring device is a terminal, the sending unit may be a transmitter or a transmitter; the receiving unit may be a receiver or a receiver.

[0054] In a seventh aspect, a computer-readable storage medium is provided, wherein a computer program or instruction is stored in the computer-readable storage medium, and when the computer program or instruction is executed, the methods described in the above aspects are implemented.

[0055] According to an eighth aspect, a computer program product comprising instructions is provided. When the instructions are executed on a model monitoring device, the model monitoring device executes the methods described in the above aspects.

[0056] In a ninth aspect, a model monitoring system is provided, which includes the model monitoring device described in the fifth aspect and the model monitoring device described in the sixth aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A simplified schematic diagram of a wireless communication system provided by an embodiment of the present application;

[0058] Figure 2A A schematic diagram of the structure of a neuron;

[0059] Figure 2B A schematic diagram of a neural network;

[0060] Figure 2C It is an AI application framework;

[0061] Figure 3 A schematic diagram of a communication system in which an AI network element is introduced;

[0062] Figure 4A to Figure 4D A schematic diagram of a network architecture of an embodiment of the present application;

[0063] Figure 5 A flow chart of a model monitoring method provided in an embodiment of the present application;

[0064] Figure 6 A flowchart of another model monitoring method provided in an embodiment of the present application;

[0065] Figure 7 A schematic diagram of predicting CSI based on an AI model according to an example of an embodiment of the present application;

[0066] Figure 8 A flowchart of another model monitoring method provided in an embodiment of the present application;

[0067] Fig. 9 A schematic diagram of reference resources for illustrative embodiments of the present application;

[0068] Fig.10 A flowchart of another model monitoring method provided in an embodiment of the present application;

[0069] Fig.11 A flowchart of another model monitoring method provided in an embodiment of the present application;

[0070] Fig.12 A flowchart of another model monitoring method provided in an embodiment of the present application;

[0071] Fig.13 A flowchart of another model monitoring method provided in an embodiment of the present application;

[0072] Fig.14 This is a schematic diagram of reporting monitoring results according to an example of an embodiment of the present application;

[0073] Fig.15 A flowchart of another model monitoring method provided in an embodiment of the present application;

[0074] Fig.16 A flowchart of another model monitoring method provided in an embodiment of the present application;

[0075] Fig.17 A flowchart of another model monitoring method provided in an embodiment of the present application;

[0076] Fig.18 A schematic diagram of the structure of a model monitoring device provided in an embodiment of the present application;

[0077] Fig.19 A schematic diagram of the structure of another model monitoring device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0078] The embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application.

[0079] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings.

[0080] The at least one (item) involved in the present application as follows indicates one (item) or more (items). More than one (item) refers to two (items) or more than two (items). "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. In addition, it should be understood that although the terms first, second, etc. may be used to describe each object in the present application, these objects should not be limited to these terms. These terms are only used to distinguish each object from each other.

[0081] The terms "including" and "having" and any variations thereof mentioned in the following description of the present application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes other steps or units that are not listed, or optionally includes other steps or units that are inherent to these processes, methods, products or devices. It should be noted that in the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any method or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other methods or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way.

[0082] The technology provided by the present application can be applied to various communication systems. For example, the communication system can be a fourth generation (4 th generation, 4G) communication systems (such as long term evolution (LTE) systems), fifth generation (5 th The 5G communication system may also be referred to as a new radio (NR) system.

[0083] A network element in a communication system can send a signal to another network element or receive a signal from another network element. The signal may include information, signaling, or data, etc. The network element may also be replaced by an entity, a network entity, a device, a terminal device, a communication module, a node, a communication node, etc. The network element is used as an example for description in this application. For example, a communication system may include at least one terminal device and at least one access network device. The access network device may send a downlink signal to the terminal device, and / or the terminal device may send an uplink signal to the access network device. In addition, it can be understood that if a plurality of terminal devices are included in the communication system, the plurality of terminal devices may also send signals to each other, that is, the signal sending network element and the signal receiving network element may both be terminal devices.

[0084] The model monitoring method provided in the embodiment of the present application can be applied to wireless communication systems such as 5G, 6G, and satellite communications. Figure 1 , Figure 1 A simplified schematic diagram of a wireless communication system provided in an embodiment of the present application. Figure 1 As shown, the wireless communication system includes a wireless access network 100. The wireless access network 100 may be a next generation (e.g., 6G or higher) wireless access network, or a traditional (e.g., 5G, 4G) wireless access network. One or more terminal devices (120a-120j, collectively referred to as 120) may be connected to each other, or to one or more network devices (110a, 110b, collectively referred to as 110) in the wireless access network 100. Optionally, Figure 1 This is just a schematic diagram. The wireless communication system may also include other devices, such as core network devices, wireless relay devices and / or wireless backhaul devices. Figure 1 Not drawn in.

[0085] Optionally, in practical applications, the wireless communication system may include multiple network devices (also referred to as access network devices) at the same time, and may also include multiple terminal devices at the same time. A network device may serve one or more terminal devices at the same time. A terminal device may also access one or more network devices at the same time. The embodiment of the present application does not limit the number of terminal devices and network devices included in the wireless communication system.

[0086] The network device may be an entity on the network side for transmitting or receiving signals. The network device may be an access device for a terminal device to access the wireless communication system in a wireless manner, such as a base station. The base station can broadly cover the following various names, or be replaced with the following names, such as: RAN node, NodeB, evolved NodeB (eNB), next generation NodeB (gNB), access network equipment in open radio access network (O-RAN), relay station, access point, transmission point (transmitting and receiving point, TRP), transmission point (transmitting point, TP), master station MeNB, secondary station SeNB, multi-standard radio (MSR) node, home base station, network controller, access node, wireless node, access point (AP), transmission node, transceiver node, baseband unit (BBU), remote radio unit (RRU), active antenna unit (AAU), remote radio head (RRH), centralized unit (CU), distributed unit (DU), radio unit (RU), centralized unit control plane (CU The base station may be a macro base station, a micro base station, a relay node, a donor node or the like, or a combination thereof. The network device may also refer to a communication module, a modem or a chip that is arranged in the aforementioned device or apparatus. The network device may also be a mobile switching center and a device that performs the base station function in device-to-device (D2D), vehicle-to-everything (V2X), and machine-to-machine (M2M) communications, a network side device in a 6G network, and a device that performs the base station function in future communication systems. The network device may support networks with the same or different access technologies. The embodiments of the present application do not limit the specific technology and specific device form adopted by the network device.

[0087] The network equipment may be fixed or mobile. For example, the base stations 110 a and 110 b are stationary and are responsible for wireless transmission and reception in one or more cells from the terminal equipment 120 . Figure 1 The helicopter or drone 120i shown in the figure can be configured to act as a mobile base station, and one or more cells can move according to the location of the mobile base station 120i. In other examples, the helicopter or drone (120i) can be configured to act as a terminal device communicating with the base station 110b.

[0088] In the present application, the communication device used to implement the above access network function can be an access network device, or a network device with some functions of accessing the network, or a device capable of supporting the implementation of the access network function, such as a chip system, a hardware circuit, a software module, or a hardware circuit plus a software module, which can be installed in the access network device or used in combination with the access network device. In the method of the present application, the communication device used to implement the access network device function is an access network device for example.

[0089] The terminal device can be an entity on the user side for receiving or transmitting signals, such as a mobile phone. The terminal device can be used to connect people, objects and machines. The terminal device can communicate with one or more core networks through a network device. The terminal device includes a handheld device with a wireless connection function, other processing devices connected to a wireless modem, or a vehicle-mounted device. The terminal device can be a portable, pocket-sized, handheld, computer-built-in or vehicle-mounted mobile device. The terminal device 120 can be widely used in various scenarios, such as cellular communication, D2D, V2X, point-to-point (P2P), machine-to-machine (M2M), machine type communication (MTC), Internet of Things (IoT), virtual reality (VR), augmented reality (AR), industrial control, autonomous driving, telemedicine, smart grid, smart furniture, smart office, smart wear, smart transportation, smart city, drone, robot, remote sensing, passive sensing, positioning, navigation and tracking, autonomous delivery and mobility, etc.Some examples of the terminal device 120 are: user equipment (UE) of 3GPP standard, fixed equipment, mobile equipment, handheld equipment, wearable equipment, cellular phone, smart phone, session initiation protocol (SIP) phone, laptop, personal computer, smart book, vehicle, satellite, global positioning system (GPS) equipment, target tracking equipment, drone, helicopter, aircraft, ship, remote control equipment, smart home equipment, industrial equipment, personal communication service (PCS) phone, wireless local loop (WLL) station, personal digital assistant (PDA), wireless network camera, tablet computer, PDA, mobile internet device (MID), wearable equipment such as smart watch, VR equipment, AR equipment, wireless terminal in industrial control, terminal in vehicle networking system, wireless terminal in self driving, wireless terminal in smart grid, wireless terminal in transportation safety, smart city, etc. The terminal device 120 may be a wireless terminal in a smart city, such as a smart gas pump, a terminal device on a high-speed rail, and a wireless terminal in a smart home, such as a smart speaker, a smart coffee machine, a smart printer, etc. The terminal device 120 may be a wireless device in the above various scenarios or a device used to be set in a wireless device, for example, a communication module, a modem or a chip in the above device. The terminal device may also be referred to as a terminal, a terminal device, a user equipment (UE), a mobile station (MS), a mobile terminal (MT), etc. The terminal device may also be a terminal device in a future wireless communication system. The terminal device may be used in a dedicated network device or a general device. The embodiments of the present application do not limit the specific technology and specific device form adopted by the terminal device.

[0090] Optionally, the terminal device may be used to act as a base station. For example, the UE may act as a scheduling entity that provides sidelink signals between UEs in V2X, D2D, or P2P, etc. Figure 1As shown, the cell phone 120a and the car 120b communicate with each other using a sidelink signal. The cell phone 120a and the smart home device 120e communicate with each other without relaying the communication signal through the base station 110b.

[0091] In the present application, the communication device for realizing the functions of the terminal device may be a terminal device, or a terminal device having some functions of the above terminal devices, or a device capable of supporting the functions of the above terminal devices, such as a chip system, which may be installed in the terminal device or used in combination with the terminal device. In the present application, the chip system may be composed of a chip, or may include a chip and other discrete devices. In the technical solution provided in the present application, the communication device is described as a terminal device or UE as an example.

[0092] Optionally, a wireless communication system is usually composed of cells, and the base station provides management of the cell. The base station provides communication services to multiple mobile stations (MS) in the cell. The base station includes a baseband unit (BBU) and a remote radio unit (RRU). The BBU and RRU can be placed in different places, for example: the RRU is remote and placed in an area with high traffic volume, and the BBU is placed in a central computer room. The BBU and RRU can also be placed in the same computer room. The BBU and RRU can also be different components under one rack. Optionally, a cell can correspond to a carrier or a component carrier.

[0093] In some deployments, the network device mentioned in the embodiments of the present application may be a device including a CU, or a DU, or a device including a CU and a DU, or a control plane CU node (central unit control plane (central unit-control plane, CU-CP)) and a user plane CU node (central unit user plane (central unit-user plane, CU-UP)) and a DU node. For example, the network device may include a gNB-CU-CP, a gNB-CU-UP, and a gNB-DU.

[0094] In some deployments, multiple RAN nodes collaborate to assist the terminal in achieving wireless access, and different RAN nodes implement part of the functions of the base station. For example, the RAN node can be a CU, DU, CU-CP, CU-UP, or RU. The CU and DU can be set separately, or can also be included in the same network element, such as a BBU. The RU can be included in a radio frequency device or a radio frequency unit, such as an RRU, AAU, or RRH.

[0095] The RAN node may support one or more types of fronthaul interfaces, and different fronthaul interfaces correspond to DUs and RUs with different functions. If the fronthaul interface between the DU and the RU is a common public radio interface (CPRI), the DU is configured to implement one or more baseband functions, and the RU is configured to implement one or more radio frequency functions. If the fronthaul interface between the DU and the RU is another interface, relative to the CPRI, part of the downlink and / or uplink baseband functions, such as, for downlink, one or more of precoding, digital beamforming (BF), or inverse fast Fourier transform (IFFT) / adding cyclic prefix (CP), are moved from the DU to the RU for implementation, and for uplink, one or more of digital beamforming (BF), or fast Fourier transform (IFFT) / removing cyclic prefix (CP), are moved from the DU to the RU for implementation. In a possible implementation, the interface may be an enhanced common public radio interface (eCPRI). In the eCPRI architecture, the division between DU and RU is different, corresponding to different types (category, Cat) of eCPRI, such as eCPRI Cat A, B, C, D, E, F.

[0096] Taking eCPRI Cat A as an example, for downlink transmission, based on layer mapping, the DU is configured to implement layer mapping and one or more functions before it (i.e., one or more functions of coding, rate matching, scrambling, modulation, and layer mapping), while other functions after layer mapping (e.g., RE mapping, digital beamforming (BF), or one or more functions of inverse fast Fourier transform (IFFT) / adding cyclic prefix (CP)) are moved to the RU for implementation. For uplink transmission, based on de-RE mapping, the DU is configured to implement de-mapping and one or more functions before it (i.e., one or more functions of decoding, de-rate matching, de-scrambling, demodulation, inverse discrete Fourier transform (IDFT), channel equalization, and de-RE mapping), while other functions after de-mapping (e.g., one or more functions of digital BF or fast Fourier transform (FFT) / removing CP) are moved to the RU for implementation. It can be understood that for the functional description of DU and RU corresponding to various types of eCPRI, reference can be made to the eCPRI protocol and will not be repeated here.

[0097] In one possible design, the processing unit for implementing the baseband function in the BBU is called a baseband high layer (BBH) unit, and the processing unit for implementing the baseband function in the RRU / AAU / RRH is called a baseband low layer (BBL) unit.

[0098] In different systems, CU (or CU-CP and CU-UP), DU or RU may also have different names, but those skilled in the art can understand their meanings. For example, in the ORAN system, CU may also be called O-CU (open CU), DU may also be called O-DU, CU-CP may also be called O-CU-CP, CU-UP may also be called O-CU-UP, and RU may also be called O-RU. Any unit in the CU (or CU-CP, CU-UP), DU and RU in this application may be implemented by a software module, a hardware module, or a combination of a software module and a hardware module.

[0099] In the embodiments of the present application, the device for realizing the function of the network device may be a network device; or it may be a device capable of supporting the network device to realize the function, such as a chip system, a hardware circuit, a software module, or a hardware circuit plus a software module. The device may be installed in the network device or used in combination with the network device. In the embodiments of the present application, only the device for realizing the function of the network device is a network device as an example for explanation, and the scheme of the embodiments of the present application is not limited.

[0100] It can be understood that the present application can be applied between access network equipment and terminal equipment.

[0101] Protocol layer structure between access network equipment and terminal equipment

[0102] The communication between the access network device and the terminal device follows a certain protocol layer structure. The protocol layer structure may include a control plane protocol layer structure and a user plane protocol layer structure. For example, the control plane protocol layer structure may include the functions of the protocol layers such as the radio resource control (RRC) layer, the packet data convergence protocol (PDCP) layer, the radio link control (RLC) layer, the medium access control (MAC) layer and the physical layer. For example, the user plane protocol layer structure may include the functions of the protocol layers such as the PDCP layer, the RLC layer, the MAC layer and the physical layer. In a possible implementation, the service data adaptation protocol (SDAP) layer may also be included above the PDCP layer.

[0103] Optionally, the protocol layer structure between the access network device and the terminal device may also include an artificial intelligence (AI) layer for transmitting data related to AI functions.

[0104] Taking data transmission between access network equipment and terminal equipment as an example, data transmission needs to pass through the user plane protocol layer, such as the SDAP layer, PDCP layer, RLC layer, MAC layer, and physical layer. Among them, the SDAP layer, PDCP layer, RLC layer, MAC layer, and physical layer can also be collectively referred to as the access layer. According to the transmission direction of the data, it is divided into sending or receiving, and each of the above layers is divided into a sending part and a receiving part. Taking downlink data transmission as an example, after the PDCP layer obtains data from the upper layer, it transmits the data to the RLC layer and the MAC layer, and then the MAC layer generates a transmission block, and then transmits it wirelessly through the physical layer. The data is encapsulated accordingly in each layer. For example, the data received by a layer from the upper layer of the layer is regarded as the service data unit (SDU) of the layer, which becomes a protocol data unit (PDU) after being encapsulated by the layer, and then passed to the next layer.

[0105] Exemplarily, the terminal device may also have an application layer and a non-access layer. The application layer may be used to provide services to applications installed in the terminal device. For example, downlink data received by the terminal device may be sequentially transmitted from the physical layer to the application layer, and then provided to the application by the application layer; for another example, the application layer may obtain data generated by the application, and sequentially transmit the data to the physical layer and send it to other communication devices. The non-access layer may be used to forward user data, such as forwarding uplink data received from the application layer to the SDAP layer, or forwarding downlink data received from the SDAP layer to the application layer.

[0106] In order to support AI technology in wireless networks, AI nodes may also be introduced into the network.

[0107] Optionally, the AI ​​node can be deployed in one or more of the following locations in the communication system: access network equipment, terminal equipment, or core network equipment, etc., or the AI ​​node can also be deployed separately, for example, deployed in a location other than any of the above devices, such as a host or cloud server in an over the top (OTT) system. The AI ​​node can communicate with other devices in the communication system, and the other devices can be, for example, one or more of the following: network equipment, terminal equipment, or network elements of the core network, etc.

[0108] It is understood that the present application does not limit the number of AI nodes. For example, when there are multiple AI nodes, the multiple AI nodes can be divided based on functions, such as different AI nodes are responsible for different functions.

[0109] It can also be understood that AI nodes can be independent devices, or they can be integrated into the same device to implement different functions, or they can be network elements in hardware devices, or they can be software functions running on dedicated hardware, or they can be virtualized functions instantiated on a platform (for example, a cloud platform). This application does not limit the specific form of the above-mentioned AI nodes.

[0110] An AI node can be an AI network element or an AI module.

[0111] One or more AI modules are provided in one or more devices of these network element nodes, such as core network equipment, access network nodes (RAN nodes), terminals or OAM. The access network node can be a separate RAN node, or it can include multiple RAN nodes, for example, including CU and DU. The CU and / or DU can also be provided with one or more AI modules. Optionally, the CU can also be split into CU-CP and CU-UP. One or more AI models are provided in the CU-CP and / or CU-UP.

[0112] The AI ​​module is used to implement the corresponding AI function. The AI ​​modules deployed in different network elements may be the same or different. The model of the AI ​​module can implement different functions according to different parameter configurations. The model of the AI ​​module can be configured based on one or more of the following parameters: structural parameters (such as the number of neural network layers, the width of the neural network, the connection relationship between layers, the weight of the neuron, the activation function of the neuron, or at least one of the biases in the activation function), input parameters (such as the type of input parameters and / or the dimension of input parameters), or output parameters (such as the type of output parameters and / or the dimension of output parameters). Among them, the bias in the activation function can also be called the bias of the neural network.

[0113] An AI module can have one or more models. A model can be inferred to obtain an output, which includes one parameter or multiple parameters. The learning process, training process, or inference process of different models can be deployed in different nodes or devices, or can be deployed in the same node or device.

[0114] The communication system includes a RAN intelligent controller (RIC). For example, the RIC may be the above-mentioned AI module, which is used to implement AI-related functions. The RIC includes a near-real time RIC (near-real time RIC, near-RT RIC) and a non-real time RIC (non-real time RIC, Non-RT RIC). Among them, the non-real time RIC mainly processes non-real-time information, such as data that is not sensitive to delay, and the delay of the data may be in the order of seconds. The real-time RIC mainly processes near-real-time information, such as data that is relatively sensitive to delay, and the delay of the data is in the order of tens of milliseconds.

[0115] Near real-time RIC is used for model training and reasoning. For example, it is used to train an AI model and use the AI ​​model for reasoning. Near real-time RIC can obtain information on the network side and / or the terminal side from RAN nodes (such as CU, CU-CP, CU-UP, DU and / or RU) and / or terminals. This information can be used as training data or reasoning data. Optionally, near real-time RIC can submit the reasoning results to the RAN node and / or terminal. Optionally, the reasoning results can be exchanged between the CU and DU, and / or between the DU and RU. For example, the near real-time RIC submits the reasoning results to the DU, and the DU sends it to the RU.

[0116] Non-real-time RIC is also used for model training and reasoning. For example, it is used to train an AI model and use the model for reasoning. Non-real-time RIC can obtain information on the network side and / or the terminal side from RAN nodes (such as CU, CU-CP, CU-UP, DU and / or RU) and / or terminals. This information can be used as training data or reasoning data, and the reasoning results can be submitted to the RAN node and / or the terminal. Optionally, the reasoning results can be exchanged between the CU and the DU, and / or between the DU and the RU. For example, the non-real-time RIC submits the reasoning results to the DU, and the DU sends it to the RU.

[0117] The near real-time RIC and the non-real-time RIC may also be separately set as a network element. Optionally, the near real-time RIC and the non-real-time RIC may also be part of other devices, for example, the near real-time RIC is set in a RAN node (for example, in a CU or DU), and the non-real-time RIC is set in an OAM, a cloud server, a core network device, or other network devices.

[0118] For example, the configuration of near real-time RIC and non-real-time RIC in the network architecture can be as follows: Figure 4A to Figure 4D As shown:

[0119] like Figure 4AAs shown in (a), in a first possible implementation, the access network device includes a near real-time RIC module for model learning and / or reasoning.

[0120] like Figure 4A As shown in (b), in a second possible implementation, in a communication system, a non-real-time RIC may be included outside the access network device. Optionally, the non-real-time RIC may be located in the OAM or in the core network device.

[0121] like Figure 4A As shown in (c), in a third possible implementation, the access network device includes a near real-time RIC, and the access network device also includes a non-real-time RIC. Optionally, the non-real-time RIC may be located in the OAM or in the core network device.

[0122] relatively Figure 4A (c) in Figure 4B The CU is separated into CU-CP and CU-UP. The near real-time RIC and non-real-time RIC are set in Figure 4A Same as (c) in .

[0123] like Figure 4C As shown, optionally, the access network device includes one or more AI entities, and the function of the AI ​​entity is similar to the above-mentioned near real-time RIC. Optionally, the OAM includes one or more AI entities, and the function of the AI ​​entity is similar to the above-mentioned non-real-time RIC. Optionally, the core network device includes one or more AI entities, and the function of the AI ​​entity is similar to the above-mentioned non-real-time RIC. When both the OAM and the core network device include AI entities, the models trained by their respective AI entities are different, and / or the models used for reasoning are different. In the present application, the difference in models may include at least one of the following differences: structural parameters of the model (such as the number of layers of the model, and / or weights, etc.), input parameters of the model, or output parameters of the model.

[0124] relatively Figure 4C , Figure 4D The access network devices in the embodiment are separated into CU and DU. Optionally, the CU may include an AI entity, and the function of the AI ​​entity is similar to the above-mentioned near real-time RIC. Optionally, the DU may include an AI entity, and the function of the AI ​​entity is similar to the above-mentioned near real-time RIC. When both the CU and the DU include AI entities, the models trained by their respective AI entities are different, and / or the models used for reasoning are different. Optionally, the Figure 4D The CU in the CU is split into CU-CP and CU-UP. Optionally, one or more AI models can be deployed in the CU-CP. And / or, one or more AI models can be deployed in the CU-UP. Optionally, Figure 4C or Figure 4DIn the example, the OAM of access network equipment and the OAM of core network equipment can be deployed separately and independently.

[0125] It should be understood that Figure 1 The number and type of each device in the communication system shown are for illustration only, and the present application is not limited thereto. In actual applications, the communication system may also include more terminal devices, more access network devices, and other network elements, such as core network devices, and / or network elements for implementing artificial intelligence functions.

[0126] It is understandable that all or part of the functions implemented by one or more of the terminal equipment, access network equipment, core network equipment, or network elements used to implement artificial intelligence functions can be virtualized, that is, implemented by one or more of the proprietary processors or general-purpose processors and the corresponding software modules. Among them, the terminal equipment and the access network equipment involve interfaces for air interface transmission, and the transceiver functions of the interfaces can be implemented by hardware. Core network equipment, such as operation administration and maintenance (OAM) network elements, can be virtualized. Optionally, one or more functions of the virtualized terminal equipment, access network equipment, core network equipment, or network elements used to implement artificial intelligence functions can be implemented by cloud devices, such as cloud devices in over the top (OTT) systems.

[0127] For ease of understanding, the AI ​​technology involved in this application is first introduced below. It is understandable that this introduction is not intended to limit this application.

[0128] (1) AI Model

[0129] AI refers to the intelligence displayed by machines created by humans. Usually, artificial intelligence refers to the technology that presents human intelligence through ordinary computer programs. Artificial intelligence can be defined as machines or computers that imitate humans and have cognitive functions related to human thinking, such as learning and problem solving. Artificial intelligence is able to learn from past experiences, make reasonable decisions, and respond quickly. The goal of artificial intelligence is to understand intelligence by building computer programs with symbolic reasoning or reasoning.

[0130] Machine learning is a way to achieve artificial intelligence, that is, to use machine learning as a means to solve problems in artificial intelligence. Machine learning theory mainly designs and analyzes some algorithms that allow computers to "learn" automatically. Machine learning algorithms are a type of algorithm that automatically analyzes data to obtain patterns and uses the patterns to predict unknown data. Because learning algorithms involve a lot of statistical theories, machine learning is particularly closely related to inferential statistics, and is also called statistical learning theory.

[0131] An AI model is an algorithm or computer program that can realize AI functions. It is the specific implementation of AI technology functions. The AI ​​model represents the mapping relationship between the input and output of the model. The type of AI model can be a neural network, linear regression model, decision tree model, support vector machine (SVM), Bayesian network, Q learning model or other machine learning (ML) model.

[0132] (2) Neural Network

[0133] Neural networks are a specific implementation of AI or machine learning technology. According to the universal approximation theorem, neural networks can theoretically approximate any continuous function, which enables neural networks to have the ability to learn any mapping. Therefore, neural networks can accurately abstract and model complex high-dimensional problems.

[0134] The idea of ​​neural network comes from the neuron structure of brain tissue. For example, each neuron performs a weighted sum operation on its input value and outputs the operation result through an activation function. Figure 2A As shown in Figure 1, it is a schematic diagram of the neuron structure. Assume that the input of the neuron is x = [x0, x1, ..., x n ], and the weights corresponding to each input are w = [w, w1, ..., w n ], where w i As x i The weight of x i Weighted. The bias of weighted summation of input values ​​according to the weight is, for example, b. The activation function can take many forms. Assuming that the activation function of a neuron is: y = f(z) = max(0, z), the output of the neuron is:

[0135] For another example, the activation function of a neuron is: y = f(z) = z, then the output of the neuron is:

[0136] Among them, b, w i 、x i It can be a decimal, an integer (such as 0, a positive integer or a negative integer), or a complex number. The activation functions of different neurons in a neural network can be the same or different.

[0137] A neural network generally includes multiple layers, and each layer may include one or more neurons. By increasing the depth and / or width of the neural network, the expressive power of the neural network can be improved, providing more powerful information extraction and abstract modeling capabilities for complex systems. Among them, the depth of a neural network can refer to the number of layers included in the neural network, and the number of neurons included in each layer can be called the width of the layer. In one implementation, the neural network includes an input layer and an output layer. The input layer of the neural network processes the received input information through neurons, and passes the processing results to the output layer, and the output layer obtains the output result of the neural network. In another implementation, the neural network includes an input layer, a hidden layer, and an output layer, which can be referenced Figure 2B Schematic diagram of a neural network. The input layer of the neural network processes the received input information through neurons and passes the processing results to the middle hidden layer. The hidden layer calculates the received processing results to obtain the calculation results. The hidden layer passes the calculation results to the output layer or the adjacent hidden layer, and finally the output layer obtains the output result of the neural network. Among them, a neural network can include one hidden layer, or include multiple hidden layers connected in sequence, without limitation.

[0138] Taking the type of AI model as a neural network as an example, the AI ​​model involved in this application can be a deep neural network (DNN). Depending on the way the network is constructed, DNN can include feedforward neural networks (FNN), convolutional neural networks (CNN) and recurrent neural networks (RNN).

[0139] (3) Training Dataset and Inference Data

[0140] The training data set is used for training the AI ​​model. The training data set may include the input of the AI ​​model, or the input and target output of the AI ​​model. Among them, the training data set includes one or more training data, and the training data may be a training sample input to the AI ​​model, or it may be the target output of the AI ​​model. Among them, the target output may also be referred to as a label or a label sample. The training data set is one of the important parts of machine learning. Model training is essentially learning certain features from the training data so that the output of the AI ​​model is as close to the target output as possible, such as the difference between the output of the AI ​​model and the target output is as small as possible. The composition and selection of the training data set can determine the performance of the trained AI model to a certain extent.

[0141] In addition, during the training process of an AI model (such as a neural network), a loss function can be defined. The loss function describes the gap or difference between the output value of the AI ​​model and the target output value. This application does not limit the specific form of the loss function. The training process of the AI ​​model is to adjust the model parameters of the AI ​​model so that the value of the loss function is less than the threshold, or the value of the loss function meets the target requirements. For example, the AI ​​model is a neural network, and adjusting the model parameters of the neural network includes adjusting at least one of the following parameters: the number of layers, width, weights of neurons, or parameters in the activation function of neurons.

[0142] Inference data can be used as input to the trained AI model for inference. During the model inference process, the inference data is input into the AI ​​model, and the corresponding output is the inference result.

[0143] (4) AI model design

[0144] The design of AI models mainly includes data collection (such as collecting training data and / or inference data), model training, and model inference. It can also include the application of inference results. Figure 2C , illustrating an AI application framework. In the aforementioned data collection link, the data source is used to provide training data sets and reasoning data. In the model training link, the AI ​​model is obtained by analyzing or training the training data provided by the data source. Among them, the AI ​​model represents the mapping relationship between the input and output of the model. Learning the AI ​​model through the model training node is equivalent to learning the mapping relationship between the input and output of the model using the training data. In the model reasoning link, the AI ​​model trained through the model training link is used to reason based on the reasoning data provided by the data source to obtain the reasoning result. This link can also be understood as: inputting the reasoning data into the AI ​​model, and obtaining the output through the AI ​​model, which is the reasoning result. The reasoning result can indicate: the configuration parameters used (executed) by the execution object, and / or the operation performed by the execution object. The reasoning result is published in the reasoning result application link, for example, the reasoning result can be uniformly planned by the execution (actor) entity, for example, the execution entity can send the reasoning result to one or more execution objects (for example, core network equipment, access network equipment, or terminal equipment, etc.) for execution. For example, the execution entity can also provide feedback on the model's performance to the data source to facilitate subsequent model update and training.

[0145] It is understandable that a network element with artificial intelligence function may be included in the communication system. The above-mentioned AI model design-related links may be performed by one or more network elements with artificial intelligence functions. In one possible design, an AI function (such as an AI module or an AI entity) may be configured in an existing network element in the communication system to implement AI-related operations, such as training and / or reasoning of an AI model. For example, the existing network element may be an access network device (such as a gNB), a terminal device, a core network device, or a network management system. Among them, the network management system may divide the network management work into three categories according to the actual needs of the operator's network operation: operation, administration, and maintenance. The network management system may also be referred to as an operation administration and maintenance (OAM) network element, referred to as OAM. Operation mainly completes the analysis, prediction, planning and configuration of daily networks and services; maintenance is mainly daily operation activities such as testing and fault management of the network and its services. The network management system can detect the network operation status, optimize network connection and performance, improve network operation stability, and reduce network maintenance costs. Or in another possible design, an independent network element can also be introduced into the communication system to perform AI-related operations, such as training AI models. The independent network element can be called an AI network element or an AI node, etc., and this application does not limit this name. The AI ​​network element can be directly connected to the access network device in the communication system, or it can be indirectly connected to the access network device through a third-party network element. Among them, the third-party network element can be a core network element, OAM, cloud server or other network element such as an authentication management function (AMF) network element and a user plane function (UPF) network element, without limitation. For example, see Figure 3 The communication system includes an access network device 110, terminal devices 120, 130, and the above Figure 1 An AI network element 140 is introduced into the communication system shown.

[0146] In this application, a model can be inferred to obtain one parameter, or multiple parameters. The training process of different models can be deployed in different devices or nodes, or in the same device or node. The reasoning process of different models can be deployed in different devices or nodes, or in the same device or node.

[0147] Among them, the model parameters may include one or more of the following structural parameters of the model (such as the number of layers of the model, and / or weights, etc.), the input parameters of the model (such as input dimension, number of input ports), or the output parameters of the model (such as output dimension, number of output ports). It can be understood that the input dimension may refer to the size of an input data. For example, when the input data is a sequence, the input dimension corresponding to the sequence may indicate the length of the sequence. The number of input ports may refer to the number of input data. Similarly, the output dimension may refer to the size of an output data. For example, when the output data is a sequence, the output dimension corresponding to the sequence may indicate the length of the sequence. The number of output ports may refer to the number of output data. In view of the fact that the prediction performance obtained based on the prediction model proposed in the background technology may not reflect the actual system performance, the present application provides a model monitoring solution, and the terminal device can obtain a specific time unit, such as a required time unit, and the corresponding truth information by obtaining one or more first indexes of one or more truth information, so as to flexibly monitor the performance of the prediction model.

[0148] In this application, "sending information to... (such as a terminal device)" or the related illustrations in the accompanying drawings can be understood as the destination end of the information is the terminal device. It can include sending information to the terminal device directly or indirectly. "Receiving information from... (such as a terminal device)" or "receiving information from... (such as a terminal device)", or the related illustrations in the accompanying drawings can be understood as the source end of the information is the terminal device, which can include receiving information from the terminal device directly or indirectly. The information may be processed as necessary between the source end and the destination end of the information transmission, such as format changes, etc., but the destination end can understand the valid information from the source end. Similar expressions in this application can be understood similarly and will not be repeated here.

[0149] The model monitoring method provided by the embodiment of the present application is described in detail below. It can be understood that the present application uses the access network device and the terminal device as an example to illustrate the execution subject of the interaction diagram, but the present application does not limit the execution subject of the interaction diagram. For example, the access network device in the method provided by the present application may also be a chip, a chip system, or a processor applied to the access network device, and may also be a logical node, a logical module, or software that can implement all or part of the access network device; the terminal device in the method provided by the present application may also be a chip, a chip system, or a processor applied to the terminal device, and may also be a logical node, a logical module, or software that can implement all or part of the terminal device functions.

[0150] like Figure 5 FIG. 1 is a flow chart of a model monitoring method provided in an embodiment of the present application. Exemplarily, the method may include the following steps:

[0151] S501. The terminal device obtains first information.

[0152] The model monitoring involved in the embodiments of the present application refers to monitoring the performance of the AI ​​model and judging whether the AI ​​model is working normally. If the performance of the AI ​​model is poor, it is necessary to switch to non-AI mode, or replace the AI ​​model, or update the AI ​​model, etc. Model monitoring can be to monitor the accuracy of the output of the AI ​​model (also referred to as an intermediate KPI). Monitoring the accuracy of the output of the AI ​​model is to judge whether the performance of the AI ​​model meets the requirements by comparing the difference between the output of the AI ​​model and the corresponding label or true value. Intermediate KPIs generally include generalized cosine similarity (GCS), square generalized cosine similarity (SGCS), minimum mean square error (MSE), normalized mean square error (NMSE), etc.

[0153] Calculating the intermediate KPI of the AI ​​model requires two types of information: one is the information to be monitored, such as the prediction information output by the AI ​​model, and the other is the label information or true value information, that is, Where H is the true value information, True value information refers to information that can be considered accurate. For example, information measured by a terminal device relative to predicted information can be considered true value information. Generally used to represent H and The difference or similarity measure between them.

[0154] In this embodiment, the terminal device is instructed which truth value information to obtain through the index of the truth value information.

[0155] The terminal device obtains the first information. The first information includes one or more first indexes corresponding to one or more true value information. Exemplarily, the terminal device may obtain the first information sent by the access network device, or may obtain the first information pre-stored by the terminal device.

[0156] S502. The terminal device obtains one or more truth value information corresponding to one or more first indexes respectively.

[0157] After the terminal device obtains one or more first indexes corresponding to one or more true value information, it measures the reference signal on the time slot corresponding to the one or more indexes to obtain one or more true value information corresponding to the one or more first indexes.

[0158] The one or more true value information is used to obtain monitoring results of the performance of the AI ​​model. The output of the AI ​​model is prediction information.

[0159] Compared with the prior art which can only periodically obtain model monitoring results on grid points, the terminal device in this embodiment can obtain the truth information corresponding to a specific time unit by obtaining one or more first indexes of one or more truth information.

[0160] According to a model monitoring method provided in an embodiment of the present application, a terminal device can obtain truth information corresponding to a specific time unit by obtaining one or more first indexes corresponding to one or more truth information, thereby flexibly monitoring the performance of a prediction model.

[0161] like Figure 6 FIG. 1 is a flow chart of another model monitoring method provided in an embodiment of the present application. Figure 5 The embodiment shown is different in that this embodiment does not directly indicate which truth information the terminal device obtains through the index of the truth information, but determines the truth information according to the time domain resource position of the reference signal used to measure the truth information. Exemplarily, the method may include the following steps:

[0162] S601. The terminal device obtains first information.

[0163] The reference signal used to measure true value information (i.e., used for model monitoring) can be the same group of reference signals as the reference signal used for model inference (as input to the model, used to predict information), i.e., belong to the same reference signal configuration; the reference signal used to measure true value information can also be a different group of reference signals from the reference signal used for model inference, i.e., a reference signal is specially configured for model monitoring.

[0164] In this embodiment, the reference signal used to measure the true value information and the reference signal used for model inference are reference signals of different groups, and the terminal device can determine the true value information according to the reference signal used to measure the true value information. Therefore, the terminal device can obtain the first information, which includes the time domain resource location of one or more reference signals used to measure the true value information.

[0165] Exemplarily, the terminal device may obtain the above-mentioned first information sent by the access network device, or may obtain the above-mentioned first information pre-stored by the terminal device.

[0166] S602. The terminal device determines one or more true value information based on the first information.

[0167] After the terminal device obtains the first information, it can determine one or more true value information based on the first information.

[0168] In one implementation, the terminal device may measure, by default, one or more reference signals at the time domain resource locations of the one or more reference signals carried by the first information to obtain one or more true value information.

[0169] In another implementation, if the access network device configures the time domain resource location of one or more reference signals for measuring true value information through the first information, the terminal device can measure the one or more reference signals at the time domain resource location of the one or more reference signals carried by the first information by default to obtain one or more true value information.

[0170] In another implementation, the terminal device may also obtain the time domain resource position of one or more reference signals used for model inference. The terminal device may measure one or more reference signals at the time domain resource position of one or more reference signals carried by the first information, and measure one or more reference signals at the time domain resource position of one or more reference signals used for model inference by default, to obtain one or more true value information.

[0171] In another implementation, the access network device may also instruct the terminal device to measure one or more reference signals at the time domain resource location of one or more reference signals carried by the first information to obtain one or more true value information, or instruct the terminal device to measure one or more reference signals at the time domain resource location of one or more reference signals used for model inference to obtain one or more true value information.

[0172] The one or more true value information is used to obtain monitoring results of the performance of the AI ​​model. The output of the AI ​​model is prediction information.

[0173] It can be seen that, in this embodiment, the time domain resource position of at least one of the one or more obtained true value information is different from the time domain resource position of the predicted information.

[0174] According to a model monitoring method provided in an embodiment of the present application, a terminal device can obtain true value information corresponding to a specific time domain resource location by obtaining the time domain resource location of one or more reference signals used to measure true value information, thereby flexibly monitoring the performance of the prediction model.

[0175] The above scheme can be used to monitor the CSI prediction model. In a communication system (for example, an LTE communication system or an NR communication system, etc.), the network equipment needs to decide the resource, MCS, and precoding configuration of the downlink data channel of the terminal device based on the CSI. It can be understood that CSI is a kind of channel information, which is information that can reflect the channel characteristics and channel quality.

[0176] CSI measurement refers to the receiving end solving the channel information based on the reference signal sent by the transmitting end, that is, estimating the channel information using the channel estimation method. Exemplarily, the reference signal may include one or more of the channel state information reference signal (CSI-RS), synchronization signal / physical broadcast channel block (SSB), channel sounding reference signal (SRS) or demodulation reference signal (DMRS). CSI-RS, SSB and DMRS can be used to measure downlink CSI. SRS and DMRS can be used to measure uplink CSI.

[0177] Taking the FDD communication scenario as an example, in the FDD communication scenario, since the uplink and downlink channels are not reciprocal or cannot be guaranteed, the network equipment usually sends a downlink reference signal to the terminal device, and the terminal device performs channel measurement and interference measurement based on the received downlink reference signal to estimate the downlink CSI. The terminal device generates a CSI report according to the protocol predefined method or the network device configuration method, and feeds it back to the network device so that it can obtain the downlink CSI.

[0178] Exemplarily, CSI may include at least one of the following: channel quality indication (CQI), precoding matrix indicator (PMI), rank indicator (RI), CSI-RS resource indicator (CRI), layer indicator (LI), reference signal receiving power (RSRP) or signal to interference plus noise ratio (SINR), etc. The signal to interference plus noise ratio may also be referred to as signal to interference plus noise ratio.

[0179] Among them, RI is used to indicate the number of downlink transmission layers recommended by the terminal device, CQI is used to indicate the modulation and coding mode supported by the current channel conditions determined by the terminal device, and PMI is used to indicate the precoding recommended by the terminal device. The number of precoding layers indicated by PMI corresponds to RI.

[0180] It should be understood that the RI, CQI and PMI indicated in the above CSI report are only recommended values ​​for the terminal device, and the network device may perform downlink transmission according to part or all of the information indicated in the CSI report. Alternatively, the network device may not perform downlink transmission according to the information indicated in the CSI report.

[0181] AI-based CSI prediction refers to inputting some historical / current CSI information into the AI ​​prediction model and outputting the predicted future CSI. Figure 7 As shown, it is a schematic diagram of predicting CSI based on an AI model as an example of an embodiment of the present application. The two historical / current CSI information of t-5 and t can be input into the AI ​​model, and the AI ​​model can output the CSI information at the future time t+5. The predicted CSI can be in any of the following forms: channel response, channel matrix, channel feature matrix, precoding matrix. Among them, the channel response and channel matrix are the channels themselves; the channel feature matrix and the precoding matrix are matrices composed of the features extracted from the channel.

[0182] Calculating the intermediate KPI of the AI ​​model requires two types of CSI information: one is the CSI to be monitored, such as the predicted CSI output by the AI ​​model, and the other is the label CSI or the true value CSI, that is, Where H is the true value CSI, The true CSI means that the CSI can be considered accurate, for example, the CSI measured by the terminal device can be considered as the true CSI relative to the predicted CSI. Generally used to represent H and The difference or similarity measure between them.

[0183] As mentioned above, for example, the KPI may be GCS, SGCS, MSE or NMSE.

[0184] When the KPI is GCS, is the predicted CSI of each resource unit, w i is the true CSI of each resource unit.

[0185] When the KPI is SGCS, is the predicted CSI of each resource unit, w i is the true CSI of each resource unit.

[0186] When the KPI is MSE, is the predicted CSI of each resource unit, w i is the true CSI of each resource unit.

[0187] When the KPI is NMSE, is the predicted CSI of each resource unit, w i is the true CSI of each resource unit.

[0188] Depending on whether the terminal device stores an AI model, different embodiments are used to describe how to monitor the CSI prediction model:

[0189] like Figure 8 As shown, it is a flow chart of another model monitoring method provided in an embodiment of the present application. In this embodiment, the terminal device does not store an AI model. Exemplarily, the method may include the following steps:

[0190] S801. The terminal device obtains first information.

[0191] In this embodiment, the index of the true value CSI is used to instruct the terminal device which true value CSI to obtain.

[0192] The terminal device obtains the first information. The first information includes one or more first indexes corresponding to one or more true value CSIs. Exemplarily, the terminal device may obtain the first information sent by the access network device, or may obtain the first information pre-stored by the terminal device.

[0193] In one implementation, the terminal device can determine the index of the true CSI by the time of the reference resource. The time of the reference resource can be defined by the protocol or configured by the access network device, and the reference resource is a time unit in the time domain (the time unit includes but is not limited to any of the following: symbol, time slot, micro time slot, subframe, radio frame, and the embodiment of the present application is described by taking the time slot as an example). Fig. 9 As shown, an example of a reference resource is given, and the time slot corresponding to the reference resource is the t time slot. Taking the time slot where the reference resource is located as a reference, for example, the index of the nth true value CSI before the time slot where the reference resource is located is -n, and the index of the nth true value CSI before and after the time slot where the reference resource is located is n-1. For another example, the index of the nth true value CSI before and before the time slot where the reference resource is located is -n+1, ​​and the index of the nth true value CSI after the time slot where the reference resource is located is n. For another example, the index of the Kth true value CSI before the time slot where the reference resource is located is 0, and the indexes of the subsequent true value CSIs increase successively, and K is a predefined parameter or a parameter configured by the access network device. For another example, the index of the Kth true value CSI after the time slot where the reference resource is located is 0, and the indexes of the previous true value CSIs increase successively.

[0194] The one or more first indexes corresponding to the one or more true value CSIs included in the first information are indexes relative to the time slot where the reference resource is located. Fig. 9In the example shown, the time slot where the reference resource is located is the t time slot, then the index of the true value CSI of the t+2 time slot is 0, and the index of the true value CSI of the t+4 time slot is 1.

[0195] The access network device sends a channel state information-reference signal (CSI-RS) to the terminal device, and the terminal device receives and measures the CSI-RS to obtain the true CSI. The CSI-RS used to measure the true CSI (i.e., for model monitoring) can be the same group of CSI-RS as the CSI-RS used for model inference (as the input of the model, used to predict CSI), that is, they belong to the same CSI-RS configuration; the CSI-RS used to measure the true CSI can also be a different group of CSI-RS from the CSI-RS used for model inference, that is, a CSI-RS is specially configured for model monitoring.

[0196] In another implementation, the index of the true CSI may also be the index of the CSI-RS used to measure the true CSI. Fig. 9 , and also illustrates the time domain position of the CSI-RS used to measure the true CSI. The CSI-RS used for model inference is a CSI-RS with a period of 5ms (for example, the CSI-RS corresponding to the t-5, t, and t+5 time slots); the CSI-RS used for model monitoring is a non-periodic or semi-static CSI-RS, which is the CSI-RS corresponding to the t+2 and t+4 time slots. The index of the true CSI of the t+2 time slot is 0, the index of the true CSI of the t+4 time slot is 1, and the index of the true CSI of the t+5 time slot is 2.

[0197] In yet another implementation, the CSI-RS used to measure the true CSI may include only the CSI-RS used for monitoring, for example, Fig. 9 In the example shown, the CSI reference resource is in time slot t, the index of the true value CSI at time t+2 is 0, and the index of the true value CSI at time t+4 is 1.

[0198] In another implementation, the CSI-RS used to measure the true CSI may also include the CSI-RS used for monitoring and the CSI-RS at the time when the predicted CSI is located, for example, Fig. 9 In the example shown, the index of the true value CSI at time t+2 is 0, the index of the true value CSI at time t+4 is 1, and the index of the true value CSI at time t+5 is 2.

[0199] S802. The terminal device obtains one or more true value CSIs corresponding to one or more first indexes.

[0200] After the terminal device obtains one or more first indexes corresponding to one or more true CSIs, it measures the CSI-RS on the time slots corresponding to the one or more indexes to obtain one or more true CSIs corresponding to the one or more first indexes.

[0201] The one or more true CSIs are used to obtain monitoring results of the performance of the AI ​​model. The output of the AI ​​model is the predicted CSI.

[0202] Compared with the prior art which can only periodically obtain model monitoring results on the grid, the terminal device in this embodiment can obtain the true value CSI corresponding to a specific time unit, such as the required time unit, by obtaining one or more first indexes of one or more true value CSIs.

[0203] Furthermore, the method may also include the following steps (indicated by dotted lines in the figure):

[0204] S803. The terminal device sends one or more true value CSIs to the access network device. Correspondingly, the access network device receives the one or more true value CSIs.

[0205] In this embodiment, the terminal device does not store the AI ​​model and cannot perform CSI prediction. Therefore, after the terminal device obtains one or more true CSI values, it sends the one or more true CSI values ​​to the access network device.

[0206] S804. The access network device obtains one or more predicted CSIs.

[0207] The access network device stores an AI model and can use the received one or more true value CSIs as input information of the model to output one or more predicted CSIs.

[0208] For example, the access network device receives the true CSI of the time slots t+2 and t+4. The access network device may use the true CSI of the time slots t+2 and t+4 as input information and output the predicted CSI of the time slot t+5.

[0209] S805. The access network device obtains one or more monitoring results of the performance of the AI ​​model based on one or more true CSIs and one or more predicted CSIs.

[0210] The monitoring results of the performance of the AI ​​model are Where H is the true value CSI, For each true CSI, the access network device may obtain one or more monitoring results according to the true CSI and one or more predicted CSIs; or for each predicted CSI, the access network device may obtain one or more monitoring results according to one or more true CSIs.

[0211] Furthermore, in order to compare the predicted CSI and the true CSI, the access network device can also use the true CSI at time T1 as the predicted CSI at time T2. For example, assuming that the CSI-RS period is 5ms, and the downlink transmission is at time t+3, if it is not in the monitoring stage, there is no CSI-RS at time t+3. The access network device can use the true CSI at time t as the predicted CSI at time t+3, and obtain the model monitoring result at time t+3: KPI (H t+3 ,H t ).

[0212] In response to the problem raised in the background technology, since the true CSI at time t+3 can be obtained, the model performance at time t+3 can be obtained, so that the predicted CSI predicted by the model can reflect the actual system performance, and the predicted CSI can be used for communication.

[0213] According to a model monitoring method provided in an embodiment of the present application, a terminal device can obtain the true value CSI corresponding to a specific time unit by obtaining one or more first indexes corresponding to one or more true value CSIs, thereby flexibly monitoring the performance of the prediction model.

[0214] like Fig.10 FIG. 1 is a flow chart of another model monitoring method provided in an embodiment of the present application. In this embodiment, the terminal device does not store an AI model. Figure 8 The embodiment shown is different in that this embodiment does not directly indicate which true CSI the terminal device acquires by the index of the true CSI, but determines the true CSI according to the time domain resource position of the CSI-RS used to measure the true CSI. Exemplarily, the method may include the following steps:

[0215] S1001. The terminal device obtains first information.

[0216] The CSI-RS used to measure the true CSI (i.e., used for model monitoring) can be the same group of CSI-RS as the CSI-RS used for model inference (as input to the model, used to predict CSI), that is, they belong to the same CSI-RS configuration; the CSI-RS used to measure the true CSI can also be a CSI-RS in a different group from the CSI-RS used for model inference, that is, a CSI-RS is specially configured for model monitoring.

[0217] Still refer to Fig. 9, and also illustrates the time domain position of the CSI-RS used to measure the true CSI. The CSI-RS used for model inference is a CSI-RS with a period of 5ms (for example, the CSI-RS corresponding to the t-5, t, and t+5 time slots); the CSI-RS used for model monitoring is a non-periodic or semi-static CSI-RS, which is the CSI-RS corresponding to the t+2 and t+4 time slots, that is, the CSI-RS used to measure the true CSI and the CSI-RS used for model inference are CSI-RS of different groups.

[0218] In this embodiment, the CSI-RS used to measure the true CSI and the CSI-RS used for model inference are CSI-RSs of different groups, and the terminal device can determine the true CSI according to the CSI-RS used to measure the true CSI. Therefore, the terminal device can obtain the first information, which includes the time domain resource position of one or more CSI-RSs used to measure the true CSI.

[0219] Exemplarily, the terminal device may obtain the above-mentioned first information sent by the access network device, or may obtain the above-mentioned first information pre-stored by the terminal device.

[0220] S1002. The terminal device determines one or more true value CSIs based on the first information.

[0221] After the terminal device obtains the first information, it can determine one or more true value CSIs based on the first information.

[0222] In one implementation, the terminal device can measure one or more CSI-RS at the time domain resource position of one or more CSI-RS carried by the first information by default to obtain one or more true CSI values. Fig. 9 In the example, the terminal device can measure the CSI-RS at time t+2 and time t+4 to obtain the corresponding true CSI respectively.

[0223] In another implementation, if the access network device configures the time domain resource position of one or more CSI-RS for measuring the true CSI through the first information, the terminal device can measure the one or more CSI-RS at the time domain resource position of the one or more CSI-RS carried by the first information by default to obtain one or more true CSI. Fig. 9 In the example, the terminal device can measure the CSI-RS at time t+2 and time t+4 to obtain the corresponding true CSI respectively.

[0224] In another implementation, the terminal device may also obtain the time domain resource position of one or more CSI-RS used for model inference. The terminal device may measure one or more CSI-RS at the time domain resource position of one or more CSI-RS carried by the first information, and measure one or more CSI-RS at the time domain resource position of one or more CSI-RS used for model inference, to obtain one or more true CSIs. For example, Fig. 9 In the example, the terminal device can measure the CSI-RS at times t+2, t+4, and t+5 to obtain the corresponding true CSI respectively.

[0225] In another implementation, the access network device may also instruct the terminal device to measure one or more CSI-RS at the time domain resource position of one or more CSI-RS carried by the first information to obtain one or more true CSIs, or instruct the terminal device to measure one or more CSI-RS at the time domain resource position of one or more CSI-RS used for model inference to obtain one or more true CSIs.

[0226] The one or more true CSIs are used to obtain monitoring results of the performance of the AI ​​model. The output of the AI ​​model is the predicted CSI.

[0227] It can be seen that, in this embodiment, the time domain resource position of at least one true value CSI among the one or more true value CSIs obtained by the above implementation manner is different from the time domain resource position of the predicted CSI.

[0228] Furthermore, the method may also include the following steps (indicated by dotted lines in the figure):

[0229] S1003. The terminal device sends one or more true value CSIs to the access network device. Correspondingly, the access network device receives the one or more true value CSIs.

[0230] The specific implementation of this step can be found in Figure 8 Step S803 of the illustrated embodiment will not be described in detail here.

[0231] S1004. The access network device obtains one or more predicted CSIs.

[0232] Among them, the one or more predicted CSIs are the outputs of the above-mentioned AI model.

[0233] The specific implementation of this step can be found in Figure 8 Step S804 of the illustrated embodiment will not be described in detail here.

[0234] S1005. The access network device obtains one or more monitoring results of the performance of the AI ​​model based on the received one or more true CSIs and one or more predicted CSIs.

[0235] The specific implementation of this step can be found in Figure 8 Step S805 of the illustrated embodiment will not be described in detail here.

[0236] According to a model monitoring method provided in an embodiment of the present application, a terminal device can obtain the true CSI corresponding to a specific time domain resource position by obtaining the time domain resource position of one or more CSI-RS used to measure the true CSI, thereby flexibly monitoring the performance of the prediction model.

[0237] like Fig.11 FIG. 1 is a flow chart of another model monitoring method provided in an embodiment of the present application. Figure 8 , Fig.10 The difference from the embodiment shown is that in this embodiment, the terminal device stores an AI model that can predict CSI. Exemplarily, the method may include the following steps:

[0238] S1101. The terminal device obtains first information.

[0239] The first information includes one or more first indexes of one or more true value CSIs.

[0240] The specific implementation of this step can be found in Figure 8 Step S801 of the illustrated embodiment will not be described in detail here.

[0241] S1102. The terminal device obtains one or more true value CSIs corresponding to one or more first indexes.

[0242] The specific implementation of this step can be found in Figure 8 Step S802 of the illustrated embodiment will not be described in detail here.

[0243] S1103. The terminal device obtains one or more predicted CSIs.

[0244] This embodiment does not limit the order of the above steps S1101, S1102 and S1103. Steps S1101 and S1102 may be performed first, and then step S1103; steps S1101, S1102 and S1103 may also be performed simultaneously; or step S1103 may be performed first, and then steps S1101 and S1102.

[0245] In this embodiment, the access network device may instruct the terminal device to obtain the predicted CSI at which time points, and the terminal device obtains one or more predicted CSIs according to the instruction of the access network device.

[0246] Specifically, the access network device may send second information to the terminal device, and the terminal device acquires one or more predicted CSIs according to the second information.

[0247] In one implementation, the second information includes one or more second indexes of one or more predicted CSIs. The one or more second indexes are based on the time unit corresponding to the reference resource. Among them, the moment of the reference resource is defined by the protocol, and the reference resource is a time unit in the time domain (the time unit includes but is not limited to any of the following: symbol, time slot, micro time slot, subframe, radio frame, the embodiment of the present application is described by taking the time slot as an example).

[0248] The time unit corresponding to the reference resource is any of the following:

[0249] The first time unit of the resource that reports the monitoring result; or

[0250] a second time unit in which downlink control signaling (DCI) monitored by the scheduling model is located; or

[0251] a third time unit determined based on the first time unit or the second time unit; or

[0252] Predefined / preconfigured fourth time unit.

[0253] In some embodiments of the present application, after the terminal device obtains the predicted CSI, it can generate a monitoring result report of the AI ​​model and report the monitoring result report to the access network device in the first time unit. The time unit corresponding to the reference resource can be the first time unit where the resource for reporting the monitoring result report is located. For example, the time slot corresponding to the reference resource is the time slot where the Ath CSI-RS is located before or after the time slot where the resource for reporting the monitoring result report is located, where A is a parameter configured or predefined by the access network device. For example, A=1.

[0254] In some embodiments of the present application, the access network device may schedule the terminal device to perform model monitoring. Specifically, the access network device sends a DCI to the terminal device in a second time unit, and the DCI is used to instruct the terminal device to perform model monitoring. The time unit corresponding to the reference resource may be the second time unit where the DCI for scheduling model monitoring is located.

[0255] The time unit corresponding to the reference resource may also be a third time unit determined based on the first time unit or the second time unit. For example, T is the first time unit or the second time unit, and the third time unit is T+Δt. Δt may be a parameter configured or predefined by the access network device.

[0256] After determining the time unit corresponding to the reference resource, in the present implementation, one or more second indexes of one or more predicted CSIs are based on the time unit corresponding to the above-mentioned reference resource. For example, the index of the nth predicted CSI before the time slot where the reference resource is located is -n, and the index of the nth predicted CSI before and after the time slot where the reference resource is located is n-1. For another example, the index of the nth predicted CSI before and before the time slot where the reference resource is located is -n+1, ​​and the index of the nth predicted CSI after the time slot where the reference resource is located is n. For another example, the index of the Kth predicted CSI before the time slot where the reference resource is located is 0, and the indexes of the subsequent predicted CSIs increase successively, where K is a predefined parameter or a parameter configured by the access network device. For another example, the index of the Kth predicted CSI after the time slot where the CSI reference resource is located is 0, and the indexes of the previous predicted CSIs increase successively.

[0257] In another implementation, the second information includes a relative time unit value of each predicted CSI in one or more predicted CSIs relative to a time unit corresponding to a reference resource. That is, the predicted CSI may also be indicated by a relative time unit value of the predicted CSI relative to a time unit corresponding to a reference resource. For example, 0 indicates the time slot where the reference resource is located, -n indicates the nth time slot before the time slot where the reference resource is located, and n indicates the nth time slot after the time slot where the CSI reference resource is located.

[0258] After receiving the second information, the terminal device can generate one or more predicted CSIs by implementing its own AI model. Fig. 9 As shown, the time slot where the reference resource is located is t, the second index included in the second information is 1, and the CSI-RS used for model inference is a CSI-RS with a period of 5ms. Therefore, the second information is used to indicate the acquisition of the predicted CSI for the t+5 time slot. The terminal device can use the CSI actually measured at the t-5 and t time slots as the input information of the model according to the implementation of its own AI model to obtain the predicted CSI for the t+5 time slot.

[0259] Furthermore, the method may also include the following steps (indicated by dotted lines in the figure):

[0260] S1104. The terminal device sends one or more true CSIs and one or more predicted CSIs to the access network device. Correspondingly, the access network device receives the one or more true CSIs and one or more predicted CSIs.

[0261] After obtaining one or more true CSIs and one or more predicted CSIs, the terminal device may send the one or more true CSIs and one or more predicted CSIs to the access network device.

[0262] S1105. The access network device obtains one or more monitoring results of the performance of the AI ​​model based on one or more true CSIs and one or more predicted CSIs.

[0263] The specific implementation of this step can be found in Figure 8 Step S805 of the illustrated embodiment will not be described in detail here.

[0264] According to a model monitoring method provided in an embodiment of the present application, a terminal device can obtain the true CSI corresponding to a specific time unit by obtaining one or more first indexes corresponding to one or more true CSIs, and can obtain the corresponding predicted CSI according to the instruction of the access network device, so that the access network device can obtain the model monitoring result of the actual system, so that the performance of the prediction model can be flexibly monitored, and the predicted performance can reflect the actual system performance.

[0265] like Fig.12 FIG. 1 is a flow chart of another model monitoring method provided in an embodiment of the present application. In this embodiment, the terminal device stores an AI model that can predict CSI. Fig.11 The difference from the embodiment shown is that this embodiment determines the true CSI according to the time domain resource position of the CSI-RS used to measure the true CSI. Exemplarily, the method may include the following steps:

[0266] S1201. The terminal device obtains first information.

[0267] The first information includes the time domain resource location of one or more CSI-RSs used to measure the true CSI.

[0268] The specific implementation of this step can be found in Fig.10 Step S1001 of the illustrated embodiment will not be described in detail here.

[0269] S1202. The terminal device determines one or more true value CSIs based on the first information.

[0270] The specific implementation of this step can be found in Fig.10 Step S1002 of the illustrated embodiment will not be described in detail here.

[0271] S1203. The terminal device obtains one or more predicted CSIs.

[0272] The specific implementation of this step can be found in Fig.11 Step S1103 of the illustrated embodiment will not be described in detail here.

[0273] S1204. The terminal device sends one or more true CSIs and one or more predicted CSIs to the access network device. Correspondingly, the access network device receives the one or more true CSIs and one or more predicted CSIs.

[0274] The specific implementation of this step can be found in Fig.11 Step S1104 of the illustrated embodiment will not be described in detail here.

[0275] This embodiment does not limit the order of the above steps S1201, S1202 and S1203. Steps S1201 and S1202 may be performed first, and then step S1203; steps S1201, S1202 and S1203 may also be performed simultaneously; or step S1203 may be performed first, and then steps S1201 and S1202.

[0276] S1205. The access network device obtains one or more monitoring results of the performance of the AI ​​model based on one or more true CSIs and one or more predicted CSIs.

[0277] The specific implementation of this step can be found in Figure 8 Step S805 of the illustrated embodiment will not be described in detail here.

[0278] According to a model monitoring method provided in an embodiment of the present application, a terminal device can obtain the true CSI corresponding to a specific time domain resource position by obtaining the time domain resource position of one or more CSI-RS used to measure the true CSI, and can obtain the corresponding predicted CSI according to the instruction of the access network device, so that the access network device can obtain the model monitoring result of the actual system, so that the performance of the prediction model can be flexibly monitored, and the predicted performance can reflect the actual system performance.

[0279] like Fig.13 As shown, it is a flow chart of another model monitoring method provided by an embodiment of the present application. Different from the above-mentioned embodiment, in this embodiment, the terminal device stores an AI model, which can predict CSI, and the terminal device can also obtain monitoring results. Exemplarily, the method may include the following steps:

[0280] S1301. The terminal device obtains the first information.

[0281] The first information includes one or more first indexes of one or more true value CSIs.

[0282] The specific implementation of this step can be found in Figure 8 Step S801 of the illustrated embodiment will not be described in detail here.

[0283] S1302. The terminal device obtains one or more true value CSIs corresponding to one or more first indexes.

[0284] The specific implementation of this step can be found in Figure 8 Step S802 of the illustrated embodiment will not be described in detail here.

[0285] S1303. The terminal device obtains one or more predicted CSIs.

[0286] The specific implementation of this step can be found in Fig.11 Step S1103 of the illustrated embodiment will not be described in detail here.

[0287] This embodiment does not limit the order of the above steps S1301, S1302 and S1303. Steps S1301 and S1302 may be performed first, and then step S1303; steps S1301, S1302 and S1303 may also be performed simultaneously; or step S1303 may be performed first, and then steps S1301 and S1302.

[0288] S1304. The terminal device obtains one or more monitoring results of the performance of the AI ​​model based on one or more true CSIs and one or more predicted CSIs.

[0289] After the terminal device obtains one or more true CSIs and one or more predicted CSIs, it can obtain one or more monitoring results for each true CSI based on the true CSI and one or more predicted CSIs; or it can also obtain one or more monitoring results for each predicted CSI based on one or more true CSIs.

[0290] For example, the terminal device obtains 2 predicted CSIs, namely and And two true value CSIs, namely H t+2 and H t+4 , then the terminal device can calculate 4 KPIs, which are and

[0291] Furthermore, the access network device may also indicate which predicted CSI and which true CSI are used to calculate each monitoring result.

[0292] S1305. The terminal device sends a monitoring result report to the access network device.

[0293] After obtaining one or more of the above monitoring results, the terminal device can send a monitoring result report to the network device.

[0294] The implementation methods of sending monitoring result reports by terminal devices include but are not limited to:

[0295] In one implementation, the monitoring result report includes the above one or more monitoring results. In this implementation, the terminal device can report all the acquired monitoring results to the access network device, so that the access network device can obtain the complete monitoring results. For example, in the above example, the terminal device can report the above four KPIs.

[0296] When reporting, the terminal device can report the above one or more monitoring results to the access network device in a preset order. For example, if the monitoring result is a KPI result, for multiple predicted CSIs and / or multiple true CSIs, they can be reported in the order indicated by the access network device, or in the order in time. For predicted CSI and true CSI, they can be reported in the order of predicted CSI first and then the order of true CSI, or in the order of true CSI first and then the order of predicted CSI. Taking 4 KPIs as and For example, according to the order from front to back in time, and first in the order of true value CSI and then in the order of CSI to be monitored, the order of the four KPIs is

[0297] Another implementation is that, in some scenarios, the access network device may only know whether there is a monitoring result indicating that the model performance is good, without obtaining all monitoring results. Therefore, the monitoring result report includes at least one monitoring result greater than or equal to the first threshold among the one or more monitoring results. The first threshold may be configured or predefined by the access network device. For example, in the above example, the terminal device may report one or more KPIs greater than or equal to the first threshold among the above four KPIs. This implementation can save signaling overhead.

[0298] Another implementation is that, in some scenarios, the access network device may only know whether there is a monitoring result indicating that the model performance is poor, without obtaining all monitoring results. Therefore, the monitoring result report may include at least one monitoring result that is less than or equal to the second threshold among the one or more monitoring results. The second threshold may be configured or predefined by the access network device. The second threshold is less than the first threshold. For example, in the above example, the terminal device may report one or more KPIs of the above four KPIs that are less than or equal to the second threshold. This implementation can save signaling overhead.

[0299] Another implementation method is that in some scenarios, the access network device may only know whether there is a monitoring result indicating that the model performance is good, without obtaining all monitoring results. Therefore, the monitoring result report may include the index of the predicted CSI corresponding to the optimal monitoring result among the above one or more monitoring results and the index of the true CSI. The access network device may have received one or more predicted CSIs and one or more true CSIs in advance, and can obtain the optimal monitoring result itself based on the index of the predicted CSI corresponding to the optimal monitoring result and the index of the true CSI. This implementation method can save signaling overhead and has low computing power requirements for the terminal device.

[0300] Alternatively, as described above, the access network device indicates, through a first message, the index of one or more predicted CSIs used to calculate the monitoring result, and the same indication method as the access network device may be used. In the monitoring result report, the predicted CSI corresponding to the best monitoring result among the above one or more monitoring results may be included in the sequence of one or more predicted CSIs indicated by the access network device; and as described above, the access network device also indicates the index of one or more true value CSIs used to calculate the monitoring result, and the same indication method as the access network device may be used. In the monitoring result report, the true value CSI corresponding to the best monitoring result among the above one or more monitoring results may be included in the sequence of one or more true value CSIs indicated by the access network device.

[0301] Another implementation method is that in some scenarios, the access network device may only know whether there is a monitoring result indicating that the model performance is poor, without obtaining all monitoring results. Therefore, the monitoring result report may include the index of the predicted CSI corresponding to the worst monitoring result among the above one or more monitoring results and the index of the true CSI. The access network device may have received one or more predicted CSIs and one or more true CSIs in advance, and can obtain the worst monitoring result itself based on the index of the predicted CSI corresponding to the worst monitoring result and the index of the true CSI. This implementation method can save signaling overhead and has low computing power requirements for the terminal device.

[0302] Alternatively, as described above, the access network device indicates, through a first message, the index of one or more predicted CSIs used to calculate the monitoring result, and the same indication method as the access network device may be used. In the monitoring result report, the predicted CSI corresponding to the worst monitoring result among the above one or more monitoring results may be included in the sequence of one or more predicted CSIs indicated by the access network device; and as described above, the access network device also indicates the index of one or more true value CSIs used to calculate the monitoring result, and the same indication method as the access network device may be used. In the monitoring result report, the true value CSI corresponding to the worst monitoring result among the above one or more monitoring results may be included in the sequence of one or more true value CSIs indicated by the access network device.

[0303] Another implementation method is to use a differential reporting method, where the monitoring result report includes the absolute value of the first monitoring result among multiple monitoring results, and the relative values ​​of other monitoring results other than the first monitoring result relative to the first monitoring result. and For example, the reported amount is ΔKPI1, ΔKPI2, ΔKPI3, where This implementation method can save signaling overhead.

[0304] Further, the above-mentioned first monitoring result can be determined based on the order of the true CSI used and / or the order of the predicted CSI used when obtaining multiple monitoring results. As mentioned above, the terminal device can obtain one or more monitoring results for each true CSI according to the true CSI and one or more predicted CSIs, and for each true CSI, the access network device indicates the index of one or more true CSIs used to calculate the monitoring result; or it can also obtain one or more monitoring results for each predicted CSI according to one or more true CSIs, and for each predicted CSI, the access network device indicates the index of one or more predicted CSIs used to calculate the monitoring result through a first message. Therefore, the generation order of one or more monitoring results can be determined according to the order in which the true CSI and / or predicted CSI are used when the terminal device generates the monitoring results, so that the first monitoring result can be determined according to the generation order. For example, the first monitoring result can be the first monitoring result or the last monitoring result among the one or more monitoring results generated sequentially.

[0305] Another implementation is that the access network device may also instruct the terminal device to report part of the monitoring results. For example, the terminal device may be instructed to report part of the above four KPI results in a bitmap manner.

[0306] Furthermore, the monitoring result report and the CSI feedback may be reported in the same CSI report, or may be reported in a different CSI report. The CSI feedback refers to the measurement result used to report the CSI-RS.

[0307] If the monitoring results can be reported in the same CSI report as the CSI feedback, then which CSI report for CSI feedback is reported in can be configured by the access network device or predefined. For example, it is predefined to be reported in the first CSI report after the last CSI-RS used to measure the true CSI, or in the first CSI report after T1 time after the last CSI-RS used to measure the true CSI. T1 is the time reserved for the terminal device to calculate the KPI, which can be configured or predefined by the access network device, and T1 may be related to the processing capability of the terminal device. It should be noted that if the monitoring result report can be reported in the same CSI report as the CSI feedback, the predicted CSI reported in the CSI report and the predicted CSI to be monitored corresponding to the monitoring result report are not the same CSI, such as Fig.14 As shown, it is a schematic diagram of reporting monitoring result reports of an example of an embodiment of the present application, in which a CSI report is reported at time t+7, and the predicted CSI reported in the CSI report at time t+7 is the predicted CSI at time t+10, which is predicted by the true CSI at time t+5 and before; and the predicted CSI to be monitored corresponding to the monitoring result report in the CSI report is the predicted CSI at time t+5, which is predicted by the true CSI at time t and before.

[0308] If the monitoring result report is independent of the CSI feedback report, the reporting resource of the monitoring result report is configured by the access network device. And the terminal device does not expect the reporting resource to be earlier than the last CSI-RS for measuring the true CSI corresponding to the monitoring result report, or the terminal device does not expect the reporting resource to be earlier than T2 time after the last CSI-RS for measuring the true CSI corresponding to the monitoring result. T2 is the time reserved for the terminal device to calculate the KPI, which can be configured or predefined by the access device, and T2 may be related to the processing capability of the terminal device.

[0309] According to a model monitoring method provided in an embodiment of the present application, a terminal device can obtain the true CSI corresponding to a specific time unit by obtaining one or more first indexes corresponding to one or more true CSIs, and can obtain the corresponding predicted CSI according to the instruction of the access network device, and obtain the model monitoring result based on the obtained true CSI and predicted CSI, so that the access network device can obtain the model monitoring result of the actual system, so that the performance of the prediction model can be flexibly monitored, and the predicted performance can reflect the actual system performance.

[0310] like Fig.15 FIG. 1 is a flow chart of another model monitoring method provided in an embodiment of the present application. Figure 8 , Fig.10 The difference from the embodiment shown is that in this embodiment, the terminal device stores an AI model that can predict CSI, and the terminal device can also obtain monitoring results. Fig.13 The difference from the embodiment shown is that this embodiment determines the true CSI according to the time domain resource position of the CSI-RS used to measure the true CSI. Exemplarily, the method may include the following steps:

[0311] S1501. The terminal device obtains first information.

[0312] The first information includes the time domain resource location of one or more CSI-RSs used to measure the true CSI.

[0313] The specific implementation of this step can be found in Fig.10 Step S1001 of the illustrated embodiment will not be described in detail here.

[0314] S1502. The terminal device determines one or more true value CSIs based on the first information.

[0315] The specific implementation of this step can be found in Fig.10 Step S1002 of the illustrated embodiment will not be described in detail here.

[0316] S1503. The terminal device obtains one or more predicted CSIs.

[0317] The specific implementation of this step can be found in Fig.11 Step S1103 of the illustrated embodiment will not be described in detail here.

[0318] S1504. The terminal device obtains one or more monitoring results of the performance of the AI ​​model based on one or more true CSIs and one or more predicted CSIs.

[0319] The specific implementation of this step can be found in Fig.13 Step S1304 of the illustrated embodiment will not be described in detail here.

[0320] S1505. The terminal device sends a monitoring result report to the access network device.

[0321] The specific implementation of this step can be found in Fig.13 Step S1305 of the illustrated embodiment will not be described in detail here.

[0322] According to a model monitoring method provided in an embodiment of the present application, a terminal device can obtain the true CSI corresponding to a specific time domain resource position by obtaining the time domain resource position of one or more CSI-RS used to measure the true CSI, and can obtain the corresponding predicted CSI according to the instruction of the access network device, and obtain the model monitoring result based on the obtained true CSI and predicted CSI, so that the access network device can obtain the model monitoring result of the actual system, so that the performance of the prediction model can be flexibly monitored, and the predicted performance can reflect the actual system performance.

[0323] The solution described in the above embodiment can also be used in beam prediction scenarios:

[0324] Beam prediction is to predict the optimal beam in the future based on historical / current beam measurement results.

[0325] The KPIs for beam prediction may be Top-1 accuracy, Top-1 accuracy with ndB margin, Top-K accuracy, reference signal receiving power (RSRP) difference, etc.

[0326] Among them, Top-1 accuracy refers to whether the predicted optimal beam is the actual optimal beam, or the proportion of the predicted optimal beam in multiple monitorings that is the actual optimal beam;

[0327] The Top1 accuracy with ndB margin refers to whether the difference between the predicted RSRP of the best beam and the actual RSRP of the best beam is less than ndB, or the proportion of the difference between the predicted RSRP of the best beam and the actual RSRP of the best beam in multiple monitorings that is less than ndB;

[0328] Top-K accuracy refers to whether the predicted optimal beam is among the actual K optimal beams, or the ratio of the predicted optimal beam in multiple monitorings to the actual K optimal beams, or whether the actual optimal beam is among the predicted K optimal beams, or the ratio of the actual optimal beam in multiple monitorings to the predicted K optimal beams;

[0329] The RSRP difference refers to the difference between the RSRP of the predicted optimal beam and the RSRP of the actual optimal beam, or the difference between the RSRP of the predicted optimal beam and the actual RSRP of the beam.

[0330] like Fig.16 FIG. 1 is a flow chart of another model monitoring method provided in an embodiment of the present application. Exemplarily, the method may include the following steps:

[0331] S1601. The terminal device obtains the first information.

[0332] Similar to CSI prediction, calculating the KPI for beam prediction also requires two types of information: one is prediction information, such as the index of the predicted optimal beam or one or more of RSRP, and the other is true value information, such as the index of the actual optimal beam or one or more of RSRP. The access network device can also instruct the terminal device which prediction information and true value information to use to calculate the KPI.

[0333] In this embodiment, the terminal device is instructed, through the index of the true value beam information, to obtain the true value beam information.

[0334] The terminal device obtains the first information. The first information includes one or more first indexes corresponding to information of one or more true value beams. Exemplarily, the terminal device may obtain the first information sent by the access network device, or may obtain the first information pre-stored by the terminal device.

[0335] The specific implementation of this step can be found in Figure 8 The specific implementation of obtaining the first information in CSI prediction in the illustrated embodiment will not be described in detail here.

[0336] S1602. The terminal device obtains information of one or more true value beams corresponding to one or more first indexes.

[0337] After the terminal device obtains one or more first indexes corresponding to the information of one or more true value beams, it measures the reference signals corresponding to the one or more indexes, such as CSI-RS, to obtain the information of one or more true value beams corresponding to the one or more first indexes.

[0338] The information of the one or more true value beams is used to obtain monitoring results of the performance of the AI ​​model. The output of the AI ​​model is the information of the predicted beam.

[0339] Compared with the prior art which can only periodically obtain model monitoring results on the grid, the terminal device in this embodiment can obtain the information of the truth beam corresponding to a specific time unit, such as the required time unit, by obtaining one or more first indexes of the information of one or more truth beams.

[0340] Furthermore, the terminal device can send information of one or more true value beams to the access network device, the access network device obtains information of one or more predicted beams, and obtains one or more monitoring results of the performance of the AI ​​model based on the information of one or more true value beams and the information of one or more predicted beams.

[0341] Alternatively, the terminal device may also obtain information about one or more prediction beams, and send information about one or more true value beams and information about one or more prediction beams to the access network device, and the access network device may obtain one or more monitoring results of the performance of the AI ​​model based on the information about one or more true value beams and the information about one or more prediction beams.

[0342] According to a model monitoring method provided in an embodiment of the present application, a terminal device can obtain information of a truth beam corresponding to a specific time unit by obtaining one or more first indexes corresponding to information of one or more truth beams, thereby flexibly monitoring the performance of a prediction model.

[0343] like Fig.17 FIG. 1 is a flow chart of another model monitoring method provided in an embodiment of the present application. Fig.16 The embodiment shown is different in that this embodiment does not directly indicate which true value beam information the terminal device acquires through the index of the true value beam information, but determines the true value beam information according to the time domain resource position of the reference signal used to measure the true value beam information. Exemplarily, the method may include the following steps:

[0344] S1701. The terminal device obtains the first information.

[0345] The reference signal used to measure the information of the true value beam (i.e., used for model monitoring) can be the same group of reference signals as the reference signal used for model inference (as input to the model, used to predict the information of the beam), i.e., they belong to the same reference signal configuration; the reference signal used to measure the information of the true value beam can also be a different group of reference signals from the reference signal used for model inference, i.e., a reference signal is specially configured for model monitoring.

[0346] In this embodiment, the reference signal used to measure the information of the true value beam and the reference signal used for model inference are reference signals of different groups, and the terminal device can determine the information of the true value beam according to the reference signal used to measure the information of the true value beam. Therefore, the terminal device can obtain the first information, which includes the time domain resource location of one or more reference signals used to measure the information of the true value beam.

[0347] Exemplarily, the terminal device may obtain the above-mentioned first information sent by the access network device, or may obtain the above-mentioned first information pre-stored by the terminal device.

[0348] The specific implementation of this step can be found in Fig.10 The specific implementation of obtaining the first information in CSI prediction in the illustrated embodiment will not be described in detail here.

[0349] S1702. The terminal device determines information of one or more true value beams based on the first information.

[0350] After the terminal device obtains the first information, it can determine the information of one or more true value beams based on the first information.

[0351] In one implementation, the terminal device can measure one or more reference signals at the time domain resource positions of one or more reference signals carried by the first information by default to obtain information of one or more true value beams.

[0352] In another implementation, if the access network device configures the time domain resource location of one or more reference signals for measuring the information of the true value beam through the first information, the terminal device can measure the one or more reference signals at the time domain resource location of the one or more reference signals carried by the first information by default to obtain the information of one or more true value beams.

[0353] In another implementation, the terminal device may also obtain the time domain resource position of one or more reference signals used for model inference. The terminal device may measure one or more reference signals at the time domain resource position of one or more reference signals carried by the first information, and measure one or more reference signals at the time domain resource position of one or more reference signals used for model inference by default, to obtain information of one or more true value beams.

[0354] In another implementation, the access network device may also instruct the terminal device to measure one or more reference signals at the time domain resource location of one or more reference signals carried by the first information to obtain information of one or more true value beams, or instruct the terminal device to measure one or more reference signals at the time domain resource location of one or more reference signals used for model inference to obtain information of one or more true value beams.

[0355] The information of the one or more true value beams is used to obtain monitoring results of the performance of the AI ​​model. The output of the AI ​​model is the information of the predicted beam.

[0356] It can be seen that, in this embodiment, the time domain resource position of at least one true value beam information among the obtained one or more true value beam information is different from the time domain resource position of the predicted beam information.

[0357] Furthermore, the terminal device can send information of one or more true value beams to the access network device, the access network device obtains information of one or more predicted beams, and obtains one or more monitoring results of the performance of the AI ​​model based on the information of one or more true value beams and the information of one or more predicted beams.

[0358] Alternatively, the terminal device may also obtain information about one or more prediction beams, and send information about one or more true value beams and information about one or more prediction beams to the access network device, and the access network device may obtain one or more monitoring results of the performance of the AI ​​model based on the information about one or more true value beams and the information about one or more prediction beams.

[0359] According to a model monitoring method provided in an embodiment of the present application, a terminal device can obtain information about a true beam corresponding to a specific time domain resource position by obtaining the time domain resource position of one or more reference signals for measuring information about the true beam, thereby flexibly monitoring the performance of the prediction model.

[0360] It can be understood that in the above embodiments, the methods and / or steps implemented by the access network device can also be implemented by components (such as chips or circuits) that can be used for the access network device; the methods and / or steps implemented by the terminal device can also be implemented by components (such as chips or circuits) that can be used for the terminal device.

[0361] The above mainly introduces the scheme provided by the embodiment of the present application from the perspective of interaction between various network elements. Accordingly, the embodiment of the present application also provides a model monitoring device, which is used to implement the above various methods. The model monitoring device can be an access network device in the above method embodiment, or a component that can be used for the access network device; or, the model monitoring device can be a terminal device in the above method embodiment, or a component that can be used for the terminal device. It can be understood that in order to implement the above functions, the model monitoring device includes a hardware structure and / or software module corresponding to each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed in this article, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0362] The embodiment of the present application can divide the functional modules of the model monitoring device according to the above method embodiment. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing unit. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical functional division. There may be other division methods in actual implementation.

[0363] Based on the same concept of the above model monitoring method, the present application also provides the following model monitoring device:

[0364] like Fig.18 , which is a schematic diagram of the structure of a model monitoring device provided in an embodiment of the present application, the model monitoring device 1800 includes a transceiver unit 1801 and a processing unit 1802; wherein:

[0365] When the model monitoring device is used to implement the functions of the terminal device in the above method embodiment, the processing unit 1802 is used to execute the following steps: Figure 6 In the embodiment shown, the operations of S601 and S602 are performed; or, the processing unit 1802 is used to perform the following Figure 7 The operations of S701 and S702 in the embodiment shown; the transceiver unit 1801 is used to perform the following Figure 8 In the embodiment shown, the operation of the terminal device in S803 and the processing unit 1802 are used to perform the following steps: Figure 8 The operations of S801 and S802 in the embodiment shown; or, the transceiver unit 1801 is used to perform the following Fig.10 In the embodiment shown, the operation of the terminal device in S1003 and the processing unit 1802 are used to perform the following steps: Fig.10 In the embodiment shown, the operations of S1001 and S1002 are performed; or, the transceiver unit 1801 is used to perform the following steps: Fig.11 In the embodiment shown, the operation of the terminal device in S1104 and the processing unit 1802 are used to perform the following steps: Fig.11 Steps S1101 to S1103 in the embodiment shown; or, the transceiver unit 1801 is used to execute Fig.12 In the embodiment shown, the operation of the terminal device in S1204 and the processing unit 1802 are used to perform the following steps: Fig.12 Steps S1201 to S1203 in the embodiment shown; or, the transceiver unit 1801 is used to execute the following steps: Fig.13 In the embodiment shown, the operation of the terminal device in S1305 and the processing unit 1802 are used to perform the following steps: Fig.13 Steps S1301 to S1304 in the embodiment shown; or, the transceiver unit 1801 is used to execute Fig.15 In the embodiment shown, the operation of the terminal device in S1505 and the processing unit 1802 are used to perform the following steps: Fig.15 Steps S1501 to S1504 in the embodiment shown; or, the processing unit 1802 is used to execute Fig.16 In the embodiment shown, the operations of S1601 and S1602 are performed; or, the processing unit 1802 is used to perform the following Fig.17 The operations of S1701 and S1702 in the illustrated embodiment.

[0366] When the model monitoring device is used to implement the function of the access network device in the above method embodiment, the transceiver unit 1801 is used to execute the following steps: Figure 8 In the embodiment shown, the operation of the access network device in S803 and the processing unit 1802 are used to perform the following steps: Figure 8 The operations of S804 and S605 in the embodiment shown; or, the transceiver unit 1801 is used to perform the following Fig.10 In the embodiment shown, the operation of the access network device in S1003 and the processing unit 1802 are used to perform the following steps: Fig.10 The operations of S1004 and S1005 in the embodiment shown; or, the transceiver unit 1801 is used to perform the following Fig.11 In the embodiment shown, the operation of the access network device in S1104 and the processing unit 1802 are used to perform the following steps: Fig.11 Step S1105 in the embodiment shown; or, the transceiver unit 1801 is used to execute Fig.12 In the embodiment shown, the operation of the access network device in S1204 and the processing unit 1802 are used to perform the following steps: Fig.12 Step S1205 in the embodiment shown; or, the transceiver unit 1801 is used to execute Fig.13 The operation of the access network device in S1305 in the embodiment shown; or, the transceiver unit 1801 is used to perform the following Fig.15 The operation of the access network device in S1505 in the illustrated embodiment.

[0367] For the specific implementation of the above-mentioned transceiver unit 1801 and the processing unit 1802, reference may be made to the description in the above-mentioned method embodiment.

[0368] like Fig.19 As shown, it is a schematic diagram of the structure of another model monitoring device provided in an embodiment of the present application, and the model monitoring device 1900 includes one or more processors 1901 (one processor is illustrated in the figure). Optionally, the model monitoring device 1900 may also include a memory 1903 (indicated by a dotted line in the figure). The memory 1903 is used to store instructions executed by the processor 1901, or to store input data required for the processor 1901 to run instructions, or to store data generated after the processor 1901 runs instructions. Optionally, the model monitoring device 1900 may also include an interface circuit 1902 (indicated by a dotted line in the figure), and the processor 1901 and the interface circuit 1902 are coupled to each other. It can be understood that the interface circuit 1902 can be a transceiver or an input-output interface. Among them, the processor 1901 is used to implement the above-mentioned Fig.18 The functions of the processing unit 1802 in the embodiment shown; and the interface circuit 1902 are used to implement the above Fig.18 Functions of the transceiver unit 1801 in the illustrated embodiment.

[0369] When the above-mentioned model monitoring device is a chip applied to an access network device, the chip implements the functions of the access network device in the above-mentioned method embodiment. The chip receives information from other modules (such as a radio frequency module or an antenna) in the access network device, and the information is sent by the terminal device to the access network device; or the chip sends information to other modules (such as a radio frequency module or an antenna) in the access network device, and the information is sent by the access network device to the terminal device.

[0370] When the above-mentioned model monitoring device is a chip applied to a terminal device, the chip implements the functions of the terminal device in the above-mentioned method embodiment. The chip receives information from other modules in the terminal device (such as a radio frequency module or an antenna), and the information is sent to the terminal device by the access network device; or the chip sends information to other modules in the terminal device (such as a radio frequency module or an antenna), and the information is sent to the access network device by the terminal device.

[0371] In addition, it should be noted that the aforementioned transceiver unit and / or processing unit can be implemented through a virtual module, for example, the processing unit can be implemented through a software function unit or a virtual device, and the transceiver unit can be implemented through a software function or a virtual device. Alternatively, the processing unit or the transceiver unit can also be implemented through a physical device, for example, if the device is implemented using a chip / chip circuit, the transceiver unit can be an input-output circuit and / or a communication interface, performing input operations (corresponding to the aforementioned receiving operations) and output operations (corresponding to the aforementioned sending operations); the processing unit is an integrated processor or microprocessor or integrated circuit.

[0372] The division of modules in this application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each example of this application may be integrated into one processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0373] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0374] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program or instruction is stored. When the computer program or instruction is executed, the method in the above embodiment is implemented.

[0375] The embodiments of the present application also provide a computer program product including instructions, which, when executed on a computer, enables the computer to execute the method in the above embodiments.

[0376] An embodiment of the present application also provides a model monitoring system, including the above-mentioned model monitoring device.

[0377] The embodiment of the present application also provides a circuit, which is coupled to a memory and is used to execute the method shown in the above embodiment. The circuit may include a chip circuit.

[0378] When the above-mentioned model monitoring device is a module applied to a base station, the base station module implements the functions of the base station in the above-mentioned method embodiment. The base station module receives information from other modules in the base station (such as a radio frequency module or an antenna), and the information is sent by the terminal device to the base station; or, the base station module sends information to other modules in the base station (such as a radio frequency module or an antenna), and the information is sent by the base station to the terminal device. The base station module here can be a baseband chip of a base station, or a CU, DU or other module, or a device under an open radio access network (O-RAN) architecture, such as an open CU, an open DU and other devices.

[0379] It should be noted that the above units or one or more of the units can be implemented by software, hardware or a combination of the two. When any of the above units or units is implemented by software, the software exists in the form of computer program instructions and is stored in a memory, and a processor can be used to execute the program instructions and implement the above method flow.

[0380] In this application, the processor may be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, or all or part of the circuits in the aforementioned devices for implementing processing functions, which may implement or execute the methods, steps and logic block diagrams disclosed in this application. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in this application may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0381] When the above units or units are implemented in hardware, the hardware can be any one or any combination of a CPU, a microprocessor, a digital signal processing (DSP) chip, a microcontroller unit (MCU), an artificial intelligence processor, an ASIC, a SoC, an FPGA, a PLD, a dedicated digital circuit, a hardware accelerator or a non-integrated discrete device, which can run the necessary software or not rely on the software to execute the above method flow.

[0382] Optionally, the embodiment of the present application further provides a chip system, including: at least one processor and an interface, the at least one processor is coupled to a memory via the interface, and when the at least one processor runs a computer program or instruction in the memory, the chip system executes a method in any of the above method embodiments. Optionally, the chip system may be composed of a chip, or may include a chip and other discrete devices, which is not specifically limited in the embodiment of the present application.

[0383] The memory in the present application may also be a circuit or any other device capable of implementing a storage function, for storing program instructions and / or data. The memory is any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. For example, the memory may be a non-volatile memory, such as a digital versatile disc (DVD), a hard disk drive (HDD) or a solid-state drive (SSD), etc., or a volatile memory (volatile memory), such as a random-access memory (RAM).

[0384] It should be understood that in the description of the present application, unless otherwise specified, " / " indicates that the objects associated before and after are in an "or" relationship, for example, A / B can represent A or B; wherein A and B can be singular or plural. Also, in the description of the present application, unless otherwise specified, "multiple" refers to two or more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, wherein a, b, c can be single or multiple. In addition, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first", "second", etc. are used to distinguish the same items or similar items with substantially the same functions and effects. Those skilled in the art can understand that the words "first", "second", etc. do not limit the quantity and execution order, and the words "first", "second", etc. do not limit them to be necessarily different. Meanwhile, in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete manner for ease of understanding.

[0385] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using a software program, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loading and executing computer program instructions on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center.

[0386] Although the present application is described herein in conjunction with various embodiments, in the process of implementing the claimed application, those skilled in the art may understand and implement other variations of the disclosed embodiments by viewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "one" or "an" does not exclude multiple situations. A single processor or other unit may implement several functions listed in a claim. Certain measures are recorded in different dependent claims, but this does not mean that these measures cannot be combined to produce good results.

[0387] It is understood that the various numbers involved in the embodiments of the present application are only for the convenience of description and are not used to limit the scope of the embodiments of the present application. The size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic.

[0388] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0389] The components in the device of the embodiment of the present application can be merged, divided and deleted according to actual needs. Those skilled in the art can combine or combine the different embodiments and features of the different embodiments described in this specification.

[0390] In the present application, under the premise of no logical contradiction, the examples may reference each other, for example, the methods and / or terms between method embodiments may reference each other, for example, the functions and / or terms between device embodiments may reference each other, for example, the functions and / or terms between device examples and method examples may reference each other.

Claims

1. A model monitoring method, characterized in that: The method comprises: Acquire first information, where the first information includes one or more first indexes corresponding to one or more pieces of true value information respectively; Obtain one or more pieces of truth value information respectively corresponding to the one or more first indexes; Among them, the one or more true value information is used to obtain monitoring results of the performance of the artificial intelligence AI model, and the output of the AI ​​model is prediction information.

2. A model monitoring method, characterized in that: The method comprises: Acquire first information, wherein the first information includes time domain resource locations of one or more reference signals used to measure true value information; Determine one or more pieces of truth information based on the first information; Among them, the one or more truth value information are used to obtain the monitoring results of the performance of the AI ​​model, the output of the AI ​​model is prediction information, and the time domain resource location of at least one of the one or more truth value information is different from the time domain resource location of the prediction information.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: The one or more truth value information are sent.

4. The method according to claim 1 or 2, characterized in that: The method further comprises: Obtain one or more prediction information, wherein the one or more prediction information is output of an artificial intelligence (AI) model; The one or more prediction information are sent.

5. The method according to claim 1 or 2, characterized in that: The method further comprises: Obtain one or more prediction information, wherein the one or more prediction information is output of an artificial intelligence (AI) model; Based on the one or more true value information and the one or more prediction information, obtain one or more monitoring results of the performance of the AI ​​model.

6. The method according to claim 4 or 5, characterized in that: The obtaining one or more prediction information includes: receiving second information, where the second information includes one or more second indexes corresponding to the one or more prediction information respectively, or the second information includes a relative time unit value of each prediction information in the one or more prediction information relative to a time unit corresponding to a reference resource; According to the second information, one or more prediction information is obtained.

7. The method according to claim 6, characterized in that The one or more first indexes and the one or more second indexes are based on a time unit corresponding to a reference resource.

8. The method according to claim 7, characterized in that The time unit corresponding to the reference resource is any one of the following: The first time unit of the resource that reports the monitoring result; or Scheduling a second time unit in which the downlink control signaling monitored by the model is located; or a third time unit determined based on the first time unit or the second time unit; or Predefined / preconfigured fourth time unit.

9. The method according to claim 5, characterized in that The method further comprises: Send monitoring result reports; Wherein, the monitoring result report includes the one or more monitoring results; or The monitoring result report includes at least one monitoring result of the one or more monitoring results that is greater than or equal to a first threshold; or The monitoring result report includes at least one monitoring result of the one or more monitoring results that is less than or equal to a second threshold; or The monitoring result report includes an index of prediction information and an index of true value information corresponding to the best monitoring result among the one or more monitoring results; or The monitoring result report includes an index of prediction information and an index of true value information corresponding to the worst monitoring result among the one or more monitoring results; or The monitoring result report includes an absolute value of a first monitoring result among a plurality of monitoring results, and relative values ​​of other monitoring results among the plurality of monitoring results except the first monitoring result with respect to the first monitoring result.

10. The method according to claim 9, characterized in that The first monitoring result is determined based on an order of the true value information used and / or an order of the prediction information used when acquiring the plurality of monitoring results.

11. A model monitoring method, characterized in that: The method comprises: Sending first information, wherein the first information includes one or more first indexes corresponding to one or more pieces of true value information respectively; Receive one or more pieces of truth value information respectively corresponding to the one or more first indexes; Among them, the one or more true value information is used to obtain monitoring results of the performance of the artificial intelligence AI model, and the output of the AI ​​model is prediction information.

12. A model monitoring method, characterized in that: The method comprises: Sending first information, wherein the first information includes time domain resource locations of one or more reference signals used to measure true value information; receiving one or more truth value information according to the first information; Among them, the one or more truth value information are used to obtain the monitoring results of the performance of the AI ​​model, the output of the AI ​​model is prediction information, and the time domain resource location of at least one of the one or more truth value information is different from the time domain resource location of the prediction information.

13. The method according to claim 11 or 12, characterized in that: The method further comprises: obtaining one or more forecast information; Based on the one or more true value information and the one or more prediction information, obtain one or more monitoring results of the performance of the AI ​​model.

14. The method according to claim 11 or 12, characterized in that: The method further comprises: A monitoring result report is received, where the monitoring result report includes one or more monitoring results, where the one or more monitoring results are obtained based on the one or more true value information and the one or more prediction information.

15. The method according to claim 13, characterized in that The method comprises: Sending second information, where the second information includes one or more second indexes corresponding to the one or more prediction information respectively, or the second information includes a relative time unit value of each prediction information in the one or more prediction information relative to a time unit corresponding to a reference resource; The one or more prediction information is received.

16. The method according to claim 15, characterized in that The one or more first indexes and the one or more second indexes are based on a time unit corresponding to a reference resource.

17. The method according to claim 16, characterized in that The time unit corresponding to the reference resource is any one of the following: The first time unit of the resource that reports the monitoring result; or Scheduling a second time unit in which the downlink control signaling monitored by the model is located; or a third time unit determined based on the first time unit or the second time unit; or Predefined / preconfigured fourth time unit.

18. The method according to claim 14, characterized in that: The monitoring result report includes the one or more monitoring results; or The monitoring result report includes at least one monitoring result of the one or more monitoring results that is greater than or equal to a first threshold; or The monitoring result report includes at least one monitoring result of the one or more monitoring results that is less than or equal to a second threshold; or The monitoring result report includes an index of prediction information and an index of true value information corresponding to the best monitoring result among the one or more monitoring results; or The monitoring result report includes an index of prediction information and an index of true value information corresponding to the worst monitoring result among the one or more monitoring results; or The monitoring result report includes an absolute value of a first monitoring result among a plurality of monitoring results, and relative values ​​of other monitoring results among the plurality of monitoring results except the first monitoring result with respect to the first monitoring result.

19. The method according to claim 18, characterized in that The first monitoring result is determined based on an order of the true value information used and / or an order of the prediction information used when acquiring the plurality of monitoring results.

20. A model monitoring device, characterized in that: Comprising modules for executing the method as claimed in any one of claims 1 to 18.

Images