Model monitoring method, apparatus and system, and storage medium and program product

Abstract

Disclosed in the present application are a model monitoring method, apparatus and system, and a storage medium and a program product. The method comprises: a terminal device acquiring first information, wherein the first information comprises one or more first indices of one or more pieces of ground-truth information; and acquiring the one or more pieces of ground-truth information corresponding to the one or more first indices, wherein the one or more pieces of ground-truth information are used for acquiring a performance monitoring result of an AI model, and the output of the AI model is prediction information. In the solution, by means of acquiring one or more first indices of one or more pieces of ground-truth information, a terminal device can acquire ground-truth information corresponding to a specific time unit, thereby flexibly monitoring the performance of a prediction model.

Description

Model monitoring method, device, system, storage medium and program product

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on July 14, 2023, with application number 202310870686.9 and invention name “Model monitoring method, device, system, storage medium and program product”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of communication technology, and in particular to a model monitoring method, device, system, storage medium and program product. Background Art

[0003] Model monitoring refers to monitoring the performance of artificial intelligence (AI) models. This determines whether the AI ​​model is functioning properly. If the AI ​​model's performance is poor, it is necessary to switch to a non-AI mode, replace the AI ​​model, or update the AI ​​model. Model monitoring can be performed by monitoring the accuracy of the AI ​​model's output (also known as an intermediate key performance indicator (KPI)). Monitoring the accuracy of the AI ​​model's output involves comparing the AI ​​model's output with the corresponding ground-truth value to determine whether the AI ​​model's performance meets requirements.

[0004] AI models can be used to predict information. For example, if the output of the AI ​​model is the predicted information at time t+5, the true value information at time t+5 is usually compared with the predicted information at time t+5. If the intermediate KPI meets the preset requirements, it means that the AI ​​model has good performance; otherwise, it means that the AI ​​model has poor performance. Considering that the AI ​​model is used in actual systems, the predicted information is used for base station scheduling downlink transmission, but the downlink transmission scheduled by the base station is not necessarily at the moment corresponding to the predicted information. For example, the information predicted by the AI ​​model is the information at time t+5, that is, if the period of the reference signal (RS) is 5ms, the AI ​​model can only predict the information on the 5ms grid, but the base station may schedule downlink transmission at any time, such as scheduling downlink transmission at time t+3. Then, only monitoring the performance of the predicted information at its corresponding moment cannot reflect the actual performance when the predicted information is used for downlink transmission.

[0005] There is currently no corresponding solution for how to flexibly monitor the performance of the prediction model so that the prediction performance can reflect the actual system performance.

[0006] Summary of the Invention

[0007] The present application provides a model monitoring method, device, system, storage medium and program product to support flexible monitoring of the performance of the prediction model so that the predicted performance can reflect the actual system performance.

[0008] In a first aspect, a model monitoring method is provided, the method comprising: obtaining first information, the first information comprising one or more first indexes corresponding to one or more true value information; and obtaining one or more true value information corresponding to the one or more first indexes; wherein the one or more true value information are used to obtain monitoring results of the performance of an artificial intelligence (AI) model, and the output of the AI ​​model is prediction information.

[0009] In this aspect, the terminal device can obtain the truth information corresponding to a specific time unit by obtaining one or more first indexes of one or more truth information, so as to flexibly monitor the performance of the prediction model.

[0010] In a second aspect, a model monitoring method is provided, the method comprising: obtaining first information, the first information comprising the time domain resource location of one or more reference signals for measuring true value information; and determining one or more true value information based on the first information; wherein the one or more true value information are used to obtain monitoring results of the performance of the AI ​​model, the output of the AI ​​model is prediction information, and the time domain resource location of at least one of the one or more true value information is different from the time domain resource location of the prediction information.

[0011] In this aspect, the terminal device can obtain the true value information corresponding to any time domain resource position by obtaining the time domain resource position of one or more reference signals used to measure the true value information, thereby flexibly monitoring the performance of the prediction model.

[0012] In a possible implementation, the method further includes: sending the one or more truth value information.

[0013] In this implementation, by sending one or more true value information to the access network device, the access network device can obtain the model monitoring results of the actual system based on the received one or more true value information, so that the performance of the prediction model can be flexibly monitored, and the prediction performance can reflect the actual system performance.

[0014] In another possible implementation, the method further includes: obtaining one or more prediction information, where the one or more prediction information are outputs of an artificial intelligence (AI) model; and sending the one or more prediction information.

[0015] In this implementation, by sending one or more true value information to the access network device, the access network device can obtain the model monitoring results of the actual system based on the received one or more true value information and one or more prediction information, so that the performance of the prediction model can be flexibly monitored, and the prediction performance can reflect the actual system performance.

[0016] In another possible implementation, the method further includes: obtaining one or more prediction information, wherein the one or more prediction information are outputs of an artificial intelligence (AI) model; and obtaining one or more monitoring results of the performance of the AI ​​model based on the one or more true value information and the one or more prediction information.

[0017] In this implementation, the terminal device stores an AI model that can predict information, and the terminal device can also obtain monitoring results, so that the access network device can obtain the model monitoring results of the actual system, making it possible to flexibly monitor the performance of the prediction model, and the prediction performance can reflect the actual system performance.

[0018] In another possible implementation, obtaining one or more prediction information includes: receiving second information, the second information including one or more second indexes corresponding to the one or more prediction information, or the second information including a relative time unit value of each of the one or more prediction information relative to the time unit corresponding to the reference resource; and obtaining one or more prediction information based on the second information.

[0019] In this implementation, the terminal device can obtain corresponding prediction information according to the instructions of the access network device.

[0020] In another possible implementation, the one or more first indexes and the one or more second indexes are based on a time unit corresponding to a reference resource.

[0021] In another possible implementation, the time unit corresponding to the reference resource is any one of the following: the first time unit where the resource for reporting the monitoring result report is located; or the second time unit where the downlink control signaling for scheduling the model monitoring is located; or a third time unit determined based on the first time unit or the second time unit; or a predefined / preconfigured fourth time unit.

[0022] In another possible implementation, a monitoring result report is sent; wherein, the monitoring result report includes the one or more monitoring results; or the monitoring result report includes at least one monitoring result among the one or more monitoring results that is greater than or equal to a first threshold; or the monitoring result report includes at least one monitoring result among the one or more monitoring results that is less than or equal to a second threshold; or the monitoring result report includes an index of prediction information and an index of true value information corresponding to the best monitoring result among the one or more monitoring results; or the monitoring result report includes an index of prediction information and an index of true value information corresponding to the worst monitoring result among the one or more monitoring results; or the monitoring result report includes the absolute value of a first monitoring result among multiple monitoring results, and the relative values ​​of other monitoring results among the multiple monitoring results except the first monitoring result relative to the first monitoring result.

[0023] In the above implementation, the terminal device can report all acquired monitoring results to the access network device, so that the access network device can obtain complete monitoring results;

[0024] In some scenarios, the access network device may only need to know whether a monitoring result indicates good model performance, without having to obtain all monitoring results. Therefore, the monitoring result report includes at least one monitoring result from the one or more monitoring results that is greater than or equal to a first threshold. This first threshold may be configured or predefined by the access network device. This implementation can reduce signaling overhead.

[0025] In some scenarios, the access network device may only need to know whether a monitoring result indicates poor model performance, without having to obtain all monitoring results. Therefore, the monitoring result report may include at least one monitoring result, from the one or more monitoring results, that is less than or equal to a second threshold. The second threshold may be configured or predefined by the access network device. The second threshold is less than the first threshold. This implementation can reduce signaling overhead.

[0026] In some scenarios, the access network device may only know whether there is a monitoring result indicating that the model performance is good, without having to obtain all monitoring results. Therefore, the monitoring result report may include the index of the prediction information and the index of the true value information corresponding to the optimal monitoring result among the one or more monitoring results mentioned above. The access network device may have received one or more prediction information and one or more true value information in advance, and can then obtain the optimal monitoring result itself based on the index of the prediction information and the index of the true value information corresponding to the optimal monitoring result. This implementation method can save signaling overhead and has low computing power requirements for the terminal device.

[0027] In some scenarios, the access network device may only know whether there is a monitoring result indicating that the model performance is poor, without having to obtain all monitoring results. Therefore, the monitoring result report may include the index of the prediction information and the index of the true value information corresponding to the worst monitoring result among the one or more monitoring results mentioned above. The access network device may have received one or more prediction information and one or more true value information in advance, and can then obtain the worst monitoring result itself based on the index of the prediction information and the index of the true value information corresponding to the worst monitoring result. This implementation method can save signaling overhead and has low computing power requirements for the terminal device.

[0028] In some scenarios, the terminal device can adopt a differential reporting method, in which the monitoring result report includes the absolute value of the first monitoring result among multiple monitoring results, and the relative values ​​of other monitoring results other than the first monitoring result relative to the first monitoring result. This implementation method can save signaling overhead.

[0029] In another possible implementation, the first monitoring result is determined based on the order of the true value information used and / or the order of the prediction information used when obtaining the multiple monitoring results.

[0030] In this implementation, the generation order of one or more monitoring results can be determined based on the order in which the terminal device uses true value information and / or prediction information when generating monitoring results, so that the first monitoring result can be determined based on the generation order.

[0031] In another possible implementation, the terminal device may report part of the monitoring results according to the instruction of the access network device. For example, the terminal device may be instructed to report part of the monitoring results among all the above monitoring results in the form of a bitmap.

[0032] In another possible implementation, the monitoring result report and the information feedback for sending the reference signal measurement result are reported in the same information report, or the monitoring result report and the information feedback for sending the reference signal measurement result are reported in different information reports.

[0033] In a third aspect, a model monitoring method is provided, which includes: sending first information, the first information including one or more first indexes corresponding to one or more true value information; and receiving one or more true value information corresponding to the one or more first indexes; wherein the one or more true value information are used to obtain monitoring results of the performance of an artificial intelligence AI model, and the output of the AI ​​model is prediction information.

[0034] In this aspect, the access network device indicates one or more first indexes of one or more truth information to the terminal device, so that the terminal device can obtain the truth information corresponding to a specific time unit, thereby flexibly monitoring the performance of the prediction model.

[0035] In a fourth aspect, a model monitoring method is provided, the method comprising: sending first information, the first information comprising the time domain resource location of one or more reference signals for measuring true value information; and receiving one or more true value information based on the first information; wherein the one or more true value information are used to obtain monitoring results of the performance of the AI ​​model, the output of the AI ​​model is prediction information, and the time domain resource location of at least one of the one or more true value information is different from the time domain resource location of the prediction information.

[0036] In this aspect, the access network device indicates to the terminal device the time domain resource location of one or more reference signals used to measure true value information, so that the terminal device can obtain the true value information corresponding to any time domain resource location, thereby flexibly monitoring the performance of the prediction model.

[0037] In one possible implementation, the method further includes: obtaining one or more prediction information; and obtaining one or more monitoring results of the performance of the AI ​​model based on the one or more true value information and the one or more prediction information.

[0038] In another possible implementation, the method further includes: receiving a monitoring result report, where the monitoring result report includes one or more monitoring results, and the one or more monitoring results are obtained based on the one or more true value information and the one or more prediction information.

[0039] In another possible implementation, the method includes: sending second information, the second information including one or more second indexes corresponding to the one or more prediction information, or the second information including a relative time unit value of each of the one or more prediction information relative to the time unit corresponding to the reference resource; and receiving the one or more prediction information.

[0040] In another possible implementation, the one or more first indexes and the one or more second indexes are based on a time unit corresponding to a reference resource.

[0041] In another possible implementation, the time unit corresponding to the reference resource is any one of the following: the first time unit where the resource for reporting the monitoring result report is located; or the second time unit where the downlink control signaling for scheduling the model monitoring is located; or a third time unit determined based on the first time unit or the second time unit; or a predefined / preconfigured fourth time unit.

[0042] In another possible implementation, the monitoring result report includes the one or more monitoring results; or the monitoring result report includes at least one monitoring result among the one or more monitoring results that is greater than or equal to a first threshold; or the monitoring result report includes at least one monitoring result among the one or more monitoring results that is less than or equal to a second threshold; or the monitoring result report includes the index of the prediction information corresponding to the optimal monitoring result among the one or more monitoring results and the index of the true value information; or the monitoring result report includes the index of the prediction information corresponding to the worst monitoring result among the one or more monitoring results and the index of the true value information; or the monitoring result report includes the absolute value of the first monitoring result among multiple monitoring results, and the relative values ​​of other monitoring results among the multiple monitoring results except the first monitoring result relative to the first monitoring result.

[0043] In another possible implementation, the first monitoring result is determined based on the order of the true value information used and / or the order of the prediction information used when obtaining the multiple monitoring results.

[0044] In another possible implementation, the access network device may also instruct the terminal device to report part of the monitoring results. For example, the terminal device may be instructed to report part of all the monitoring results in a bitmap manner.

[0045] In another possible implementation, the monitoring result report and the information feedback for sending the reference signal measurement result are reported in the same information report, or the monitoring result report and the information feedback for sending the reference signal measurement result are reported in different information reports.

[0046] In any one of the above-mentioned aspects 1 to 4, the predicted information is predicted channel state information (CSI), and the true information is true CSI; or the predicted information is predicted beam information, and the true information is true beam information.

[0047] The predicted beam information may include one or more of the following: a predicted optimal beam or a reference signal received power (RSRP) of the predicted optimal beam. The true beam information may include one or more of the following: an actual optimal beam or the RSRP of the actual optimal beam. In other words, the true beam information includes at least one of the following: a true optimal beam and a true reference signal received power.

[0048] The methods of the first and second aspects above can be executed by a terminal device, or by a module applied to the terminal device (such as a processor, chip, or chip system, etc.), or by a logical node, logical module or software that can realize all or part of the terminal device functions.

[0049] The methods of the third and fourth aspects mentioned above can be executed by an access network device, or by a module (such as a processor, chip, or chip system, etc.) applied to the access network device, or by a logical node, logical module or software that can realize all or part of the functions of the access network device.

[0050] In a fifth aspect, a model monitoring device is provided for implementing the model monitoring method in the above-mentioned first aspect or any one of the implementations of the first aspect; or for implementing the model monitoring method in the above-mentioned second aspect or any one of the implementations of the second aspect. The device can be a terminal device, or a module applied to a terminal device (such as a processor, a chip, or a chip system, etc.), or a logical node, a logical module or software that can implement all or part of the functions of the terminal device. In one implementation, the model monitoring device can include a sending unit, a receiving unit, and can also include a processing unit. The sending unit and the receiving unit can be independent or combined together (can be referred to as a "transceiver unit").

[0051] In a sixth aspect, a model monitoring device is provided for implementing the model monitoring method in the third aspect or any one of the implementations of the third aspect; or for implementing the model monitoring method in the fourth aspect or any one of the implementations of the fourth aspect. The device can be an access network device, or a module applied to an access network device (such as a processor, a chip, or a chip system, etc.), or a logical node, a logical module, or software that can implement all or part of the functions of the access network device. In one implementation, the model monitoring device can include a sending unit, a receiving unit, and can also include a processing unit. The sending unit and the receiving unit can be independent or combined together (can be referred to as a "transceiver unit").

[0052] In a possible implementation, the model monitoring device in the fifth to sixth aspects includes a module for respectively executing the method in any one of the first to fourth aspects or any one of their implementations.

[0053] In another possible implementation, the model monitoring device in the fifth and sixth aspects includes a processor coupled to a memory; the processor is configured to enable the device to perform the corresponding functions in the above-mentioned model monitoring method. The memory is used to couple with the processor and store the necessary programs (instructions) and / or data for the device. Optionally, the model monitoring device may further include a communication interface for enabling communication between the device and other network elements. Optionally, the memory may be located inside or outside the model monitoring device.

[0054] In another possible implementation, the model monitoring device in the fifth to sixth aspects includes a processor and a transceiver, wherein the processor is coupled to the transceiver, and the processor is used to execute a computer program or instruction to control the transceiver to receive and send information; when the processor executes the computer program or instruction, the processor is also used to implement the above method through a logic circuit or execute code instructions. The transceiver can be a transceiver, a transceiver circuit, or an input-output interface, which is used to receive signals from other model monitoring devices other than the model monitoring device and transmit them to the processor, or to send signals from the processor to other model monitoring devices other than the model monitoring device. When the model monitoring device is a chip, the transceiver is a transceiver circuit or an input-output interface.

[0055] When the model monitoring device in the fifth and sixth aspects is a chip, the sending unit may be an output unit, such as an output circuit or a communication interface; and the receiving unit may be an input unit, such as an input circuit or a communication interface. When the model monitoring device is a terminal, the sending unit may be a transmitter or a transmitter; and the receiving unit may be a receiver or a receiver.

[0056] In a seventh aspect, a computer-readable storage medium is provided, in which a computer program or instruction is stored. When the computer program or instruction is executed, the methods described in the above aspects are implemented.

[0057] In an eighth aspect, a computer program product comprising instructions is provided. When the instructions are executed on a model monitoring device, the model monitoring device is caused to execute the methods described in the above aspects.

[0058] In a ninth aspect, a model monitoring system is provided, which includes the model monitoring device described in the fifth aspect and the model monitoring device described in the sixth aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] FIG1 is a simplified schematic diagram of a wireless communication system provided by an embodiment of the present application;

[0060] FIG2A is a schematic diagram of a neuron structure;

[0061] FIG2B is a schematic diagram of a neural network;

[0062] Figure 2C shows an AI application framework;

[0063] FIG3 is a schematic diagram of a communication system incorporating an AI network element;

[0064] 4A to 4D are schematic diagrams of a network architecture according to an embodiment of the present application;

[0065] FIG5 is a flow chart of a model monitoring method provided in an embodiment of the present application;

[0066] FIG6 is a flow chart of another model monitoring method provided in an embodiment of the present application;

[0067] FIG7 is a schematic diagram of an example of an embodiment of the present application for predicting CSI based on an AI model;

[0068] FIG8 is a flow chart of another model monitoring method provided in an embodiment of the present application;

[0069] FIG9 is a schematic diagram of reference resources according to an embodiment of the present application;

[0070] FIG10 is a flow chart of another model monitoring method provided in an embodiment of the present application;

[0071] FIG11 is a flow chart of another model monitoring method provided in an embodiment of the present application;

[0072] FIG12 is a flow chart of another model monitoring method provided in an embodiment of the present application;

[0073] FIG13 is a flow chart of another model monitoring method provided in an embodiment of the present application;

[0074] FIG14 is a schematic diagram of a monitoring result report according to an embodiment of the present application;

[0075] FIG15 is a flow chart of another model monitoring method provided in an embodiment of the present application;

[0076] FIG16 is a flow chart of another model monitoring method provided in an embodiment of the present application;

[0077] FIG17 is a flow chart of another model monitoring method provided in an embodiment of the present application;

[0078] FIG18 is a schematic structural diagram of a model monitoring device provided in an embodiment of the present application;

[0079] FIG19 is a schematic structural diagram of another model monitoring device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0080] The embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application.

[0081] In order to make the purpose, technical solutions and advantages of this application clearer, this application will be further described in detail below with reference to the accompanying drawings.

[0082] The following at least one (item) involved in this application indicates one (item) or more (items). More than one (item) refers to two (items) or more than two (items). "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. In addition, it should be understood that although the terms first, second, etc. may be used to describe each object in this application, these objects should not be limited to these terms. These terms are only used to distinguish each object from each other.

[0083] The terms "including" and "having" and any variations thereof mentioned in the following description of this application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include other steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices. It should be noted that, in this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any method or design described in this application as "exemplary" or "for example" should not be interpreted as being more preferred or more advantageous than other methods or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way.

[0084] The technology provided by this application can be applied to various communication systems. For example, the communication system can be a fourth generation (4G) th generation, 4G) communication systems (such as long term evolution (LTE) systems), fifth generation (5 thThe 5G communication system may also be referred to as a new radio (NR) system.

[0085] A network element in a communication system can send a signal to another network element or receive a signal from another network element. The signal may include information, signaling, or data, etc. The network element can also be replaced by an entity, a network entity, a device, a terminal device, a communication module, a node, a communication node, etc. The present application uses the network element as an example for description. For example, the communication system may include at least one terminal device and at least one access network device. The access network device can send a downlink signal to the terminal device, and / or the terminal device can send an uplink signal to the access network device. In addition, it can be understood that if the communication system includes multiple terminal devices, the multiple terminal devices can also send signals to each other, that is, the signal sending network element and the signal receiving network element can both be terminal devices.

[0086] The model monitoring method provided in the embodiment of the present application can be applied to wireless communication systems such as 5G, 6G, and satellite communications. Referring to Figure 1, Figure 1 is a simplified schematic diagram of the wireless communication system provided in the embodiment of the present application. As shown in Figure 1, the wireless communication system includes a wireless access network 100. The wireless access network 100 can be a next-generation (e.g., 6G or higher) wireless access network, or a traditional (e.g., 5G, 4G) wireless access network. One or more terminal devices (120a-120j, collectively referred to as 120) can be connected to each other, or connected to one or more network devices (110a, 110b, collectively referred to as 110) in the wireless access network 100. Optionally, Figure 1 is only a schematic diagram, and the wireless communication system may also include other devices, such as core network devices, wireless relay devices and / or wireless backhaul devices, etc., which are not drawn in Figure 1.

[0087] Optionally, in actual applications, the wireless communication system may include multiple network devices (also called access network devices) and multiple terminal devices at the same time. A network device can serve one or more terminal devices at the same time. A terminal device can also access one or more network devices at the same time. The embodiments of the present application do not limit the number of terminal devices and network devices included in the wireless communication system.

[0088] The network device may be an entity on the network side for transmitting or receiving signals. The network device may be an access device for a terminal device to access the wireless communication system in a wireless manner, such as a base station. The base station can broadly cover various names as follows, or be replaced with the following names, such as: RAN node, NodeB, evolved NodeB (eNB), next generation NodeB (gNB), access network equipment in open radio access network (O-RAN), relay station, access point, transmission point (TRP), transmitting point (TP), master station MeNB, secondary station SeNB, multi-standard radio (MSR) node, home base station, network controller, access node, wireless node, access point (AP), transmission node, transceiver node, building baseband unit (BBU), remote radio unit (RRU), active antenna unit (AAU), remote radio head (RRH), centralized unit (CU), distributed unit (DU), radio unit (RU), centralized unit control plane (CU control plane) A base station may be a macro base station, a micro base station, a relay node, a donor node or the like, or a combination thereof. A network device may also refer to a communication module, a modem or a chip provided in the aforementioned device or apparatus. A network device may also be a mobile switching center and a device to device (D2D), vehicle-to-everything (V2X), or machine-to-machine (M2M) communication device that performs base station functions, a network side device in a 6G network, or a device that performs base station functions in a future communication system. The network device may support networks with the same or different access technologies. The embodiments of the present application do not limit the specific technology and specific device form adopted by the network device.

[0089] Network devices can be fixed or mobile. For example, base stations 110a and 110b are stationary and are responsible for wireless transmission and reception in one or more cells from terminal device 120. The helicopter or drone 120i shown in Figure 1 can be configured to act as a mobile base station, and one or more cells can move according to the location of the mobile base station 120i. In other examples, the helicopter or drone (120i) can be configured to act as a terminal device communicating with base station 110b.

[0090] In this application, the communication device used to implement the above-mentioned access network function can be an access network device, a network device having some of the access network functions, or a device capable of supporting the implementation of the access network function, such as a chip system, a hardware circuit, a software module, or a hardware circuit and a software module. The device can be installed in the access network device or used in combination with the access network device. In the method of this application, the communication device used to implement the access network device function is described as an access network device.

[0091] A terminal device may be an entity on the user side for receiving or transmitting signals, such as a mobile phone. The terminal device may be used to connect people, objects, and machines. The terminal device may communicate with one or more core networks through a network device. The terminal device includes a handheld device with wireless connection capabilities, other processing devices connected to a wireless modem, or a vehicle-mounted device. The terminal device may be a portable, pocket-sized, handheld, computer-built-in, or vehicle-mounted mobile device. The terminal device 120 may be widely used in various scenarios, such as cellular communication, D2D, V2X, point-to-point (P2P), machine-to-machine (M2M), machine type communication (MTC), Internet of Things (IoT), virtual reality (VR), augmented reality (AR), industrial control, autonomous driving, telemedicine, smart grid, smart furniture, smart office, smart wearable, smart transportation, smart city, drones, robots, remote sensing, passive sensing, positioning, navigation and tracking, autonomous delivery and mobility, etc.Some examples of the terminal device 120 include: user equipment (UE) of the 3GPP standard, fixed equipment, mobile equipment, handheld equipment, wearable equipment, cellular phones, smart phones, session initiated protocol (SIP) phones, laptops, personal computers, smart books, vehicles, satellites, global positioning system (GPS) equipment, target tracking equipment, drones, helicopters, aircraft, ships, remote control equipment, smart home equipment, industrial equipment, personal communication service (PCS) phones, wireless local loop (WLL) stations, personal digital assistants (PDAs), wireless network cameras, tablet computers, handheld computers, mobile internet devices (MIDs), wearable devices such as smart watches, VR devices, AR devices, wireless terminals in industrial control, terminals in vehicle networking systems, wireless terminals in self-driving cars, wireless terminals in smart grids, wireless terminals in transportation safety, and smart cities. The terminal device 120 may be a wireless terminal in the above various scenarios or a device for being set up in a wireless device, for example, a communication module, a modem or a chip in the above device. The terminal device may also be referred to as a terminal, a terminal device, a user equipment (UE), a mobile station (MS), a mobile terminal (MT), etc. The terminal device may also be a terminal device in a future wireless communication system. The terminal device may be used in a dedicated network device or a general device. The embodiments of the present application do not limit the specific technology and specific device form adopted by the terminal device.

[0092] Alternatively, a terminal device can function as a base station. For example, a UE can act as a dispatching entity, providing sidelink signals between UEs in V2X, D2D, or P2P scenarios. As shown in Figure 1, a cell phone 120a and a car 120b communicate with each other using sidelink signals. Cell phone 120a and smart home device 120e communicate without relaying the communication signals through base station 110b.

[0093] In this application, the communication device used to implement the functions of the terminal device can be a terminal device, or a terminal device with some of the functions of the above terminal devices, or a device that can support the implementation of the functions of the above terminal devices, such as a chip system, which can be installed in the terminal device or used in combination with the terminal device. In this application, the chip system can be composed of chips, or it can include chips and other discrete devices. In the technical solution provided in this application, the communication device is described as a terminal device or UE as an example.

[0094] Optionally, a wireless communication system is typically composed of cells, with base stations providing cell management and communication services to multiple mobile stations (MS) in the cell. The base station includes a baseband unit (BBU) and a remote radio unit (RRU). The BBU and RRU can be placed in different locations, for example: the RRU is remote and placed in an area with high traffic volume, while the BBU is placed in a central computer room. The BBU and RRU can also be placed in the same computer room. The BBU and RRU can also be different components under the same rack. Optionally, a cell can correspond to a carrier or component carrier.

[0095] In some deployments, the network devices mentioned in the embodiments of the present application may include a CU, a DU, or both a CU and a DU, or a control plane CU node (central unit-control plane (CU-CP)), a user plane CU node (central unit-user plane (CU-UP)), and a DU node. For example, the network devices may include a gNB-CU-CP, a gNB-CU-UP, and a gNB-DU.

[0096] In some deployments, multiple RAN nodes collaborate to assist terminals in achieving wireless access, with different RAN nodes implementing portions of the base station's functionality. For example, a RAN node can be a CU, DU, CU-CP, CU-UP, or RU. The CU and DU can be separate or included in the same network element, such as the BBU. The RU can be included in a radio frequency device or radio unit, such as an RRU, AAU, or RRH.

[0097] The RAN node may support one or more types of fronthaul interfaces, with different fronthaul interfaces corresponding to DUs and RUs with different functions. If the fronthaul interface between the DU and the RU is a common public radio interface (CPRI), the DU is configured to implement one or more baseband functions, and the RU is configured to implement one or more radio frequency functions. If the fronthaul interface between the DU and the RU is another type of interface, relative to the CPRI, some of the downlink and / or uplink baseband functions, such as precoding, digital beamforming (BF), or one or more of inverse fast Fourier transform (IFFT) / cyclic prefix (CP) for downlink, are moved from the DU to the RU for implementation; and for uplink, one or more of digital beamforming (BF), or fast Fourier transform (IFFT) / cyclic prefix (CP) removal, are moved from the DU to the RU for implementation. In one possible implementation, the interface may be an enhanced common public radio interface (eCPRI). In the eCPRI architecture, the division between the DU and RU is different, corresponding to different types (category, Cat) of eCPRI, such as eCPRI Cat A, B, C, D, E, and F.

[0098] Taking eCPRI Cat A as an example, for downlink transmission, based on layer mapping, the DU is configured to implement layer mapping and one or more functions preceding it (i.e., one or more of coding, rate matching, scrambling, modulation, and layer mapping). Other functions after layer mapping (e.g., RE mapping, digital beamforming (BF), or one or more of inverse fast Fourier transform (IFFT) / cyclic prefix (CP) addition) are moved to the RU for implementation. For uplink transmission, based on RE demapping, the DU is configured to implement demapping and one or more functions preceding it (i.e., one or more of decoding, rate matching, descrambling, demodulation, inverse discrete Fourier transform (IDFT), channel equalization, and RE demapping). Other functions after demapping (e.g., one or more of digital BF or fast Fourier transform (FFT) / CP removal) are moved to the RU for implementation. It is understandable that for the functional description of DU and RU corresponding to various types of eCPRI, reference can be made to the eCPRI protocol, which will not be described in detail here.

[0099] In one possible design, the processing unit for implementing baseband functions in the BBU is called a baseband high layer (BBH) unit, and the processing unit for implementing baseband functions in the RRU / AAU / RRH is called a baseband low layer (BBL) unit.

[0100] In different systems, CU (or CU-CP and CU-UP), DU or RU may also have different names, but those skilled in the art can understand their meanings. For example, in the ORAN system, CU may also be called O-CU (Open CU), DU may also be called O-DU, CU-CP may also be called O-CU-CP, CU-UP may also be called O-CU-UP, and RU may also be called O-RU. Any unit of CU (or CU-CP, CU-UP), DU and RU in this application can be implemented by a software module, a hardware module, or a combination of a software module and a hardware module.

[0101] In the embodiments of the present application, the device for implementing the functions of the network device can be a network device; it can also be a device that can support the network device to implement the functions, such as a chip system, a hardware circuit, a software module, or a hardware circuit and a software module. The device can be installed in the network device or used in conjunction with the network device. In the embodiments of the present application, only the device for implementing the functions of the network device is used as an example to illustrate, and does not constitute a limitation on the solutions of the embodiments of the present application.

[0102] It can be understood that the present application can be applied between access network equipment and terminal equipment.

[0103] Protocol layer structure between access network equipment and terminal equipment

[0104] The communication between the access network device and the terminal device follows a certain protocol layer structure. The protocol layer structure may include a control plane protocol layer structure and a user plane protocol layer structure. For example, the control plane protocol layer structure may include the functions of protocol layers such as the radio resource control (RRC) layer, the packet data convergence protocol (PDCP) layer, the radio link control (RLC) layer, the medium access control (MAC) layer, and the physical layer. For example, the user plane protocol layer structure may include the functions of protocol layers such as the PDCP layer, the RLC layer, the MAC layer, and the physical layer. In one possible implementation, a service data adaptation protocol (SDAP) layer may also be included above the PDCP layer.

[0105] Optionally, the protocol layer structure between the access network device and the terminal device may also include an artificial intelligence (AI) layer for transmitting data related to AI functions.

[0106] Taking data transmission between access network equipment and terminal devices as an example, data transmission needs to pass through the user plane protocol layers, such as the SDAP layer, PDCP layer, RLC layer, MAC layer, and physical layer. The SDAP layer, PDCP layer, RLC layer, MAC layer, and physical layer can also be collectively referred to as the access layer. Data transmission is divided into sending or receiving based on the direction of transmission, and each of these layers is further divided into a sending part and a receiving part. Taking downlink data transmission as an example, after the PDCP layer obtains data from the upper layer, it transmits the data to the RLC layer and MAC layer. The MAC layer then generates a transport block, which is then wirelessly transmitted through the physical layer. Data is encapsulated accordingly in each layer. For example, data received by a layer from the layer above it is considered a service data unit (SDU) of that layer. After encapsulation by that layer, it becomes a protocol data unit (PDU) and is then passed to the next layer.

[0107] For example, a terminal device may also include an application layer and a non-access layer. The application layer can be used to provide services to applications installed in the terminal device. For example, downlink data received by the terminal device can be sequentially transmitted from the physical layer to the application layer, which then provides it to the application. For another example, the application layer can obtain data generated by the application and sequentially transmit the data to the physical layer for transmission to other communication devices. The non-access layer can be used to forward user data, such as forwarding uplink data received from the application layer to the SDAP layer, or forwarding downlink data received from the SDAP layer to the application layer.

[0108] In order to support AI technology in wireless networks, AI nodes may also be introduced into the network.

[0109] Optionally, the AI ​​node can be deployed in one or more of the following locations in the communication system: access network equipment, terminal equipment, or core network equipment. Alternatively, the AI ​​node can be deployed separately, for example, in a location other than any of the above devices, such as a host or cloud server in an over-the-top (OTT) system. The AI ​​node can communicate with other devices in the communication system, such as one or more of the following: network equipment, terminal equipment, or network elements of the core network.

[0110] It is understood that this application does not limit the number of AI nodes. For example, when there are multiple AI nodes, the multiple AI nodes can be divided based on function, such as different AI nodes are responsible for different functions.

[0111] It can also be understood that AI nodes can be independent devices, or they can be integrated into the same device to implement different functions, or they can be network elements in hardware devices, or they can be software functions running on dedicated hardware, or they can be virtualized functions instantiated on a platform (for example, a cloud platform). This application does not limit the specific form of the above-mentioned AI nodes.

[0112] An AI node can be an AI network element or an AI module.

[0113] One or more AI modules are provided in one or more of these network element nodes, such as core network equipment, access network nodes (RAN nodes), terminals or OAM devices. The access network node can be a separate RAN node, or it can include multiple RAN nodes, for example, including CU and DU. The CU and / or DU can also be provided with one or more AI modules. Optionally, the CU can also be split into CU-CP and CU-UP. One or more AI models are provided in the CU-CP and / or CU-UP.

[0114] The AI ​​module is used to implement the corresponding AI function. The AI ​​modules deployed in different network elements may be the same or different. The model of the AI ​​module can implement different functions according to different parameter configurations. The model of the AI ​​module can be configured based on one or more of the following parameters: structural parameters (such as the number of neural network layers, the width of the neural network, the connection relationship between layers, the weight of the neuron, the activation function of the neuron, or at least one of the bias in the activation function), input parameters (such as the type of input parameters and / or the dimension of the input parameters), or output parameters (such as the type of output parameters and / or the dimension of the output parameters). Among them, the bias in the activation function can also be called the bias of the neural network.

[0115] An AI module can have one or more models. A model can infer an output, which includes one or more parameters. The learning, training, or inference processes of different models can be deployed on different nodes or devices, or on the same node or device.

[0116] The communication system includes a RAN intelligent controller (RIC). For example, the RIC can be the above-mentioned AI module, which is used to implement AI-related functions. The RIC includes a near-real-time RIC (near-real time RIC, near-RT RIC) and a non-real-time RIC (non-real time RIC, Non-RT RIC). Among them, the non-real-time RIC mainly processes non-real-time information, such as data that is not sensitive to delay, and the delay of the data can be in the order of seconds. The real-time RIC mainly processes near-real-time information, such as data that is relatively sensitive to delay, and the delay of the data is in the order of tens of milliseconds.

[0117] Near real-time RIC is used for model training and reasoning. For example, it is used to train an AI model and use the AI ​​model for reasoning. Near real-time RIC can obtain network-side and / or terminal-side information from RAN nodes (e.g., CU, CU-CP, CU-UP, DU, and / or RU) and / or terminals. This information can be used as training data or reasoning data. Optionally, near real-time RIC can deliver the reasoning results to the RAN node and / or terminal. Optionally, the reasoning results can be exchanged between the CU and DU, and / or between the DU and RU. For example, the near real-time RIC delivers the reasoning results to the DU, and the DU sends it to the RU.

[0118] Non-real-time RIC is also used for model training and reasoning. For example, it is used to train AI models and use the models for reasoning. Non-real-time RIC can obtain network-side and / or terminal-side information from RAN nodes (such as CU, CU-CP, CU-UP, DU and / or RU) and / or terminals. This information can be used as training data or reasoning data, and the reasoning results can be submitted to the RAN node and / or terminal. Optionally, the reasoning results can be exchanged between the CU and DU, and / or between the DU and RU. For example, the non-real-time RIC submits the reasoning results to the DU, and the DU sends it to the RU.

[0119] The near-real-time RIC and non-real-time RIC can also be set up as separate network elements. Optionally, the near-real-time RIC and non-real-time RIC can also be part of other devices. For example, the near-real-time RIC is set up in a RAN node (e.g., a CU or DU), while the non-real-time RIC is set up in an OAM, a cloud server, a core network device, or other network devices.

[0120] For example, the configuration of near real-time RIC and non-real-time RIC in the network architecture may be as shown in FIG4A to FIG4D :

[0121] As shown in (a) of FIG4A , in a first possible implementation, the access network device includes a near real-time RIC module for performing model learning and / or reasoning.

[0122] As shown in (b) of FIG4A , in a second possible implementation, in a communication system, a non-real-time RIC may be included outside the access network device. Optionally, the non-real-time RIC may be located in the OAM or in the core network device.

[0123] As shown in (c) of Figure 4A, in a third possible implementation, the access network device includes a near real-time RIC, and a non-real-time RIC is also included outside the access network device. Optionally, the non-real-time RIC can be located in the OAM or core network device.

[0124] Compared to (c) in Figure 4A, the CU is separated into CU-CP and CU-UP in Figure 4B. The settings of near real-time RIC and non-real-time RIC are the same as those in (c) in Figure 4A.

[0125] As shown in Figure 4C, optionally, the access network device includes one or more AI entities, and the function of the AI ​​entity is similar to the above-mentioned near real-time RIC. Optionally, the OAM includes one or more AI entities, and the function of the AI ​​entity is similar to the above-mentioned non-real-time RIC. Optionally, the core network device includes one or more AI entities, and the function of the AI ​​entity is similar to the above-mentioned non-real-time RIC. When both the OAM and the core network device include AI entities, the models trained by their respective AI entities are different, and / or the models used for reasoning are different. In the present application, the difference in models may include at least one of the following differences: structural parameters of the model (such as the number of layers and / or weights of the model, etc.), input parameters of the model, or output parameters of the model.

[0126] Relative to Figure 4C, the access network device in Figure 4D is separated into CU and DU. Optionally, the CU may include an AI entity, and the function of the AI ​​entity is similar to the above-mentioned near real-time RIC. Optionally, the DU may include an AI entity, and the function of the AI ​​entity is similar to the above-mentioned near real-time RIC. When both the CU and the DU include AI entities, the models trained by their respective AI entities are different, and / or the models used for reasoning are different. Optionally, the CU in Figure 4D can be further split into CU-CP and CU-UP. Optionally, one or more AI models can be deployed in the CU-CP. And / or, one or more AI models can be deployed in the CU-UP. Optionally, in Figure 4C or Figure 4D, the OAM of the access network device and the OAM of the core network device can be deployed separately and independently.

[0127] It should be understood that the number and type of each device in the communication system shown in Figure 1 are for illustration only, and the present application is not limited to this. In actual applications, the communication system may also include more terminal devices, more access network devices, and other network elements, such as core network devices, and / or network elements for implementing artificial intelligence functions.

[0128] It is understandable that all or part of the functions implemented by one or more of the terminal equipment, access network equipment, core network equipment, or network elements for implementing artificial intelligence functions can be virtualized, that is, implemented by one or more of the proprietary processors or general-purpose processors and the corresponding software modules. Among them, since the terminal equipment and the access network equipment involve interfaces for air interface transmission, the transceiver functions of the interfaces can be implemented by hardware. Core network equipment, such as operation administration and maintenance (OAM) network elements, can be virtualized. Optionally, one or more functions of the virtualized terminal equipment, access network equipment, core network equipment, or network elements for implementing artificial intelligence functions can be implemented by cloud devices, such as cloud devices in over the top (OTT) systems.

[0129] For ease of understanding, the following first introduces the AI ​​technology involved in this application. It should be understood that this introduction does not limit this application.

[0130] (1) AI Model

[0131] AI refers to the intelligence exhibited by machines created by humans. Generally, AI refers to the technology that replicates human intelligence through ordinary computer programs. AI can be defined as machines or computers that mimic humans and possess cognitive functions associated with human thinking, such as learning and problem-solving. AI is able to learn from past experiences, make rational decisions, and respond quickly. The goal of AI is to understand intelligence by building computer programs capable of symbolic reasoning or deduction.

[0132] Machine learning is a path to artificial intelligence (AI), specifically using machine learning to solve AI problems. Machine learning theory primarily involves the design and analysis of algorithms that enable computers to automatically "learn." Machine learning algorithms automatically analyze data to identify patterns and use these patterns to make predictions about unknown data. Because learning algorithms involve extensive statistical theory, machine learning is particularly closely linked to inferential statistics, also known as statistical learning theory.

[0133] An AI model is an algorithm or computer program that implements AI functions. It is the concrete implementation of AI technology functions. An AI model represents the mapping relationship between the model's input and output. AI models can be neural networks, linear regression models, decision tree models, support vector machines (SVMs), Bayesian networks, Q-learning models, or other machine learning (ML) models.

[0134] (2) Neural Network

[0135] Neural networks are a specific implementation of AI or machine learning technology. According to the universal approximation theorem, neural networks can theoretically approximate any continuous function, enabling them to learn arbitrary mappings. Therefore, neural networks can accurately abstractly model complex, high-dimensional problems.

[0136] The idea of ​​a neural network originates from the neuronal structure of the brain. For example, each neuron performs a weighted sum operation on its input values ​​and outputs the result through an activation function. Figure 2A shows a schematic diagram of a neuron structure. Assume that the input of a neuron is x = [x0, x1, ..., x n ], and the weights corresponding to each input are w=[w,w1,…,w n ], where w i As x i The weight of x i Weighted. The bias of the weighted sum of the input values ​​according to the weight is, for example, b. The activation function can take many forms. Assuming that the activation function of a neuron is: y = f(z) = max(0,z), then the output of the neuron is:

[0137] For another example, if the activation function of a neuron is: y = f(z) = z, then the output of the neuron is:

[0138] Among them, b, w i 、x i It can be a decimal, an integer (such as 0, a positive integer or a negative integer), or a complex number. The activation functions of different neurons in a neural network can be the same or different.

[0139] A neural network generally includes multiple layers, each of which may include one or more neurons. By increasing the depth and / or width of a neural network, the neural network's expressive power can be improved, providing more powerful information extraction and abstract modeling capabilities for complex systems. The depth of a neural network can refer to the number of layers it comprises, and the number of neurons in each layer can be referred to as the width of that layer. In one implementation, the neural network includes an input layer and an output layer. The input layer of the neural network processes the input information it receives through neurons, passes the processing results to the output layer, and the output layer obtains the output of the neural network. In another implementation, the neural network includes an input layer, a hidden layer, and an output layer. The schematic diagram of the neural network is shown in Figure 2B. The input layer of the neural network processes the input information it receives through neurons, passes the processing results to an intermediate hidden layer, which calculates the received processing results to obtain a calculation result. The hidden layer then passes the calculation results to the output layer or an adjacent hidden layer, and the output layer ultimately obtains the output of the neural network. A neural network can include one hidden layer or multiple hidden layers connected in sequence, without limitation.

[0140] Taking the AI ​​model type as a neural network as an example, the AI ​​model involved in this application can be a deep neural network (DNN). Depending on the network construction method, DNN can include feedforward neural networks (FNN), convolutional neural networks (CNN), and recurrent neural networks (RNN).

[0141] (3) Training dataset and inference data

[0142] The training dataset is used to train the AI ​​model. The training dataset may include the input of the AI ​​model, or the input and target output of the AI ​​model. The training dataset includes one or more training data. The training data may be a training sample input to the AI ​​model or the target output of the AI ​​model. The target output may also be referred to as a label or a label sample. The training dataset is an important part of machine learning. Model training is essentially learning certain features from the training data so that the output of the AI ​​model is as close as possible to the target output, such as the difference between the output of the AI ​​model and the target output is as small as possible. The composition and selection of the training dataset can, to a certain extent, determine the performance of the trained AI model.

[0143] In addition, during the training process of an AI model (such as a neural network), a loss function can be defined. The loss function describes the gap or difference between the output value of the AI ​​model and the target output value. This application does not limit the specific form of the loss function. The training process of the AI ​​model is a process of adjusting the model parameters of the AI ​​model so that the value of the loss function is less than the threshold, or the value of the loss function meets the target requirements. For example, the AI ​​model is a neural network, and adjusting the model parameters of the neural network includes adjusting at least one of the following parameters: the number of layers, width, weights of neurons, or parameters in the activation function of neurons.

[0144] Inference data can be used as input to a trained AI model for inference. During the model inference process, the inference data is input into the AI ​​model, and the corresponding output is the inference result.

[0145] (4) AI model design

[0146] The design of an AI model mainly includes a data collection phase (e.g., collecting training data and / or inference data), a model training phase, and a model inference phase. It can further include an inference result application phase. Referring to Figure 2C, an AI application framework is illustrated. In the aforementioned data collection phase, a data source is used to provide a training data set and inference data. In the model training phase, an AI model is obtained by analyzing or training the training data provided by the data source. The AI ​​model represents the mapping relationship between the model's input and output. Learning an AI model through a model training node is equivalent to learning the mapping relationship between the model's input and output using training data. In the model inference phase, the AI ​​model trained through the model training phase is used to perform inference based on the inference data provided by the data source to obtain an inference result. This phase can also be understood as: inputting inference data into the AI ​​model, obtaining an output through the AI ​​model, and the output being the inference result. The inference result can indicate: configuration parameters used (executed) by the execution object, and / or the operation performed by the execution object. Inference results are published during the application phase. For example, inference results can be centrally planned by an actor entity. For example, the actor entity can send the inference results to one or more execution targets (e.g., core network equipment, access network equipment, or terminal devices) for execution. Furthermore, the actor entity can provide feedback on model performance to the data source to facilitate subsequent model updates and training.

[0147] It is understood that the communication system may include network elements with artificial intelligence capabilities. The aforementioned AI model design-related steps can be performed by one or more network elements with AI capabilities. In one possible design, AI capabilities (such as AI modules or AI entities) can be configured within existing network elements in the communication system to implement AI-related operations, such as AI model training and / or inference. For example, the existing network elements may be access network equipment (such as gNBs), terminal equipment, core network equipment, or network management (NM). NMs can divide network management tasks into three categories based on the actual needs of the operator's network operations: operations, administration, and maintenance. NMs are also referred to as operation administration and maintenance (OAM) network elements, or OAM for short. Operations primarily perform routine network and service analysis, forecasting, planning, and configuration; maintenance primarily involves daily operational activities such as testing and fault management of the network and its services. NMs can monitor network operating status, optimize network connectivity and performance, improve network operational stability, and reduce network maintenance costs. Or in another possible design, an independent network element can also be introduced into the communication system to perform AI-related operations, such as training AI models. The independent network element can be called an AI network element or an AI node, etc., and this application does not limit this name. The AI ​​network element can be directly connected to the access network device in the communication system, or it can be indirectly connected to the access network device through a third-party network element. Among them, the third-party network element can be a core network element such as an authentication management function (AMF) network element, a user plane function (UPF) network element, an OAM, a cloud server or other network element, without limitation. For example, referring to Figure 3, the communication system includes an access network device 110, terminal devices 120, 130, and an AI network element 140 is also introduced into the communication system shown in Figure 1 above.

[0148] In this application, a model can be inferred to obtain a single parameter or multiple parameters. The training process of different models can be deployed on different devices or nodes, or on the same device or node. The inference process of different models can be deployed on different devices or nodes, or on the same device or node.

[0149] Among them, the model parameters may include one or more of the following structural parameters of the model (such as the number of layers of the model, and / or weights, etc.), the input parameters of the model (such as input dimension, number of input ports), or the output parameters of the model (such as output dimension, number of output ports). It can be understood that the input dimension may refer to the size of an input data. For example, when the input data is a sequence, the input dimension corresponding to the sequence may indicate the length of the sequence. The number of input ports may refer to the number of input data. Similarly, the output dimension may refer to the size of an output data. For example, when the output data is a sequence, the output dimension corresponding to the sequence may indicate the length of the sequence. The number of output ports may refer to the number of output data. In response to the problem proposed in the background technology that the prediction performance obtained based on the prediction model may not reflect the actual system performance, the present application provides a model monitoring solution. The terminal device can obtain the truth information corresponding to a specific time unit, such as the required time unit, by obtaining one or more first indexes of one or more truth value information, so that the performance of the prediction model can be flexibly monitored.

[0150] In this application, "sending information to... (e.g., a terminal device)" or the related illustrations in the accompanying drawings can be understood as the destination end of the information being the terminal device. This can include sending information to the terminal device directly or indirectly. "Receiving information from... (e.g., a terminal device)" or "receiving information from... (e.g., a terminal device)", or the related illustrations in the accompanying drawings can be understood as the source end of the information being the terminal device, which can include receiving information from the terminal device directly or indirectly. The information may be processed as necessary between the source end and the destination end of the information transmission, such as format changes, etc., but the destination end can understand the valid information from the source end. Similar expressions in this application can be understood similarly and will not be repeated here.

[0151] The model monitoring method provided by the embodiment of the present application is described in detail below. It can be understood that the present application uses the access network device and the terminal device as an example to illustrate the execution subject of the interaction diagram, but the present application does not limit the execution subject of the interaction diagram. For example, the access network device in the method provided by the present application can also be a chip, chip system, or processor applied to the access network device, and can also be a logical node, logic module or software that can realize all or part of the access network device; the terminal device in the method provided by the present application can also be a chip, chip system, or processor applied to the terminal device, and can also be a logical node, logic module or software that can realize all or part of the terminal device functions.

[0152] FIG5 is a flow chart of a model monitoring method provided in an embodiment of the present application. Exemplarily, the method may include the following steps:

[0153] S501. The terminal device obtains first information.

[0154] The model monitoring involved in the embodiments of the present application refers to monitoring the performance of the AI ​​model and determining whether the AI ​​model is working properly. If the performance of the AI ​​model is poor, it is necessary to switch to non-AI mode, or replace the AI ​​model, or update the AI ​​model, etc. Model monitoring can be monitoring the accuracy of the AI ​​model output (also known as an intermediate KPI). Monitoring the accuracy of the AI ​​model output is to determine whether the performance of the AI ​​model meets the requirements by comparing the difference between the output of the AI ​​model and the corresponding label or true value. Intermediate KPIs generally include generalized cosine similarity (GCS), square generalized cosine similarity (SGCS), minimum mean square error (MSE), normalized mean square error (NMSE), etc.

[0155] Calculating the intermediate KPI of the AI ​​model requires two types of information: one is the information to be monitored, such as the prediction information output by the AI ​​model, and the other is the label information or true value information, that is, Where H is the true value information, True information refers to information that can be considered accurate. For example, information measured by a terminal device can be considered true information relative to predicted information. Generally used to represent H and The difference or similarity measure between them.

[0156] In this embodiment, the index of the truth value information is used to instruct the terminal device which truth value information to obtain.

[0157] The terminal device obtains first information. The first information includes one or more first indexes corresponding to one or more pieces of truth value information. Exemplarily, the terminal device may obtain the first information sent by the access network device or may obtain the first information pre-stored by the terminal device.

[0158] S502. The terminal device obtains one or more pieces of truth value information corresponding to the one or more first indexes.

[0159] After the terminal device obtains one or more first indexes corresponding to one or more true value information, it measures the reference signal on the time slot corresponding to the one or more indexes to obtain one or more true value information corresponding to the one or more first indexes.

[0160] The one or more true value information is used to obtain monitoring results of the performance of the AI ​​model. The output of the AI ​​model is prediction information.

[0161] Compared with the prior art which can only periodically obtain model monitoring results on a grid point, the terminal device in this embodiment can obtain the truth information corresponding to a specific time unit by obtaining one or more first indexes of one or more truth information.

[0162] According to a model monitoring method provided in an embodiment of the present application, a terminal device can obtain the truth information corresponding to a specific time unit by obtaining one or more first indexes corresponding to one or more truth information, thereby flexibly monitoring the performance of the prediction model.

[0163] Figure 6 shows a flow chart of another model monitoring method provided by an embodiment of the present application. Unlike the embodiment shown in Figure 5, this embodiment does not directly indicate which true value information the terminal device should obtain by the index of the true value information, but instead determines the true value information based on the time domain resource location of the reference signal used to measure the true value information. Exemplarily, the method may include the following steps:

[0164] S601. The terminal device obtains first information.

[0165] The reference signal used to measure true value information (i.e., for model monitoring) can be the same group of reference signals as the reference signal used for model inference (as input to the model, used to predict information), that is, they belong to the same reference signal configuration; the reference signal used to measure true value information can also be a different group of reference signals from the reference signal used for model inference, that is, a reference signal is specially configured for model monitoring.

[0166] In this embodiment, the reference signals used to measure the true value information and the reference signals used for model inference are from different groups of reference signals. The terminal device can determine the true value information based on the reference signals used to measure the true value information. Therefore, the terminal device can obtain first information including the time domain resource locations of one or more reference signals used to measure the true value information.

[0167] Exemplarily, the terminal device may obtain the above-mentioned first information sent by the access network device, or may obtain the above-mentioned first information pre-stored by the terminal device.

[0168] S602. The terminal device determines one or more pieces of truth value information based on the first information.

[0169] After the terminal device obtains the first information, it can determine one or more true value information based on the first information.

[0170] In one implementation, the terminal device may measure, by default, one or more reference signals at the time domain resource locations of the one or more reference signals carried by the first information to obtain one or more true value information.

[0171] In another implementation, if the access network device configures the time domain resource location of one or more reference signals for measuring true value information through the first information, the terminal device can measure one or more reference signals at the time domain resource location of one or more reference signals carried by the first information by default to obtain one or more true value information.

[0172] In another implementation, the terminal device may further obtain the time domain resource locations of one or more reference signals used for model inference. By default, the terminal device may measure one or more reference signals at the time domain resource locations of the one or more reference signals carried by the first information, and one or more reference signals at the time domain resource locations of the one or more reference signals used for model inference, to obtain one or more pieces of true value information.

[0173] In another implementation, the access network device may also instruct the terminal device to measure one or more reference signals at the time domain resource location of one or more reference signals carried by the first information to obtain one or more true value information, or instruct the terminal device to measure one or more reference signals at the time domain resource location of one or more reference signals used for model inference to obtain one or more true value information.

[0174] The one or more true value information is used to obtain monitoring results of the performance of the AI ​​model. The output of the AI ​​model is prediction information.

[0175] It can be seen that, in this embodiment, the time domain resource location of at least one of the one or more obtained true value information is different from the time domain resource location of the predicted information.

[0176] According to a model monitoring method provided in an embodiment of the present application, a terminal device can obtain the true value information corresponding to a specific time domain resource position by obtaining the time domain resource position of one or more reference signals used to measure the true value information, thereby flexibly monitoring the performance of the prediction model.

[0177] The above scheme can be used to monitor the CSI prediction model. In a communication system (e.g., an LTE communication system or a NR communication system), network equipment needs to determine the resource, MCS, and precoding configuration of the downlink data channel of the terminal device based on the CSI. It can be understood that CSI is a type of channel information that can reflect channel characteristics and channel quality.

[0178] CSI measurement refers to the receiver solving the channel information based on the reference signal sent by the transmitter, that is, estimating the channel information using the channel estimation method. Exemplarily, the reference signal may include one or more of the channel state information reference signal (CSI-RS), synchronization signal / physical broadcast channel block (SSB), sounding reference signal (SRS), or demodulation reference signal (DMRS). CSI-RS, SSB, and DMRS can be used to measure downlink CSI. SRS and DMRS can be used to measure uplink CSI.

[0179] Taking FDD communication scenarios as an example, in FDD communication scenarios, because uplink and downlink channels are not reciprocal or cannot be guaranteed, network equipment typically transmits a downlink reference signal to the terminal device. The terminal device performs channel and interference measurements based on the received downlink reference signal to estimate the downlink CSI. The terminal device generates a CSI report based on a protocol predefined method or a network device configuration method and feeds it back to the network device to obtain the downlink CSI.

[0180] Exemplarily, the CSI may include at least one of the following: channel quality indication (CQI), precoding matrix indicator (PMI), rank indicator (RI), CSI-RS resource indicator (CRI), layer indicator (LI), reference signal receiving power (RSRP), or signal to interference plus noise ratio (SINR). The signal to interference plus noise ratio may also be referred to as the signal to interference plus noise ratio.

[0181] The RI indicates the number of downlink transmission layers recommended by the terminal device, the CQI indicates the modulation and coding scheme supported by the current channel conditions as determined by the terminal device, and the PMI indicates the precoding recommended by the terminal device. The number of precoding layers indicated by the PMI corresponds to the RI.

[0182] It should be understood that the RI, CQI, and PMI indicated in the above CSI report are only recommended values ​​for the terminal device, and the network device may perform downlink transmission according to part or all of the information indicated in the CSI report. Alternatively, the network device may not perform downlink transmission according to the information indicated in the CSI report.

[0183] AI-based CSI prediction refers to inputting several historical / current CSI information into the AI ​​prediction model and outputting the predicted future CSI. As shown in Figure 7, it is a schematic diagram of an example of an embodiment of the present application for predicting CSI based on an AI model. The two historical / current CSI information of t-5 and t can be input into the AI ​​model, and the AI ​​model can output the CSI information at the future time t+5. The predicted CSI can be in any of the following forms: channel response, channel matrix, channel feature matrix, precoding matrix. Among them, the channel response and channel matrix are the channel itself; the channel feature matrix and precoding matrix are matrices composed of the features extracted from the channel.

[0184] Calculating the intermediate KPI of the AI ​​model requires two types of CSI information: one is the CSI to be monitored, such as the predicted CSI output by the AI ​​model, and the other is the label CSI or the true value CSI, that is, Where H is the true value CSI, The true CSI refers to the CSI that can be considered accurate. For example, the CSI measured by the terminal device can be considered as the true CSI relative to the predicted CSI. Generally used to represent H and The difference or similarity measure between them.

[0185] As mentioned above, for example, the KPI may be GCS, SGCS, MSE or NMSE.

[0186] When the KPI is GCS, is the predicted CSI of each resource unit, w i is the true CSI of each resource unit.

[0187] When the KPI is SGCS, is the predicted CSI of each resource unit, w i is the true CSI of each resource unit.

[0188] When the KPI is MSE, is the predicted CSI of each resource unit, w i is the true CSI of each resource unit.

[0189] When the KPI is NMSE, is the predicted CSI of each resource unit, w i is the true CSI of each resource unit.

[0190] Depending on whether the terminal device stores an AI model, different embodiments are used to describe how to monitor the CSI prediction model:

[0191] As shown in Figure 8, a flow chart of another model monitoring method provided in an embodiment of the present application is provided. In this embodiment, the terminal device does not store an AI model. Exemplarily, the method may include the following steps:

[0192] S801. The terminal device obtains first information.

[0193] In this embodiment, the index of the true value CSI is used to instruct the terminal device which true value CSI to obtain.

[0194] The terminal device obtains first information. The first information includes one or more first indexes corresponding to one or more true CSIs. For example, the terminal device may obtain the first information sent by the access network device or may obtain the first information pre-stored by the terminal device.

[0195] In one implementation, the terminal device can determine the index of the true value CSI by the time of the reference resource. The time of the reference resource can be defined by the protocol or configured by the access network device, and the reference resource is a time unit in the time domain (the time unit includes but is not limited to any of the following: symbol, time slot, mini-time slot, subframe, radio frame, and the embodiment of the present application is described by taking the time slot as an example). As shown in Figure 9, the reference resource is illustrated, and the time slot corresponding to the reference resource is t time slot. Taking the time slot where the reference resource is located as a reference, for example, the index of the nth true value CSI before the time slot where the reference resource is located is -n, and the index of the nth true value CSI before and after the time slot where the reference resource is located is n-1. For another example, the index of the nth true value CSI before and before the time slot where the reference resource is located is -n+1, ​​and the index of the nth true value CSI after the time slot where the reference resource is located is n. For another example, the index of the Kth true value CSI before the time slot where the reference resource is located is 0, and the index of the subsequent true value CSI increases successively, and K is a predefined parameter or a parameter configured by the access network device. For another example, the index of the K-th true value CSI after the time slot where the reference resource is located is 0, and the indexes of the previous true value CSIs increase sequentially.

[0196] The one or more first indexes corresponding to the one or more true CSIs included in the first information are indexes relative to the time slot where the reference resource is located. For example, in the example shown in FIG9 , where the time slot where the reference resource is located is time slot t, the index of the true CSI for time slot t+2 is 0, and the index of the true CSI for time slot t+4 is 1.

[0197] The access network equipment sends a channel state information-reference signal (CSI-RS) to the terminal device. The terminal device receives and measures the CSI-RS to obtain the true CSI. The CSI-RS used to measure the true CSI (i.e., for model monitoring) can be from the same group as the CSI-RS used for model inference (as model input, used to predict CSI), that is, they belong to the same CSI-RS configuration. The CSI-RS used to measure the true CSI can also be from a different group than the CSI-RS used for model inference, that is, a CSI-RS is configured specifically for model monitoring.

[0198] In another implementation, the index of the true CSI can also be the index of the CSI-RS used to measure the true CSI. Still referring to Figure 9, the time domain position of the CSI-RS used to measure the true CSI is also illustrated. The CSI-RS used for model inference is a CSI-RS with a period of 5ms (for example, the CSI-RS corresponding to the t-5, t, and t+5 time slots); the CSI-RS used for model monitoring is a non-periodic or semi-static CSI-RS, which is the CSI-RS corresponding to the t+2 and t+4 time slots. The index of the true CSI of the t+2 time slot is 0, the index of the true CSI of the t+4 time slot is 1, and the index of the true CSI of the t+5 time slot is 2.

[0199] In another implementation, the CSI-RS used to measure the true CSI may include only the CSI-RS used for monitoring. For example, in the example shown in FIG9 , the CSI reference resource is in time slot t, the index of the true CSI at time t+2 is 0, and the index of the true CSI at time t+4 is 1.

[0200] In another implementation, the CSI-RS used to measure the true CSI may also include the CSI-RS used for monitoring and the CSI-RS at the time when the CSI is predicted. For example, in the example shown in Figure 9, the index of the true CSI at time t+2 is 0, the index of the true CSI at time t+4 is 1, and the index of the true CSI at time t+5 is 2.

[0201] S802. The terminal device obtains one or more true value CSIs corresponding to one or more first indexes.

[0202] After obtaining one or more first indexes corresponding to one or more true CSIs, the terminal device measures the CSI-RS on the time slots corresponding to the one or more indexes to obtain one or more true CSIs corresponding to the one or more first indexes.

[0203] The one or more true CSIs are used to obtain monitoring results of the performance of the AI ​​model. The output of the AI ​​model is the predicted CSI.

[0204] Compared with the existing technology that can only periodically obtain model monitoring results on the grid, the terminal device in this embodiment can obtain the true value CSI corresponding to a specific time unit, such as the required time unit, by obtaining one or more first indexes of one or more true value CSIs.

[0205] Furthermore, the method may further include the following steps (indicated by dotted lines in the figure):

[0206] S803. The terminal device sends one or more true CSI values ​​to the access network device. Correspondingly, the access network device receives the one or more true CSI values.

[0207] In this embodiment, the terminal device does not store the AI ​​model and cannot perform CSI prediction. Therefore, after obtaining one or more true CSI values, the terminal device sends the one or more true CSI values ​​to the access network device.

[0208] S804. The access network device obtains one or more predicted CSIs.

[0209] The access network device stores an AI model and can use the received one or more true CSIs as input information of the model to output one or more predicted CSIs.

[0210] For example, the access network device receives the true CSI of time slots t+2 and t+4. The access network device can use the true CSI of time slots t+2 and t+4 as input information and output the predicted CSI of time slot t+5.

[0211] S805. The access network device obtains one or more monitoring results of the performance of the AI ​​model based on one or more true CSIs and one or more predicted CSIs.

[0212] The monitoring results of the AI ​​model’s performance are as follows: Where H is the true value CSI, For each true CSI, the access network device may obtain one or more monitoring results based on the true CSI and one or more predicted CSIs. Alternatively, for each predicted CSI, the access network device may obtain one or more monitoring results based on one or more true CSIs.

[0213] Furthermore, in order to compare the accuracy of the predicted CSI and the true CSI, the access network device can also use the true CSI at time T1 as the predicted CSI at time T2. For example, assuming that the CSI-RS period is 5ms and the downlink transmission is at time t+3, if it is not in the monitoring phase, there is no CSI-RS at time t+3. The access network device can use the true CSI at time t as the predicted CSI at time t+3, and obtain the model monitoring result at time t+3: KPI (H t+3 ,H t ).

[0214] In response to the problem raised in the background technology, since the true CSI at time t+3 can be obtained, the model performance at time t+3 can be obtained, so that the predicted CSI predicted by the model can reflect the actual system performance, and the predicted CSI can be used for communication.

[0215] According to a model monitoring method provided in an embodiment of the present application, a terminal device can obtain the true value CSI corresponding to a specific time unit by obtaining one or more first indexes corresponding to one or more true value CSIs, thereby flexibly monitoring the performance of the prediction model.

[0216] As shown in Figure 10, a flow chart of another model monitoring method provided in an embodiment of the present application is provided. In this embodiment, the terminal device does not store an AI model. Unlike the embodiment shown in Figure 8, this embodiment does not directly indicate which true CSI the terminal device obtains by the index of the true CSI, but determines the true CSI based on the time domain resource position of the CSI-RS used to measure the true CSI. Exemplarily, the method may include the following steps:

[0217] S1001. The terminal device obtains first information.

[0218] The CSI-RS used to measure the true CSI (i.e., for model monitoring) can be from the same group as the CSI-RS used for model inference (as model input, used to predict CSI), that is, they belong to the same CSI-RS configuration. The CSI-RS used to measure the true CSI can also be from a different group than the CSI-RS used for model inference, that is, a CSI-RS is configured specifically for model monitoring.

[0219] Still referring to Figure 9, the time domain position of the CSI-RS used to measure the true CSI is also illustrated. The CSI-RS used for model inference is a CSI-RS with a period of 5ms (for example, the CSI-RS corresponding to the time slots t-5, t, and t+5); the CSI-RS used for model monitoring is a non-periodic or semi-static CSI-RS, which is the CSI-RS corresponding to the time slots t+2 and t+4, that is, the CSI-RS used to measure the true CSI and the CSI-RS used for model inference are CSI-RSs from different groups.

[0220] In this embodiment, the CSI-RS used to measure the true CSI and the CSI-RS used for model inference are from different groups of CSI-RS. The terminal device can determine the true CSI based on the CSI-RS used to measure the true CSI. Therefore, the terminal device can obtain first information including the time-domain resource locations of one or more CSI-RS used to measure the true CSI.

[0221] Exemplarily, the terminal device may obtain the above-mentioned first information sent by the access network device, or may obtain the above-mentioned first information pre-stored by the terminal device.

[0222] S1002. The terminal device determines one or more true value CSIs based on the first information.

[0223] After the terminal device obtains the first information, it can determine one or more true value CSIs based on the first information.

[0224] In one implementation, the terminal device may, by default, measure one or more CSI-RSs at the time-domain resource locations of the one or more CSI-RSs carried in the first information to obtain one or more true CSI values. For example, in the example of FIG9 , the terminal device may measure the CSI-RSs at time t+2 and time t+4 to obtain the corresponding true CSI values, respectively.

[0225] In another implementation, if the access network device configures the time domain resource locations of one or more CSI-RSs for measuring true CSI through the first information, the terminal device can, by default, measure the one or more CSI-RSs at the time domain resource locations of the one or more CSI-RSs carried in the first information to obtain one or more true CSIs. For example, in the example of Figure 9, the terminal device can measure the CSI-RSs at times t+2 and t+4 to obtain the corresponding true CSIs, respectively.

[0226] In another implementation, the terminal device may also obtain the time domain resource location of one or more CSI-RSs used for model inference. By default, the terminal device may measure one or more CSI-RSs at the time domain resource location of the one or more CSI-RSs carried by the first information, as well as one or more CSI-RSs at the time domain resource location of the one or more CSI-RSs used for model inference, to obtain one or more true CSIs. For example, in the example of Figure 9, the terminal device may measure the CSI-RSs at times t+2, t+4, and t+5, respectively, to obtain the corresponding true CSIs.

[0227] In another implementation, the access network device may also instruct the terminal device to measure one or more CSI-RSs at the time domain resource positions of one or more CSI-RSs carried by the first information to obtain one or more true CSIs, or instruct the terminal device to measure one or more CSI-RSs at the time domain resource positions of one or more CSI-RSs used for model inference to obtain one or more true CSIs.

[0228] The one or more true CSIs are used to obtain monitoring results of the performance of the AI ​​model. The output of the AI ​​model is the predicted CSI.

[0229] It can be seen that, in this embodiment, the time domain resource position of at least one true CSI among the one or more true CSIs obtained through the above implementation manner is different from the time domain resource position of the predicted CSI.

[0230] Furthermore, the method may further include the following steps (indicated by dotted lines in the figure):

[0231] S1003. The terminal device sends one or more true CSI values ​​to the access network device. Correspondingly, the access network device receives the one or more true CSI values.

[0232] For the specific implementation of this step, reference may be made to step S803 of the embodiment shown in FIG8 , and details thereof will not be repeated here.

[0233] S1004. The access network device obtains one or more predicted CSIs.

[0234] The one or more predicted CSIs are outputs of the above-mentioned AI model.

[0235] For the specific implementation of this step, reference may be made to step S804 of the embodiment shown in FIG8 , and details thereof will not be repeated here.

[0236] S1005. The access network device obtains one or more monitoring results of the performance of the AI ​​model based on the received one or more true CSIs and one or more predicted CSIs.

[0237] For the specific implementation of this step, reference may be made to step S805 of the embodiment shown in FIG8 , and details thereof will not be repeated here.

[0238] According to a model monitoring method provided in an embodiment of the present application, a terminal device can obtain the true CSI corresponding to a specific time domain resource position by obtaining the time domain resource position of one or more CSI-RS used to measure the true CSI, thereby flexibly monitoring the performance of the prediction model.

[0239] FIG11 is a flow chart of another model monitoring method provided in an embodiment of the present application. Unlike the embodiments shown in FIG8 and FIG10, in this embodiment, the terminal device stores an AI model that can predict CSI. Exemplarily, the method may include the following steps:

[0240] S1101. The terminal device obtains the first information.

[0241] The first information includes one or more first indexes of one or more true value CSIs.

[0242] For the specific implementation of this step, reference may be made to step S801 of the embodiment shown in FIG8 , and details thereof will not be repeated here.

[0243] S1102. The terminal device obtains one or more true value CSIs corresponding to one or more first indexes.

[0244] For the specific implementation of this step, reference may be made to step S802 of the embodiment shown in FIG8 , and details thereof will not be repeated here.

[0245] S1103. The terminal device obtains one or more predicted CSIs.

[0246] This embodiment does not limit the order of the above steps S1101, S1102 and S1103. Steps S1101 and S1102 may be performed first, and then step S1103; steps S1101, S1102 and S1103 may also be performed simultaneously; or step S1103 may be performed first, and then steps S1101 and S1102.

[0247] In this embodiment, the access network device may instruct the terminal device to obtain the predicted CSI at which time points, and the terminal device obtains one or more predicted CSIs according to the instruction of the access network device.

[0248] Specifically, the access network device may send second information to the terminal device, and the terminal device may obtain one or more predicted CSIs based on the second information.

[0249] In one implementation, the second information includes one or more second indices of one or more predicted CSIs. The one or more second indices are based on the time unit corresponding to the reference resource. The time of the reference resource is defined by the protocol, and the reference resource is a time unit in the time domain (a time unit includes but is not limited to any of the following: a symbol, a time slot, a mini-time slot, a subframe, and a radio frame. The embodiment of the present application is described using a time slot as an example).

[0250] The time unit corresponding to this reference resource is any of the following:

[0251] The first time unit of the resource that reports the monitoring results; or

[0252] a second time unit in which downlink control signaling (DCI) monitored by the scheduling model is located; or

[0253] a third time unit determined based on the first time unit or the second time unit; or

[0254] Predefined / preconfigured fourth time unit.

[0255] In some embodiments of the present application, after the terminal device obtains the predicted CSI, it can generate a monitoring result report of the AI ​​model and report the monitoring result report to the access network device in the first time unit. The time unit corresponding to the reference resource can be the first time unit where the resource for reporting the monitoring result report is located. For example, the time slot corresponding to the reference resource is the time slot where the Ath CSI-RS is located before or after the time slot where the resource for reporting the monitoring result report is located, where A is a parameter configured or predefined by the access network device. For example, A=1.

[0256] In some embodiments of the present application, an access network device may schedule a terminal device to perform model monitoring. Specifically, the access network device sends a DCI to the terminal device in a second time unit, where the DCI is used to instruct the terminal device to perform model monitoring. The time unit corresponding to the reference resource may be the second time unit where the DCI for scheduling model monitoring is located.

[0257] The time unit corresponding to the reference resource may also be a third time unit determined based on the first time unit or the second time unit. For example, T is the first time unit or the second time unit, and the third time unit is T+Δt. Δt may be a parameter configured or predefined by the access network device.

[0258] After determining the time unit corresponding to the reference resource, in this implementation, one or more second indexes of one or more predicted CSIs are based on the time unit corresponding to the above-mentioned reference resource. For example, the index of the n-th predicted CSI before the time slot where the reference resource is located is -n, and the index of the n-th predicted CSI before and after the time slot where the reference resource is located is n-1. For another example, the index of the n-th predicted CSI before and before the time slot where the reference resource is located is -n+1, ​​and the index of the n-th predicted CSI after the time slot where the reference resource is located is n. For another example, the index of the K-th predicted CSI before the time slot where the reference resource is located is 0, and the indexes of the subsequent predicted CSIs increase successively, where K is a predefined parameter or a parameter configured by the access network device. For another example, the index of the K-th predicted CSI after the time slot where the CSI reference resource is located is 0, and the indexes of the previous predicted CSIs increase successively.

[0259] In another implementation, the second information includes a relative time unit value of each of the one or more predicted CSIs relative to the time unit corresponding to the reference resource. That is, the predicted CSI may also be indicated by the relative time unit value of the predicted CSI relative to the time unit corresponding to the reference resource. For example, 0 represents the time slot where the reference resource is located, -n represents the nth time slot before the time slot where the reference resource is located, and n represents the nth time slot after the time slot where the CSI reference resource is located.

[0260] After receiving the above-mentioned second information, the terminal device can generate one or more predicted CSIs based on the implementation of its own AI model. For example, as shown in Figure 9, the time slot where the reference resource is located is t, the second index included in the second information is 1, and the CSI-RS used for model inference is a CSI-RS with a period of 5ms. Therefore, the second information is used to indicate the acquisition of the predicted CSI for the t+5 time slot. The terminal device can use the CSI obtained from the actual measurements of the t-5 and t time slots as the input information of the model based on the implementation of its own AI model to obtain the predicted CSI for the t+5 time slot.

[0261] Furthermore, the method may further include the following steps (indicated by dotted lines in the figure):

[0262] S1104. The terminal device sends one or more true CSIs and one or more predicted CSIs to the access network device. Correspondingly, the access network device receives the one or more true CSIs and one or more predicted CSIs.

[0263] After obtaining one or more true CSIs and one or more predicted CSIs, the terminal device may send the one or more true CSIs and one or more predicted CSIs to the access network device.

[0264] S1105. The access network device obtains one or more monitoring results of the performance of the AI ​​model based on one or more true CSIs and one or more predicted CSIs.

[0265] For the specific implementation of this step, reference may be made to step S805 of the embodiment shown in FIG8 , and details thereof will not be repeated here.

[0266] According to a model monitoring method provided in an embodiment of the present application, a terminal device can obtain the true value CSI corresponding to a specific time unit by obtaining one or more first indexes corresponding to one or more true value CSIs, and can obtain the corresponding predicted CSI according to the instructions of the access network device, so that the access network device can obtain the model monitoring results of the actual system, so that the performance of the prediction model can be flexibly monitored, and the predicted performance can reflect the actual system performance.

[0267] As shown in Figure 12, a flow chart of another model monitoring method provided in an embodiment of the present application is provided. In this embodiment, the terminal device stores an AI model that can predict CSI. Unlike the embodiment shown in Figure 11, this embodiment determines the true CSI based on the time domain resource position of the CSI-RS used to measure the true CSI. Exemplarily, the method may include the following steps:

[0268] S1201. The terminal device obtains the first information.

[0269] The first information includes the time domain resource location of one or more CSI-RSs used to measure the true value CSI.

[0270] For the specific implementation of this step, reference may be made to step S1001 of the embodiment shown in FIG10 , which will not be described in detail here.

[0271] S1202. The terminal device determines one or more true value CSIs based on the first information.

[0272] For the specific implementation of this step, reference may be made to step S1002 of the embodiment shown in FIG10 , which will not be described in detail here.

[0273] S1203. The terminal device obtains one or more predicted CSIs.

[0274] For the specific implementation of this step, please refer to step S1103 of the embodiment shown in FIG11 , which will not be described in detail here.

[0275] S1204. The terminal device sends one or more true CSIs and one or more predicted CSIs to the access network device. Correspondingly, the access network device receives the one or more true CSIs and one or more predicted CSIs.

[0276] For the specific implementation of this step, please refer to step S1104 of the embodiment shown in FIG11 , which will not be described in detail here.

[0277] This embodiment does not limit the order of the above steps S1201, S1202 and S1203. Steps S1201 and S1202 can be executed first, and then step S1203; steps S1201, S1202 and S1203 can also be executed simultaneously; or step S1203 can be executed first, and then steps S1201 and S1202.

[0278] S1205. The access network device obtains one or more monitoring results of the performance of the AI ​​model based on one or more true CSIs and one or more predicted CSIs.

[0279] For the specific implementation of this step, reference may be made to step S805 of the embodiment shown in FIG8 , and details thereof will not be repeated here.

[0280] According to a model monitoring method provided in an embodiment of the present application, a terminal device can obtain the true CSI corresponding to a specific time domain resource position by obtaining the time domain resource position of one or more CSI-RSs used to measure the true CSI, and can obtain the corresponding predicted CSI according to the instruction of the access network device, so that the access network device can obtain the model monitoring results of the actual system, so that the performance of the prediction model can be flexibly monitored, and the predicted performance can reflect the actual system performance.

[0281] As shown in Figure 13, a flow chart of another model monitoring method provided in an embodiment of the present application is provided. Unlike the previous embodiment, in this embodiment, the terminal device stores an AI model that can predict CSI, and the terminal device can also obtain monitoring results. Exemplarily, the method may include the following steps:

[0282] S1301. The terminal device obtains the first information.

[0283] The first information includes one or more first indexes of one or more true value CSIs.

[0284] For the specific implementation of this step, reference may be made to step S801 of the embodiment shown in FIG8 , and details thereof will not be repeated here.

[0285] S1302. The terminal device obtains one or more true value CSIs corresponding to one or more first indexes.

[0286] For the specific implementation of this step, reference may be made to step S802 of the embodiment shown in FIG8 , and details thereof will not be repeated here.

[0287] S1303. The terminal device obtains one or more predicted CSIs.

[0288] For the specific implementation of this step, please refer to step S1103 of the embodiment shown in FIG11 , which will not be described in detail here.

[0289] This embodiment does not limit the order of the above steps S1301, S1302 and S1303. Steps S1301 and S1302 can be performed first, and then step S1303; steps S1301, S1302 and S1303 can also be performed simultaneously; or step S1303 can be performed first, and then steps S1301 and S1302.

[0290] S1304. The terminal device obtains one or more monitoring results of the performance of the AI ​​model based on one or more true CSIs and one or more predicted CSIs.

[0291] After the terminal device obtains one or more true CSIs and one or more predicted CSIs, it can obtain one or more monitoring results for each true CSI based on the true CSI and one or more predicted CSIs; or it can also obtain one or more monitoring results for each predicted CSI based on one or more true CSIs.

[0292] For example, the terminal device obtains 2 predicted CSIs, namely and And 2 true value CSIs, namely H t+2 and H t+4 , then the terminal device can calculate 4 KPIs, which are and

[0293] Furthermore, the access network device may also indicate which predicted CSI and which true CSI are used to calculate each monitoring result.

[0294] S1305. The terminal device sends a monitoring result report to the access network device.

[0295] After obtaining one or more of the above monitoring results, the terminal device can send a monitoring result report to the network device.

[0296] The implementation methods of sending monitoring result reports by terminal devices include but are not limited to:

[0297] In one implementation, the monitoring result report includes one or more of the aforementioned monitoring results. In this implementation, the terminal device can report all acquired monitoring results to the access network device, allowing the access network device to obtain complete monitoring results. For example, in the above example, the terminal device can report the aforementioned four KPIs.

[0298] When reporting, the terminal device can report the above one or more monitoring results to the access network device in a preset order. For example, if the monitoring result is a KPI result, for multiple predicted CSIs and / or multiple true CSIs, they can be reported in the order indicated by the access network device, or in the order of their time. For the predicted CSI and the true CSI, they can be reported in the order of the predicted CSI first and then the true CSI, or in the order of the true CSI first and then the predicted CSI. and For example, according to the order from front to back in time, and first in the order of true value CSI and then in the order of CSI to be monitored, the order of the four KPIs is

[0299] Another implementation is that, in certain scenarios, the access network device may only know whether there are monitoring results indicating good model performance, without having to obtain all monitoring results. Therefore, the monitoring result report includes at least one monitoring result greater than or equal to a first threshold among the one or more monitoring results mentioned above. The first threshold can be configured or predefined by the access network device. For example, in the above example, the terminal device can report one or more KPIs greater than or equal to the first threshold among the four KPIs mentioned above. This implementation can save signaling overhead.

[0300] Another implementation is that, in certain scenarios, the access network device may only know whether there are monitoring results indicating poor model performance, without having to obtain all monitoring results. Therefore, the monitoring result report may include at least one monitoring result that is less than or equal to the second threshold among the one or more monitoring results mentioned above. The second threshold may be configured or predefined by the access network device. The second threshold is less than the first threshold. For example, in the above example, the terminal device may report one or more KPIs among the above four KPIs that are less than or equal to the second threshold. This implementation can save signaling overhead.

[0301] Another implementation method is that, in certain scenarios, the access network device may only know whether there is a monitoring result indicating that the model performance is good, without having to obtain all monitoring results. Therefore, the monitoring result report may include the index of the predicted CSI and the index of the true value CSI corresponding to the optimal monitoring result among the above one or more monitoring results. The access network device may have received one or more predicted CSIs and one or more true value CSIs in advance, and can obtain the optimal monitoring result itself based on the index of the predicted CSI and the index of the true value CSI corresponding to the optimal monitoring result. This implementation method can save signaling overhead and has low computing power requirements for the terminal device.

[0302] Alternatively, as described above, the access network device indicates the index of one or more predicted CSIs used to calculate the monitoring results through the first message, and the same indication method as the access network device may be used, and the monitoring result report may include the sequence number of the predicted CSI corresponding to the optimal monitoring result among the above one or more monitoring results in the sequence of one or more predicted CSIs indicated by the access network device; and as described above, the access network device also indicates the index of one or more true value CSIs used to calculate the monitoring results, and the same indication method as the access network device may be used, and the monitoring result report may include the sequence number of the true value CSI corresponding to the optimal monitoring result among the above one or more monitoring results in the sequence of one or more true value CSIs indicated by the access network device.

[0303] Another implementation method is that, in certain scenarios, the access network device may only know whether there is a monitoring result indicating that the model performance is poor, without having to obtain all monitoring results. Therefore, the monitoring result report may include the index of the predicted CSI corresponding to the worst monitoring result among the one or more monitoring results and the index of the true value CSI. The access network device may have received one or more predicted CSIs and one or more true value CSIs in advance, and can obtain the worst monitoring result itself based on the index of the predicted CSI corresponding to the worst monitoring result and the index of the true value CSI. This implementation method can save signaling overhead and has low computing power requirements for the terminal device.

[0304] Alternatively, as described above, the access network device indicates the index of one or more predicted CSIs used to calculate the monitoring results through the first message, and the same indication method as the access network device may be used, and the monitoring result report may include the sequence number of the predicted CSI corresponding to the worst monitoring result among the above one or more monitoring results in the sequence of one or more predicted CSIs indicated by the access network device; and as described above, the access network device also indicates the index of one or more true value CSIs used to calculate the monitoring results, and the same indication method as the access network device may be used, and the monitoring result report may include the sequence number of the true value CSI corresponding to the worst monitoring result among the above one or more monitoring results in the sequence of one or more true value CSIs indicated by the access network device.

[0305] Another implementation is to use a differential reporting method, where the monitoring result report includes the absolute value of the first monitoring result among multiple monitoring results, and the relative values ​​of the other monitoring results except the first monitoring result relative to the first monitoring result. and For example, the reported amount is ΔKPI1, ΔKPI2, ΔKPI3, where This implementation can save signaling overhead.

[0306] Furthermore, the above-mentioned first monitoring result can be determined based on the order of the true CSI used and / or the order of the predicted CSI used when obtaining multiple monitoring results. As mentioned above, the terminal device can obtain one or more monitoring results for each true CSI based on the true CSI and one or more predicted CSIs, and for each true CSI, the access network device indicates the index of one or more true CSIs used to calculate the monitoring result; or it can also obtain one or more monitoring results for each predicted CSI based on one or more true CSIs, and for each predicted CSI, the access network device indicates the index of one or more predicted CSIs used to calculate the monitoring result through a first message. Therefore, the generation order of one or more monitoring results can be determined based on the order in which the true CSI and / or predicted CSI are used when the terminal device generates the monitoring result, so that the first monitoring result can be determined based on the generation order. For example, the first monitoring result can be the first monitoring result or the last monitoring result among the one or more monitoring results generated sequentially.

[0307] In another implementation, the access network device may also instruct the terminal device to report part of the monitoring results. For example, the terminal device may be instructed to report part of the above four KPI results in a bitmap manner.

[0308] Furthermore, the monitoring result report and the CSI feedback may be reported in the same CSI report, or may be reported in a different CSI report. The CSI feedback refers to the measurement result for reporting the CSI-RS.

[0309] If the monitoring results can be reported in the same CSI report as the CSI feedback, the CSI report used for CSI feedback can be configured by the access network device or predefined. For example, it is predefined to report in the first CSI report after the last CSI-RS used to measure the true value CSI, or in the first CSI report after T1 time after the last CSI-RS used to measure the true value CSI. T1 is the time reserved for the terminal device to calculate the KPI, which can be configured by the access network device or predefined. T1 may be related to the processing capability of the terminal device. It should be noted that if the monitoring result report and the CSI feedback can be reported in the same CSI report, the predicted CSI reported in the CSI report and the predicted CSI to be monitored corresponding to the monitoring result report are not the same CSI. As shown in Figure 14, it is a schematic diagram of reporting the monitoring result report of an example embodiment of the present application. The CSI report is reported at time t+7, and the predicted CSI reported in the CSI report at time t+7 is the predicted CSI at time t+10, which is predicted by the true value CSI at time t+5 and before; and the predicted CSI to be monitored corresponding to the monitoring result report in the CSI report is the predicted CSI at time t+5, which is predicted by the true value CSI at time t and before.

[0310] If the monitoring result report is independent of the CSI feedback report, the reporting resource for the monitoring result report is configured by the access network device. The terminal device does not expect the reporting resource to be earlier than the last CSI-RS for measuring the true value CSI corresponding to the monitoring result report, or the terminal device does not expect the reporting resource to be earlier than T2 time after the last CSI-RS for measuring the true value CSI corresponding to the monitoring result. T2 is the time reserved for the terminal device to calculate the KPI, which can be configured or predefined by the access device. T2 may be related to the processing capability of the terminal device.

[0311] According to a model monitoring method provided in an embodiment of the present application, a terminal device can obtain the true value CSI corresponding to a specific time unit by obtaining one or more first indexes corresponding to one or more true value CSIs, and can obtain the corresponding predicted CSI according to the instruction of the access network device, and obtain the model monitoring result based on the obtained true value CSI and predicted CSI, so that the access network device can obtain the model monitoring result of the actual system, so that the performance of the prediction model can be flexibly monitored, and the predicted performance can reflect the actual system performance.

[0312] As shown in Figure 15, a flow chart of another model monitoring method provided in an embodiment of the present application is shown. Unlike the embodiments shown in Figures 8 and 10, in this embodiment, the terminal device stores an AI model that can predict CSI, and the terminal device can also obtain monitoring results. Unlike the embodiment shown in Figure 13, this embodiment determines the true CSI based on the time domain resource position of the CSI-RS used to measure the true CSI. Exemplarily, the method may include the following steps:

[0313] S1501. The terminal device obtains the first information.

[0314] The first information includes the time domain resource location of one or more CSI-RSs used to measure the true value CSI.

[0315] For the specific implementation of this step, reference may be made to step S1001 of the embodiment shown in FIG10 , which will not be described in detail here.

[0316] S1502. The terminal device determines one or more true value CSIs based on the first information.

[0317] For the specific implementation of this step, reference may be made to step S1002 of the embodiment shown in FIG10 , and details thereof will not be repeated here.

[0318] S1503. The terminal device obtains one or more predicted CSIs.

[0319] For the specific implementation of this step, please refer to step S1103 of the embodiment shown in FIG11 , which will not be described in detail here.

[0320] S1504. The terminal device obtains one or more monitoring results of the performance of the AI ​​model based on one or more true CSIs and one or more predicted CSIs.

[0321] For the specific implementation of this step, please refer to step S1304 of the embodiment shown in FIG13 , which will not be described in detail here.

[0322] S1505. The terminal device sends a monitoring result report to the access network device.

[0323] For the specific implementation of this step, please refer to step S1305 of the embodiment shown in FIG13 , which will not be described in detail here.

[0324] According to a model monitoring method provided in an embodiment of the present application, a terminal device can obtain the true CSI corresponding to a specific time domain resource position by obtaining the time domain resource position of one or more CSI-RSs used to measure the true CSI, and can obtain the corresponding predicted CSI according to the instruction of the access network device, and obtain the model monitoring result based on the obtained true CSI and predicted CSI, so that the access network device can obtain the model monitoring result of the actual system, so that the performance of the prediction model can be flexibly monitored, and the predicted performance can reflect the actual system performance.

[0325] The solution described in the above embodiment can also be used in beam prediction scenarios:

[0326] Beam prediction is to predict the optimal beam in the future based on historical / current beam measurement results.

[0327] The KPIs for beam prediction can be Top-1 accuracy, Top-1 accuracy with ndB margin, Top-K accuracy, reference signal receiving power (RSRP) difference, etc.

[0328] Among them, Top-1 accuracy refers to whether the predicted optimal beam is the actual optimal beam, or the proportion of the predicted optimal beam in multiple monitorings that is the actual optimal beam;

[0329] Top1 accuracy with ndB margin refers to whether the difference between the predicted RSRP of the best beam and the actual RSRP of the best beam is less than ndB, or the proportion of the difference between the predicted RSRP of the best beam and the actual RSRP of the best beam in multiple monitorings that is less than ndB;

[0330] Top-K accuracy refers to whether the predicted optimal beam is among the actual K optimal beams, or the ratio of the predicted optimal beam to the actual K optimal beams in multiple monitorings, or whether the actual optimal beam is among the predicted K optimal beams, or the ratio of the actual optimal beam to the predicted K optimal beams in multiple monitorings;

[0331] The RSRP difference refers to the difference between the RSRP of the predicted optimal beam and the RSRP of the actual optimal beam, or the difference between the RSRP of the predicted optimal beam and the actual RSRP of the beam.

[0332] FIG16 is a flow chart of another model monitoring method provided in an embodiment of the present application. Exemplarily, the method may include the following steps:

[0333] S1601. The terminal device obtains the first information.

[0334] Similar to CSI prediction, beam prediction KPI calculation requires two types of information: prediction information, such as the predicted optimal beam index or one or more RSRP values, and ground truth information, such as the actual optimal beam index or one or more RSRP values. The access network device can also instruct the terminal device which prediction information and ground truth information to use for KPI calculation.

[0335] In this embodiment, the index of the true value beam information is used to indicate to the terminal device which true value beam information to obtain.

[0336] The terminal device obtains first information. The first information includes one or more first indexes corresponding to information of one or more true value beams. Exemplarily, the terminal device may obtain the first information sent by the access network device or may obtain the first information pre-stored by the terminal device.

[0337] For the specific implementation of this step, reference may be made to the specific implementation of obtaining the first information in CSI prediction in the embodiment shown in FIG8 , and details thereof will not be repeated here.

[0338] S1602. The terminal device obtains information of one or more true value beams corresponding to one or more first indexes.

[0339] After the terminal device obtains the one or more first indexes corresponding to the information of one or more true value beams, it measures the reference signals corresponding to the one or more indexes, such as CSI-RS, to obtain the one or more true value beams corresponding to the one or more first indexes.

[0340] The information of the one or more true beams is used to obtain monitoring results of the performance of the AI ​​model. The output of the AI ​​model is the information of the predicted beam.

[0341] Compared with the existing technology that can only periodically obtain model monitoring results on the grid, the terminal device in this embodiment can obtain the information of the truth beam corresponding to a specific time unit, such as the required time unit, by obtaining one or more first indexes of the information of one or more truth beams.

[0342] Furthermore, the terminal device can send information of one or more true value beams to the access network device, the access network device obtains information of one or more predicted beams, and obtains one or more monitoring results of the performance of the AI ​​model based on the information of one or more true value beams and the information of one or more predicted beams.

[0343] Alternatively, the terminal device may also obtain information about one or more prediction beams, and send information about one or more true value beams and information about one or more prediction beams to the access network device, and the access network device may obtain one or more monitoring results of the performance of the AI ​​model based on the information about one or more true value beams and the information about one or more prediction beams.

[0344] According to a model monitoring method provided in an embodiment of the present application, the terminal device can obtain the information of the truth beam corresponding to a specific time unit by obtaining one or more first indexes corresponding to the information of one or more truth beams, thereby flexibly monitoring the performance of the prediction model.

[0345] FIG17 is a flow chart of another model monitoring method provided by an embodiment of the present application. Unlike the embodiment shown in FIG16 , this embodiment does not directly indicate which true beam information the terminal device should obtain by using the index of the true beam information. Instead, the true beam information is determined based on the time domain resource location of the reference signal used to measure the true beam information. Exemplarily, the method may include the following steps:

[0346] S1701. The terminal device obtains the first information.

[0347] The reference signal used to measure the true beam information (i.e., used for model monitoring) can be the same group of reference signals as the reference signal used for model inference (as input to the model, used to predict beam information), that is, they belong to the same reference signal configuration; the reference signal used to measure the true beam information can also be a different group of reference signals from the reference signal used for model inference, that is, a reference signal is specially configured for model monitoring.

[0348] In this embodiment, the reference signals used to measure the true beam information and the reference signals used for model inference are from different groups of reference signals. The terminal device can determine the true beam information based on the reference signals used to measure the true beam information. Therefore, the terminal device can obtain first information including the time-domain resource locations of one or more reference signals used to measure the true beam information.

[0349] Exemplarily, the terminal device may obtain the above-mentioned first information sent by the access network device, or may obtain the above-mentioned first information pre-stored by the terminal device.

[0350] For the specific implementation of this step, reference may be made to the specific implementation of obtaining the first information in CSI prediction in the embodiment shown in FIG10 , which will not be described in detail here.

[0351] S1702. The terminal device determines information of one or more true value beams based on the first information.

[0352] After the terminal device obtains the first information, it can determine the information of one or more true value beams based on the first information.

[0353] In one implementation, the terminal device can measure one or more reference signals at the time domain resource positions of the one or more reference signals carried by the first information by default to obtain information of one or more true value beams.

[0354] In another implementation, if the access network device configures the time domain resource location of one or more reference signals for measuring the information of the true value beam through the first information, the terminal device can measure the one or more reference signals at the time domain resource location of the one or more reference signals carried by the first information by default to obtain the information of one or more true value beams.

[0355] In another implementation, the terminal device may further obtain the time domain resource locations of one or more reference signals used for model inference. By default, the terminal device may measure one or more reference signals at the time domain resource locations of the one or more reference signals carried by the first information, and one or more reference signals at the time domain resource locations of the one or more reference signals used for model inference, to obtain information about one or more true beams.

[0356] In another implementation, the access network device may also instruct the terminal device to measure one or more reference signals at the time domain resource position of one or more reference signals carried by the first information to obtain information of one or more true value beams, or instruct the terminal device to measure one or more reference signals at the time domain resource position of one or more reference signals used for model inference to obtain information of one or more true value beams.

[0357] The information of the one or more true beams is used to obtain monitoring results of the performance of the AI ​​model. The output of the AI ​​model is the information of the predicted beam.

[0358] It can be seen that, in this embodiment, the time domain resource position of at least one true value beam among the obtained one or more true value beam information is different from the time domain resource position of the predicted beam information.

[0359] Furthermore, the terminal device can send information of one or more true value beams to the access network device, the access network device obtains information of one or more predicted beams, and obtains one or more monitoring results of the performance of the AI ​​model based on the information of one or more true value beams and the information of one or more predicted beams.

[0360] Alternatively, the terminal device may also obtain information about one or more prediction beams, and send information about one or more true value beams and information about one or more prediction beams to the access network device, and the access network device may obtain one or more monitoring results of the performance of the AI ​​model based on the information about one or more true value beams and the information about one or more prediction beams.

[0361] According to a model monitoring method provided in an embodiment of the present application, a terminal device can obtain the information of the true value beam corresponding to a specific time domain resource position by obtaining the time domain resource position of one or more reference signals for measuring the information of the true value beam, thereby flexibly monitoring the performance of the prediction model.

[0362] It can be understood that in the above embodiments, the methods and / or steps implemented by the access network device can also be implemented by components (such as chips or circuits) that can be used for the access network device; the methods and / or steps implemented by the terminal device can also be implemented by components (such as chips or circuits) that can be used for the terminal device.

[0363] The above description primarily describes the solutions provided by the embodiments of the present application from the perspective of interactions between various network elements. Accordingly, the embodiments of the present application also provide a model monitoring device, which is used to implement the various methods described above. The model monitoring device can be the access network device described in the method embodiments described above, or a component that can be used in the access network device; alternatively, the model monitoring device can be the terminal device described in the method embodiments described above, or a component that can be used in the terminal device. It will be understood that, to implement the aforementioned functions, the model monitoring device includes hardware structures and / or software modules corresponding to the respective functions. Those skilled in the art will readily appreciate that, in conjunction with the various exemplary units and algorithm steps described in the embodiments disclosed herein, the present application can be implemented in hardware or a combination of hardware and computer software. Whether a function is implemented in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0364] In the embodiment of the present application, the functional modules of the model monitoring device can be divided according to the above method embodiment. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing unit. The above integrated modules can be implemented in the form of hardware or software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical functional division. In actual implementation, there may be other division methods.

[0365] Based on the same concept of the above model monitoring method, this application also provides the following model monitoring device:

[0366] FIG18 is a schematic diagram showing the structure of a model monitoring device provided in an embodiment of the present application. The model monitoring device 1800 includes a transceiver unit 1801 and a processing unit 1802.

[0367] When the model monitoring device is used to implement the functions of the terminal device in the above method embodiment, the processing unit 1802 is used to perform the operations of S601 and S602 in the embodiment shown in Figure 6; or the processing unit 1802 is used to perform the operations of S701 and S702 in the embodiment shown in Figure 7; the transceiver unit 1801 is used to perform the operation of the terminal device in S803 in the embodiment shown in Figure 8, and the processing unit 1802 is used to perform the operations of S801 and S802 in the embodiment shown in Figure 8; or the transceiver unit 1801 is used to perform the operation of the terminal device in S1003 in the embodiment shown in Figure 10, and the processing unit 1802 is used to perform the operations of S1001 and S1002 in the embodiment shown in Figure 10; or the transceiver unit 1801 is used to perform the operation of the terminal device in S1104 in the embodiment shown in Figure 11, and the processing unit 1802 is used to perform steps S1101 to S1102 in the embodiment shown in Figure 11. S1103; or, the transceiver unit 1801 is used to perform the operation of the terminal device in S1204 in the embodiment shown in Figure 12, and the processing unit 1802 is used to perform steps S1201 to S1203 in the embodiment shown in Figure 12; or, the transceiver unit 1801 is used to perform the operation of the terminal device in S1305 in the embodiment shown in Figure 13, and the processing unit 1802 is used to perform steps S1301 to S1304 in the embodiment shown in Figure 13; or, the transceiver unit 1801 is used to perform the operation of the terminal device in S1505 in the embodiment shown in Figure 15, and the processing unit 1802 is used to perform steps S1501 to S1504 in the embodiment shown in Figure 15; or, the processing unit 1802 is used to perform the operations of S1601 and S1602 in the embodiment shown in Figure 16; or, the processing unit 1802 is used to perform the operations of S1701 and S1702 in the embodiment shown in Figure 17.

[0368] When the model monitoring device is used to implement the functions of the access network device in the above-mentioned method embodiment, the transceiver unit 1801 is used to perform the operation of the access network device in S803 in the embodiment shown in FIG8 , and the processing unit 1802 is used to perform the operations of S804 and S605 in the embodiment shown in FIG8 ; or, the transceiver unit 1801 is used to perform the operation of the access network device in S1003 in the embodiment shown in FIG10 , and the processing unit 1802 is used to perform the operations of S1004 and S1005 in the embodiment shown in FIG10 ; or, the transceiver unit 1801 is used to perform the operation of the access network device in S1003 in the embodiment shown in FIG10 . The operation of the access network device in S1104 in the example, and the processing unit 1802 is used to execute step S1105 in the embodiment shown in Figure 11; or, the transceiver unit 1801 is used to execute the operation of the access network device in S1204 in the embodiment shown in Figure 12, and the processing unit 1802 is used to execute step S1205 in the embodiment shown in Figure 12; or, the transceiver unit 1801 is used to execute the operation of the access network device in S1305 in the embodiment shown in Figure 13; or, the transceiver unit 1801 is used to execute the operation of the access network device in S1505 in the embodiment shown in Figure 15.

[0369] For the specific implementation of the above-mentioned transceiver unit 1801 and the processing unit 1802, reference may be made to the description in the above-mentioned method embodiment.

[0370] As shown in Figure 19, it is a schematic diagram of the structure of another model monitoring device provided in an embodiment of the present application. The model monitoring device 1900 includes one or more processors 1901 (one processor is illustrated in the figure). Optionally, the model monitoring device 1900 may also include a memory 1903 (indicated by a dotted line in the figure). The memory 1903 is used to store instructions executed by the processor 1901, or to store input data required for the processor 1901 to run instructions, or to store data generated after the processor 1901 runs instructions. Optionally, the model monitoring device 1900 may also include an interface circuit 1902 (indicated by a dotted line in the figure), and the processor 1901 and the interface circuit 1902 are coupled to each other. It can be understood that the interface circuit 1902 can be a transceiver or an input and output interface. Among them, the processor 1901 is used to implement the function of the processing unit 1802 in the embodiment shown in Figure 18 above; and the interface circuit 1902 is used to implement the function of the transceiver unit 1801 in the embodiment shown in Figure 18 above.

[0371] When the model monitoring device is a chip implemented in an access network device, the chip implements the functions of the access network device in the above-described method embodiments. The chip receives information from other modules in the access network device (e.g., a radio frequency module or antenna), which is information sent by the terminal device to the access network device; or the chip sends information to other modules in the access network device (e.g., a radio frequency module or antenna), which is information sent by the access network device to the terminal device.

[0372] When the model monitoring device is a chip implemented in a terminal device, the chip implements the functions of the terminal device in the above-described method embodiments. The chip receives information from other modules in the terminal device (such as a radio frequency module or antenna), which is sent by the access network device to the terminal device; or the chip sends information to other modules in the terminal device (such as a radio frequency module or antenna), which is sent by the terminal device to the access network device.

[0373] In addition, it should be noted that the aforementioned transceiver unit and / or processing unit may be implemented through virtual modules, for example, the processing unit may be implemented through a software function unit or a virtual device, and the transceiver unit may be implemented through a software function or a virtual device. Alternatively, the processing unit or transceiver unit may also be implemented through a physical device, for example, if the device is implemented using a chip / chip circuit, the transceiver unit may be an input / output circuit and / or a communication interface, performing input operations (corresponding to the aforementioned receiving operations) and output operations (corresponding to the aforementioned sending operations); the processing unit is an integrated processor or microprocessor or integrated circuit.

[0374] The division of modules in this application is illustrative and represents only a logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the examples of this application may be integrated into a single processor, exist physically as separate modules, or two or more modules may be integrated into a single module. The aforementioned integrated modules may be implemented in either hardware or software functional modules.

[0375] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0376] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program or instruction is stored. When the computer program or instruction is executed, the method in the above embodiment is implemented.

[0377] An embodiment of the present application further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method in the above embodiment.

[0378] An embodiment of the present application also provides a model monitoring system, including the above-mentioned model monitoring device.

[0379] The present application also provides a circuit, which is coupled to a memory and is used to execute the method shown in the above embodiment. The circuit may include a chip circuit.

[0380] When the above-mentioned model monitoring device is a module applied to a base station, the base station module implements the functions of the base station in the above-mentioned method embodiment. The base station module receives information from other modules in the base station (such as a radio frequency module or an antenna), and the information is sent by the terminal device to the base station; or, the base station module sends information to other modules in the base station (such as a radio frequency module or an antenna), and the information is sent by the base station to the terminal device. The base station module here can be the baseband chip of the base station, or it can be a CU, DU or other module, or it can be a device under the open radio access network (O-RAN) architecture, such as an open CU, open DU and other devices.

[0381] It should be noted that the above units or one or more of the units can be implemented by software, hardware, or a combination of the two. When any of the above units or units is implemented by software, the software exists in the form of computer program instructions and is stored in a memory, and a processor can be used to execute the program instructions and implement the above method flow.

[0382] In this application, a processor may be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, or all or part of the circuitry in the aforementioned devices used to implement processing functions, which may implement or execute the various methods, steps, and logic block diagrams disclosed in this application. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in this application may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0383] When the above units or units are implemented in hardware, the hardware can be any one or any combination of a CPU, a microprocessor, a digital signal processing (DSP) chip, a microcontroller unit (MCU), an artificial intelligence processor, an ASIC, a SoC, an FPGA, a PLD, a dedicated digital circuit, a hardware accelerator or a non-integrated discrete device, which can run the necessary software or not rely on the software to execute the above method flow.

[0384] Optionally, an embodiment of the present application further provides a chip system, comprising: at least one processor and an interface, wherein the at least one processor is coupled to a memory via the interface, and when the at least one processor executes a computer program or instruction in the memory, the chip system executes the method in any of the above method embodiments. Optionally, the chip system may be composed of a chip, or may include a chip and other discrete devices, which is not specifically limited in the embodiments of the present application.

[0385] The memory in the present application can also be a circuit or any other device capable of realizing a storage function, for storing program instructions and / or data. A memory is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. For example, the memory can be a non-volatile memory, such as a digital versatile disc (DVD), a hard disk drive (HDD) or a solid-state drive (SSD), etc., or a volatile memory (volatile memory), such as a random-access memory (RAM).

[0386] It should be understood that in the description of this application, unless otherwise specified, " / " indicates that the objects associated with each other are in an "or" relationship. For example, A / B can mean A or B; where A and B can be singular or plural. Also, in the description of this application, unless otherwise specified, "multiple" means two or more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural. In addition, to facilitate the clear description of the technical solutions of the embodiments of this application, in the embodiments of this application, words such as "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. Those skilled in the art will understand that words such as "first" and "second" do not limit the quantity or execution order, and words such as "first" and "second" do not necessarily mean different. At the same time, in the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner to facilitate understanding.

[0387] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using a software program, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0388] Although the present application is described herein in conjunction with various embodiments, in the process of implementing the claimed application, those skilled in the art can understand and implement other changes to the disclosed embodiments by reviewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple situations. A single processor or other unit can implement several functions listed in the claims. Certain measures are recorded in different dependent claims, but this does not mean that these measures cannot be combined to produce good results.

[0389] It is understood that the various numbers used in the embodiments of this application are merely for ease of description and are not intended to limit the scope of the embodiments of this application. The order of the sequence numbers of the above-mentioned processes does not necessarily imply a specific order of execution; the order of execution of the processes should be determined by their functions and inherent logic.

[0390] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0391] The components in the device of the embodiment of the present application can be merged, divided, or deleted according to actual needs. Those skilled in the art can combine or combine the different embodiments and features of the different embodiments described in this specification.

[0392] In this application, under the premise of no logical contradiction, the examples can reference each other, for example, the methods and / or terms between method embodiments can reference each other, for example, the functions and / or terms between device embodiments can reference each other, for example, the functions and / or terms between device examples and method examples can reference each other.

Claims

1. A model monitoring method, characterized in that: The method comprises: Acquire first information, where the first information includes one or more first indexes corresponding to one or more pieces of true value information respectively; Obtain one or more pieces of truth value information respectively corresponding to the one or more first indexes; Among them, the one or more true value information is used to obtain monitoring results of the performance of the artificial intelligence AI model, and the output of the AI ​​model is prediction information.

2. A model monitoring method, characterized in that: The method comprises: Acquire first information, wherein the first information includes time domain resource locations of one or more reference signals used to measure true value information; Determine one or more pieces of truth information based on the first information; Among them, the one or more truth value information are used to obtain the monitoring results of the performance of the AI ​​model, the output of the AI ​​model is prediction information, and the time domain resource location of at least one of the one or more truth value information is different from the time domain resource location of the prediction information.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: The one or more truth value information are sent.

4. The method according to claim 1 or 2, characterized in that: The method further comprises: Obtain one or more prediction information, wherein the one or more prediction information is output of an artificial intelligence (AI) model; The one or more prediction information are sent.

5. The method according to claim 1 or 2, characterized in that: The method further comprises: Obtain one or more prediction information, wherein the one or more prediction information is output of an artificial intelligence (AI) model; Based on the one or more true value information and the one or more prediction information, obtain one or more monitoring results of the performance of the AI ​​model.

6. The method according to claim 4 or 5, characterized in that: The obtaining one or more prediction information includes: receiving second information, where the second information includes one or more second indexes corresponding to the one or more prediction information respectively, or the second information includes a relative time unit value of each prediction information in the one or more prediction information relative to a time unit corresponding to a reference resource; According to the second information, one or more prediction information is obtained.

7. The method according to claim 6, characterized in that The one or more first indexes and the one or more second indexes are based on a time unit corresponding to a reference resource.

8. The method according to claim 7, characterized in that The time unit corresponding to the reference resource is any one of the following: The first time unit of the resource that reports the monitoring result; or Scheduling a second time unit in which the downlink control signaling monitored by the model is located; or a third time unit determined based on the first time unit or the second time unit; or Predefined / preconfigured fourth time unit.

9. The method according to claim 5, characterized in that The method further comprises: Send monitoring result reports; Wherein, the monitoring result report includes the one or more monitoring results; or The monitoring result report includes at least one monitoring result of the one or more monitoring results that is greater than or equal to a first threshold; or The monitoring result report includes at least one monitoring result of the one or more monitoring results that is less than or equal to a second threshold; or The monitoring result report includes an index of prediction information and an index of true value information corresponding to the best monitoring result among the one or more monitoring results; or The monitoring result report includes an index of prediction information and an index of true value information corresponding to the worst monitoring result among the one or more monitoring results; or The monitoring result report includes an absolute value of a first monitoring result among a plurality of monitoring results, and relative values ​​of other monitoring results among the plurality of monitoring results except the first monitoring result with respect to the first monitoring result.

10. The method according to claim 9, characterized in that The first monitoring result is determined based on an order of the true value information used and / or an order of the prediction information used when acquiring the plurality of monitoring results.

11. The method according to claim 9 or 10, characterized in that The monitoring result report includes some monitoring results among all monitoring results.

12. The method according to any one of claims 9 to 11, characterized in that The monitoring result report and the information feedback for sending the reference signal measurement result are reported in the same information report, or the monitoring result report and the information feedback for sending the reference signal measurement result are reported in different information reports.

13. A model monitoring method, characterized in that: The method comprises: Sending first information, wherein the first information includes one or more first indexes corresponding to one or more pieces of true value information respectively; Receive one or more pieces of truth value information respectively corresponding to the one or more first indexes; Among them, the one or more true value information is used to obtain monitoring results of the performance of the artificial intelligence AI model, and the output of the AI ​​model is prediction information.

14. A model monitoring method, characterized in that: The method comprises: Sending first information, wherein the first information includes time domain resource locations of one or more reference signals used to measure true value information; receiving one or more truth value information according to the first information; Among them, the one or more truth value information are used to obtain the monitoring results of the performance of the AI ​​model, the output of the AI ​​model is prediction information, and the time domain resource location of at least one of the one or more truth value information is different from the time domain resource location of the prediction information.

15. The method according to claim 13 or 14, characterized in that The method further comprises: obtaining one or more forecast information; Based on the one or more true value information and the one or more prediction information, obtain one or more monitoring results of the performance of the AI ​​model.

16. The method according to claim 13 or 14, characterized in that The method further comprises: A monitoring result report is received, where the monitoring result report includes one or more monitoring results, where the one or more monitoring results are obtained based on the one or more true value information and the one or more prediction information.

17. The method according to claim 15, characterized in that The method comprises: Sending second information, where the second information includes one or more second indexes corresponding to the one or more prediction information respectively, or the second information includes a relative time unit value of each prediction information in the one or more prediction information relative to a time unit corresponding to a reference resource; The one or more prediction information is received.

18. The method according to claim 17, characterized in that The one or more first indexes and the one or more second indexes are based on a time unit corresponding to a reference resource.

19. The method according to claim 18, characterized in that The time unit corresponding to the reference resource is any one of the following: The first time unit of the resource that reports the monitoring result; or Scheduling a second time unit in which the downlink control signaling monitored by the model is located; or a third time unit determined based on the first time unit or the second time unit; or Predefined / preconfigured fourth time unit.

20. The method according to claim 16, characterized in that: The monitoring result report includes the one or more monitoring results; or The monitoring result report includes at least one monitoring result of the one or more monitoring results that is greater than or equal to a first threshold; or The monitoring result report includes at least one monitoring result of the one or more monitoring results that is less than or equal to a second threshold; or The monitoring result report includes an index of prediction information and an index of true value information corresponding to the best monitoring result among the one or more monitoring results; or The monitoring result report includes an index of prediction information and an index of true value information corresponding to the worst monitoring result among the one or more monitoring results; or The monitoring result report includes an absolute value of a first monitoring result among a plurality of monitoring results, and relative values ​​of other monitoring results among the plurality of monitoring results except the first monitoring result with respect to the first monitoring result.

21. The method according to claim 20, characterized in that The first monitoring result is determined based on an order of the true value information used and / or an order of the prediction information used when acquiring the plurality of monitoring results.

22. The method according to claim 20 or 21, characterized in that The monitoring result report includes some monitoring results among all monitoring results.

23. The method according to any one of claims 20 to 22, characterized in that The monitoring result report and the information feedback for sending the reference signal measurement result are reported in the same information report, or the monitoring result report and the information feedback for sending the reference signal measurement result are reported in different information reports.

24. The method according to any one of claims 1 to 23, characterized in that The predicted information is predicted channel state information CSI, and the true value information is true value CSI; or the predicted information is predicted beam information, and the true value information is true value beam information.

25. A model monitoring device, characterized in that: Comprising modules for executing the method as claimed in any one of claims 1-24.

26. A model monitoring device, characterized in that: The device comprises a processor coupled to a memory, wherein the processor is configured to enable the device to perform corresponding functions in the method according to any one of claims 1 to 24.

27. A computer-readable storage medium having a computer program or instruction stored therein, characterized in that: When the computer program or instruction is executed, the method according to any one of claims 1 to 24 is implemented.

28. A computer program product comprising instructions, characterized in that When the instructions are executed on a model monitoring device, the model monitoring device is caused to execute the method according to any one of claims 1 to 24.

Images