Model monitoring methods, apparatus, and systems, storage media, and program products

Abstract

This application discloses a model monitoring method, apparatus, and system, storage medium, and program product. The method includes the steps of a terminal device obtaining first information, wherein the first information includes one or more first indices of one or more ground truth information, and obtaining one or more ground truth information corresponding to the one or more first indices, wherein the one or more ground truth information is used to obtain monitoring results of the performance of an AI model, and the output of the AI ​​model is predictive information. In this solution, the terminal device can obtain ground truth information corresponding to a specific time unit by obtaining one or more first indices of one or more ground truth information, and therefore the performance of the predictive model can be flexibly monitored.

Description

[Technical Field]

[0001] This application claims priority to Chinese Patent Application No. 202310870686.9, titled "Model Monitoring Method, Apparatus, and System, Storage Medium, and Program Product," filed with the China National Intellectual Property Administration on 14 July 2023, which is incorporated herein by reference in its entirety.

[0002] This application relates to the field of communications technology, and more particularly to model monitoring methods, apparatus, and systems, storage media, and program products. [Background technology]

[0003] Model monitoring refers to monitoring the performance of an artificial intelligence (AI) model to determine whether it is functioning correctly. If the AI ​​model's performance is poor, it may be necessary to switch to a non-AI mode, replace or update the AI ​​model. Model monitoring can involve monitoring the accuracy of the AI ​​model's output (sometimes called a key performance indicator, or KPI). The accuracy of the AI ​​model's output is monitored by comparing the AI ​​model's output to the corresponding ground truth to determine whether the AI ​​model's performance meets the requirements.

[0004] AI models can be used for information prediction. For example, the output of an AI model is predicted information at time t+5. In this case, comparisons are typically made using ground truth information at time t+5 and predicted information at time t+5. If intermediate KPIs meet the specified requirements, this means the AI ​​model is performing well. If intermediate KPIs do not meet the specified requirements, this means the AI ​​model is performing poorly. Considering that AI models are used in actual systems, predicted information is used by base stations to schedule downlink transmissions. However, downlink transmissions scheduled by base stations do not necessarily correspond precisely to the time points of the predicted information. For example, the information predicted by the AI ​​model is information at time t+5. Specifically, if the period of the reference signal (RS) is 5 ms, the AI ​​model can only predict information for a 5 ms grid. However, base stations may schedule downlink transmissions at any time, for example, they may schedule a downlink transmission at time t+3. In this case, monitoring the performance of the predicted information only at the time point corresponding to the predicted information will not reflect the actual performance during downlink transmissions performed using the predicted information.

[0005] Currently, there is no corresponding solution for flexibly monitoring the performance of predictive models so that their predictive performance reflects actual system performance. [Overview of the project]

[0006] This application provides a model monitoring method, apparatus, and system, storage medium, and program product to support flexible monitoring of the performance of a predictive model so that predictive performance reflects actual system performance. [Means for solving the problem]

[0007] According to the first aspect, a model monitoring method is provided. This method includes the steps of obtaining first information, wherein the first information includes one or more first indices corresponding to one or more ground truth information, and obtaining one or more ground truth information corresponding to one or more first indices, wherein the one or more ground truth information is used to obtain monitoring results of the performance of an artificial intelligence (AI) model, and the output of the AI ​​model is predictive information.

[0008] In this embodiment, the terminal device can obtain ground truth information corresponding to a specific time unit by obtaining one or more first indices of one or more ground truth pieces of information, and therefore the performance of the predictive model can be flexibly monitored.

[0009] A second aspect provides a model monitoring method. This method includes the steps of: obtaining first information, wherein the first information includes the time-domain resource location of one or more reference signals used to measure ground truth information; and determining one or more pieces of ground truth information based on the first information, wherein the one or more pieces of ground truth information are used to obtain monitoring results of the performance of an AI model, the output of the AI ​​model is predictive information, and the time-domain resource location of at least one of the one or more pieces of ground truth information is different from the time-domain resource location of the predictive information.

[0010] In this embodiment, the terminal device can obtain ground truth information corresponding to any time-domain resource location by obtaining the time-domain resource location of one or more reference signals used to measure ground truth information, and therefore the performance of the predictive model can be flexibly monitored.

[0011] In one possible implementation, this method further includes the step of transmitting one or more pieces of ground truth information.

[0012] In this implementation, one or more ground truth information is sent to the access network device, and therefore the access network device can obtain model monitoring results of the actual system based on the one or more ground truth information received, and thus the performance of the predictive model can be flexibly monitored, and the predictive performance can reflect the actual system performance.

[0013] In another possible implementation, the method further includes the steps of obtaining one or more prediction information, wherein one or more prediction information is the output of an artificial intelligence (AI) model, and transmitting one or more prediction information.

[0014] In this implementation, one or more ground truth information is sent to the access network device, and therefore the access network device can obtain model monitoring results of the actual system based on the one or more ground truth information and the one or more prediction information received, and therefore the performance of the prediction model can be flexibly monitored, and the prediction performance can reflect the actual system performance.

[0015] In yet another possible implementation, the method further includes the steps of obtaining one or more predictive information, wherein one or more predictive information is the output of an artificial intelligence (AI) model, and obtaining one or more monitoring results of the AI ​​model's performance based on one or more ground truth information and one or more predictive information.

[0016] In this implementation, terminal devices store AI models that can predict information, and terminal devices can obtain further monitoring results. Therefore, access network devices can obtain model monitoring results of the actual system. Thus, the performance of the predictive model can be flexibly monitored, and the predictive performance can reflect the actual system performance.

[0017] In yet another possible implementation, the step of obtaining one or more prediction information includes the step of receiving second information, wherein the second information includes one or more second indices corresponding to one or more prediction information, or the second information includes a relative time unit value for each of the one or more prediction information with respect to a time unit corresponding to a reference resource, and the step of obtaining one or more prediction information based on the second information.

[0018] In this implementation, terminal devices can obtain corresponding predictive information based on instructions from access network devices.

[0019] In yet another possible implementation, one or more first indexes and one or more second indexes are based on time units corresponding to the base resource.

[0020] In yet another possible implementation, the time unit corresponding to the reference resource is one of the following: a first time unit in which the resource for reporting monitoring results reports is located; a second time unit in which the downlink control signaling for scheduling model monitoring is located; a third time unit determined based on the first or second time unit; or a predetermined / pre-configured fourth time unit.

[0021] In yet another possible implementation, a monitoring result report is sent, the monitoring result report includes one or more monitoring results, the monitoring result report includes at least one monitoring result that is greater than or equal to a first threshold among the one or more monitoring results, the monitoring result report includes at least one monitoring result that is less than or equal to a second threshold among the one or more monitoring results, the monitoring result report includes indexes of prediction information and ground truth information corresponding to the optimal monitoring result among the one or more monitoring results, the monitoring result report includes indexes of prediction information and ground truth information corresponding to the worst monitoring result among the one or more monitoring results, or the monitoring result report includes the absolute value of a first monitoring result among a plurality of monitoring results and the relative value of the monitoring results other than the first monitoring result among the plurality of monitoring results with respect to the first monitoring result.

[0022] In the foregoing implementation, the terminal device can report all the obtained monitoring results to the access network device, and thus, the access network device can obtain the complete monitoring results.

[0023] In some scenarios, the access network device only needs to know whether there is a monitoring result indicating that the performance of the model is good, and does not need to obtain all the monitoring results. Therefore, the monitoring result report includes at least one monitoring result that is greater than or equal to a first threshold among the one or more monitoring results. The first threshold may be configured by the access network device or may be predefined. In this implementation, signaling overhead can be reduced.

[0024] In some scenarios, the access network device only needs to know whether there is a monitoring result indicating that the performance of the model is poor, and does not need to obtain all the monitoring results. Therefore, the monitoring result report may include at least one monitoring result that is below the second threshold among one or more monitoring results. The second threshold may be configured by the access network device or may be predefined, and the second threshold is smaller than the first threshold. In this implementation, the signaling overhead can be reduced.

[0025] In some scenarios, the access network device only needs to know whether there is a monitoring result indicating that the performance of the model is good, and does not need to obtain all the monitoring results. Therefore, the monitoring result report may include the indexes of the prediction information and the ground truth information corresponding to the optimal monitoring result among one or more monitoring results. The access network device may have received one or more prediction information and one or more ground truth information in advance. In this case, the access network device can obtain the optimal monitoring result based on the indexes of the prediction information and the ground truth information corresponding to the optimal monitoring result. In this implementation, the signaling overhead can be reduced, and the requirement for the computing power of the terminal device is low.

[0026] In some scenarios, an access network device only needs to know if there are monitoring results indicating poor model performance, and does not need to obtain all monitoring results. Therefore, the monitoring results report only needs to include an index of predictive information and an index of ground truth information corresponding to the worst monitoring result among one or more monitoring results. The access network device may have already received one or more predictive information and one or more ground truth information. In this case, the access network device can obtain the worst monitoring result based on the index of predictive information and the index of ground truth information corresponding to the worst monitoring result. This implementation reduces signaling overhead and places a low demand on the computing power of the terminal device.

[0027] In some scenarios, terminal devices can use a differential reporting method. The monitoring result report includes the absolute value of the first monitoring result among multiple monitoring results, and the relative values ​​of the other monitoring results relative to the first monitoring result. This implementation can reduce signaling overhead.

[0028] In yet another possible implementation, the first monitoring result may be determined based on the order of the ground truth information used and / or the order of the predictive information used when obtaining multiple monitoring results.

[0029] In this implementation, the order in which one or more monitoring results are generated may be determined based on the order of ground truth information and / or the order of predictive information used during the generation of monitoring results by the terminal device, and therefore, the first monitoring result may be determined based on its generation order.

[0030] In yet another possible implementation, the terminal device may report only a portion of the monitoring results based on instructions from the access network device. For example, the terminal device may be instructed to report only a portion of all monitoring results using a bitmap format.

[0031] In yet another possible implementation, the monitoring results report and the information feedback used to transmit the measurement results of the reference signal are reported in the same information report, or the monitoring results report and the information feedback used to transmit the measurement results of the reference signal are reported in separate information reports.

[0032] According to a third aspect, a model monitoring method is provided. This method includes the steps of: transmitting first information, wherein the first information includes one or more first indices corresponding to one or more ground truth information; and receiving one or more ground truth information corresponding to one or more first indices, wherein the one or more ground truth information is used to obtain monitoring results of the performance of an artificial intelligence (AI) model, and the output of the AI ​​model is predictive information.

[0033] In this embodiment, the access network device instructs the terminal device to provide one or more first indices of one or more ground truth information, so that the terminal device can obtain ground truth information corresponding to a specific time unit, and thus the performance of the predictive model can be flexibly monitored.

[0034] A fourth aspect provides a model monitoring method. This method includes the steps of: transmitting first information, the first information including the time-domain resource locations of one or more reference signals used to measure ground truth information; and receiving one or more ground truth information based on the first information, the one or more ground truth information being used to obtain monitoring results of the performance of an AI model, the output of the AI ​​model being predictive information, and the time-domain resource locations of at least one of the one or more ground truth information being different from the time-domain resource locations of the predictive information.

[0035] In this embodiment, the access network device instructs the terminal device on the time-domain resource locations of one or more reference signals used to measure ground truth information, so that the terminal device can obtain ground truth information corresponding to any time-domain resource location, and thus the performance of the predictive model can be flexibly monitored.

[0036] In one possible implementation, the method further includes the steps of obtaining one or more predictive information and obtaining one or more monitoring results of the performance of an AI model based on one or more ground truth information and one or more predictive information.

[0037] In another possible implementation, the method further includes the step of receiving a monitoring results report, wherein the monitoring results report includes one or more monitoring results, and one or more monitoring results are obtained based on one or more ground truth information and one or more predictive information.

[0038] In yet another possible implementation, the method further includes the steps of transmitting a second piece of information, wherein the second piece of information includes one or more second indices corresponding to one or more prediction pieces of information, or the second piece of information includes a relative time unit value for each of the one or more prediction pieces of information with respect to a time unit corresponding to a reference resource; and receiving one or more prediction pieces of information.

[0039] In yet another possible implementation, one or more first indexes and one or more second indexes are based on time units corresponding to the base resource.

[0040] In yet another possible implementation, the time unit corresponding to the reference resource is one of the following: a first time unit in which the resource for reporting monitoring results reports is located; a second time unit in which the downlink control signaling for scheduling model monitoring is located; a third time unit determined based on the first or second time unit; or a predetermined / pre-configured fourth time unit.

[0041] In yet another possible implementation, the monitoring results report includes one or more monitoring results, the monitoring results report includes at least one monitoring result that is greater than or equal to a first threshold among the one or more monitoring results, the monitoring results report includes at least one monitoring result that is less than or equal to a second threshold among the one or more monitoring results, the monitoring results report includes an index of predictive information and an index of ground truth information corresponding to the best monitoring result among the one or more monitoring results, the monitoring results report includes an index of predictive information and an index of ground truth information corresponding to the worst monitoring result among the one or more monitoring results, or the monitoring results report includes the absolute value of the first monitoring result among multiple monitoring results and the relative values ​​of the other monitoring results among multiple monitoring results with respect to the first monitoring result.

[0042] In yet another possible implementation, the first monitoring result may be determined based on the order of the ground truth information used and / or the order of the predictive information used when obtaining multiple monitoring results.

[0043] In yet another possible implementation, the access network device may instead instruct the terminal device to report only a portion of the monitoring results. For example, the terminal device may be instructed in a bitmap format to report only a portion of all monitoring results.

[0044] In yet another possible implementation, the monitoring results report and the information feedback used to transmit the measurement results of the reference signal are reported in the same information report, or the monitoring results report and the information feedback used to transmit the measurement results of the reference signal are reported in separate information reports.

[0045] In any one of the first to fourth embodiments, the prediction information is prediction channel state information (CSI) and the ground truth information is ground truth CSI, or the prediction information is prediction beam information and the ground truth information is ground truth beam information.

[0046] Predicted beam information may be one or more of the predicted optimal beam or the reference signal received power (RSRP) of the predicted optimal beam. Ground truth beam information may be one or more of the actual optimal beam or the RSRP of the actual optimal beam. In other words, ground truth beam information includes at least one of the following: ground truth optimal beam or ground truth reference signal received power.

[0047] The methods of the first and second embodiments may be performed by a terminal device, by a module used within the terminal device (e.g., a processor, chip, or chip system), or by a logical node, logical module, or software capable of performing all or part of the functions of the terminal device.

[0048] The methods of the third and fourth embodiments may be performed by an access network device, by a module used within the access network device (e.g., a processor, chip, or chip system), or by a logical node, logical module, or software capable of performing all or part of the functions of the access network device.

[0049] According to a fifth aspect, a model monitoring device is provided, configured to implement one of the model monitoring methods of the first aspect or an implementation of the first aspect, or configured to implement one of the model monitoring methods of the second aspect or an implementation of the second aspect. The device may be a terminal device, a module used within a terminal device (e.g., a processor, chip, or chip system), or a logical node, logical module, or software capable of performing all or part of the functions of a terminal device. In one implementation, the model monitoring device may include a transmit unit and a receive unit, and may further include a processing unit. The transmit unit and the receive unit may be independent of each other or combined together (this may be called a “transceiver unit”).

[0050] According to the sixth aspect, a model monitoring device is provided, configured to implement one of the model monitoring methods of the third aspect or an implementation of the third aspect, or configured to implement one of the model monitoring methods of the fourth aspect or an implementation of the fourth aspect. The device may be an access network device, a module used within an access network device (e.g., a processor, chip, or chip system), or a logical node, logical module, or software capable of performing all or part of the functions of an access network device. In one implementation, the model monitoring device may include a transmit unit and a receive unit, and may further include a processing unit. The transmit unit and the receive unit may be independent of each other or combined together (this may be called a “transceiver unit”).

[0051] In one possible implementation, the model monitoring device of the fifth and sixth embodiments includes a module configured to separately perform one of the first through fourth embodiments or one of the implementations of the first through fourth embodiments.

[0052] In other possible implementations, the model monitoring device of the fifth and sixth embodiments includes a processor coupled to memory, the processor configured to perform the corresponding functions in a model monitoring method, the method being executed by the device. The memory is configured to be coupled to the processor and stores the programs (instructions) and / or data required by the device. Optionally, the model monitoring device may further include a communication interface to enable communication between the device and another network element. Optionally, the memory may be located inside or outside the model monitoring device.

[0053] In yet another possible implementation, the model monitoring device of the fifth and sixth embodiments includes a processor and a transceiver device. The processor is coupled to the transceiver device and is configured to execute a computer program or instructions to control the transceiver device to send and receive information. Once the processor has executed the computer program or instructions, it is further configured to carry out the aforementioned method by using logic circuits or by executing code instructions. The transceiver device may be a transceiver, a transceiver circuit, or an input / output interface and is configured to receive signals from model monitoring devices other than this model monitoring device and to transmit signals to the processor, or to transmit signals from the processor to model monitoring devices other than this model monitoring device. If the model monitoring device is a chip, the transceiver device is a transceiver circuit or an input / output interface.

[0054] When the model monitoring device of the fifth and sixth embodiments is a chip, the transmitting unit may be an output unit, such as an output circuit or a communication interface, and the receiving unit may be an input unit, such as an input circuit or a communication interface. When the model monitoring device is a terminal, the transmitting unit may be a transmitter or transmitter, and the receiving unit may be a receiver or receiver.

[0055] According to the seventh aspect, a computer-readable storage medium is provided which stores a computer program or instruction. When the computer program or instruction is executed, the method according to the preceding aspect is carried out.

[0056] According to the eighth aspect, a computer program product including instructions is provided. When the instructions are executed on a model monitoring device, the model monitoring device is made to perform the method according to the preceding aspects.

[0057] According to the ninth aspect, a model monitoring system is provided, which includes a model monitoring device according to the fifth aspect and a model monitoring device according to the sixth aspect. [Brief explanation of the drawing]

[0058] [Figure 1] This is a simplified diagram of a wireless communication system according to one embodiment of this application. [Figure 2A] This is a diagram of the neuron structure. [Figure 2B] This is a diagram of a neural network. [Figure 2C] This is an AI application configuration. [Figure 3] This is a diagram of a communication system with AI network elements implemented. [Figure 4A] This is a diagram of a network architecture according to one embodiment of the present application. [Figure 4B] This is a diagram of a network architecture according to one embodiment of the present application. [Figure 4C] This is a diagram of a network architecture according to one embodiment of the present application. [Figure 4D] This is a diagram of a network architecture according to one embodiment of the present application. [Figure 5] This is a schematic flowchart of a model monitoring method according to one embodiment of this application. [Figure 6] This is a schematic flowchart of another model monitoring method according to one embodiment of this application. [Figure 7] This figure shows an example of predicting CSI based on an AI model according to one embodiment of this application. [Figure 8] This is a schematic flowchart of yet another model monitoring method according to one embodiment of this application. [Figure 9] This is a diagram illustrating an example of a reference resource according to one embodiment of this application. [Figure 10] This is a schematic flowchart of another model monitoring method according to one embodiment of this application. [Figure 11] This is a schematic flowchart of yet another model monitoring method according to one embodiment of this application. [Figure 12] This is a schematic flowchart of yet another model monitoring method according to one embodiment of this application. [Figure 13] This is a schematic flowchart of yet another model monitoring method according to one embodiment of this application. [Figure 14] This figure shows an example of a monitoring results report according to one embodiment of the present application. [Figure 15] This is a schematic flowchart of yet another model monitoring method according to one embodiment of this application. [Figure 16] This is a schematic flowchart of yet another model monitoring method according to one embodiment of this application. [Figure 17] This is a schematic flowchart of yet another model monitoring method according to one embodiment of this application. [Figure 18] This is a diagram showing the structure of a model monitoring device according to one embodiment of this application. [Figure 19] This is a diagram showing the structure of another model monitoring device according to one embodiment of this application. [Modes for carrying out the invention]

[0059] Hereinafter, embodiments of this application will be described with reference to the accompanying drawings of embodiments of this application.

[0060] To further clarify the purpose, technical solution, and advantages of this application, the application will be described in more detail below with reference to the attached drawings.

[0061] Hereinafter, “at least one piece (item)” in this application refers to one piece (item) or more pieces (items). More pieces (items) means two or more pieces (items). The term “and / or” describes the relationship between related objects and indicates that three relationships may exist. For example, A and / or B may refer to the following three cases: that only A exists, that both A and B exist, and that only B exists. The letter “ / ” generally indicates an “or” relationship between related objects. In addition, while terms such as “first” and “second” may be used in this application to describe objects, it should be understood that these objects are not limited to these terms. These terms are used only to distinguish objects from one another.

[0062] The terms “includes,” “have,” and any variations thereof as used in the following description of this application are intended to encompass non-exclusive inclusion. For example, a process, method, system, product, or device comprising a set of steps or units may, at their discretion, further include, but is not limited to, other steps or units not listed, or at their discretion, further include other specific steps or units of the process, method, product, or device. Note that in this application, words such as “example” or “for example” are used to indicate that an example, illustration, or explanation is being given. No method or design described as “example” or “for example” in this application should be construed as being preferable to or having more advantages than another method or design. Strictly speaking, the use of words such as “example” or “for example” is intended to present the relevant concepts in a particular manner.

[0063] The technology provided in this application can be applied to various communication systems. For example, communication systems can be fourth generation (4 th Fifth generation (5G) communication systems (e.g., Long Term Evolution (LTE) systems), 5th generation (5 th This could be a 5G generation communication system, a worldwide interoperability for microwave access (WiMAX) or wireless local area network (WLAN) system, an integrated system of multiple systems, or a future communication system, such as a 6G communication system. A 5G communication system is sometimes also called a new radio (NR) system.

[0064] Network elements within a communication system can transmit signals to other network elements or receive signals from other network elements. These signals may include information, signaling, data, etc. Alternatively, network elements can be replaced by entities, network entities, devices, terminal devices, communication modules, nodes, communication nodes, etc. In this application, network elements are used as an example. For example, a communication system may include at least one terminal device and at least one access network device. The access network device can transmit downlink signals to the terminal device, and / or the terminal device can transmit uplink signals to the access network device. Furthermore, it will be understood that if a communication system includes multiple terminal devices, these multiple terminal devices can also transmit signals to each other. That is, both the signal-transmitting network element and the signal-receiving network element can be terminal devices.

[0065] The model monitoring method provided in the embodiments of this application may be applied to wireless communication systems, such as 5G systems, 6G systems, or satellite communication systems. Figure 1 is a simplified diagram of a wireless communication system according to one embodiment of this application. As shown in Figure 1, the wireless communication system includes a radio access network 100. The radio access network 100 may be a next-generation (e.g., 6G or higher version) radio access network or a conventional (e.g., 5G or 4G) radio access network. One or more terminal devices (120a to 120j, collectively referred to as 120) may be interconnected or connected to one or more network devices (110a and 110b, collectively referred to as 110) within the radio access network 100. Optionally, Figure 1 is merely a diagram. The wireless communication system may further include other devices, such as core network devices, radio relay devices, and / or radio backhaul devices not shown in Figure 1.

[0066] Optionally, in actual use, a wireless communication system may include multiple network devices (also called access network devices) or multiple terminal devices. One network device may serve one or more terminal devices. One terminal device may also access one or more network devices. The number of terminal devices and network devices included in a wireless communication system is not limited in the embodiments of this application.

[0067] A network device can be an entity configured to transmit or receive signals on the network side. A network device can also be an access device through which a terminal device accesses a wireless communication system wirelessly. For example, a network device may be a base station. A base station can broadly cover the following various names, or the following names, for example: RAN node, Node B, evolved Node B (eNB), next generation Node B (gNB), access network device in an open radio access network (O-RAN), relay station, access point, transmitting and receiving point (TRP), transmitting point (TP), main station (MeNB), secondary station (SeNB), multi-standard radio (MSR) node, home base station, network controller, access node, radio node, access point (AP), transmitting node, transceiver node, building baseband unit (BBU), remote radio unit (RRU), active antenna unit (AAU), remote radio head (RRH), central unit (CU), distributed unit (DU), radio unit (radio A base station may be replaced by a unit (RU), a central unit control plane (CU-CP) node, a central unit user plane (CU-UP) node, or a positioning node. A base station may be a macro base station, a micro base station, a relay node, a donor node, or a combination thereof. Alternatively, a network device may be a communication module, modem, or chip located within the aforementioned device or apparatus.Alternatively, the network device may be a mobile switching center, a device that functions as a base station in device-to-device (D2D), vehicle-to-everything (V2X), and machine-to-machine (M2M) communications, a network-side device in a 6G network, or a device that functions as a base station in a future communication system. The network device can support networks with the same or different access technologies. The specific technologies and specific device forms employed by the network device are not limited to the embodiments of this application.

[0068] Network devices may be stationary or mobile. For example, base stations 110a and 110b are stationary and responsible for radio transmission and reception in one or more cells from terminal device 120. The helicopter or unmanned aerial vehicle 120i shown in Figure 1 may be configured to function as a mobile base station, and one or more cells may move based on the location of the mobile base station 120i. In another example, the helicopter or unmanned aerial vehicle (120i) may be configured to function as a terminal device communicating with base station 110b.

[0069] In this application, a communication device configured to perform the functions of an access network may be an access network device, a network device having some of the functions of an access network, or a device capable of supporting the performance of access network functions, such as a chip system, hardware circuitry, software module, or hardware circuitry and software module. The device may be mounted on an access network device or used together with an access network device. In the method of this application, an example is used in which the communication device configured to perform the functions of an access network device is an access network device.

[0070] A terminal device may be an entity configured on the user side to receive or transmit signals, such as a mobile phone. A terminal device may be configured to connect to people, objects, and machines. A terminal device may communicate with one or more core networks via a network device. A terminal device may include a handheld device with wireless connectivity, other processing devices connected to a wireless modem, and in-vehicle devices. A terminal device may be portable, pocket-sized, handheld, computer-integrated, or in-vehicle mobile device. Terminal device 120 can be widely used in a variety of scenarios, including cellular communication, D2D, V2X, point-to-point (P2P), machine-to-machine (M2M), machine-type communication (MTC), Internet of Things (IoT), virtual reality (VR), augmented reality (AR), industrial control, autonomous driving, telemedicine, smart grids, smart furniture, smart offices, smart wearables, smart transportation, smart cities, unmanned aerial vehicles, robots, remote sensing, passive sensing, positioning, navigation and tracking, as well as autonomous delivery and mobility.Some examples of terminal devices 120 include 3GPP® standard user equipment (UE), fixed devices, mobile devices, handheld devices, wearable devices, cellular phones, smartphones, session initiation protocol (SIP) phones, notebook computers, personal computers, smartbooks, vehicles, satellites, global positioning system (GPS) devices, target tracking devices, unmanned aerial vehicles, helicopters, flights, ships, remote control devices, smart home devices, industrial devices, personal communication service (PCS) phones, wireless local loop (WLL) stations, personal digital assistants (PDAs), wireless network cameras, tablet computers, palmtop computers, mobile internet devices (MIDs), wearable devices such as smartwatches, VR devices, AR devices, wireless terminals for industrial control, terminals for Internet of Vehicles systems, wireless terminals for autonomous driving, and smart grids. These include wireless terminals for grids, wireless terminals for transportation safety, wireless terminals for smart cities (e.g., smart fuel dispensers), terminal devices on high-speed trains, and wireless terminals for smart homes (e.g., smart speakers, smart coffee machines, or smart printers). Terminal device 120 may be a wireless device in the aforementioned scenarios, or a device configured to be placed within a wireless device, such as a communication module, modem, or chip within the aforementioned device.Terminal devices are sometimes also called terminals, terminal equipment, user equipment (UE), mobile stations (MS), or mobile terminals (MT). Alternatively, a terminal device may be a terminal device in a future wireless communication system. A terminal device may be used within a dedicated network device or a general-purpose device. The specific technologies and device forms employed by the terminal device are not limited to those described in this application.

[0071] Optionally, terminal devices may be configured to function as base stations. For example, a UE may function as a scheduling entity providing sidelink signals between UEs in V2X, D2D, P2P, etc. As shown in Figure 1, the cellular phone 120a and the car 120b communicate with each other using sidelink signals. The cellular phone 120a communicates with the smart home device 120e without relaying communication signals via the base station 110b.

[0072] In this application, a communication device configured to perform the functions of a terminal device may be a terminal device, a terminal device having some of the functions of a terminal device, or a device capable of supporting the performance of the aforementioned functions of a terminal device, such as a chip system. The device may be mounted on a terminal device or used together with a terminal device. In this application, a chip system may include a chip or include a chip and other discrete devices. In the technical solutions provided in this application, an example is used in which the communication device is a terminal device or UE.

[0073] Optionally, a wireless communication system typically includes a cell, with a base station providing cell management and communication services to multiple mobile stations (MS) within the cell. A base station includes a baseband unit (BBU) and a remote radio unit (RRU). The BBU and RRU may be located in separate locations. For example, the RRU may be remotely deployed in a high-traffic area, while the BBU is located in a central equipment room. Alternatively, the BBU and RRU may be located in the same equipment room. Alternatively, the BBU and RRU may be separate components within the same rack. Optionally, a single cell may correspond to a single carrier or component carrier.

[0074] In some deployments, the network devices referred to in the embodiments of this application may be devices including a CU or DU, devices including a CU and a DU, or devices including a CU control plane (central unit-control plane, CU-CP) node, a CU user plane (central unit-user plane, CU-UP) node, and a DU node. For example, the network device may include gNB-CU-CP, gNB-CU-UP, and gNB-DU.

[0075] In some deployments, multiple RAN nodes cooperate to help terminals perform radio access, with each RAN node performing some of the functions of a base station. For example, a RAN node could be a CU, DU, CU-CP, CU-UP, or RU. A CU and DU may be located separately or may be contained within the same network element, for example, within a BBU. An RU may be contained within a radio frequency device or radio frequency unit, for example, an RRU, AAU, or RRH.

[0076] A RAN node can support one or more categories of fronthaul interfaces, each corresponding to a DU and RU with separate functions. If the fronthaul interface between the DU and RU is a common public radio interface (CPRI), the DU is configured to perform one or more baseband functions, and the RU is configured to perform one or more radio frequency functions. If the fronthaul interface between the DU and RU is a different type of interface, there is a transfer of some downlink and / or uplink baseband functions compared to a CPRI. For example, for downlink, one or more of precoding, digital beamforming (BF), or inverse fast Fourier transform (IFFT) / cyclic prefix (CP) addition are transferred from the DU to the RU for implementation, and for uplink, one or more of digital beamforming (BF) or fast Fourier transform (IFFT) / cyclic prefix (CP) removal are transferred from the DU to the RU for implementation. In one possible implementation, the interface may be an enhanced common public radio interface (eCPRI). In the eCPRI architecture, the partitioning method between the DU and RU differs, corresponding to each eCPRI category (Cat), such as eCPRI Cat A, B, C, D, E, and F.

[0077] eCPRI Cat A is used as an example. For downlink transmissions, splitting is performed by layer mapping. The DU is configured to perform layer mapping and one or more pre-layer mapping functions (specifically, one or more of coding, rate matching, scrambling, modulation, and layer mapping), while other post-layer mapping functions (e.g., one or more of RE mapping, digital beamforming (BF), or inverse fast Fourier transform (IFFT) / cyclic prefix (CP) addition) are transferred to the RU for implementation. For uplink transmissions, splitting is performed by RE demapping. The DU is configured to perform demapping and one or more pre-demapping functions (specifically, one or more of the following functions: decoding, derate matching, descrambling, demodulation, inverse discrete Fourier transform (IDFT), channel equalization, and RE demapping), while other post-demapping functions (e.g., one or more of digital BF or fast Fourier transform (FFT) / CP rejection) are transferred to the RU for implementation. It will be understood that for functional descriptions of DUs and RUs corresponding to various categories of eCPRI, one should refer to the eCPRI protocol; details are not provided here.

[0078] In one possible design, a processing unit configured to perform baseband functions in the BBU is called a baseband high (BBH) unit, and a processing unit configured to perform baseband functions in the RRU / AAU / RRH is called a baseband low (BBL) unit.

[0079] In different systems, CU (or CU-CP and CU-UP), DU, or RU may also have different names. However, those skilled in the art will understand their meaning. For example, in the ORAN system, CU may be called O-CU (Open CU), DU may be called O-DU, CU-CP may be called O-CU-CP, CU-UP may be called O-CU-UP, and RU may be called O-RU. In this application, any one of CU (or CU-CP and CU-UP), DU, and RU may be implemented by a software module, a hardware module, or a combination of a software module and a hardware module.

[0080] In embodiments of this application, the device configured to perform the functions of a network device may be a network device, or a device capable of supporting a network device in performing its functions, such as a chip system, hardware circuitry, software modules, or a combination of hardware circuitry and software modules. The device may be mounted on or used in conjunction with a network device. The example in embodiments of this application where the device configured to perform the functions of a network device is a network device is used for illustrative purposes only and does not constitute a limitation on the solutions of embodiments of this application.

[0081] It will be understood that this application may be applicable between access network devices and terminal devices.

[0082] Protocol layer structure between access network devices and terminal devices

[0083] Communication between access network devices and terminal devices follows a specific protocol layer structure. This protocol layer structure may include a control plane protocol layer structure and a user plane protocol layer structure. For example, the control plane protocol layer structure may include the functions of protocol layers such as the radio resource control (RRC) layer, the packet data convergence protocol (PDCP) layer, the radio link control (RLC) layer, the medium access control (MAC) layer, and the physical layer. Similarly, the user plane protocol layer structure may include the functions of protocol layers such as the PDCP layer, the RLC layer, the MAC layer, and the physical layer. In one possible implementation, a service data adaptation protocol (SDAP) layer may be included on top of the PDCP layer.

[0084] Optionally, the protocol layer structure between the access network device and the terminal device may further include an artificial intelligence (AI) layer for transmitting data related to AI functionality.

[0085] Let's take data transmission between an access network device and a terminal device as an example. Data transmission must pass through user plane protocol layers, such as the SDAP layer, PDCP layer, RLC layer, MAC layer, or physical layer. The SDAP layer, PDCP layer, RLC layer, MAC layer, and physical layer are sometimes collectively called the access layer. Data transmission directions are divided into transmission and reception, and each layer is further divided into a transmitting section and a receiving section. Let's take downlink data transmission as an example. The PDCP layer receives data from the upper layer and then transmits that data to the RLC layer and MAC layer. The MAC layer then generates a transport block, and wireless transmission takes place through the physical layer. Each layer performs the corresponding encapsulation of the data. For example, data received by a particular layer from an upper layer is considered a service data unit (SDU) of that layer, encapsulated in a protocol data unit (PDU) by that layer, and then forwarded to the next layer.

[0086] For example, a terminal device may further have an application layer and an accessless layer. The application layer may be used to provide services to applications installed on the terminal device. For example, downlink data received by the terminal device may be sequentially transmitted from the physical layer to the application layer and then provided to the application by the application layer. As another example, the application layer may receive data generated by the application, sequentially transmit the data to the physical layer, and transmit the data to another communication device. The accessless layer may be used to transfer user data, for example, to transfer uplink data received from the application layer to the SDAP layer, or to transfer downlink data received from the SDAP layer to the application layer.

[0087] To support AI technology over wireless networks, more AI nodes may be introduced into the network.

[0088] Optionally, an AI node may be deployed in one or more of the following locations within the communication system: access network devices, terminal devices, core network devices, etc. Alternatively, an AI node may be deployed independently, for example, in a location other than one of the above devices, such as a host or cloud server in an over-the-top (OTT) system. An AI node can communicate with other devices within the communication system. For example, another device may be one or more of the following: network devices, terminal devices, or network elements of the core network.

[0089] It should be understood that the number of AI nodes is not limited in this application. For example, if there are multiple AI nodes, they may be divided based on their functions. For instance, each AI node may be responsible for a different function.

[0090] It will be further understood that an AI node may be an independent device, may be integrated into the same device to perform different functions, may be a network element within a hardware device, may be a software function running on dedicated hardware, or may be a virtualization function whose instances are created on a platform (e.g., a cloud platform). The specific form of an AI node is not limited in this application.

[0091] An AI node may be an AI network element or an AI module.

[0092] One or more AI modules are placed on one or more devices of these network element nodes, such as core network devices, access network nodes (RAN nodes), terminals, or OAMs. An access network node can function as an independent RAN node or may contain multiple RAN nodes, for example, a CU and a DU. One or more AI modules may be placed on the CU and / or DU. Optionally, the CU may be further divided into a CU-CP and a CU-UP. One or more AI models are placed on the CU-CP and / or CU-UP.

[0093] AI modules are configured to perform corresponding AI functions. AI modules deployed in separate network elements may be the same or different. The model of an AI module is constructed based on various parameters, and the AI ​​module can perform a variety of functions. The model of an AI module may be constructed based on one or more of the following parameters: structural parameters (e.g., at least one of the following: the number of neural network layers, the width of the neural network, the connectivity between layers, the weights of neurons, the activation function of neurons, or the bias of the activation function), input parameters (e.g., the type and / or dimensions of the input parameters), or output parameters (e.g., the type and / or dimensions of the output parameters). The bias in the activation function is sometimes called the bias of the neural network.

[0094] A single AI module may have one or more models. A single model can obtain an output through inference, and the output may include one or more parameters. The learning process, training process, or inference process for each model may be deployed on separate nodes or devices, or on the same node or device.

[0095] The communication system includes a RAN intelligent controller (RIC). For example, the RIC may be an AI module and configured to perform AI-related functions. The RIC includes near-real-time RICs (near-RT RICs) and non-real-time RICs (non-RT RICs). Non-real-time RICs primarily process non-real-time information, such as latency-insensitive data, with latency of several seconds. Real-time RICs primarily process near-real-time information, such as latency-sensitive data, with latency of tens of milliseconds.

[0096] A quasi-real-time RIC is configured to perform model training and inference, for example, to train an AI model and perform inference using that AI model. The quasi-real-time RIC can obtain network-side and / or terminal-side information from RAN nodes (e.g., CU, CU-CP, CU-UP, DU, and / or RU) and / or terminals. This information can be used as training data or inference data. Optionally, the quasi-real-time RIC can distribute inference results to RAN nodes and / or terminals. Optionally, inference results can be exchanged between CU and DU and / or between DU and RU. For example, the quasi-real-time RIC distributes inference results to DU, and DU sends inference results to RU.

[0097] Non-real-time RICs are also configured to perform model training and inference, for example, to train an AI model and perform inference using that model. Non-real-time RICs can obtain network-side and / or terminal-side information from RAN nodes (e.g., CU, CU-CP, CU-UP, DU, and / or RU) and / or terminals. This information can be used as training data or inference data, and the inference results can be delivered to RAN nodes and / or terminals. Optionally, inference results can be exchanged between CUs and DUs and / or between DUs and RUs. For example, a non-real-time RIC delivers inference results to a DU, and the DU sends the inference results to the RU.

[0098] Alternatively, the quasi-real-time RIC and non-real-time RIC may be deployed independently as network elements. Optionally, the quasi-real-time RIC and non-real-time RIC may instead function as part of another device. For example, the quasi-real-time RIC may be located on a RAN node (e.g., a CU or DU), while the non-real-time RIC may be located within an OAM, cloud server, core network device, or another network device.

[0099] For example, the arrangement of quasi-real-time RICs and non-real-time RICs in a network architecture may be as shown in Figures 4A to 4D.

[0100] As shown in Figure 4A(a), in the first possible implementation, the access network device includes a quasi-real-time RIC module for performing model training and / or inference.

[0101] As shown in Figure 4A(b), in the second possible implementation, the non-real-time RIC may be located outside the access network device in the communication system. Optionally, this non-real-time RIC may be located within the OAM or core network device.

[0102] As shown in Figure 4A(c), in the third possible implementation, the access network device includes a quasi-real-time RIC, and a non-real-time RIC is further included outside the access network device. Optionally, this non-real-time RIC may be located within the OAM or core network device.

[0103] In contrast to Figure 4A(c), in Figure 4B, the CU is separated into CU-CP and CU-UP. The quasi-real-time RIC and non-real-time RIC are arranged in the same way as in Figure 4A(c).

[0104] As shown in Figure 4C, optionally, an access network device includes one or more AI entities whose functionality is similar to that of a quasi-real-time RIC. Optionally, an OAM includes one or more AI entities whose functionality is similar to that of a non-real-time RIC. Optionally, a core network device includes one or more AI entities whose functionality is similar to that of a non-real-time RIC. If both the OAM and the core network device include AI entities, the models obtained by training with the AI ​​entities of the OAM and the core network device will be different, and / or the models configured to perform inference will be different. In this application, different models include at least one of the following: different structural parameters of the models (e.g., number of layers and / or model weights), different input parameters of the models, or different output parameters of the models.

[0105] Compared to Figure 4C, the access network device in Figure 4D is separated into CU and DU. Optionally, the CU may contain AI entities whose functionality is similar to that of a quasi-real-time RIC. Optionally, the DU may contain AI entities whose functionality is similar to that of a quasi-real-time RIC. If the CU and DU each contain AI entities, the models obtained by training with the AI ​​entities in the CU and DU will be different, and / or the models configured to perform inference will be different. Optionally, the CU in Figure 4D may be further divided into CU-CP and CU-UP. Optionally, one or more AI models may be deployed in CU-CP, and / or one or more AI models may be deployed in CU-UP. Optionally, in Figure 4C or Figure 4D, the OAM for the access network device and the OAM for the core network device may be deployed separately and independently.

[0106] It should be understood that the number and types of devices in the communication system shown in Figure 1 are used only as an example. This application is not limited thereto. In actual use, the communication system may further include more terminal devices and more access network devices, and may further include other network elements, for example, a core network device and / or network element configured to perform artificial intelligence functions.

[0107] It will be understood that all or part of the functions performed by one or more terminal devices, access network devices, core network devices, or network elements configured to perform artificial intelligence functions may be virtualized, that is, they may be performed using one or more dedicated processors or general-purpose processors and corresponding software modules. Since terminal devices and access network devices relate to air interface transmission interfaces, the transceiver functions of the interface may be performed by hardware. Core network devices, such as operation administration and maintenance (OAM) network elements, can all be virtualized. Optionally, one or more functions of a virtualized terminal device, access network device, core network device, or network element configured to perform artificial intelligence functions may be performed by a cloud device, for example, by a cloud device in an over-the-top (OTT) system.

[0108] To facilitate understanding, the AI ​​technology used in this application will be explained below. It should be understood that this explanation is not intended to limit the scope of this application.

[0109] (1) AI model AI refers to intelligence exhibited by machines created by humans. Typically, artificial intelligence is the technology that uses conventional computer programs to demonstrate human intelligence. Artificial intelligence can be defined as a machine or computer that mimics humans and possesses cognitive functions related to human thought, such as learning and problem-solving. Artificial intelligence can learn from past experiences, make rational decisions, and respond quickly. The goal of artificial intelligence is to understand intelligence by constructing symbolic reasoning or computer programs for reasoning.

[0110] Machine learning is a method for realizing artificial intelligence, that is, a method of solving problems using artificial intelligence by employing machine learning as a means. Machine learning theory primarily involves designing and analyzing several algorithms that enable computers to automatically "learn." Machine learning algorithms are algorithms that automatically analyze data to derive rules and use those rules to predict unknown data. Because learning algorithms involve a large amount of statistical theory, machine learning is closely related to inferential statistics in particular, and is also called statistical learning theory.

[0111] An AI model is an algorithm or computer program capable of performing AI functions, and is a concrete implementation of AI technology. An AI model represents the mapping relationship between the model's inputs and outputs. The types of AI models may include neural networks, linear regression models, decision tree models, support vector machines (SVMs), Bayesian networks, Q-learning models, or other machine learning (ML) models.

[0112] (2) Neural Networks A neural network is a specific implementation of AI or machine learning technology. According to the universal approximation theorem, a neural network can theoretically approximate any continuous function, and therefore has the ability to learn any mapping. Consequently, neural networks can accurately perform abstract modeling for complex, high-dimensional problems.

[0113] The concept of neural networks originates from the neuronal structure of brain tissue. For example, each neuron performs weighted addition on the input values ​​of the neuron and outputs the result using an activation function. Figure 2A is a diagram of the neuronal structure. The input of the neuron is x=[x0,x1,...,x n ] and the weights corresponding to the inputs are w=[w,w1,...,w n ] and w i x iis used as the weight of x i is assumed to be used for weighting i . The bias for performing weighted addition on the input value based on the weight is, for example, b. The activation function can have multiple forms. Assume that the activation function of one neuron is y = f(z) = max(0, z). In this case, the output of the neuron is

[0114]

Equation

[0117] For example, the type of AI model is a neural network. The AI ​​model in this application may be a deep neural network (DNN). Depending on the network construction method, the DNN may include a feedforward neural network (FNN), a convolutional neural network (CNN), and a recurrent neural network (RNN).

[0118] (3) Training dataset and inference data A training dataset is used to train an AI model. A training dataset may contain the inputs to the AI ​​model, or it may contain both the inputs and the target outputs of the AI ​​model. A training dataset contains one or more training data points. These training data points can be training samples that are input to the AI ​​model, or they can be the target outputs of the AI ​​model. Target outputs are sometimes called labels or label samples. The training dataset is a crucial part of machine learning. Essentially, model training involves learning certain features from the training data so that the output of the AI ​​model is as close as possible to the target output. For example, the difference between the output of the AI ​​model and the target output is minimized as much as possible. The performance of the AI ​​model obtained through training can be determined to some extent by the composition and selection of the training dataset.

[0119] In addition, a loss function may be defined in the training process of an AI model (e.g., a neural network). The loss function describes the gap or difference between the output value of the AI ​​model and the target output value. The specific form of the loss function is not limited in this application. The training process of an AI model is the process of adjusting the model parameters of the AI ​​model so that the value of the loss function is below a threshold or so that the value of the loss function satisfies a target requirement. For example, the AI ​​model is a neural network, and adjusting the model parameters of the neural network includes adjusting at least one of the following parameters: the number of layers in the neural network, the width of the neural network, the weights of the neurons, or the parameters of the activation function of the neurons.

[0120] Inference data can be used as input to an AI model that has been trained for inference. During model inference, the inference data is input to the AI ​​model, and a corresponding output, i.e., the inference result, can be obtained.

[0121] (4) AI model design AI model design primarily involves a data collection phase (e.g., collecting training data and / or inference data), a model training phase, and a model inference phase. An inference result application phase may also be included. Figure 2C shows an AI application configuration. In the data collection phase, a data source is used to provide training datasets and inference data. In the model training phase, the training data provided by the data source is analyzed or trained, resulting in an AI model. The AI ​​model represents the mapping relationship between the model's inputs and outputs. Obtaining an AI model through learning using the model training node is equivalent to obtaining the mapping relationship between the model's inputs and outputs through learning using the training data. In the model inference phase, the AI ​​model obtained through training in the model training phase is used to perform inference based on the inference data provided by the data source to obtain inference results. This phase can also be understood as the AI ​​model being input with inference data in order to obtain an output, which is the inference result. The inference result may represent the configuration parameters used (executed) by the actor object, and / or the operations performed by the actor object. The inference result is exposed in the inference result application phase. For example, inference results may be uniformly planned by an actor entity. For instance, an actor entity can send its inference results to one or more actor objects (e.g., core network devices, access network devices, or terminal devices) for execution. As another example, an actor entity can further feed back the model's performance to a data source to facilitate subsequent model updates and training.

[0122] It will be understood that a communication system may include network elements with artificial intelligence capabilities. The aforementioned stages related to AI model design may be performed by one or more network elements with artificial intelligence capabilities. In one possible design, AI functionality (e.g., AI modules or AI entities) may be configured in existing network elements within the communication system to perform AI-related operations, such as AI model training and / or inference. For example, existing network elements may be access network devices (e.g., gNBs), terminal devices, core network devices, network management systems, etc. Network management systems can classify network management work into three categories: Operation, Administration, and Maintenance, based on the actual requirements of the operator's network operations. Network management systems are sometimes called operation administration and maintenance (OAM) network elements, or OAM for short. Operation mainly refers to routine analysis, forecasting, planning, and configuration for the network and services. Maintenance mainly refers to routine operational activities for testing and fault management of the network and network services. A network management system can detect the operational status of the network, optimize network connectivity and performance, improve network operational stability, and reduce network maintenance costs. Alternatively, in another possible design, a separate network element may be introduced into the communication system to perform AI-related operations, such as AI model training. This separate network element may be called an AI network element or an AI node, etc. This naming convention is not limited to this application. The AI ​​network element may be directly connected to an access network device in the communication system, or it may be indirectly connected to the access network device via a third-party network element.Third-party network elements may be, but are not limited to, core network elements such as an authentication management function (AMF) network element or a user plane function (UPF) network element, OAM, a cloud server, or another network element. See, for example, Figure 3. The communication system includes an access network device 110, terminal devices 120 and 130, and the communication system shown in Figure 1 further incorporates an AI network element 140.

[0123] In this application, one or more parameters can be obtained by inference using a single model. The training process for each model may be deployed on separate devices or nodes, or on the same device or node. The inference process for each model may be deployed on separate devices or nodes, or on the same device or node.

[0124] Model parameters may include one or more of the following: model structure parameters (e.g., number of layers and / or weights), model input parameters (e.g., input dimension or number of input ports), or model output parameters (e.g., output dimension or number of output ports). It will be understood that the input dimension may be the size of a single input data. For example, if the input data is a sequence, the input dimension corresponding to that sequence may indicate the length of the sequence. The number of input ports may be the amount of input data. Similarly, the output dimension may be the size of a single output data. For example, if the output data is a sequence, the output dimension corresponding to that sequence may indicate the length of the sequence. The number of output ports may be the amount of output data. Due to the possibility proposed in the background art that the predictive performance obtained based on a predictive model may not reflect the actual system performance, this application provides a model monitoring solution. A terminal device can obtain ground truth information corresponding to a specific time unit, e.g., a required time unit, by obtaining one or more first indices of one or more ground truth information, and therefore the performance of the predictive model can be flexibly monitored.

[0125] In this application, “transmitting information to (e.g., a terminal device)” or the relevant description in the attached drawings may be understood to mean that the destination of the information is a terminal device and may include directly or indirectly transmitting information to a terminal device; and “receiving information from (e.g., a terminal device)” or the relevant description in the attached drawings may be understood to mean that the source of the information is a terminal device and may include directly or indirectly receiving information from a terminal device. The information may undergo necessary processing, such as formatting changes, between the source transmitting the information and the destination. However, the destination may understand valid information from the source. Similar expressions in this application may be understood in the same way. Further details are not provided here.

[0126] The following describes in detail the model monitoring method provided in embodiments of this application. It will be understood that in this application, an example is used in which an access network device and a terminal device function as the entities that perform the interaction example. However, the entities that perform the interaction example are not limited in this application. For example, the access network device in the method provided in this application may instead be a chip, chip system, or processor used within the access network device, or a logical node, logical module, or software capable of performing all or part of the functions of the access network device. The terminal device in the method provided in this application may instead be a chip, chip system, or processor used within the terminal device, or a logical node, logical module, or software capable of performing all or part of the functions of the terminal device.

[0127] Figure 5 is a schematic flowchart of a model monitoring method according to one embodiment of the present application. For example, the method may include the following steps:

[0128] S501: The terminal device obtains the first piece of information.

[0129] Model monitoring in this embodiment of the present application refers to monitoring the performance of an AI model to determine whether the AI ​​model is functioning correctly. If the AI ​​model is performing poorly, it may be necessary to switch to a non-AI mode, replace or update the AI ​​model, etc. Model monitoring may be monitoring the accuracy of the AI ​​model's output (sometimes called an intermediate KPI). The accuracy of the AI ​​model's output is monitored by comparing the AI ​​model's output to the corresponding label or ground truth to determine whether the AI ​​model's performance meets the requirements. Intermediate KPIs typically include generalized cosine similarity (GCS), square generalized cosine similarity (SGCS), mean square error (MSE), normalized mean square error (NMSE), etc.

[0130] Two types of information are needed to calculate the intermediate KPIs of an AI model. One type is the information to be monitored, for example, the predictive information output by the AI ​​model. The other type is label information or ground truth information, i.e.

number

number

number

number

[0131] In this embodiment, the ground truth information index indicates specific ground truth information obtained by the terminal device.

[0132] The terminal device obtains first information, which includes one or more first indices corresponding to one or more ground truth pieces of information. For example, the terminal device may obtain first information transmitted by an access network device, or it may obtain first information that has been previously stored by the terminal device.

[0133] S502: The terminal device obtains one or more pieces of ground truth information corresponding to one or more first indices.

[0134] The terminal device, after obtaining one or more first indices corresponding to one or more ground truth information, measures a reference signal in the slot corresponding to each of the one or more indices in order to obtain one or more ground truth information corresponding to the one or more first indices. One or more ground truth data points are used to obtain monitoring results for the performance of the AI ​​model, and the output of the AI ​​model is predictive information.

[0135] Compared to conventional techniques that can only periodically obtain model monitoring results in a grid, in this embodiment, the terminal device can obtain ground truth information corresponding to a specific time unit by obtaining one or more first indices of one or more pieces of ground truth information.

[0136] According to the model monitoring method provided in this embodiment of the present application, a terminal device can obtain ground truth information corresponding to a specific time unit by obtaining one or more first indices corresponding to one or more ground truth information, and therefore the performance of the predictive model can be flexibly monitored.

[0137] Figure 6 is a schematic flowchart of another model monitoring method according to one embodiment of the present application. Unlike the embodiment shown in Figure 5, this embodiment determines the ground truth information based on the time-domain resource location of a reference signal used to measure the ground truth information, rather than instructing a terminal device to obtain specific ground truth information by directly using an index of ground truth information. For example, the method may include the following steps:

[0138] S601: The terminal device obtains the first piece of information.

[0139] The reference signal used to measure ground truth information (i.e., used for model monitoring) and the reference signal used for model inference (used as input to the model for information prediction) may be reference signals within the same group, i.e., they may belong to the same reference signal configuration. Alternatively, the reference signal used to measure ground truth information and the reference signal used for model inference may be reference signals within different groups, i.e., the reference signal may be specially configured for model monitoring.

[0140] In this embodiment, the reference signals used to measure ground truth information and the reference signals used for model inference are reference signals in different groups. A terminal device can determine ground truth information based on the reference signals used to measure ground truth information. Thus, the terminal device can obtain first information, which includes the time-domain resource locations of one or more reference signals used to measure ground truth information.

[0141] For example, a terminal device may obtain first information transmitted by an access network device, or it may obtain first information that has been previously stored by the terminal device.

[0142] S602: The terminal device determines one or more pieces of ground truth information based on the first piece of information.

[0143] After obtaining the first piece of information, the terminal device can determine one or more pieces of ground truth information based on that first piece of information.

[0144] In one implementation, by default, a terminal device can measure one or more reference signals at the time-domain resource locations of one or more reference signals carried with the first piece of information in order to obtain one or more pieces of ground truth information.

[0145] In another implementation, if an access network device uses the first information to configure the time-domain resource locations of one or more reference signals used to measure ground truth information, by default, a terminal device can measure one or more reference signals at the time-domain resource locations of one or more reference signals carried with the first information in order to obtain one or more ground truth information.

[0146] In yet another implementation, the terminal device can obtain the time-domain resource locations of one or more reference signals used for model inference. By default, the terminal device can measure one or more reference signals at the time-domain resource locations of one or more reference signals carried with the first information in order to obtain one or more ground truth information, and can measure one or more reference signals at the time-domain resource locations of one or more reference signals used for model inference.

[0147] In yet another implementation, the access network device can instead instruct the terminal device to measure one or more reference signals at the time-domain resource locations of one or more reference signals carried with the first information in order to obtain one or more ground truth information, or to measure one or more reference signals at the time-domain resource locations of one or more reference signals used for model inference in order to obtain one or more ground truth information. One or more ground truth data points are used to obtain monitoring results for the performance of the AI ​​model, and the output of the AI ​​model is predictive information.

[0148] In this embodiment, it is found that the time-domain resource location of at least one of the obtained ground truth information is different from the time-domain resource location of the prediction information.

[0149] According to the model monitoring method provided in this embodiment of the present application, a terminal device can obtain ground truth information corresponding to a specific time-domain resource location by obtaining the time-domain resource location of one or more reference signals used to measure ground truth information, and therefore the performance of the predictive model can be flexibly monitored.

[0150] The aforementioned solution can be used to monitor CSI prediction models. In communication systems (e.g., LTE or NR communication systems), network devices need to determine configurations such as resource allocation, MCS, and precoding for scheduling downlink data channels for terminal devices based on CSI. It will be understood that CSI is channel information, and that it can reflect the characteristics and quality of the channel.

[0151] CSI measurement means that the receiver obtains channel information based on a reference signal transmitted by the transmitter, that is, it estimates channel information using a channel estimation method. For example, the reference signal may include one or more of the following: channel state information reference signal (CSI-RS), synchronization signal / physical broadcast channel block (SSB), sounding reference signal (SRS), demodulation reference signal (DMRS), etc. CSI-RS, SSB, DMRS, etc., can be used to measure downlink CSI. SRS, DMRS, etc., can be used to measure uplink CSI.

[0152] As an example, let's consider an FDD communication scenario. In an FDD communication scenario, the uplink channel and downlink channel are not reciprocal; in other words, reciprocity between the uplink channel and downlink channel cannot be guaranteed. Therefore, the network device typically transmits a downlink reference signal to the terminal device, and the terminal device performs channel measurement or interference measurement based on the received downlink reference signal in order to estimate the downlink CSI. The terminal device generates a CSI report in a manner predetermined by the protocol or configured by the network device, and feeds the CSI report back to the network device, which then obtains the downlink CSI.

[0153] For example, CSI may include at least one of the following: channel quality indicator (CQI), precoding matrix indicator (PMI), rank indicator (RI), CSI-RS resource indicator (CRI), layer indicator (LI), reference signal received power (RSRP), signal-to-interference plus noise ratio (SINR), etc. The signal-to-interference plus noise ratio is sometimes also called the signal-to-interference-plus-noise ratio.

[0154] The RI indicates the number of downlink transmission layers recommended by the terminal device. The CQI indicates the modulation and coding schemes that can be supported under the current channel conditions, as determined by the terminal device. The PMI indicates the precoding recommended by the terminal device. The number of precoding layers indicated by the PMI corresponds to the RI.

[0155] Please understand that the RI, CQI, PMI, etc., shown in the CSI report are merely recommended values ​​for terminal devices. Network devices may perform downlink transmissions based on some or all of the information shown in the CSI report, or they may perform downlink transmissions without referring to the information shown in the CSI report.

[0156] AI-based CSI prediction refers to inputting some past / present CSI information into an AI prediction model to output a predicted future CSI. Figure 7 is an example of predicting CSI based on an AI model according to one embodiment of the present application. Past / present CSI information for two slots, t-5 and t, may be input into the AI ​​model, and the AI ​​model may output CSI information at the future time t+5. The predicted CSI may take any of the following forms: channel response, channel matrix, channel feature matrix, or precoding matrix. The channel response and channel matrix are channels, and the channel feature matrix and precoding matrix are matrices composed of features extracted from the channels.

[0157] To calculate the intermediate KPIs of an AI model, two types of CSI information are needed. One type is the monitored CSI, for example, the predicted CSI output by the AI ​​model. The other type is the labeled CSI or ground truth CSI, i.e.

number

number

number

number

[0158] As mentioned above, for example, the KPI may be GCS, SGCS, MSE, or NMSE.

[0159] If the KPI is GCS,

number

number

[0160] If the KPI is SGCS,

number

number

[0161] If the KPI is MSE,

number

number

[0162] If the KPI is NMSE,

number

number

[0163] Whether a terminal device remembers the AI ​​model and how the CSI prediction model is monitored will be described separately with reference to each embodiment.

[0164] Figure 8 is a schematic flowchart of yet another model monitoring method according to one embodiment of the present application. In this embodiment, the terminal device does not store the AI ​​model. For example, the method may include the following steps:

[0165] S801: The terminal device obtains the first piece of information.

[0166] In this embodiment, the ground truth CSI index indicates a specific ground truth CSI obtained by the terminal device.

[0167] The terminal device obtains first information, which includes one or more first indices corresponding to one or more ground truth CSIs. For example, the terminal device may obtain first information transmitted by an access network device, or it may obtain first information that has been pre-stored by the terminal device.

[0168] In one implementation, a terminal device can determine the index of a ground truth CSI using the time point of a reference resource. The time point of the reference resource may be defined by a protocol or configured by an access network device. The reference resource is a time unit in the time domain (a time unit includes, but is not limited to, any one of the following: a symbol, a slot, a minislot, a subframe, and a radio frame; in this embodiment of the present application, a slot is used as an example for illustrative purposes). As shown in Figure 9, a reference resource is used as an example, and the slot corresponding to the reference resource is slot t. The slot in which the reference resource is located is used as the reference. For example, the index of the nth ground truth CSI before the slot in which the reference resource is located is -n, and the index of the nth ground truth CSI within the slot in which the reference resource is located and the nth ground truth CSI after the slot in which the reference resource is located is n-1. As another example, the index of the nth Ground Truth CSI in the slot where the reference resource is located and the nth Ground Truth CSI before the slot where the reference resource is located is -n+1, ​​and the index of the nth Ground Truth CSI after the slot where the reference resource is located is n. As yet another example, the index of the Kth Ground Truth CSI before the slot where the reference resource is located is 0, and the index of the Ground Truth CSI after the slot where the reference resource is located increases sequentially, where K is a parameter configured by the access network device or a predefined parameter. As yet another example, the index of the Kth Ground Truth CSI after the slot where the reference resource is located is 0, and the index of the Ground Truth CSI before the slot where the reference resource is located increases sequentially.

[0169] One or more first indices corresponding to one or more ground truth CSIs included in the first piece of information are indices relative to the slots where the reference resource is located. For example, in the example shown in Figure 9, the slot where the reference resource is located is slot t. In this case, the index of the ground truth CSI in slot t+2 is 0, and the index of the ground truth CSI in slot t+4 is 1.

[0170] An access network device transmits a channel state information-reference signal (CSI-RS) to a terminal device, which receives the CSI-RS and performs measurements to obtain ground truth CSI. The CSI-RS used to measure ground truth CSI (i.e., used for model monitoring) and the CSI-RS used for model inference (used as input to the model for CSI prediction) may be in the same group, i.e., they may belong to the same CSI-RS configuration. Alternatively, the CSI-RS used to measure ground truth CSI and the CSI-RS used for model inference may be in different groups, i.e., the CSI-RS may be specially configured for model monitoring.

[0171] In an alternative implementation, the index of the ground truth CSI may instead be the index of the CSI-RS used to measure the ground truth CSI. See Figure 9 again. We will further use the time-domain position of the CSI-RS used to measure the ground truth CSI as an example. The CSI-RS used for model inference is a CSI-RS with a period of 5 ms (e.g., the CSI-RS corresponding to the t-5, t, or t+5 slot), and the CSI-RS used for model monitoring is a non-periodic or quasi-static CSI-RS, corresponding to the t+2 or t+4 slot. The index of the ground truth CSI in the t+2 slot is 0, the index of the ground truth CSI in the t+4 slot is 1, and the index of the ground truth CSI in the t+5 slot is 2.

[0172] In yet another implementation, the CSI-RS used to measure the ground truth CSI may only include the CSI-RS used for monitoring. For example, in the example shown in Figure 9, the CSI reference resource is in slot t, the ground truth CSI index at time t+2 is 0, and the ground truth CSI index at time t+4 is 1.

[0173] In yet another implementation, the CSI-RS used to measure the ground truth CSI may instead include the CSI-RS used for monitoring and the CSI-RS used to predict the time point in time when the CSI is located. For example, in the example shown in Figure 9, the index of the ground truth CSI at time t+2 is 0, the index of the ground truth CSI at time t+4 is 1, and the index of the ground truth CSI at time t+5 is 2.

[0174] S802: The terminal device obtains one or more ground truth CSIs corresponding to one or more first indices.

[0175] The terminal device measures the CSI-RS in the slots corresponding to each of the indices in order to obtain one or more ground truth CSIs corresponding to the one or more first indices, after obtaining one or more first indices corresponding to the one or more first indices.

[0176] One or more ground truth CSIs are used to obtain performance monitoring results for the AI ​​model, and the output of the AI ​​model is a predictive CSI.

[0177] Compared to conventional techniques that can only periodically obtain model monitoring results in a grid, in this embodiment, the terminal device can obtain ground truth CSIs corresponding to a specific time unit, for example, a required time unit, by obtaining one or more first indices of one or more ground truth CSIs.

[0178] Furthermore, the method may also include the following steps (shown by dashed lines in the diagram).

[0179] S803: A terminal device transmits one or more ground truth CSIs to an access network device. Correspondingly, the access network device receives one or more ground truth CSIs.

[0180] In this embodiment, the terminal device does not store the AI ​​model and cannot perform CSI prediction. Therefore, after obtaining one or more ground truth CSIs, the terminal device transmits one or more ground truth CSIs to the access network device.

[0181] S804: Access network device obtains one or more predicted CSIs.

[0182] The access network device can store an AI model and use one or more received ground truth CSIs as input information for the model to output one or more predicted CSIs.

[0183] For example, an access network device can receive the ground truth CSI in slot t+2 and the ground truth CSI in slot t+4, and use the ground truth CSI in slots t+2 and t+4 as input information to output the predicted CSI in slot t+5.

[0184] S805: An access network device obtains one or more monitoring results of the AI ​​model's performance based on one or more ground truth CSIs and one or more predictive CSIs.

[0185] The results of monitoring the performance of the AI ​​model

number

number

[0186] Furthermore, to compare the accuracy of predicted CSI and ground truth CSI, the access network device may instead use the ground truth CSI at time T1 as the predicted CSI at time T2. For example, assume that the period of CSI-RS is 5 ms and a downlink transmission is performed at time t+3. In this case, if the downlink transmission is not in the monitoring phase, CSI-RS does not exist at time t+3. The access network device can use the ground truth CSI at time t as the predicted CSI at time t+3 to obtain the model monitoring result at time t+3: KPI(H t+3H t ).

[0187] In this case, regarding the problem raised in the background technology, since the ground truth CSI at time t+3 can be obtained, the model performance at time t+3 can be obtained, and therefore the predicted CSI predicted by the model can reflect the actual system performance, and therefore the predicted CSI can be used for communication.

[0188] According to the model monitoring method provided in this embodiment of the present application, a terminal device can obtain ground truth CSIs corresponding to a specific time unit by obtaining one or more first indices corresponding to one or more ground truth CSIs, and therefore the performance of the predictive model can be flexibly monitored.

[0189] Figure 10 is a schematic flowchart of another model monitoring method according to one embodiment of the present application. In this embodiment, the terminal device does not store the AI ​​model. Unlike the embodiment shown in Figure 8, in this embodiment, the ground truth CSI is determined based on the time-domain resource location of the CSI-RS used to measure the ground truth CSI, rather than instructing the terminal device to obtain a specific ground truth CSI by directly using the index of the ground truth CSI. For example, the method may include the following steps:

[0190] S1001: The terminal device obtains the first piece of information.

[0191] The CSI-RS used to measure ground truth CSI (i.e., used for model monitoring) and the CSI-RS used for model inference (used as input to the model for CSI prediction) may be CSI-RS within the same group, i.e., they may belong to the same CSI-RS configuration. Alternatively, the CSI-RS used to measure ground truth CSI and the CSI-RS used for model inference may be CSI-RS within different groups, i.e., the CSI-RS may be specially configured for model monitoring.

[0192] Please refer to Figure 9 for further details. As an example, we will use the time-domain position of the CSI-RS used to measure ground truth CSI, and the CSI-RS used for model inference, which is a CSI-RS with a period of 5 ms (e.g., a CSI-RS corresponding to the t-5, t, or t+5 slot). The CSI-RS used for model monitoring is a non-periodic or quasi-static CSI-RS, corresponding to the t+2 or t+4 slot. In other words, the CSI-RS used to measure ground truth CSI and the CSI-RS used for model inference are CSI-RS in different groups.

[0193] In this embodiment, the CSI-RS used to measure the ground truth CSI and the CSI-RS used for model inference are CSI-RS in different groups. The terminal device can determine the ground truth CSI based on the CSI-RS used to measure the ground truth CSI. Thus, the terminal device can obtain first information, which includes the time-domain resource locations of one or more CSI-RS used to measure the ground truth CSI.

[0194] For example, a terminal device may obtain first information transmitted by an access network device, or it may obtain first information that has been previously stored by the terminal device.

[0195] S1002: The terminal device determines one or more ground truth CSIs based on the first piece of information.

[0196] After obtaining the first piece of information, the terminal device determines one or more ground truth CSIs based on that information.

[0197] In one implementation, by default, a terminal device can measure one or more CSI-RSs at the time-domain resource locations of one or more CSI-RSs carried with the first information in order to obtain one or more ground truth CSIs. For example, in the example in Figure 9, the terminal device can measure the CSI-RS at time t+2 and the CSI-RS at time t+4 to obtain the corresponding ground truth CSIs separately.

[0198] In another implementation, if an access network device uses first information to configure one or more time-domain resource locations of CSI-RS used to measure ground truth CSI, by default, a terminal device can measure one or more CSI-RS at the time-domain resource locations of one or more CSI-RS carried by the first information in order to obtain one or more ground truth CSIs. For example, in the example in Figure 9, the terminal device can measure the CSI-RS at time t+2 and the CSI-RS at time t+4 to obtain the corresponding ground truth CSIs separately.

[0199] In yet another implementation, the terminal device can obtain one or more time-domain resource locations of CSI-RS used for model inference. By default, the terminal device can measure one or more CSI-RS at the time-domain resource locations of one or more CSI-RS carried in the first information, and one or more CSI-RS at the time-domain resource locations of one or more CSI-RS used for model inference, in order to obtain one or more ground truth CSIs. For example, in the example in Figure 9, the terminal device can measure the CSI-RS at time t+2, the CSI-RS at time t+4, and the CSI-RS at time t+5 to obtain the corresponding ground truth CSIs separately.

[0200] In yet another implementation, the access network device can instead instruct the terminal device to measure one or more CSI-RS at the time-domain resource locations of one or more CSI-RS carried with the first information in order to obtain one or more ground truth CSIs, or to measure one or more CSI-RS at the time-domain resource locations of one or more CSI-RS used for model inference in order to obtain one or more ground truth CSIs.

[0201] One or more ground truth CSIs are used to obtain performance monitoring results for the AI ​​model, and the output of the AI ​​model is a predictive CSI.

[0202] In this embodiment, it can be seen that the time-domain resource location of at least one of the one or more ground-truth CSIs obtained in the aforementioned implementation is different from the time-domain resource location of the predicted CSI.

[0203] Furthermore, the method may also include the following steps (shown by dashed lines in the diagram).

[0204] S1003: A terminal device transmits one or more ground truth CSIs to an access network device. The access network device receives one or more ground truth CSIs accordingly.

[0205] For a detailed implementation of this step, please refer to step S803 of the embodiment shown in Figure 8. Further details will not be explained here.

[0206] S1004: Access network device obtains one or more predicted CSIs, One or more predicted CSIs are outputs of the AI ​​model.

[0207] For a detailed implementation of this step, please refer to step S804 of the embodiment shown in Figure 8. Further details will not be explained here.

[0208] S1005: The access network device obtains one or more monitoring results of the AI ​​model's performance based on one or more received ground truth CSIs and one or more received predictive CSIs.

[0209] For a detailed implementation of this step, please refer to step S805 of the embodiment shown in Figure 8. Further details will not be explained here.

[0210] According to the model monitoring method provided in this embodiment of the present application, a terminal device can obtain a ground truth CSI corresponding to a specific time-domain resource location by obtaining the time-domain resource locations of one or more CSI-RS used to measure the ground truth CSI, and therefore the performance of the predictive model can be flexibly monitored.

[0211] Figure 11 is a schematic flowchart of yet another model monitoring method according to one embodiment of the present application. Unlike the embodiments shown in Figures 8 and 10, in this embodiment, the terminal device stores an AI model that can predict CSI. For example, the method may include the following steps:

[0212] S1101: The terminal device obtains the first piece of information. The first piece of information includes one or more first indices of one or more Ground Truth CSIs.

[0213] For a detailed implementation of this step, please refer to step S801 of the embodiment shown in Figure 8. Further details will not be explained here.

[0214] S1102: The terminal device obtains one or more ground truth CSIs corresponding to one or more first indices.

[0215] For a detailed implementation of this step, please refer to step S802 of the embodiment shown in Figure 8. Further details will not be explained here.

[0216] S1103: The terminal device obtains one or more predicted CSIs.

[0217] The order of steps S1101, S1102, and S1103 is not limited in this embodiment. Steps S1101 and S1102 may be performed first, followed by step S1103, or steps S1101, S1102, and S1103 may be performed simultaneously, or step S1103 may be performed first, followed by steps S1101 and S1102.

[0218] In this embodiment, an access network device can instruct a terminal device to obtain a predicted CSI at a specific point in time, and the terminal device obtains one or more predicted CSIs based on the instructions from the access network device.

[0219] Specifically, the access network device can send second information to the terminal device, and the terminal device obtains one or more predicted CSIs based on the second information.

[0220] In one implementation, the second information includes one or more second indices of one or more predicted CSIs. The one or more second indices are based on time units corresponding to a reference resource. The time point of the reference resource is defined by the protocol. The reference resource is a time unit in the time domain (a time unit includes, but is not limited to, any one of the following: a symbol, a slot, a minislot, a subframe, and a wireless frame; in this embodiment of the present application, a slot is used as an example for illustrative purposes).

[0221] The time units corresponding to the reference resource are as follows: The first time unit in which the resources for reporting the monitoring results are located, The second time unit in which downlink control signaling (DCI) for scheduling model monitoring is located, A third time unit determined based on the first or second time unit, or It is one of the predetermined / pre-configured fourth time units.

[0222] In some embodiments of this application, after obtaining a predicted CSI, the terminal device can generate a monitoring result report of the AI ​​model and report the monitoring result report to the access network device in a first time unit. In this case, the time unit corresponding to the reference resource may be the first time unit in which the resource for reporting the monitoring result report is located. For example, the slot corresponding to the reference resource is the slot before or after the slot in which the A-th CSI-RS is located and the resource for reporting the monitoring result report is located, where A is a parameter configured by the access network device or a predefined parameter. For example, A=1.

[0223] In some embodiments of this application, an access network device can schedule a terminal device to perform model monitoring. Specifically, the access network device sends a DCI to the terminal device in a second time unit, and the DCI instructs the terminal device to perform model monitoring. In this case, the time unit corresponding to the reference resource may be the second time unit in which the DCI for scheduling the model monitoring is located.

[0224] Alternatively, the time unit corresponding to the reference resource may be a third time unit determined based on a first or second time unit. For example, T may be a first or second time unit, the third time unit may be T + Δt, where Δt may be a parameter configured by the access network device or a predefined parameter.

[0225] After the time unit corresponding to the reference resource is determined, in this implementation, one or more second indices of one or more predicted CSIs are based on the time unit corresponding to the reference resource. For example, the index of the nth predicted CSI before the slot where the reference resource is located is -n, and the index of the nth predicted CSI within the slot where the reference resource is located and the nth predicted CSI after the slot where the reference resource is located is n-1. As another example, the index of the nth predicted CSI within the slot where the reference resource is located and the nth predicted CSI before the slot where the reference resource is located is -n+1, ​​and the index of the nth predicted CSI after the slot where the reference resource is located is n. As yet another example, the index of the Kth predicted CSI before the slot where the reference resource is located is 0, and the indices of the predicted CSI after the slot where the reference resource is located increase sequentially, where K is a parameter configured by the access network device or a predefined parameter. As yet another example, the index of the Kth predicted CSI after the slot where the CSI reference resource is located is 0, and the indices of the predicted CSI before the slot where the CSI reference resource is located increase sequentially.

[0226] In an alternative implementation, the second piece of information includes the relative time unit value of each of the one or more predicted CSIs relative to the time unit corresponding to the reference resource. In other words, the predicted CSI may instead be indicated using the relative time unit value of the predicted CSI relative to the time unit corresponding to the reference resource. For example, 0 indicates the slot where the reference resource is located, -n indicates the nth slot before the slot where the reference resource is located, and n indicates the nth slot after the slot where the CSI reference resource is located.

[0227] After receiving the second piece of information, the terminal device can generate one or more predicted CSIs based on the execution of the terminal device's AI model. For example, as shown in Figure 9, the slot where the reference resource is located is t, the second index included in the second piece of information is 1, and the CSI-RS used for model inference is a CSI-RS with a period of 5 ms. Thus, the second piece of information instructs to obtain a predicted CSI in the t+5 slot. Based on the execution of the terminal device's AI model, the terminal device can obtain a predicted CSI in the t+5 slot by using the CSIs obtained by actual measurements in the t-5 and t slots as input information for the model.

[0228] Furthermore, the method may also include the following steps (shown by dashed lines in the diagram).

[0229] S1104: The terminal device transmits one or more ground truth CSIs and one or more predictive CSIs to the access network device. Accordingly, the access network device receives one or more ground truth CSIs and one or more predictive CSIs.

[0230] The terminal device obtains one or more ground truth CSIs and one or more predictive CSIs, and the terminal device can transmit one or more ground truth CSIs and one or more predictive CSIs to the access network device.

[0231] S1105: An access network device obtains one or more monitoring results of the AI ​​model's performance based on one or more ground truth CSIs and one or more predictive CSIs.

[0232] For a detailed implementation of this step, please refer to step S805 of the embodiment shown in Figure 8. Further details will not be explained here.

[0233] According to the model monitoring method provided in this embodiment of the present application, a terminal device can obtain a ground truth CSI corresponding to a specific time unit by obtaining one or more first indices corresponding to one or more ground truth CSIs, and obtain a corresponding predicted CSI based on instructions from an access network device, so that the access network device can obtain model monitoring results of the actual system, so that the performance of the predictive model can be flexibly monitored and the predictive performance can reflect the actual system performance.

[0234] Figure 12 is a schematic flowchart of yet another model monitoring method according to one embodiment of the present application. In this embodiment, a terminal device stores an AI model that can predict the CSI. Unlike the embodiment shown in Figure 11, in this embodiment, the ground truth CSI is determined based on the time-domain resource location of the CSI-RS used to measure the ground truth CSI. For example, the method may include the following steps:

[0235] S1201: The terminal device obtains the first piece of information. The first piece of information includes the time-domain resource locations of one or more CSI-RSs used to measure ground truth CSI.

[0236] For a detailed implementation of this step, please refer to step S1001 of the embodiment shown in Figure 10. Further details will not be explained here.

[0237] S1202: The terminal device determines one or more ground truth CSIs based on the first information.

[0238] For the specific implementation of this step, please refer to step S1002 of the embodiment shown in FIG. 10. Details will not be described again here.

[0239] S1203: The terminal device obtains one or more predicted CSIs.

[0240] For the specific implementation of this step, please refer to step S1103 of the embodiment shown in FIG. 11. Details will not be described again here.

[0241] S1204: The terminal device transmits one or more ground truth CSIs and one or more predicted CSIs to the access network device. Correspondingly, the access network device receives one or more ground truth CSIs and one or more predicted CSIs.

[0242] For the specific implementation of this step, please refer to step S1104 of the embodiment shown in FIG. 11. Details will not be described again here.

[0243] The order of step S1201, step S1202, and step S1203 is not limited in this embodiment. Step S1201 and step S1202 may be executed first, then step S1203 may be executed, step S1201, step S1202, and step S1203 may be executed simultaneously, or step S1203 may be executed first, and then step S1201 and step S1202 may be executed.

[0244] S1205: The access network device obtains one or more monitoring results of the performance of the AI model based on one or more ground truth CSIs and one or more predicted CSIs.

[0245] For a detailed implementation of this step, please refer to step S805 of the embodiment shown in Figure 8. Further details will not be explained here.

[0246] According to the model monitoring method provided in this embodiment of the present application, a terminal device can obtain a ground truth CSI corresponding to a specific time-domain resource location by obtaining the time-domain resource location of one or more CSI-RS used to measure ground truth CSI, and obtain a corresponding predicted CSI based on instructions from an access network device. Thus, the access network device can obtain model monitoring results of the actual system, and therefore, the performance of the predictive model can be flexibly monitored, and the predictive performance can reflect the actual system performance.

[0247] Figure 13 is a schematic flowchart of yet another model monitoring method according to one embodiment of the present application. Unlike the previously described embodiment, in this embodiment, the terminal device stores an AI model that can predict CSI, and the terminal device can obtain further monitoring results. For example, the method may include the following steps:

[0248] S1301: The terminal device obtains the first piece of information. The first piece of information includes one or more first indices of one or more Ground Truth CSIs.

[0249] For a detailed implementation of this step, please refer to step S801 of the embodiment shown in Figure 8. Further details will not be explained here.

[0250] S1302: The terminal device obtains one or more ground truth CSIs corresponding to one or more first indices.

[0251] For a detailed implementation of this step, please refer to step S802 of the embodiment shown in Figure 8. Further details will not be explained here.

[0252] S1303: The terminal device obtains one or more predicted CSIs.

[0253] For a detailed implementation of this step, please refer to step S1103 of the embodiment shown in Figure 11. Further details will not be explained here.

[0254] The order of steps S1301, S1302, and S1303 is not limited in this embodiment. Steps S1301 and S1302 may be performed first, followed by step S1303, or steps S1301, S1302, and S1303 may be performed simultaneously, or step S1303 may be performed first, followed by steps S1301 and S1302.

[0255] S1304: The terminal device obtains one or more monitoring results of the AI ​​model's performance based on one or more ground truth CSIs and one or more predictive CSIs.

[0256] After obtaining one or more ground truth CSIs and one or more predictive CSIs, the terminal device can obtain one or more monitoring results for each ground truth CSI based on the ground truth CSI and the one or more predictive CSIs, or it can obtain one or more monitoring results for each predictive CSI based on the one or more ground truth CSIs.

[0257] For example, each terminal device

number

number

Number

Number

Number

Number

[0258] Furthermore, the access network device may alternatively indicate a case where each monitoring result can be obtained by calculation using specific predicted CSI and specific ground truth CSI.

[0259] S1305: The terminal device sends a monitoring result report to the access network device.

[0260] After obtaining one or more monitoring results, the terminal device can send a monitoring result report to the network device.

[0261] The implementation for the terminal device to send a monitoring result report includes, but is not limited to, the following.

[0262] In one implementation, the monitoring result report includes one or more monitoring results. In this implementation, the terminal device can report all the obtained monitoring results to the access network device, and thus, the access network device can obtain the complete monitoring results. For example, in the above example, the terminal device can report the above four KPIs.

[0263] When reporting is performed, the terminal device can report one or more monitoring results to the access network device in a predetermined order. For example, if the monitoring results are KPI results, for multiple predictive CSIs and / or multiple ground truth CSIs, reporting may be performed in an order instructed by the access network device or in chronological order of the monitoring results, and for predictive CSIs and ground truth CSIs, reporting may be performed first in the order of predictive CSIs, then in the order of ground truth CSIs, or first in the order of ground truth CSIs, then in the order of predictive CSIs. For example, four KPIs:

number

number

number

number

number

number

number

number

[0264] In an alternative implementation, in some scenarios, the access network device only needs to know if there are monitoring results indicating good model performance, and does not need to obtain all monitoring results. Therefore, the monitoring results report includes at least one monitoring result that is above a first threshold among one or more monitoring results. The first threshold may be configured by the access network device or may be predefined. For example, in the above example, the terminal device only needs to report one or more KPIs that are above the first threshold among the four KPIs. This implementation can reduce signaling overhead.

[0265] In yet another implementation, in some scenarios, the access network device only needs to know if there are any monitoring results indicating poor model performance, and does not need to obtain all monitoring results. Therefore, the monitoring results report only needs to include at least one monitoring result that is below a second threshold among one or more monitoring results. The second threshold may be configured by the access network device or predefined, and it is smaller than the first threshold. For example, in the above example, the terminal device only needs to report one or more KPIs that are below the second threshold among the four KPIs. This implementation can reduce signaling overhead.

[0266] In yet another implementation, in some scenarios, the access network device only needs to know if there are monitoring results indicating good model performance, and does not need to obtain all monitoring results. Therefore, the monitoring results report only needs to include an index of the predicted CSI and an index of the ground truth CSI corresponding to the best monitoring result among one or more monitoring results. The access network device may have already received one or more predicted CSIs and one or more ground truth CSIs. In this case, the access network device can obtain the best monitoring result based on the index of the predicted CSI and the index of the ground truth CSI corresponding to the best monitoring result. This implementation reduces signaling overhead and has a low computational power requirement for the terminal device.

[0267] Alternatively, as described above, the access network device may use the first message to specify one or more predicted CSI indices used to calculate the monitoring results, using the same specifying method as the access network device, and the monitoring results report may include sequential numbers of predicted CSIs corresponding to the best monitoring result among the one or more monitoring results, in the order of the one or more predicted CSIs specified by the access network device. In addition, as described above, the access network device may further specify one or more ground truth CSI indices used to calculate the monitoring results, using the same specifying method as the access network device, and the monitoring results report may include sequential numbers of ground truth CSIs corresponding to the best monitoring result among the one or more monitoring results, in the order of the one or more ground truth CSIs specified by the access network device.

[0268] In yet another implementation, in some scenarios, the access network device only needs to know if there are monitoring results indicating poor model performance, and does not need to obtain all monitoring results. Therefore, the monitoring results report only needs to include an index of the predicted CSI and an index of the ground truth CSI corresponding to the worst monitoring result among one or more monitoring results. The access network device may have already received one or more predicted CSIs and one or more ground truth CSIs. In this case, the access network device can obtain the worst monitoring result based on the index of the predicted CSI and the index of the ground truth CSI corresponding to the worst monitoring result. This implementation reduces signaling overhead and has a low computational power requirement for the terminal device.

[0269] Alternatively, as described above, the access network device may use the first message to specify one or more predicted CSI indices used to calculate the monitoring results, using the same specifying method as the access network device, and the monitoring results report may include sequential numbers of predicted CSIs corresponding to the worst monitoring result among the one or more monitoring results, in the order of the one or more predicted CSIs specified by the access network device. In addition, as described above, the access network device may further specify one or more ground truth CSI indices used to calculate the monitoring results, using the same specifying method as the access network device, and the monitoring results report may include sequential numbers of ground truth CSIs corresponding to the worst monitoring result among the one or more monitoring results, in the order of the one or more ground truth CSIs specified by the access network device.

[0270] In yet another implementation, a differential reporting method may be used. The monitoring result report includes the absolute value of the first monitoring result among multiple monitoring results, and the relative values ​​of the other monitoring results relative to the first monitoring result. For example, four KPIs to be reported:

number

number

number

number

number

number

number

number

number

number

number

[0271] Furthermore, the first monitoring result may be determined based on the order of the ground truth CSIs used and / or the order of the predictive CSIs used when obtaining multiple monitoring results. As described above, for each ground truth CSI, a terminal device may obtain one or more monitoring results based on the ground truth CSI and one or more predictive CSIs, and for each ground truth CSI, the access network device indicates the index of one or more ground truth CSIs used to calculate the monitoring result, or for each predictive CSI, a terminal device may obtain one or more monitoring results based on one or more ground truth CSIs, and for each predictive CSI, the access network device indicates the index of one or more predictive CSIs used to calculate the monitoring result by using the first message. Therefore, the order in which one or more monitoring results are generated may be determined based on the order of the ground truth CSIs used and / or the order of the predictive CSIs used during the generation of monitoring results by the terminal device, and thus the first monitoring result may be determined based on that generation order. For example, the first monitoring result may be the first or last monitoring result among one or more sequentially generated monitoring results.

[0272] In yet another implementation, the access network device may instead instruct the terminal device to report only a portion of the monitoring results. For example, the terminal device may be instructed to report some of the KPI results from the four KPI results mentioned above in a bitmap format.

[0273] Furthermore, the monitoring results report and CSI feedback may be reported in the same CSI report, or they may be reported in separate CSI reports, with the CSI feedback being the measurement result used to report the CSI-RS.

[0274] If monitoring results and CSI feedback are reported in the same CSI report, the access network device may configure or pre-specify that the reporting is done in a specific CSI report used for CSI feedback. For example, it may be pre-specified that the reporting is done in the first CSI report after the last CSI-RS used to measure ground truth CSI, or in the first CSI report T1 time after the last CSI-RS used to measure ground truth CSI. T1 is the time allocated for the terminal device to calculate the KPI, and may be configured or pre-specified by the access network device. T1 may be related to the processing capacity of the terminal device. Note that if monitoring results reports and CSI feedback can be reported in the same CSI report, the predicted CSI reported in the CSI report and the predicted CSI of the monitored target corresponding to the monitoring results report are not the same CSI. Figure 14 is an example diagram of reporting a monitoring results report according to one embodiment of the present application. The CSI report is submitted at time t+7, and the predicted CSI reported in the CSI report at time t+7 is the predicted CSI at time t+10, which is predicted using the ground truth CSI at time t+5 and the ground truth CSI prior to time t+5. However, the predicted CSI of the monitored target corresponding to the monitoring results report within the CSI report is the predicted CSI at time t+5, which is predicted using the ground truth CSI at time t and the ground truth CSI prior to time t.

[0275] If the monitoring results report is reported independently of the CSI feedback, the resources for reporting the monitoring results report are configured by the access network device. In addition, the terminal device does not expect the reporting resources to be available earlier than the last CSI-RS corresponding to the monitoring results report used to measure the ground truth CSI, or the terminal device does not expect the reporting resources to be available earlier than T2 time after the last CSI-RS corresponding to the monitoring results report used to measure the ground truth CSI. T2 is the time that the terminal device is allowed to calculate the KPI, and may be configured by the access device or predefined. T2 may be related to the processing capacity of the terminal device.

[0276] According to the model monitoring method provided in this embodiment of the present application, a terminal device can obtain a ground truth CSI corresponding to a specific time unit by obtaining one or more first indices corresponding to one or more ground truth CSIs, obtain a corresponding predicted CSI based on instructions from an access network device, and obtain a model monitoring result based on the obtained ground truth CSI and the obtained predicted CSI. Therefore, the access network device can obtain a model monitoring result of the actual system, and therefore, the performance of the predictive model can be flexibly monitored, and the predictive performance can reflect the actual system performance.

[0277] Figure 15 is a schematic flowchart of yet another model monitoring method according to one embodiment of the present application. Unlike the embodiments shown in Figures 8 and 10, in this embodiment, a terminal device stores an AI model that can predict the CSI, and the terminal device can further obtain monitoring results. Unlike the embodiment shown in Figure 13, in this embodiment, the ground truth CSI is determined based on the time-domain resource location of the CSI-RS used to measure the ground truth CSI. For example, the method may include the following steps:

[0278] S1501: The terminal device obtains the first piece of information. The first piece of information includes the time-domain resource locations of one or more CSI-RSs used to measure ground truth CSI.

[0279] For a detailed implementation of this step, please refer to step S1001 of the embodiment shown in Figure 10. Further details will not be explained here.

[0280] S1502: The terminal device determines one or more ground truth CSIs based on the first piece of information.

[0281] For a detailed implementation of this step, please refer to step S1002 of the embodiment shown in Figure 10. Further details will not be explained here.

[0282] S1503: The terminal device obtains one or more predicted CSIs.

[0283] For a detailed implementation of this step, please refer to step S1103 of the embodiment shown in Figure 11. Further details will not be explained here.

[0284] S1504: The terminal device obtains one or more monitoring results of the AI ​​model's performance based on one or more ground truth CSIs and one or more predictive CSIs.

[0285] For a detailed implementation of this step, please refer to step S1304 of the embodiment shown in Figure 13. Further details will not be explained here.

[0286] S1505: The terminal device sends a monitoring result report to the access network device.

[0287] For a detailed implementation of this step, please refer to step S1305 of the embodiment shown in Figure 13. Further details will not be explained here.

[0288] According to the model monitoring method provided in this embodiment of the present application, a terminal device can obtain a ground truth CSI corresponding to a specific time-domain resource location by obtaining one or more CSI-RS time-domain resource locations used to measure ground truth CSI, obtain a corresponding predicted CSI based on instructions from an access network device, and obtain a model monitoring result based on the obtained ground truth CSI and the obtained predicted CSI. Thus, the access network device can obtain a model monitoring result of the actual system, and therefore, the performance of the predictive model can be flexibly monitored, and the predictive performance can reflect the actual system performance.

[0289] The solutions described in the above embodiments can also be applied to beam prediction scenarios.

[0290] Beam prediction involves forecasting the optimal beam configuration for the future based on past and present beam measurement results.

[0291] Key performance indicators (KPIs) for beam prediction may include Top-1 accuracy, Top-1 accuracy with ndB margin, Top-K accuracy, and reference signal received power (RSRP) difference.

[0292] Top-1 accuracy refers to whether the predicted optimal beam is the actual optimal beam, or the proportion of the actual optimal beam to the predicted optimal beam across multiple monitoring cycles.

[0293] Top1 accuracy with an ndB margin refers to whether the difference between the RSRP of the predicted optimal beam and the RSRP of the actual optimal beam is less than ndB, or the proportion of predicted optimal beams among those predicted over multiple monitoring cycles in which the RSRP is less than ndB from the RSRP of the actual optimal beam.

[0294] Top-K accuracy refers to whether the predicted optimal beam is actually among the K optimal beams, or the proportion of the predicted optimal beams among the K optimal beams over multiple monitoring cycles, or whether the actual optimal beam is among the K predicted optimal beams, or the proportion of the actual optimal beam among the K predicted optimal beams over multiple monitoring cycles.

[0295] The RSRP difference is the difference between the predicted optimal beam's RSRP and the actual optimal beam's RSRP, or the difference between the predicted optimal beam's RSRP and the beam's actual RSRP.

[0296] Figure 16 is a schematic flowchart of yet another model monitoring method according to one embodiment of the present application. For example, the method may include the following steps:

[0297] S1601: The terminal device obtains the first piece of information.

[0298] Similar to CSI forecasting, two types of information are needed to calculate the KPIs for beam forecasting. One type is forecasting information, such as one or more of the predicted optimal beam index or RSRP. The other type is ground truth information, such as one or more of the actual optimal beam index or RSRP. Alternatively, an access network device may instruct terminal devices to use specific forecasting and ground truth information to calculate the KPIs.

[0299] In this embodiment, the index of the ground truth beam information indicates specific ground truth beam information obtained by the terminal device.

[0300] The terminal device obtains first information, which includes one or more first indices corresponding to one or more ground truth beam pieces of information. For example, the terminal device may obtain first information transmitted by an access network device, or it may obtain first information that has been previously stored by the terminal device.

[0301] For a specific implementation of this step, please refer to the embodiment shown in Figure 8, which demonstrates a concrete implementation of obtaining the first information using CSI prediction. Further details will not be explained here.

[0302] S1602: The terminal device obtains one or more pieces of ground truth beam information corresponding to one or more first indices.

[0303] After obtaining one or more first indices corresponding to one or more ground truth beam information, the terminal device measures a reference signal, such as CSI-RS, corresponding to each of the one or more indices in order to obtain one or more ground truth beam information corresponding to the one or more first indices.

[0304] One or more ground truth beam data points are used to obtain monitoring results for the AI ​​model's performance, and the AI ​​model's output is predicted beam data.

[0305] Compared to conventional techniques that can only periodically obtain model monitoring results in a grid, in this embodiment, the terminal device can obtain ground truth beam information corresponding to a specific time unit, for example, a required time unit, by obtaining one or more first indices of one or more ground truth beam pieces of information.

[0306] Furthermore, the terminal device can transmit one or more ground truth beam information to the access network device. The access network device obtains one or more predicted beam information and, based on the one or more ground truth beam information and the one or more predicted beam information, obtains one or more monitoring results of the AI ​​model's performance.

[0307] Alternatively, the terminal device may obtain one or more prediction beam information items and transmit one or more ground truth beam information items and one or more prediction beam information items to the access network device. In addition, the access network device obtains one or more monitoring results of the AI ​​model's performance based on one or more ground truth beam information items and one or more prediction beam information items.

[0308] According to the model monitoring method provided in this embodiment of the present application, a terminal device can obtain ground truth beam information corresponding to a specific time unit by obtaining one or more first indices corresponding to one or more ground truth beam information, and therefore the performance of the predictive model can be flexibly monitored.

[0309] Figure 17 is a schematic flowchart of yet another model monitoring method according to one embodiment of the present application. Unlike the embodiment shown in Figure 16, this embodiment determines the ground truth beam information based on the time-domain resource position of a reference signal used to measure the ground truth beam information, rather than instructing a terminal device to obtain specific ground truth beam information by directly using an index of ground truth beam information. For example, the method may include the following steps:

[0310] S1701: The terminal device obtains the first piece of information.

[0311] The reference signal used to measure ground truth beam information (i.e., used for model monitoring) and the reference signal used for model inference (used as input to the model for beam information prediction) may be reference signals within the same group, i.e., they may belong to the same reference signal configuration. Alternatively, the reference signal used to measure ground truth beam information and the reference signal used for model inference may be reference signals within different groups, i.e., the reference signal may be specially configured for model monitoring.

[0312] In this embodiment, the reference signals used to measure ground truth beam information and the reference signals used for model inference are reference signals in different groups. The terminal device can determine ground truth beam information based on the reference signals used to measure ground truth beam information. Thus, the terminal device can obtain first information, which includes the time-domain resource positions of one or more reference signals used to measure ground truth beam information.

[0313] For example, a terminal device may obtain first information transmitted by an access network device, or it may obtain first information that has been previously stored by the terminal device.

[0314] For a specific implementation of this step, please refer to the specific implementation of obtaining the first information using CSI prediction in the embodiment shown in Figure 10. Further details will not be explained here.

[0315] S1702: The terminal device determines one or more pieces of ground truth beam information based on the first piece of information.

[0316] After obtaining the first piece of information, the terminal device can determine one or more pieces of ground truth beam information based on the first piece of information.

[0317] In one implementation, by default, the terminal device can measure one or more reference signals at the time-domain resource locations of one or more reference signals carried with the first information in order to obtain one or more ground truth beam information.

[0318] In another implementation, if an access network device uses first information to configure the time-domain resource locations of one or more reference signals used to measure ground truth beam information, by default, a terminal device can measure one or more reference signals at the time-domain resource locations of one or more reference signals carried with first information in order to obtain one or more ground truth beam information.

[0319] In yet another implementation, the terminal device can obtain the time-domain resource locations of one or more reference signals used for model inference. By default, the terminal device can measure one or more reference signals at the time-domain resource locations of one or more reference signals carried with the first information in order to obtain one or more ground truth beam information, and can measure one or more reference signals at the time-domain resource locations of one or more reference signals used for model inference.

[0320] In yet another implementation, the access network device can instruct the terminal device to measure one or more reference signals at the time-domain resource locations of one or more reference signals carried with the first information in order to obtain one or more ground truth beam information, or to measure one or more reference signals at the time-domain resource locations of one or more reference signals used for model inference in order to obtain one or more ground truth beam information.

[0321] One or more ground truth beam data points are used to obtain monitoring results for the AI ​​model's performance, and the AI ​​model's output is predicted beam data.

[0322] In this embodiment, it is found that the time-domain resource location of at least one of the obtained ground truth beam information is different from the time-domain resource location of the predicted beam information.

[0323] Furthermore, the terminal device can transmit one or more ground truth beam information to the access network device. The access network device obtains one or more predicted beam information and, based on the one or more ground truth beam information and the one or more predicted beam information, obtains one or more monitoring results of the AI ​​model's performance.

[0324] Alternatively, the terminal device may obtain one or more prediction beam information items and transmit one or more ground truth beam information items and one or more prediction beam information items to the access network device. In addition, the access network device obtains one or more monitoring results of the AI ​​model's performance based on one or more ground truth beam information items and one or more prediction beam information items.

[0325] According to the model monitoring method provided in this embodiment of the present application, a terminal device can obtain ground truth beam information corresponding to a specific time-domain resource location by obtaining the time-domain resource location of one or more reference signals used to measure ground truth beam information, and therefore the performance of the predictive model can be flexibly monitored.

[0326] It will be understood that in the embodiments described above, methods and / or steps performed by an access network device may instead be performed by components (e.g., chips or circuits) used within the access network device, and methods and / or steps performed by a terminal device may instead be performed by components (e.g., chips or circuits) used within the terminal device.

[0327] The foregoing describes the solutions provided in the embodiments of this application primarily from the perspective of the interaction between network elements. Accordingly, embodiments of this application further provide a model monitoring device, which is configured to carry out the aforementioned method. The model monitoring device may be an access network device or a component that can be used within an access network device in the embodiments of the aforementioned method, or it may be a terminal device or a component that can be used within a terminal device in the embodiments of the aforementioned method. It will be understood that in order to carry out the aforementioned functions, the model monitoring device includes corresponding hardware structures and / or corresponding software modules for performing each function. Those skilled in the art will readily realize, in combination with the example units and algorithmic steps described in the embodiments disclosed herein, that this application can be implemented in hardware or in combination of hardware and computer software. Whether the functions are performed by hardware or by hardware driven by computer software depends on the specific application and design constraints of the technical solution. Those skilled in the art can implement the functions described using various methods for each specific application. However, such implementations should not be considered to exceed the scope of this application.

[0328] In embodiments of this application, the model monitoring device may be divided into functional modules based on embodiments of the method described above. For example, each functional module may be divided based on corresponding functions, or two or more functions may be integrated into a single processing unit. The integrated module may be implemented in hardware form or in the form of a software functional module. Note that in embodiments of this application, the module division is an example and merely a logical functional division. Other division methods may be possible in actual implementation.

[0329] Based on the same concept as the aforementioned model monitoring method, this application further provides the following model monitoring device.

[0330] Figure 18 is a diagram showing the structure of a model monitoring device according to one embodiment of the present application. The model monitoring device 1800 includes a transceiver unit 1801 and a processing unit 1802.

[0331] When the model monitoring device is configured to perform the functions of a terminal device in an embodiment of the method described above, the processing unit 1802 is configured to perform operations S601 and S602 in the embodiment shown in Figure 6, the processing unit 1802 is configured to perform operations S701 and S702 in the embodiment shown in Figure 7, the transceiver unit 1801 is configured to perform operations of a terminal device at S803 in the embodiment shown in Figure 8, the processing unit 1802 is configured to perform operations S801 and S802 in the embodiment shown in Figure 8, the transceiver unit 1801 is configured to perform operations of a terminal device at S1003 in the embodiment shown in Figure 10, the processing unit 1802 is configured to perform operations S1001 and S1002 in the embodiment shown in Figure 10, the transceiver unit 1801 is configured to perform operations of a terminal device at S1104 in the embodiment shown in Figure 11, and the processing unit 1802 is configured in the embodiment shown in Figure 11 In the embodiment shown in Figure 12, the transceiver unit 1801 is configured to perform steps S1101 to S1103, and in the embodiment shown in Figure 12, the transceiver unit 1801 is configured to perform an operation on a terminal device in S1204, and in the embodiment shown in Figure 12, the processing unit 1802 is configured to perform steps S1201 to S1203, and in the embodiment shown in Figure 13, the transceiver unit 1801 is configured to perform an operation on a terminal device in S1305, and in the embodiment shown in Figure 13, the processing unit 1802 is configured to perform steps S1301 to S1304, and in the embodiment shown in Figure 15, the transceiver unit 1801 is configured to perform an operation on a terminal device in S1505, and in the embodiment shown in Figure 15, the processing unit 1802 is configured to perform steps S1501 to S1504, and in the embodiment shown in Figure 16, the processing unit 1802 is configured to perform operations S1601 and S1602, or the processing unit 1802 is configuredThe embodiment shown in Figure 17 is configured to perform operations S1701 and S1702.

[0332] When the model monitoring device is configured to perform the functions of an access network device in an embodiment of the method described above, the transceiver unit 1801 is configured to perform operations on the access network device in S803 in the embodiment shown in Figure 8, the processing unit 1802 is configured to perform operations S804 and S805 in the embodiment shown in Figure 8, the transceiver unit 1801 is configured to perform operations on the access network device in S1003 in the embodiment shown in Figure 10, the processing unit 1802 is configured to perform operations S1004 and S1005 in the embodiment shown in Figure 10, and the transceiver unit 1801 is configured in S1104 in the embodiment shown in Figure 11. The transceiver unit 1801 is configured to perform operations on an access network device, and in the embodiment shown in Figure 11, the processing unit 1802 is configured to perform step S1105, the transceiver unit 1801 is configured to perform operations on an access network device in step S1204, the processing unit 1802 is configured to perform step S1205, the transceiver unit 1801 is configured to perform operations on an access network device in step S1305, the transceiver unit 1801 is configured to perform operations on an access network device in step S1305, or the transceiver unit 1801 is configured to perform operations on an access network device in step S1505, the embodiment shown in Figure 15.

[0333] For specific implementations of the transceiver unit 1801 and the processing unit 1802, please refer to the description of the embodiment of the method above.

[0334] Figure 19 shows the structure of another model monitoring device according to one embodiment of the present application. The model monitoring device 1900 includes one or more processors 1901 (an example of one processor is shown in the figure). Optionally, the model monitoring device 1900 may further include a memory 1903 (shown by a dashed line in the figure). The memory 1903 is configured to store instructions executed by the processors 1901, to store input data required by the processors 1901 to execute the instructions, or to store data generated by the execution of instructions by the processors 1901. Optionally, the model monitoring device 1900 may further include an interface circuit 1902 (shown by a dashed line in the figure), the processors 1901 and the interface circuit 1902 being coupled to each other. It will be understood that the interface circuit 1902 may be a transceiver or an input / output interface. The processor 1901 is configured to perform the functions of the processing unit 1802 in the embodiment shown in Figure 18, and the interface circuit 1902 is configured to perform the functions of the transceiver unit 1801 in the embodiment shown in Figure 18.

[0335] If the model monitoring device is a chip used within an access network device, the chip performs the functions of the access network device in the embodiments of the method described above. The chip receives information from another module within the access network device (e.g., a radio frequency module or antenna), and this information is transmitted to the access network device by a terminal device. Alternatively, the chip transmits information to another module within the access network device (e.g., a radio frequency module or antenna), and this information is transmitted to a terminal device by the access network device.

[0336] If the model monitoring device is a chip used within a terminal device, the chip performs the functions of the terminal device in the embodiments of the method described above. The chip receives information from another module within the terminal device (e.g., a radio frequency module or an antenna), and this information is transmitted to the terminal device by the access network device. Alternatively, the chip transmits information to another module within the terminal device (e.g., a radio frequency module or an antenna), and this information is transmitted to the access network device by the terminal device.

[0337] In addition, it should be noted that the transceiver unit and / or processing unit may be implemented using virtual modules. For example, the processing unit may be implemented using a software function unit or virtual device, and the transceiver unit may be implemented using a software function or virtual device. Alternatively, the processing unit or transceiver unit may be implemented using a physical device. For example, if the device is implemented using a chip / chip circuit, the transceiver unit may be an input / output circuit and / or a communication interface that performs input operations (corresponding to the receive operations described above) and output operations (corresponding to the transmit operations described above). The processing unit is an integrated processor, microprocessor, or integrated circuit.

[0338] In this application, the module division is merely an example and represents only a logical functional division. Other division methods are possible in actual implementation. In addition, the functional modules in the examples of this application may be integrated into a single processor, each module may exist physically independently, or two or more modules may be integrated into a single module. The integrated module may be implemented in hardware form or in the form of a software functional module.

[0339] It will be understood that the processor in the embodiments of this application may be a central processing unit (CPU), or another general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or another programmable logic device, a transistor logic device, a hardware component, or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0340] One embodiment of this application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program or instruction. When the computer program or instruction is executed, the method of the above-described embodiment is carried out.

[0341] One embodiment of this application further provides a computer program product including instructions. When the instructions are executed on a computer, the computer is made to perform the method of the above-described embodiment.

[0342] One embodiment of this application further provides a model monitoring system including the aforementioned model monitoring device.

[0343] One embodiment of this application further provides a circuit, which is coupled to a memory and configured to perform the method described in the above-described embodiment. The circuit may include a chip circuit.

[0344] If the model monitoring device is a module used within a base station, the module within the base station performs the functions of the base station in the embodiments of the method described above. The module within the base station receives information from another module within the base station (e.g., a radio frequency module or antenna), and this information is transmitted to the base station by a terminal device. Alternatively, the module within the base station transmits information to another module within the base station (e.g., a radio frequency module or antenna), and this information is transmitted to a terminal device by the base station. The module within the base station here may be the baseband chip, CU, DU, or another module of the base station, or it may be a device within an open radio access network (O-RAN) architecture, such as an open CU or open DU.

[0345] It should be noted that the aforementioned units or one or more of the units may be implemented by software, hardware, or a combination thereof. If the aforementioned units or any one of the units are implemented by software, the software exists in the form of computer program instructions and is stored in memory. The processor may be configured to execute program instructions and carry out the aforementioned method procedures.

[0346] In this application, the processor may be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field-programmable gate array or another programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, or all or part of the circuitry in the aforementioned device configured to perform processing functions, which can implement or execute the methods, steps, and logic block diagrams disclosed in this application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed with reference to this application may be performed and completed directly by a hardware processor, or by a combination of hardware and software modules within the processor.

[0347] Where the aforementioned unit or the unit is implemented by hardware, that hardware may be one or any combination of a CPU, microprocessor, digital signal processing (DSP) chip, microcontroller unit (MCU), artificial intelligence processor, ASIC, SoC, FPGA, PLD, dedicated digital circuitry, hardware accelerator, or non-integrated discrete device. The hardware may be capable of running the necessary software or may not be software-dependent in performing the aforementioned method and procedure.

[0348] Optionally, embodiments of the present application further provide a chip system comprising at least one processor and an interface. The at least one processor is coupled to memory through the interface. When the at least one processor executes a computer program or instruction in memory, the chip system is made to execute one of the embodiments of the method described above. Optionally, the chip system may comprise a chip, or a chip and another discrete device. This is not specifically limited in these embodiments of the present application.

[0349] Memory in this application may instead be a circuit or any other device capable of performing a storage function and configured to store program instructions and / or data. Memory is any other medium that can carry or store program code expected in the form of instructions or data structures and that can be accessed by a computer, but is not limited to these. For example, memory may be non-volatile memory, such as a digital versatile disc (DVD), a hard disk drive (HDD), or a solid-state drive (SSD), or volatile memory, such as random-access memory (RAM).

[0350] Unless otherwise specified, please understand that in the description of this application, " / " indicates an "or" relationship between the related subjects. For example, A / B may refer to A or B, and A and B may be singular or plural. In addition, in the description of this application, unless otherwise specified, "plural" means two or more. "At least one of the following items (pieces)" or similar expressions means any combination of these items, including a singular item (piece) or any combination of plural items (pieces). For example, at least one of a, b, or c may refer to a, b, c, ab, ac, bc, or abc, and a, b, and c may be singular or plural. In addition, in order to clearly illustrate the technical solutions of the embodiments of this application, terms such as "first" and "second" are used in the embodiments of this application to distinguish the same or similar items that have essentially the same function or purpose. Those skilled in the art will understand that terms such as “first” and “second” do not limit the quantity or order of execution, nor do they limit any clear difference. In addition, in embodiments of this application, terms such as “example” or “for example” are used to indicate that an example, illustration, or explanation is being given. No embodiment or design described as “example” or “for example” in embodiments of this application should be construed as being preferable to another embodiment or design, or having more advantages than another embodiment or design. Strictly speaking, the use of terms such as “example” or “for example” is intended to present the relevant concepts in a particular manner for ease of understanding.

[0351] All or part of the embodiments described above may be implemented using software, hardware, firmware, or any combination thereof. If a software program is used to implement an embodiment, all or part of the embodiment may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the procedures or functions according to the embodiments of this application are generated. The computer may be a general-purpose computer, a dedicated computer, a computer network, or another programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wired means (e.g., coaxial cable, optical fiber, or digital subscriber line (DSL)) or wireless means (e.g., infrared, radio waves, or microwaves).

[0352] While this application is described with reference to embodiments, a person skilled in the art can understand and implement other variations of the disclosed embodiments by looking at the accompanying drawings, the disclosed content and the accompanying claims in the process of implementing this application for which protection is claimed. In the claims, “comprising” does not exclude another component or step, and “one” or “one” does not exclude multiple cases. A single processor or another unit may perform some of the functions enumerated in the claims. Several means are recorded in different dependent claims. However, this does not mean that these means cannot be combined to produce a better effect.

[0353] It will be understood that the various numbers in the embodiments of this application are used merely for distinction to facilitate explanation and are not used to limit the scope of the embodiments of this application. The sequential numbering of the processes described above does not imply an execution order, and the execution order of the processes should be determined based on the function and internal logic of the processes.

[0354] In the embodiments described above, each embodiment has its own focus. For aspects not described in detail in one embodiment, please refer to the relevant descriptions in other embodiments.

[0355] The components within the apparatus in the embodiments of this application may be combined, separated, and removed based on actual requirements. Those skilled in the art can combine or integrate the various embodiments or features of various embodiments described herein.

[0356] In this application, cross-referencing is possible between examples without logical inconsistencies. For example, cross-referencing is possible between methods and / or terms in embodiments of methods, between functions and / or terms in embodiments of apparatuses, and between functions and / or terms in examples of apparatuses and examples of methods. [Explanation of Symbols]

[0357] 100 Wireless Access Networks 120a~120j Terminal Devices 110a and 110b network devices 120 terminal devices 110 Network Devices 120a cellular phone 120b car 110a base station 110b base station 120e Smart Home Device 110 Access Network Devices 120 and 130 terminal devices 140 AI Network Elements 1800 Model Monitoring Device 1801 Transceiver Unit 1802 Processing Unit 1900 Model Monitoring Device 1901 Processor 1903 memory 1902 Interface Circuit

Claims

1. A model monitoring method, wherein the method is A step of obtaining first information, wherein the first information includes one or more first indices corresponding to one or more ground truth pieces of information, The steps include obtaining one or more pieces of ground truth information corresponding to one or more first indices, The method wherein one or more pieces of ground truth information are used to obtain monitoring results of the performance of an artificial intelligence (AI) model, and the output of the AI ​​model is predictive information.

2. A model monitoring method, wherein the method is A step of obtaining first information, wherein the first information includes the time-domain resource locations of one or more reference signals used to measure ground truth information; The process includes the step of determining one or more pieces of ground truth information based on the first piece of information, A method wherein the one or more ground truth pieces are used to obtain monitoring results of the performance of an AI model, the output of the AI ​​model is predictive information, and the time-domain resource location of at least one of the one or more ground truth pieces is different from the time-domain resource location of the predictive information.

3. The aforementioned method, The method according to claim 1 or 2, further comprising the step of transmitting one or more pieces of ground truth information.

4. The aforementioned method, A step of obtaining one or more pieces of predictive information, wherein the one or more pieces of predictive information are the outputs of the artificial intelligence (AI) model, The step of transmitting one or more pieces of predictive information. The method according to claim 1 or 2, further comprising:

5. The aforementioned method, A step of obtaining one or more pieces of predictive information, wherein the one or more pieces of predictive information are the outputs of the artificial intelligence (AI) model, The method according to claim 1 or 2, further comprising the step of obtaining one or more monitoring results of the performance of the AI ​​model based on the one or more ground truth information and the one or more prediction information.

6. The step of obtaining one or more pieces of predictive information is: A step of receiving second information, wherein the second information includes one or more second indices corresponding to one or more forecast information, or the second information includes a relative time unit value for each of the one or more forecast information with respect to a time unit corresponding to a reference resource, The method according to claim 4 or 5, further comprising the step of obtaining one or more pieces of predictive information based on the second piece of information.

7. The method according to claim 6, wherein the one or more first indexes and the one or more second indexes are based on the time units corresponding to the reference resource.

8. The time unit corresponding to the aforementioned reference resource is as follows: The first time unit in which the resources for reporting the monitoring results are located, The second time unit in which downlink control signaling for scheduling model monitoring is located, A third time unit determined based on the first time unit or the second time unit, or The method according to claim 7, wherein the fourth time unit is one of a predetermined / pre-configured fourth time unit.

9. The aforementioned method, The process further includes the step of sending a monitoring results report, The aforementioned monitoring results report includes one or more of the aforementioned monitoring results, The monitoring result report includes at least one monitoring result that is equal to or greater than a first threshold among the one or more monitoring results, The monitoring result report includes at least one monitoring result among the one or more monitoring results that is below a second threshold, The monitoring results report includes an index of predictive information and an index of ground truth information corresponding to the optimal monitoring result among the one or more monitoring results. The monitoring results report includes an index of predictive information and an index of ground truth information corresponding to the worst monitoring result among the one or more monitoring results, or The method according to claim 5, wherein the monitoring result report includes the absolute value of the first monitoring result among a plurality of monitoring results and the relative values ​​of the monitoring results other than the first monitoring result among the plurality of monitoring results with respect to the first monitoring result.

10. The method according to claim 9, wherein the first monitoring result is determined based on the order of the ground truth information used and / or the order of the prediction information used when obtaining the plurality of monitoring results.

11. The method according to claim 9 or 10, wherein the monitoring results report includes a portion of all monitoring results.

12. The method according to any one of claims 9 to 11, wherein the information feedback used to transmit the monitoring result report and the measurement result of the reference signal is reported in the same information report, or the information feedback used to transmit the monitoring result report and the measurement result of the reference signal is reported in separate information reports.

13. A model monitoring method, wherein the method is A step of transmitting first information, wherein the first information includes one or more first indices corresponding to one or more ground truth pieces of information, The steps include receiving one or more pieces of ground truth information corresponding to one or more first indices, The method wherein one or more pieces of ground truth information are used to obtain monitoring results of the performance of an artificial intelligence (AI) model, and the output of the AI ​​model is predictive information.

14. A model monitoring method, wherein the method is A step of transmitting first information, wherein the first information includes the time-domain resource location of one or more reference signals used to measure ground truth information; The process includes the step of receiving one or more pieces of ground truth information based on the first piece of information, A method wherein the one or more pieces of ground truth information are used to obtain monitoring results of the performance of the AI ​​model, the output of the AI ​​model is predictive information, and the time-domain resource location of at least one of the one or more pieces of ground truth information is different from the time-domain resource location of the predictive information.

15. The aforementioned method, Steps to obtain one or more predictive pieces of information, The method according to claim 13 or 14, further comprising the step of obtaining one or more monitoring results of the performance of the AI ​​model based on the one or more ground truth pieces of information and the one or more prediction pieces of information.

16. The aforementioned method, The method according to claim 13 or 14, further comprising the step of receiving a monitoring results report, wherein the monitoring results report includes one or more monitoring results, and the one or more monitoring results are obtained based on one or more ground truth information and one or more predictive information.

17. The aforementioned method, A step of transmitting second information, wherein the second information includes one or more second indices corresponding to one or more forecast information, or the second information includes a relative time unit value for each of the one or more forecast information with respect to a time unit corresponding to a reference resource, The method according to claim 15, further comprising the step of receiving one or more pieces of predictive information.

18. The method according to claim 17, wherein the one or more first indexes and the one or more second indexes are based on the time units corresponding to the reference resource.

19. The time unit corresponding to the aforementioned reference resource is as follows: The first time unit in which the resources for reporting the monitoring results are located, The second time unit in which the downlink control signaling for scheduling the aforementioned model monitoring is located, A third time unit determined based on the first time unit or the second time unit, or The method according to claim 18, wherein the fourth time unit is one of a predetermined / pre-configured fourth time unit.

20. The aforementioned monitoring results report includes one or more of the aforementioned monitoring results, The monitoring result report includes at least one monitoring result that is equal to or greater than a first threshold among the one or more monitoring results, The monitoring result report includes at least one monitoring result among the one or more monitoring results that is below a second threshold, The monitoring results report includes an index of predictive information and an index of ground truth information corresponding to the optimal monitoring result among the one or more monitoring results. The monitoring results report includes an index of predictive information and an index of ground truth information corresponding to the worst monitoring result among the one or more monitoring results, or The aforementioned monitoring result report includes the absolute value of the first monitoring result among a plurality of monitoring results and the relative values ​​of the monitoring results other than the first monitoring result among the plurality of monitoring results with respect to the first monitoring result. The method according to claim 16.

21. The method according to claim 20, wherein the first monitoring result is determined based on the order of the ground truth information used and / or the order of the prediction information used when obtaining the plurality of monitoring results.

22. The method according to claim 20 or 21, wherein the monitoring results report includes a portion of all monitoring results.

23. The method according to any one of claims 20 to 22, wherein the information feedback used to transmit the monitoring result report and the measurement result of the reference signal is reported in the same information report, or the information feedback used to transmit the monitoring result report and the measurement result of the reference signal is reported in separate information reports.

24. The method according to any one of claims 1 to 23, wherein the prediction information is prediction channel state information CSI, the ground truth information is ground truth CSI, or the prediction information is prediction beam information and the ground truth information is ground truth beam information.

25. A model monitoring device comprising a module configured to perform the method described in any one of claims 1 to 24.

26. A model monitoring device comprising a processor coupled to memory, wherein the processor is configured to perform a corresponding function in a manner performed by the device according to any one of claims 1 to 24.

27. A computer-readable storage medium wherein the computer-readable storage medium stores a computer program or instruction, and when the computer program or instruction is executed, the method according to any one of claims 1 to 24 is performed.

28. A computer program product comprising instructions, wherein when the instructions are executed on a model monitoring device, the model monitoring device is caused to perform the method according to any one of claims 1 to 24.

Images