Performance monitoring method and device of AI model, equipment and storage medium

Abstract

The invention discloses a performance monitoring method and device of an AI model, equipment and a storage medium, and relates to the technical field of communication. The AI model comprises a first model and a second model, the first model and the second model are deployed on terminal equipment, the second model is deployed on network equipment, and the method comprises the following steps: reporting a performance monitoring value of the second model, or reporting related information used for calculating the performance monitoring value, or reporting performance monitoring indication information. The method can provide a solution for performance monitoring of the AI model. The terminal equipment reports the performance monitoring value of the second model in the AI model or reports the relevant information for calculating the performance monitoring value or reports the performance monitoring indication information, so that the network equipment or the terminal equipment monitors the performance of the AI model, and the model performance of the AI model is ensured.

Description

Performance monitoring method, device, equipment and storage medium of AI model Technical Field

[0001] The embodiments of the present disclosure relate to the field of communication technologies, and in particular to a performance monitoring method, apparatus, device, and storage medium for an AI (Artificial Intelligence) model. Background Art

[0002] Currently, artificial intelligence (AI) can also be applied in wireless communications. By deploying a CSI (Channel Status Information) compression model in terminal devices and a CSI recovery model in network equipment, an AI / ML (Machine Learning) model is formed to compress and recover downlink channel information, reducing compression overhead on terminal devices or improving CSI compression accuracy.

[0003] During the application of AI / ML models, the performance of the trained AI / ML models will also change due to changes in the channel environment. When the model performance is too low, the compression overhead of the terminal device will increase or the compression accuracy of the CSI will be reduced.

[0004] Summary of the Invention

[0005] The embodiments of the present disclosure provide a method, apparatus, device, and storage medium for monitoring the performance of an AI model, which can provide a solution for monitoring the performance of an AI model. The technical solution is as follows:

[0006] According to one aspect of an embodiment of the present disclosure, a method for monitoring the performance of an AI model is provided. The method is executed by a terminal device. The AI ​​model includes a first model and a second model. The first model and the second model are deployed on the terminal device, and the second model is deployed on a network device. The method includes:

[0007] Reporting the performance monitoring value of the second model to the network device, or reporting relevant information used to calculate the performance monitoring value, or reporting performance monitoring indication information.

[0008] According to one aspect of an embodiment of the present disclosure, a method for monitoring the performance of an AI model is provided. The method is executed by a terminal device. The AI ​​model includes a first model and a second model. The first model is deployed on the terminal device, and the second model is deployed on a network device. The method includes:

[0009] Report relevant information used to calculate the performance monitoring value of the second model to the network device.

[0010] According to one aspect of an embodiment of the present disclosure, a performance monitoring method for an AI model is provided. The method is performed by a network device. The AI ​​model includes a first model and a second model. The first model and the second model are deployed on the terminal device, and the second model is deployed on the network device. The method includes:

[0011] receiving a performance monitoring value of the second model reported by the terminal device, or

[0012] receiving relevant information for calculating the performance monitoring value; calculating the performance monitoring value of the second model based on the relevant information; or,

[0013] Receive performance monitoring indication information.

[0014] According to one aspect of an embodiment of the present disclosure, a performance monitoring method for an AI model is provided. The method is executed by a network device. The AI ​​model includes a first model and a second model. The first model is deployed on a terminal device, and the second model is deployed on the network device. The method includes:

[0015] receiving relevant information reported by the terminal device for calculating the performance monitoring value of the second model;

[0016] The performance monitoring value of the second model is calculated based on the relevant information.

[0017] According to one aspect of an embodiment of the present disclosure, a performance monitoring device for an AI model is provided, wherein the AI ​​model includes a first model and a second model, wherein the first model is deployed on a terminal device, and the second model is deployed on a network device. The device includes:

[0018] The reporting module is used to report the performance monitoring value of the second model, or to report relevant information used to calculate the performance monitoring value.

[0019] According to one aspect of an embodiment of the present disclosure, a performance monitoring device for an AI model is provided, wherein the AI ​​model includes a first model and a second model, the first model and the second model are deployed in the device, and the second model is deployed in a network device, and the device includes:

[0020] A reporting module is used to report the performance monitoring value of the second model to the network device, or to report relevant information used to calculate the performance monitoring value, or to report performance monitoring indication information.

[0021] According to one aspect of an embodiment of the present disclosure, a performance monitoring device for an AI model is provided. The AI ​​model includes a first model and a second model. The first model is deployed in the device, and the second model is deployed in a network device. The device includes:

[0022] A reporting module is used to report relevant information used to calculate the performance monitoring value of the second model to the network device.

[0023] According to one aspect of an embodiment of the present disclosure, a performance monitoring device for an AI model is provided. The AI ​​model includes a first model and a second model. The first model and the second model are deployed on a terminal device, and the second model is deployed on the device. The device includes:

[0024] a receiving module, configured to receive the performance monitoring value of the second model reported by the terminal device, or

[0025] The receiving module is further configured to receive relevant information for calculating the performance monitoring value; calculate the performance monitoring value of the second model based on the relevant information; or,

[0026] The receiving module is further configured to receive performance monitoring indication information.

[0027] According to one aspect of an embodiment of the present disclosure, a performance monitoring device for an AI model is provided. The AI ​​model includes a first model and a second model. The first model is deployed on a terminal device, and the second model is deployed on the device. The device includes:

[0028] a receiving module, configured to receive relevant information reported by the terminal device for calculating the performance monitoring value of the second model;

[0029] A calculation module is used to calculate the performance monitoring value of the second model based on the relevant information.

[0030] According to one aspect of an embodiment of the present disclosure, a terminal device is provided, the terminal device including:

[0031] processor;

[0032] a transceiver connected to the processor;

[0033] The processor is configured to load and execute executable instructions to implement the performance monitoring method of the above-mentioned AI model.

[0034] According to one aspect of an embodiment of the present disclosure, a network device is provided, the network device including:

[0035] processor;

[0036] a transceiver connected to the processor;

[0037] The processor is configured to load and execute executable instructions to implement the performance monitoring method of the above-mentioned AI model.

[0038] According to one aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is used for execution by a processor to implement the performance monitoring method of the above-mentioned AI model.

[0039] According to one aspect of an embodiment of the present disclosure, a computer program product or computer program is provided, wherein the computer program product or computer program includes computer instructions, wherein the computer instructions are stored in a computer-readable storage medium, and a processor reads and executes the computer instructions from the computer-readable storage medium to implement the above-mentioned AI model performance monitoring method.

[0040] The technical solutions provided by the embodiments of the present disclosure can bring the following beneficial effects:

[0041] A solution is provided for AI model performance monitoring. When a terminal device deploys a first model and a second model, the terminal device reports the performance monitoring value of the second model in the AI ​​model, reports related information about the calculated performance monitoring value, or reports performance monitoring indication information, enabling the network device or the terminal device to monitor the performance of the AI ​​model to ensure the model performance of the AI ​​model. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0043] FIG1 is a schematic diagram of a method for monitoring the performance of an AI model provided by an exemplary embodiment of the present disclosure;

[0044] FIG2 is a schematic diagram of a communication system provided by an exemplary embodiment of the present disclosure;

[0045] FIG3 is a flowchart of a method for monitoring the performance of an AI model provided by an exemplary embodiment of the present disclosure;

[0046] FIG4 is a flowchart of a method for monitoring the performance of an AI model provided by an exemplary embodiment of the present disclosure;

[0047] FIG5 is a flowchart of a method for monitoring the performance of an AI model provided by an exemplary embodiment of the present disclosure;

[0048] FIG6 is a flowchart of a method for monitoring the performance of an AI model provided by an exemplary embodiment of the present disclosure;

[0049] FIG7 is a flowchart of a method for monitoring the performance of an AI model provided by an exemplary embodiment of the present disclosure;

[0050] FIG8 is a flowchart of interaction between a terminal and a network device provided by an exemplary embodiment of the present disclosure;

[0051] FIG9 is a flowchart of interaction between a terminal and a network device provided by an exemplary embodiment of the present disclosure;

[0052] FIG10 is a structural block diagram of a performance monitoring device for an AI model provided by an exemplary embodiment of the present disclosure;

[0053] FIG11 is a structural block diagram of a performance monitoring device for an AI model provided by an exemplary embodiment of the present disclosure;

[0054] FIG12 is a structural block diagram of a performance monitoring device for an AI model provided by an exemplary embodiment of the present disclosure;

[0055] FIG13 is a structural block diagram of a performance monitoring device for an AI model provided by one embodiment of the present disclosure;

[0056] FIG14 is a schematic structural diagram of a communication device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0057] In order to make the objectives, technical solutions and advantages of the present disclosure more clear, the embodiments of the present disclosure will be further described in detail below with reference to the accompanying drawings.

[0058] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

[0059] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. As used in this disclosure and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0060] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0061] It should be understood that although the terms first, second, etc. may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, a first parameter may also be referred to as a second parameter, and similarly, a second parameter may also be referred to as a first parameter without departing from the scope of this disclosure. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0062] First, some technical knowledge involved in this disclosure is introduced:

[0063] At present, AI technology can also be applied to the field of wireless communications. Compared with traditional technical means, AI technology can also achieve good results. Current research and simulation results show that by adopting AI technology, the feedback overhead of the terminal can be reduced or the CSI feedback accuracy can be improved. In the 3GPP standardization research, a bilateral AI / ML model based on the CSI compression model on the terminal device side and the CSI recovery model on the network device side has been carried out to respectively realize the compression feedback and recovery of CSI. Schematically, please refer to Figure 1, which shows a flowchart of the performance monitoring of the AI ​​model provided in an embodiment of the present application. As shown in Figure 1, the terminal device 110 compresses the input downlink channel information H through the CSI compression model 111 and then quantizes it into a binary bit stream and sends it to the network device 120. The network device 120 recovers H' that is approximate to the original downlink information through the CSI recovery model 121.

[0064] The CSI compression model and CSI recovery model need to be trained using the collected data set. Currently, there are three types of CSI compression models and CSI recovery models:

[0065] Complete the training of a partial model (CSI compression model or CSI recovery model) on a single device (such as a terminal device or network device), and then send the trained partial model to another device;

[0066] Jointly train the CSI compression model and CSI recovery model on the terminal device and network equipment respectively;

[0067] First, complete the training of part of the model on a single device, and then send the training data or other auxiliary information to another device to complete the training of another part of the model. Among them, this training process can be divided into the following two forms:

[0068] 1. NW-first training: The network device first trains the CSI compression model and CSI recovery model, and then sends the dataset used to train the CSI compression model and / or other auxiliary information to the terminal device.

[0069] 2. UE-first training: The terminal device first trains the CSI compression model and CSI recovery model, and then sends the dataset for training the CSI recovery model and / or other auxiliary information to the network device.

[0070] After bilateral model training is completed, it is deployed on terminal devices and / or network equipment. During application, the performance of the trained bilateral model may change due to changes in the channel environment. To monitor the performance of the bilateral model, there are currently two methods:

[0071] Performance monitoring by network devices: The network devices activate / deactivate / update / switch the bilateral model based on the monitored performance.

[0072] Performance monitoring by terminal devices: The terminal devices monitor performance and report the monitoring information to the network device, which then activates / deactivates / updates / switches the bilateral model.

[0073] The performance of the bilateral network model can be monitored according to the following performance monitoring criteria or methods:

[0074] Intermediate key performance indicators (KPIs), such as the square of cosine similarity;

[0075] The terminal device can report the target input CSI (i.e., input-CSI-NW), or the network device can send the target output CSI (i.e., output-CSI-UE) to implement performance monitoring of the network device or terminal device respectively.

[0076] Final KPIs, such as bit error rate (BLER), hypothesized bit error rate, and acknowledgement (ACK) / negative acknowledgement (NACK)

[0077] Based on traditional CSI feedback reporting;

[0078] Traditional CSI feedback reporting includes CSI measurement and reporting based on Type I or Type II codebooks. The codebook type or codebook parameters are configured by the network device to the terminal device.

[0079] The distribution of input or output data of the bilateral model, such as the data drift between the training data and the output data.

[0080] FIG2 shows a block diagram of a communication system provided by an exemplary embodiment of the present disclosure. The communication system may include: an access network 22 , a user terminal 24 , and a core network device 26 .

[0081] The access network 22 includes several network devices 220. The network device 220 can be a base station, which is a device deployed in the access network to provide wireless communication functions for user terminals (referred to as "terminals") 24. Base stations can include various forms of macro base stations, micro base stations, relay stations, access points, etc. In systems using different wireless access technologies, the names of devices with base station functions may be different. For example, in the Long Term Evolution (LTE) system, it is called eNodeB or eNB; in the 5G NR system, it is called gNodeB or gNB. As communication technology evolves, the description of "base station" may change. The access network 22 can be constructed using at least one of the above-mentioned dual connection technologies of NGEN-DC and EN-DC.

[0082] The core network includes several core network devices 26. The core network is responsible for information exchange, routing, user data management, security, and other functions within the system, as well as information exchange and transmission with other communication systems. The main network functions of 5GC include: Access and Mobility Management Function (AMF), Session Management Function (SMF), User Plane Function (UPF), Network Repository Function (NRF), Network Exposure Function (NEF), Unified Data Management (UDM), Location Management Function (LMF), and other network functions.

[0083] For the convenience of description in the embodiments of the present disclosure, the above-mentioned apparatuses providing wireless communication functions for the user terminal 24 are collectively referred to as network devices. Exemplarily, the network devices include at least one of the network device 220 and the core network device 26.

[0084] User terminals 24 may include various handheld devices with wireless communication capabilities, vehicle-mounted devices, wearable devices, computing devices, or other processing devices connected to a wireless modem, as well as various forms of user equipment, mobile stations (MSs), terminal devices, and the like. For ease of description, the aforementioned devices are collectively referred to as user terminals. Network device 220 and user terminals 24 communicate with each other via an air interface technology, such as a Uu interface.

[0085] The technical solutions of the embodiments of the present disclosure can be applied to various communication systems, such as: Global System of Mobile Communication (GSM) system, Code Division Multiple Access (CDMA) system, Wideband Code Division Multiple Access (WCDMA) system, General Packet Radio Service (GPRS), Long Term Evolution (LTE) system, LTE Frequency Division Duplex (FDD) system, LTE Time Division Duplex (TDD) system, Advanced Long Term Evolution (LTE-A) system, New Radio (NR) system, NR system evolution system, LTE-based access to Unlicensed spectrum (LTE-U) system, NR-U system, Universal Mobile Telecommunication System (UMTS), Worldwide Interoperability for Microwave Access (WiMAX) communication system, Wireless Local Area Network (WLAN) system. Networks, WLAN), Wireless Fidelity (WiFi), next generation communication systems or other communication systems, etc.

[0086] Generally speaking, traditional communication systems support a limited number of connections and are easy to implement. However, with the development of communication technology, mobile communication systems will not only support traditional communication, but also support, for example, device-to-device (D2D) communication, machine-to-machine (M2M) communication, machine-type communication (MTC), vehicle-to-vehicle (V2V) communication, and vehicle-to-everything (V2X) systems. The embodiments of the present disclosure can also be applied to these communication systems.

[0087] Please refer to Figure 3, which shows a flow chart of a performance monitoring method for an AI model provided by an embodiment of the present disclosure. This embodiment is illustrated by applying the method to a terminal device in the communication system shown in Figure 2. The method may include the following steps:

[0088] Step 301: Report the performance monitoring value of the second model to the network device, or report relevant information used to calculate the performance monitoring value, or report performance monitoring indication information.

[0089] Schematically, the AI ​​model includes a first model and a second model, the first model and the second model are deployed on the terminal device, and the second model is deployed on the network device.

[0090] In some embodiments, the first model and the second model constitute a bilateral model, wherein the first model is used to process the input sample and output an intermediate value, which is input into the second model, and the intermediate value is processed by the second model to output an inference value, wherein the inference value is as identical or similar to the input sample as possible, and the input sample is data or information related to the mobile communication system, such as channel information.

[0091] Schematically, the performance monitoring value is used to indicate the quality of the model performance of the AI ​​model.

[0092] Optionally, the performance monitoring value is used to perform performance monitoring on the first model; or, used to perform performance monitoring on the second model; or, used to perform performance monitoring on both the first model and the second model.

[0093] In some embodiments, the performance monitoring value is determined based on the first data and the second data, or the performance monitoring value is determined based on the third data and the fourth data.

[0094] Among them, the first data is determined based on the inference value of the first input value of the second model, and the second data is determined based on the output value corresponding to the second input value when the second model is not used. The first input value and the second input value correspond to the same input sample, the input sample is the input of the first model, and the first input value is the output value of the first model for the input sample.

[0095] Illustratively, the first input value is realized as the output result of the first model, that is, the intermediate value. The first input value is input into the second model, and the output result is the inference value corresponding to the first input value.

[0096] Illustratively, the second input value refers to a result obtained by measuring the input sample using a specified configuration without using the second model.

[0097] The third data is determined based on an average value, a maximum value, or a minimum value of the first data corresponding to at least two frequency domain units, and the fourth data is determined based on an average value, a maximum value, or a minimum value of the second data corresponding to at least two frequency domain units;

[0098] Illustratively, the third data is obtained by an average value / maximum value / minimum value of a plurality of first data obtained within a historical time range corresponding to at least two frequency domain units.

[0099] Illustratively, the fourth data is obtained by an average value / maximum value / minimum value of a plurality of second data obtained within a historical time range corresponding to at least two frequency domain units.

[0100] In some embodiments, the at least two frequency domain unit data N are configured by a network device; or, are predefined by a communication protocol.

[0101] Optionally, the performance monitoring value includes at least one of the following monitoring value types:

[0102] 1) The difference between the first data and the second data;

[0103] 2) the ratio of the first data to the second data;

[0104] 3) the difference between the third data and the fourth data;

[0105] 4) a ratio of the third data to the fourth data;

[0106] 5) a weighted average of the first data and the second data;

[0107] 6) the average of the first data and the second data;

[0108] 7) a weighted average of the third data and the fourth data;

[0109] 8) The average value of the third data and the fourth data.

[0110] Optionally, the relevant information of the performance monitoring value includes at least one of the following information:

[0111] 1) first data;

[0112] 2) second data;

[0113] 3) third data;

[0114] 4) fourth data;

[0115] 5) used to calculate the correlation value of the first data;

[0116] The method for calculating the correlation value of the first data includes at least one of an input sample and a first input value.

[0117] 6) Used to calculate the correlation value of the second data.

[0118] The method for calculating the correlation value of the second data includes at least one of an input sample and a second input value.

[0119] Illustratively, the performance monitoring indication information indicates the quality of the performance parameters of the current first model and / or second model.

[0120] In one example, bit information is used as performance monitoring indication information. When the bit information takes a value of 1, it indicates that the performance parameters of the first model and / or the second model are better. When the bit information takes a value of 0, it indicates that the performance parameters of the first model and / or the second model are worse. Or, when the bit information takes a value of 0, it indicates that the performance parameters of the first model are better. When the bit information takes a value of 1, it indicates that the performance parameters of the second model are worse.

[0121] In a feasible situation, the terminal device reports the performance monitoring value of the second model to the network device. The network device directly determines the model performance of the second model in the current frequency domain unit based on the performance monitoring value and performs management operations on the AI ​​model.

[0122] In another feasible situation, the terminal device reports relevant information used to calculate the performance monitoring value of the second model to the network device. After the network device calculates the performance monitoring value of the second model based on the relevant information, it determines the model performance of the first model and / or the second model in the current frequency domain unit based on the performance monitoring indication information, and performs management operations on the AI ​​model.

[0123] In another feasible situation, the terminal device reports performance monitoring indication information to the network device, and the network device determines the model performance of the first model and / or the second model in the current frequency domain unit based on the performance monitoring indication information, and performs management operations on the AI ​​model.

[0124] The management operations on AI models include at least one of the following types of operations:

[0125] Switching operations of AI models;

[0126] Activation of AI models;

[0127] Deactivation of AI models;

[0128] Update the model parameters of the AI ​​model;

[0129] Falling back to traditional communication operations that do not use AI models.

[0130] In an illustrative manner, the network device presets a first threshold value. If the difference / ratio between the first data and the second data, or the difference / ratio between the third data and the fourth data, reaches the first threshold value, the switching operation of the AI ​​model is implemented. If the first threshold value is not reached, the AI ​​model is not switched.

[0131] In an illustrative manner, the network device presets a second threshold value. If the difference / ratio between the first data and the second data, or the difference / ratio between the third data and the fourth data, is less than the second threshold value, at least one of the following operations is implemented: activation of the AI ​​model, deactivation of the AI ​​model, and fallback to a traditional communication operation that does not use the AI ​​model.

[0132] In summary, the technical solution provided in this embodiment provides a solution for AI model performance monitoring. When a terminal device deploys a first model and a second model, the terminal device reports the performance monitoring value of the second model in the AI ​​model, reports related information on the calculated performance monitoring value, or reports performance monitoring indication information, enabling the network device or terminal device to monitor the performance of the AI ​​model to ensure the model performance of the AI ​​model.

[0133] Please refer to Figure 4, which shows a flow chart of a performance monitoring method for an AI model provided by an embodiment of the present disclosure. This embodiment is illustrated by applying the method to a terminal device in the communication system shown in Figure 2. The method may include the following steps:

[0134] Step 401: Report relevant information used to calculate the performance monitoring value of the second model to the network device.

[0135] Schematically, the AI ​​model includes a first model and a second model, the first model is deployed on the terminal device, and the second model is deployed on the network device.

[0136] In some embodiments, the first model and the second model constitute a bilateral model, wherein the first model is used to process the input sample and output an intermediate value, which is input into the second model, and the intermediate value is processed by the second model to output an inference value, wherein the inference value is as identical or similar to the input sample as possible, and the input sample is data or information related to the mobile communication system, such as channel information.

[0137] Illustratively, the relevant information of the performance monitoring value is used to calculate the performance monitoring value of the second model.

[0138] Optionally, the relevant information of the performance monitoring value includes at least one of the following information:

[0139] 1) first data;

[0140] 2) second data;

[0141] 3) third data;

[0142] 4) fourth data;

[0143] 5) used to calculate the correlation value of the first data;

[0144] The method for calculating the correlation value of the first data includes at least one of an input sample and a first input value.

[0145] 6) Used to calculate the correlation value of the second data.

[0146] The method for calculating the correlation value of the second data includes at least one of an input sample and a second input value.

[0147] Among them, the first data is determined based on the inference value of the first input value of the second model, and the second data is determined based on the output value corresponding to the second input value when the second model is not used. The first input value and the second input value correspond to the same input sample, the input sample is the input of the first model, and the first input value is the output value of the first model for the input sample.

[0148] Illustratively, the first input value is realized as the output result of the first model, that is, the intermediate value. The first input value is input into the second model, and the output result is the inference value corresponding to the first input value.

[0149] Illustratively, the second input value refers to a result obtained by measuring the input sample using a specified configuration without using the second model.

[0150] The third data is determined based on an average value, a maximum value, or a minimum value of the first data corresponding to at least two frequency domain units, and the fourth data is determined based on an average value, a maximum value, or a minimum value of the second data corresponding to at least two frequency domain units;

[0151] Illustratively, the third data is obtained by an average value / maximum value / minimum value of a plurality of first data obtained within a historical time range corresponding to at least two frequency domain units.

[0152] Illustratively, the fourth data is obtained by an average value / maximum value / minimum value of a plurality of second data obtained within a historical time range corresponding to at least two frequency domain units.

[0153] In some embodiments, the at least two frequency domain unit data N are configured by a network device; or, are predefined by a communication protocol.

[0154] In another feasible situation, the terminal device reports relevant information used to calculate the performance monitoring value of the second model to the network device. After the network device calculates the performance monitoring value of the second model based on the relevant information, it determines the model performance of the first model and / or the second model in the current frequency domain unit based on the performance monitoring indication information, and performs management operations on the AI ​​model.

[0155] The management operations on AI models include at least one of the following types of operations:

[0156] Switching operations of AI models;

[0157] Activation of AI models;

[0158] Deactivation of AI models;

[0159] Update the model parameters of the AI ​​model;

[0160] Falling back to traditional communication operations that do not use AI models.

[0161] In an illustrative manner, the network device presets a first threshold value. If the difference / ratio between the first data and the second data, or the difference / ratio between the third data and the fourth data, reaches the first threshold value, the switching operation of the AI ​​model is implemented. If the first threshold value is not reached, the AI ​​model is not switched.

[0162] In an illustrative manner, the network device presets a second threshold value. If the difference / ratio between the first data and the second data, or the difference / ratio between the third data and the fourth data, is less than the second threshold value, at least one of the following operations is implemented: activation of the AI ​​model, deactivation of the AI ​​model, and fallback to a traditional communication operation that does not use the AI ​​model.

[0163] In one implementation, the terminal device reports the relevant value of the performance monitoring value of the second model to the network device. The network device first calculates the performance monitoring value through the relevant value of the performance monitoring value, and then performs the management operations mentioned above on the AI ​​model based on the performance monitoring value.

[0164] In summary, the technical solution provided in this embodiment provides a solution for AI model performance monitoring. When a terminal device deploys a first model and a second model, the terminal device reports the performance monitoring value of the second model in the AI ​​model, reports related information on the calculated performance monitoring value, or reports performance monitoring indication information, enabling the network device or terminal device to monitor the performance of the AI ​​model to ensure the model performance of the AI ​​model.

[0165] In one embodiment, the bilateral model is applied to a channel compression scenario to illustrate the performance monitoring method of the above-mentioned AI model. Schematically, the application of the above-mentioned steps 301 and 401 to the channel compression scenario is explained in detail.

[0166] First, the AI ​​model includes a first model and a second model.

[0167] In some embodiments, the first model and the second model constitute a bilateral model.

[0168] In scenarios where the channel is compressed before transmission, the first model is implemented as a CSI compression model, which performs channel compression on input samples (e.g., downlink channel information H) and outputs an intermediate value. In one example, the downlink channel information H is input into the first model, which compresses the downlink channel information H and then quantizes it to obtain a binary bit stream as the intermediate value. This serves as the output value corresponding to the input sample and also as the input value for the second model. That is, the binary bit stream serves as the first input value.

[0169] When the first model is implemented as a CSI compression model, the second model is implemented as a CSI recovery model, which is used to perform channel recovery processing on the binary bit stream output by the first model, and output an output value corresponding to the binary bit stream, that is, the recovered downlink channel information H', wherein the downlink channel information H and the recovered downlink channel information H' are as identical or similar as possible.

[0170] Illustratively, the first model is deployed at least on a terminal device, and the second model is deployed at least on a network device. That is, the deployment methods of the first model and the second model include at least one of the following forms:

[0171] The first model and the second model are deployed on the terminal device, and the second model is deployed on the network device;

[0172] The first model is deployed on the terminal device, and the second model is deployed on the network device;

[0173] The first model is deployed on the terminal device, and the first and second models are deployed on the network device;

[0174] The first model and the second model are deployed on the terminal device, and the first model and the second model are deployed on the network device.

[0175] Among them, when the second model is deployed in both the terminal device and the network device, the terminal device reports the performance monitoring value of the second model to the network device; or reports relevant information of the performance monitoring value; or reports performance monitoring indication information.

[0176] In which case, when the second model is deployed only on the network device, the terminal device reports relevant information used to calculate the performance monitoring value of the second model to the network device.

[0177] Optionally, the performance monitoring method for the AI ​​model includes at least one of the following methods:

[0178] Performance monitoring based on intermediate key performance indicators (KPIs);

[0179] For example: Square Generalized Cosine Similarity (SGCS).

[0180] ·Performance monitoring based on final KPIs;

[0181] For example, at least one of the throughput, bit error rate (Block Error Rate, BLER), assumed bit error rate, and acknowledgement bit (ACK) / not acknowledgement bit (NACK) corresponding to the second model.

[0182] Performance monitoring based on CSI feedback reporting;

[0183] For example, CSI measurement and reporting is performed based on Type I or Type II codebook parameters, where the codebook parameters or codebook type are configured by the network device to the terminal device.

[0184] Performance monitoring based on the input data distribution or output data distribution of the bilateral model.

[0185] For example: data drift changes between input samples and output values ​​(including output values ​​corresponding to the first input value or the output value corresponding to the second input value), where data drift changes refer to the degree of deviation of the data distribution between the input samples and the input values.

[0186] Optionally, the performance monitoring value is used to perform performance monitoring on the first model; or, used to perform performance monitoring on the second model; or, used to perform performance monitoring on both the first model and the second model.

[0187] Schematically, the performance monitoring value is used to indicate the quality of the model performance of the AI ​​model.

[0188] Optionally, the performance monitoring value includes at least one of the following monitoring value types:

[0189] 1) The difference between the first data and the second data;

[0190] The first data is determined based on the inference value of the first input value of the second model, and the second data is determined based on the output value corresponding to the second input value when the second model is not used. The first input value and the second input value correspond to the same input sample, and the input sample is the input of the first model.

[0191] Illustratively, the first input value is realized as the output result of the first model, that is, the intermediate value. The first input value is input into the second model, and the output result is the inference value corresponding to the first input value.

[0192] Illustratively, the second input value refers to a result obtained by measuring the input sample using a specified configuration without using the second model.

[0193] 2) the ratio of the first data to the second data;

[0194] 3) the difference between the third data and the fourth data;

[0195] The third data is determined based on an average value, a maximum value, or a minimum value of the first data corresponding to at least two frequency domain units, and the fourth data is determined based on an average value, a maximum value, or a minimum value of the second data corresponding to at least two frequency domain units;

[0196] Illustratively, the third data is obtained by an average value / maximum value / minimum value of a plurality of first data obtained within a historical time range corresponding to at least two frequency domain units.

[0197] Illustratively, the fourth data is obtained by an average value / maximum value / minimum value of a plurality of second data obtained within a historical time range corresponding to at least two frequency domain units.

[0198] 4) The ratio of the third data to the fourth data.

[0199] Optionally, the relevant information of the performance monitoring value includes at least one of the following information:

[0200] 1) first data;

[0201] 2) second data;

[0202] 3) third data;

[0203] 4) fourth data;

[0204] 5) used to calculate the correlation value of the first data;

[0205] The method for calculating the correlation value of the first data includes at least one of an input sample and a first input value.

[0206] 6) Used to calculate the correlation value of the second data.

[0207] The method for calculating the correlation value of the second data includes at least one of an input sample and a second input value.

[0208] Illustratively, the performance monitoring indication information indicates the quality of the performance parameters of the current first model and / or second model.

[0209] In one example, bit information is used as performance monitoring indication information. When the bit information takes a value of 1, it indicates that the performance parameters of the first model and / or the second model are good. When the bit information takes a value of 0, it indicates that the performance parameters of the first model and / or the second model are poor. Alternatively, when the bit information takes a value of 0, it indicates that the performance parameters of the first model are good. When the bit information takes a value of 1, it indicates that the performance parameters of the second model are poor. In a feasible scenario, the terminal device reports the performance monitoring value of the second model to the network device. The network device directly determines the model performance of the second model in the current frequency domain unit based on the performance monitoring value and performs management operations on the AI ​​model.

[0210] It is worth noting that the above-mentioned first data, second data, third data and fourth data are all intermediate KPIs.

[0211] The management operations on AI models include at least one of the following types of operations:

[0212] Switching operations of AI models;

[0213] Activation of AI models;

[0214] Deactivation of AI models;

[0215] Update the model parameters of the AI ​​model;

[0216] Falling back to traditional communication operations that do not use AI models.

[0217] In summary, the technical solution provided in this embodiment provides a solution for AI model performance monitoring. When a terminal device deploys a first model and a second model, the terminal device reports the performance monitoring value of the second model in the AI ​​model, reports related information on the calculated performance monitoring value, or reports performance monitoring indication information, enabling the network device or terminal device to monitor the performance of the AI ​​model to ensure the model performance of the AI ​​model.

[0218] In an optional embodiment, the frequency domain unit is implemented as a subband, the number of subbands configured by the network device is N3, the number of data transmission layers is 1, and when both the CSI compression model and the CSI recovery model are deployed in the terminal device, the input sample of the CSI compression model is input-CSI-NW, and the model output value of the CSI recovery model is the inference value output-CSI-UE. That is, at least two frequency domain unit data N are configured by the network or predefined by the communication protocol.

[0219] Among them, input-CSI-NW and output-CSI-UE refer to at least one eigenrepresentation v contained in the V matrix obtained after performing singular value decomposition (SVD) on the channel matrix corresponding to the downlink channel information. SVD is used to perform eigendecomposition on the channel matrix. For input-CSI-NW, the channel matrix is ​​the first channel information (such as uplink channel information or downlink channel information). For output-CSI-UE, the channel matrix is ​​the model output value output-CSI-UE (such as the recovered uplink channel information or the recovered downlink channel information).

[0220] Therefore, input-CSI-NW is expressed as output-CSI-UE is expressed as Where dim(v) represents the vector dimension of feature representation v. Let e i It represents the feature representation corresponding to the i-th subband in the CSI recovery model.

[0221] In addition, the network device also configures codebook parameters (such as codebook parameters corresponding to eType II) for the terminal device.

[0222] The network device estimates the downlink channel information of different sub-bands based on the received Channel Status Information-Reference Signal (CSI-RS) and calculates the feature representation corresponding to each sub-band. Let v i represents the feature representation corresponding to the i-th subband, that is, the estimated channel information.

[0223] Below, based on the above conditions, different deployment scenarios of the first model and the second model in the terminal device and the network device are described in detail.

[0224] In the first type, the first model and the second model are deployed on the terminal device, and the second model is deployed on the network device.

[0225] In this embodiment, step 301 is described in detail.

[0226] In some embodiments, the inferred value is inferred by a second model in the terminal device.

[0227] Schematically, the first model is a CSI compression model, and the second model is a CSI recovery model.

[0228] In this embodiment, the inference value is implemented as a model output value output-CSI-UE of the CSI model.

[0229] In some embodiments, the first input value is the output value of the first model for the input sample, and the number of input samples is the input of the first model. When the first model is implemented as a CSI compression model, the first input value is implemented as a binary bit stream, and the number of input samples is input-CSI-NW.

[0230] In this embodiment, the performance monitoring method of the AI ​​model includes the following steps.

[0231] In the first step, the terminal device calculates the first data.

[0232] For the i-th subband channel, the terminal device inputs the input sample input-CSI-NW into the CSI compression model and outputs a binary bit stream as the first input value corresponding to the CSI recovery model. The CSI recovery model infers the first input value to obtain the inference value output-CSI-UE.

[0233] The terminal device represents e according to the feature corresponding to the i-th subband i and the estimated channel information v corresponding to the i-th subband i Calculate the square generalized cosine similarity (SGCS) K 1,i , please refer to Formula 1 for details.

[0234] Formula 1:

[0235] Among them, E represents the calculation of the expected value or average value in the brackets, and H represents the conjugate transpose.

[0236] When obtaining the square of cosine similarity K i,1 Then, the first data is obtained according to the following formula 2.

[0237] Formula 2:

[0238] Among them, K1 is the first data.

[0239] It is worth noting that in the above method, the first data can be calculated based on the inferred value output-CSI-UE and the estimated channel information, or can be directly calculated based on the inferred value output-CSI-UE.

[0240] Optionally, the first data K1 is implemented as an intermediate KPI corresponding to at least two frequency domain units, or as an average value of the intermediate KPIs corresponding to at least two frequency domain units.

[0241] Where K1 is implemented as the average of the intermediate KPIs corresponding to at least two frequency domain units, the terminal device calculates the average value of K1 in the at least two frequency domain units to obtain K1' as the third data. Wherein, the at least two frequency domain unit data N are configured by the network or predefined by the communication protocol.

[0242] In the second step, the terminal device calculates the second data.

[0243] For the i-th subband channel, the terminal device calculates the second input value using the following formula 3 based on the estimated channel information and the eType II codebook parameters configured by the network device.

[0244] Formula 3:

[0245] Wherein, W represents the second input value. In this embodiment, the second data is implemented as precoding matrix indication (PMI) information W i .

[0246] Then, the terminal device calculates W according to the above formula 1 i and V i SCGS K 2,i , calculated according to the above formula 2 to get K 2,i The corresponding second data K2.

[0247] It is worth noting that the second data can be calculated based on the estimated channel information and the codebook parameters, or can be directly obtained based on the codebook parameters, or can be directly obtained based on the estimated channel information.

[0248] Optionally, the second data K2 is implemented as an intermediate KPI corresponding to at least two frequency domain units, or as an average value of the intermediate KPIs corresponding to at least two frequency domain units.

[0249] Where K2 is implemented as the average of the intermediate KPIs corresponding to at least two frequency domain units, the terminal device calculates the average value of K2 in the at least two frequency domain units to obtain K2' as the fourth data. Wherein, the at least two frequency domain unit data N are configured by the network or predefined by the communication protocol.

[0250] Optionally, the frequency domain unit selected for the third data and the frequency domain unit selected for the fourth data are the same or different.

[0251] In the third step, the terminal device reports the performance monitoring value / related information of the performance monitoring value / performance monitoring indication information to the network device.

[0252] In one feasible case, the terminal device first calculates the difference between K1 and K2 and reports it to the network device as a performance monitoring value; or, the terminal device first calculates the ratio of K1 and K2 and reports it to the network device as a performance monitoring value.

[0253] In another feasible case, the terminal device reports K1 and K2 directly to the network device.

[0254] In another feasible case, the terminal device reports the relevant value of the first data K1 to the network device, including at least one of estimated channel information, inference value output-CSI-UE, and binary bit stream.

[0255] In another feasible case, the terminal device reports a relevant value of the second data K2 to the network device, including at least one of input-CSI-NW, estimated channel information, and PMI information.

[0256] In another feasible scenario, the terminal device reports the performance monitoring indication information to the network device.

[0257] In the fourth step, the network device performs performance monitoring on the AI ​​model based on the performance monitoring value / related information of the performance monitoring value.

[0258] Optionally, the network device directly performs performance monitoring on the AI ​​model after receiving the performance monitoring value sent by the terminal device.

[0259] In some embodiments, management operations of the AI ​​model are performed based on the performance monitoring value of the second model; the management operations of the AI ​​model include: switching operations of the AI ​​model, activation operations of the AI ​​model, deactivation operations of the AI ​​model, and falling back to traditional communication operations that do not use the AI ​​model.

[0260] In this embodiment, the network device presets a first threshold value. If the difference / ratio between the first data and the second data, or the difference / ratio between the third data and the fourth data, reaches the first threshold value, the AI ​​model switching operation is implemented. If the first threshold value is not reached, the AI ​​model is not switched.

[0261] In this embodiment, the network device presets a second threshold value. If the difference / ratio between the first data and the second data, or the difference / ratio between the third data and the fourth data, is less than the second threshold value, at least one of the following operations is implemented: activation operation of the AI ​​model / deactivation operation of the AI ​​model / falling back to a traditional communication operation that does not use the AI ​​model.

[0262] The second type is that the first model is deployed on the terminal device and the second model is deployed on the network device.

[0263] In the case where the second model is not deployed on the terminal device, the following two cases are also included.

[0264] 2-1. The network device sends the inference value to the terminal device.

[0265] Schematically, please refer to Figure 5, which shows a flow chart of a performance monitoring method for an AI model provided by an exemplary embodiment of the present application. As shown in Figure 5, the method includes the following steps.

[0266] Step 501: Receive an inference value sent by a network device.

[0267] The inferred value is inferred by a second model in the network device.

[0268] Optionally, when the first model is not deployed in the network device, the terminal device inputs the input sample into the deployed first model, outputs a binary bit stream as the first input value, and sends the first input value to the network device. The network device infers the first input value through the second model to obtain an inference value output-CSI-UE corresponding to the first input value and sends it to the terminal device.

[0269] Optionally, when the first model is deployed in the network device, the network device inputs the input sample into the deployed first model, outputs a binary bit stream as the first input value, inputs the first input value into the second model, infers the first input value to obtain the inference value output-CSI-UE corresponding to the first input value, and sends it to the terminal device.

[0270] Step 502: Report relevant information used to calculate the performance monitoring value of the second model to the network device.

[0271] Schematically, after the terminal device receives the inference value, the performance monitoring method of the AI ​​model is implemented according to the following steps.

[0272] In the first step, the terminal device calculates the first data.

[0273] The terminal device calculates the square generalized cosine similarity (SGCS) K according to the feature representation corresponding to the i-th subband and the estimated channel information corresponding to the i-th subband using the above formula 1 3,i , in obtaining K 3,i Then, the first data K3 is obtained according to the above formula 2.

[0274] It is worth noting that in the above method, the first data can be calculated based on the inferred value output-CSI-UE and the estimated channel information, or can be directly calculated based on the inferred value output-CSI-UE.

[0275] Optionally, the first data K3 is implemented as an intermediate KPI corresponding to at least two frequency domain units, or as an average value of the intermediate KPIs corresponding to at least two frequency domain units.

[0276] Where K3 is implemented as the average of the intermediate KPIs corresponding to at least two frequency domain units, the terminal device calculates the average value of K3 in the at least two frequency domain units to obtain K3' as the third data. Wherein, the at least two frequency domain unit data N are configured by the network or predefined by the communication protocol.

[0277] In the second step, the terminal device calculates the second data.

[0278] For the i-th subband channel, the terminal device calculates the second input value using the above formula three according to the estimated channel information and the eType II codebook parameters configured by the network device.

[0279] Then, the terminal device calculates the SCGS K corresponding to the codebook parameters and the estimated channel information according to the above formula 1 2,i , calculated according to the above formula 2 to get K 2,i The corresponding second data K2.

[0280] It is worth noting that the second data can be calculated based on the estimated channel information and the codebook parameters, or can be directly obtained based on the codebook parameters, or can be directly obtained based on the estimated channel information.

[0281] Optionally, the second data K2 is implemented as an intermediate KPI corresponding to at least two frequency domain units, or as an average value of the intermediate KPIs corresponding to at least two frequency domain units.

[0282] Where K2 is implemented as the average of the intermediate KPIs corresponding to at least two frequency domain units, the terminal device calculates the average value of K2 in the at least two frequency domain units to obtain K2' as the fourth data. Wherein, the at least two frequency domain unit data N are configured by the network or predefined by the communication protocol.

[0283] Optionally, the frequency domain unit selected for the third data and the frequency domain unit selected for the fourth data are the same or different.

[0284] In the third step, the terminal device reports relevant information of the performance monitoring value to the network device.

[0285] In a feasible case, the terminal device reports K3 and K2 directly to the network device.

[0286] In another feasible case, the terminal device reports the relevant value of the first data K3 to the network device, including at least one of estimated channel information, input-out-NW, inference value output-CSI-UE, and binary bit stream.

[0287] In another feasible case, the terminal device reports a relevant value of the second data K2 to the network device, including at least one of input-out-NW, estimated channel information, and PMI information.

[0288] In another feasible case, the terminal device reports the third data K1 ′ to the network device.

[0289] In another feasible case, the terminal device reports the fourth data K2 ′ to the network device.

[0290] In the fourth step, the network device monitors the performance of the AI ​​model based on the relevant information of the performance monitoring value.

[0291] In some embodiments, after receiving relevant information about the performance monitoring value sent by the terminal device, the network device calculates the performance monitoring value of the second model based on the relevant information, and performs performance monitoring on the AI ​​model based on the performance monitoring value.

[0292] In some embodiments, management operations of the AI ​​model are performed based on the performance monitoring value of the second model; the management operations of the AI ​​model include: switching operations of the AI ​​model, activation operations of the AI ​​model, deactivation operations of the AI ​​model, and falling back to traditional communication operations that do not use the AI ​​model.

[0293] In this embodiment, the network device presets a first threshold value. If the difference / ratio between the first data and the second data, or the difference / ratio between the third data and the fourth data, reaches the first threshold value, the AI ​​model switching operation is implemented. If the first threshold value is not reached, the AI ​​model is not switched.

[0294] In this embodiment, the network device presets a second threshold value. If the difference / ratio between the first data and the second data, or the difference / ratio between the third data and the fourth data, is less than the second threshold value, at least one of the following operations is implemented: activation operation of the AI ​​model / deactivation operation of the AI ​​model / falling back to a traditional communication operation that does not use the AI ​​model.

[0295] 2-2. The terminal device sends input samples to the network device.

[0296] Optionally, when the first model is not deployed in the network device, the terminal device sends the input sample input-CSI-NW and the first input value (binary bit stream) to the network device.

[0297] Optionally, when the first model is deployed in the network device, the terminal device sends the input sample input-CSI-NW to the network device.

[0298] Schematically, the performance monitoring method of the AI ​​model is implemented according to the following steps.

[0299] In the first step, input samples are sent to the network device.

[0300] Optionally, the terminal device sends the input sample input-CSI-NW and the second data K2 or fourth data K2′ calculated in the above steps to the network device simultaneously.

[0301] Optionally, the terminal device simultaneously sends the input sample input-CSI-NW and the second input value (PMI information) to the network device. In this case, the network device calculates the second data K2 using the above formula 2 based on the PMI information and the input sample input-CSI-NW, or calculates the fourth data K2' based on the second data in at least two frequency domain units.

[0302] In the second step, the network device calculates the first data.

[0303] The network device calculates first data K4 using Formula 1 based on the received input sample input-CSI-NW and the inferred value output-CSI-UE corresponding to the second model deployed in the network device. Alternatively, after calculating the first data K4, the network device calculates third data K4' based on the average of the first data in at least two frequency domains. The at least two frequency domain unit data N are configured by the network or predefined by the communication protocol.

[0304] In the third step, the network device monitors the performance of the AI ​​model based on the relevant information of the performance monitoring value.

[0305] In some embodiments, after receiving relevant information about the performance monitoring value sent by the terminal device, the network device calculates the performance monitoring value of the second model based on the relevant information and performs performance monitoring on the AI ​​model based on the performance monitoring value. The performance monitoring value includes the ratio or difference between the first data and the second data.

[0306] In some embodiments, management operations of the AI ​​model are performed based on the performance monitoring value of the second model; the management operations of the AI ​​model include: switching operations of the AI ​​model, activation operations of the AI ​​model, deactivation operations of the AI ​​model, and falling back to traditional communication operations that do not use the AI ​​model.

[0307] In this embodiment, the network device presets a first threshold value. If the difference / ratio between the first data and the second data, or the difference / ratio between the third data and the fourth data, reaches the first threshold value, the AI ​​model switching operation is implemented. If the first threshold value is not reached, the AI ​​model is not switched.

[0308] In this embodiment, the network device presets a second threshold value. If the difference / ratio between the first data and the second data, or the difference / ratio between the third data and the fourth data, is less than the second threshold value, at least one of the following operations is implemented: activation operation of the AI ​​model / deactivation operation of the AI ​​model / falling back to a traditional communication operation that does not use the AI ​​model.

[0309] It is worth noting that in the embodiments of the present application, K1, K2, K3, K4, K1', K2', K3' and K4' are used to represent the first data / second data / third data / fourth data calculated under different circumstances.

[0310] In some embodiments, when reporting indication information sent by a network device is received, relevant information for calculating a performance monitoring value is reported; or, when a target event is triggered, relevant information for calculating a performance monitoring value is reported.

[0311] The target event is determined based on the data distribution corresponding to the input data of the first model; or the target event is determined based on the data distribution corresponding to the output data of the first model.

[0312] In this embodiment, when the second model is only deployed on the network device, the relevant information used to calculate the performance monitoring value of the second model needs to be reported to the network device when the reporting indication information sent by the network device is received or the target event is triggered in the terminal device.

[0313] Illustratively, the reporting indication information sent by the network device is used to instruct the terminal device to report relevant information used to calculate the performance monitoring value of the second model within a specified time range.

[0314] Schematically, the data distribution of the input data of the first model is used to indicate the correlation between multiple input data. The correlation between the multiple input data is determined by calculating the cosine similarity between the multiple input data. When the correlation is higher, it indicates that the difference between the input data is smaller.

[0315] Optionally, a third threshold is set, and when the data distribution of the input data of the first model reaches the third threshold, the performance monitoring value of the second model is reported or relevant information used to calculate the performance monitoring value is reported.

[0316] Schematically, the data distribution of the output data of the first model is used to indicate the correlation between multiple output data. The correlation between the multiple output data is determined by calculating the cosine similarity between the multiple output data. When the correlation is higher, it indicates that the difference in the output data is smaller.

[0317] Optionally, a fourth threshold is set, and when the data distribution of the output data of the first model reaches the fourth threshold, the performance monitoring value of the second model or the relevant information used to calculate the performance monitoring value is reported.

[0318] In summary, the technical solution provided in this embodiment provides a solution for AI model performance monitoring. When a terminal device deploys a first model and a second model, the terminal device reports the performance monitoring value of the second model in the AI ​​model, reports related information on the calculated performance monitoring value, or reports performance monitoring indication information, enabling the network device or terminal device to monitor the performance of the AI ​​model to ensure the model performance of the AI ​​model.

[0319] The embodiment of the present disclosure also provides a method in which a network device receives the performance monitoring value of a second model of a terminal device, or receives relevant information for calculating the performance monitoring value, or accepts performance monitoring indication information, thereby calculating the performance monitoring value of the second model based on the relevant information.

[0320] Please refer to Figure 6, which shows a flow chart of a performance monitoring method for an AI model provided by an embodiment of the present disclosure. This embodiment is illustrated by applying the method to a network device in the communication system shown in Figure 2. The method may include the following steps:

[0321] Step 601: Receive a performance monitoring value of a second model reported by a terminal device.

[0322] Schematically, the AI ​​model includes a first model and a second model, the first model and the second model are deployed on the terminal device, and the second model is deployed on the network device.

[0323] In some embodiments, the first model and the second model constitute a bilateral model, wherein the first model is used to process the input sample and output an intermediate value, which is input into the second model, and the intermediate value is processed by the second model to output an inference value, wherein the inference value is as identical or similar to the input sample as possible, and the input sample is data or information related to the mobile communication system, such as channel information.

[0324] Schematically, the performance monitoring value is used to indicate the quality of the model performance of the AI ​​model.

[0325] Optionally, the performance monitoring value is used to perform performance monitoring on the first model; or, used to perform performance monitoring on the second model; or, used to perform performance monitoring on both the first model and the second model.

[0326] In some embodiments, the performance monitoring value is determined based on the first data and the second data, or the performance monitoring value is determined based on the third data and the fourth data.

[0327] Among them, the first data is determined based on the inference value of the first input value of the second model, and the second data is determined based on the output value corresponding to the second input value when the second model is not used. The first input value and the second input value correspond to the same input sample, the input sample is the input of the first model, and the first input value is the output value of the first model for the input sample.

[0328] Illustratively, the first input value is realized as the output result of the first model, that is, the intermediate value. The first input value is input into the second model, and the output result is the inference value corresponding to the first input value.

[0329] Illustratively, the second input value refers to a result obtained by measuring the input sample using a specified configuration without using the second model.

[0330] The third data is determined based on an average value, a maximum value, or a minimum value of the first data corresponding to at least two frequency domain units, and the fourth data is determined based on an average value, a maximum value, or a minimum value of the second data corresponding to at least two frequency domain units;

[0331] Illustratively, the third data is obtained by an average value / maximum value / minimum value of a plurality of first data obtained within a historical time range corresponding to at least two frequency domain units.

[0332] Illustratively, the fourth data is obtained by an average value / maximum value / minimum value of a plurality of second data obtained within a historical time range corresponding to at least two frequency domain units.

[0333] In some embodiments, the at least two frequency domain unit data N are configured by a network device; or, are predefined by a communication protocol.

[0334] Optionally, the performance monitoring value includes at least one of the following monitoring value types:

[0335] 1) The difference between the first data and the second data;

[0336] 2) the ratio of the first data to the second data;

[0337] 3) the difference between the third data and the fourth data;

[0338] 4) a ratio of the third data to the fourth data;

[0339] 5) a weighted average of the first data and the second data;

[0340] 6) the average of the first data and the second data;

[0341] 7) a weighted average of the third data and the fourth data;

[0342] 8) The average value of the third data and the fourth data.

[0343] Step 602: Receive relevant information for calculating performance monitoring values.

[0344] Optionally, the relevant information of the performance monitoring value includes at least one of the following information:

[0345] 1) first data;

[0346] 2) second data;

[0347] 3) third data;

[0348] 4) fourth data;

[0349] 5) used to calculate the correlation value of the first data;

[0350] The method for calculating the correlation value of the first data includes at least one of an input sample and a first input value.

[0351] 6) Used to calculate the correlation value of the second data.

[0352] The method for calculating the correlation value of the second data includes at least one of an input sample and a second input value.

[0353] Illustratively, the performance monitoring indication information indicates the quality of the performance parameters of the current first model and / or second model.

[0354] In one example, bit information is used as performance monitoring indication information. When the bit information takes a value of 1, it indicates that the performance parameters of the first model and / or the second model are better. When the bit information takes a value of 0, it indicates that the performance parameters of the first model and / or the second model are worse. Or, when the bit information takes a value of 0, it indicates that the performance parameters of the first model are better. When the bit information takes a value of 1, it indicates that the performance parameters of the second model are worse.

[0355] Step 603: Calculate the performance monitoring value of the second model based on the relevant information.

[0356] Optionally, calculating the performance monitoring value of the second model includes at least one of the following calculation methods:

[0357] Calculating the first data based on the correlation value used to calculate the first data;

[0358] Calculating the second data based on the correlation value used to calculate the second data;

[0359] Calculating third data based on the average value, maximum value, or minimum value of the first data corresponding to at least two frequency domain units;

[0360] Calculating fourth data based on an average value, a maximum value, or a minimum value of the second data corresponding to at least two frequency domain units;

[0361] Calculate the ratio of the first data to the second data;

[0362] Calculate the difference between the first data and the second data;

[0363] Calculate the difference between the third data and the fourth data;

[0364] Calculate the ratio of the third data to the fourth data.

[0365] Step 604: Receive performance monitoring indication information.

[0366] Illustratively, the performance monitoring indication information indicates the quality of the performance parameters of the current first model and / or second model.

[0367] In one example, bit information is used as performance monitoring indication information. When the bit information takes a value of 1, it indicates that the performance parameters of the first model and / or the second model are good. When the bit information takes a value of 0, it indicates that the performance parameters of the first model and / or the second model are poor. Alternatively, when the bit information takes a value of 0, it indicates that the performance parameters of the first model are good. When the bit information takes a value of 1, it indicates that the performance parameters of the second model are poor. In a feasible scenario, the terminal device reports the performance monitoring value of the second model to the network device. The network device directly determines the model performance of the second model in the current frequency domain unit based on the performance monitoring value and performs management operations on the AI ​​model.

[0368] In some embodiments, the terminal device reports relevant information used to calculate the performance monitoring value of the second model to the network device. After the network device calculates the performance monitoring value of the second model based on the relevant information, it determines the model performance of the first model and / or the second model in the current frequency domain unit based on the performance monitoring indication information, and performs management operations on the AI ​​model.

[0369] In some embodiments, the terminal device reports the performance monitoring value of the second model to the network device, and the network device determines the model performance of the first model and / or the second model in the current frequency domain unit based on the performance monitoring value, and performs management operations on the AI ​​model.

[0370] In some embodiments, the terminal device reports performance monitoring indication information to the network device, and the network device determines the model performance of the first model and / or the second model in the current frequency domain unit based on the performance monitoring indication information, and performs management operations on the AI ​​model.

[0371] In some embodiments, management operations of the AI ​​model are performed based on the performance monitoring value of the second model; the management operations of the AI ​​model include: switching operations of the AI ​​model, activation operations of the AI ​​model, deactivation operations of the AI ​​model, and falling back to traditional communication operations that do not use the AI ​​model.

[0372] The management operations on AI models include at least one of the following types of operations:

[0373] Switching operations of AI models;

[0374] Activation of AI models;

[0375] Deactivation of AI models;

[0376] Update the model parameters of the AI ​​model;

[0377] Falling back to traditional communication operations that do not use AI models.

[0378] In an illustrative manner, the network device presets a first threshold value. If the difference / ratio between the first data and the second data, or the difference / ratio between the third data and the fourth data, reaches the first threshold value, the switching operation of the AI ​​model is implemented. If the first threshold value is not reached, the AI ​​model is not switched.

[0379] In an illustrative manner, the network device presets a second threshold value. If the difference / ratio between the first data and the second data, or the difference / ratio between the third data and the fourth data, is less than the second threshold value, at least one of the following operations is implemented: activation of the AI ​​model, deactivation of the AI ​​model, and fallback to a traditional communication operation that does not use the AI ​​model.

[0380] In summary, the technical solution provided in this embodiment provides a solution for AI model performance monitoring. When a terminal device deploys a first model and a second model, the terminal device reports the performance monitoring value of the second model in the AI ​​model, reports related information on the calculated performance monitoring value, or reports performance monitoring indication information, enabling the network device or terminal device to monitor the performance of the AI ​​model to ensure the model performance of the AI ​​model.

[0381] Please refer to Figure 7, which shows a flow chart of a method for monitoring the performance of an AI model provided by an embodiment of the present disclosure. This embodiment is illustrated by applying the method to a network device in the communication system shown in Figure 2. The method may include the following steps:

[0382] Step 701: Receive relevant information reported by a terminal device for calculating a performance monitoring value of a second model.

[0383] Schematically, the AI ​​model includes a first model and a second model, the first model is deployed on the terminal device, and the second model is deployed on the network device.

[0384] In some embodiments, the first model and the second model constitute a bilateral model, wherein the first model is used to process the input sample and output an intermediate value, which is input into the second model, and the intermediate value is processed by the second model to output an inference value, wherein the inference value is as identical or similar to the input sample as possible, and the input sample is data or information related to the mobile communication system, such as channel information.

[0385] Illustratively, the relevant information of the performance monitoring value is used to calculate the performance monitoring value of the second model.

[0386] Optionally, the relevant information of the performance monitoring value includes at least one of the following information:

[0387] 1) first data;

[0388] 2) second data;

[0389] 3) third data;

[0390] 4) fourth data;

[0391] 5) used to calculate the correlation value of the first data;

[0392] The method for calculating the correlation value of the first data includes at least one of an input sample and a first input value.

[0393] 6) Used to calculate the correlation value of the second data.

[0394] The method for calculating the correlation value of the second data includes at least one of an input sample and a second input value.

[0395] Among them, the first data is determined based on the inference value of the first input value of the second model, and the second data is determined based on the output value corresponding to the second input value when the second model is not used. The first input value and the second input value correspond to the same input sample, the input sample is the input of the first model, and the first input value is the output value of the first model for the input sample.

[0396] Illustratively, the first input value is realized as the output result of the first model, that is, the intermediate value. The first input value is input into the second model, and the output result is the inference value corresponding to the first input value.

[0397] Illustratively, the second input value refers to a result obtained by measuring the input sample using a specified configuration without using the second model.

[0398] The third data is determined based on an average value, a maximum value, or a minimum value of the first data corresponding to at least two frequency domain units, and the fourth data is determined based on an average value, a maximum value, or a minimum value of the second data corresponding to at least two frequency domain units;

[0399] Illustratively, the third data is obtained by an average value / maximum value / minimum value of a plurality of first data obtained within a historical time range corresponding to at least two frequency domain units.

[0400] Illustratively, the fourth data is obtained by an average value / maximum value / minimum value of a plurality of second data obtained within a historical time range corresponding to at least two frequency domain units.

[0401] In some embodiments, the at least two frequency domain unit data N are configured by a network device; or, are predefined by a communication protocol.

[0402] Step 702: Calculate the performance monitoring value of the second model based on the relevant information.

[0403] In another feasible situation, the terminal device reports relevant information used to calculate the performance monitoring value of the second model to the network device. After the network device calculates the performance monitoring value of the second model based on the relevant information, it determines the model performance of the first model and / or the second model in the current frequency domain unit based on the performance monitoring indication information, and performs management operations on the AI ​​model.

[0404] The management operations on AI models include at least one of the following types of operations:

[0405] Switching operations of AI models;

[0406] Activation of AI models;

[0407] Deactivation of AI models;

[0408] Update the model parameters of the AI ​​model;

[0409] Falling back to traditional communication operations that do not use AI models.

[0410] In an illustrative manner, the network device presets a first threshold value. If the difference / ratio between the first data and the second data, or the difference / ratio between the third data and the fourth data, reaches the first threshold value, the switching operation of the AI ​​model is implemented. If the first threshold value is not reached, the AI ​​model is not switched.

[0411] In an illustrative manner, the network device presets a second threshold value. If the difference / ratio between the first data and the second data, or the difference / ratio between the third data and the fourth data, is less than the second threshold value, at least one of the following operations is implemented: activation of the AI ​​model, deactivation of the AI ​​model, and fallback to a traditional communication operation that does not use the AI ​​model.

[0412] In one implementation, the terminal device reports the relevant value of the performance monitoring value of the second model to the network device. The network device first calculates the performance monitoring value through the relevant value of the performance monitoring value, and then performs the management operations mentioned above on the AI ​​model based on the performance monitoring value.

[0413] In summary, the technical solution provided in this embodiment provides a solution for AI model performance monitoring. When a terminal device deploys a first model and a second model, the terminal device reports the performance monitoring value of the second model in the AI ​​model, reports related information on the calculated performance monitoring value, or reports performance monitoring indication information, enabling the network device or terminal device to monitor the performance of the AI ​​model to ensure the model performance of the AI ​​model.

[0414] In an optional embodiment, the frequency domain unit is implemented as a subband, the number of subbands configured by the network device is N3, the number of data transmission layers is 1 (or other values), and when both the CSI compression model and the CSI recovery model are deployed in the terminal device, the input sample of the CSI compression model is input-CSI-NW, and the model output value of the CSI recovery model is the inference value output-CSI-UE. That is, at least two frequency domain unit data N are configured by the network or predefined by the communication protocol.

[0415] Among them, input-CSI-NW and output-CSI-UE refer to at least one eigenrepresentation v contained in the V matrix obtained after performing singular value decomposition (SVD) on the channel matrix corresponding to the downlink channel information. SVD is used to perform eigendecomposition on the channel matrix. For input-CSI-NW, the channel matrix is ​​the first channel information (such as uplink channel information or downlink channel information). For output-CSI-UE, the channel matrix is ​​the model output value output-CSI-UE (such as the recovered uplink channel information or the recovered downlink channel information).

[0416] Therefore, input-CSI-NW is expressed as output-CSI-UE is expressed as Where dim(v) represents the vector dimension of feature representation v. Let e i It represents the feature representation corresponding to the i-th subband in the CSI recovery model.

[0417] In addition, the network device also configures codebook parameters (such as codebook parameters corresponding to eType II) for the terminal device.

[0418] The network device estimates the downlink channel information of different sub-bands based on the received Channel Status Information-Reference Signal (CSI-RS) and calculates the feature representation corresponding to each sub-band. Let v i represents the feature representation corresponding to the i-th subband, that is, the estimated channel information.

[0419] Below, based on the above conditions, different deployment scenarios of the first model and the second model in the terminal device and the network device are described in detail.

[0420] In the first type, the first model and the second model are deployed on the terminal device, and the second model is deployed on the network device.

[0421] In some embodiments, the inferred value is inferred by a second model in the terminal device.

[0422] Schematically, the first model is a CSI compression model, and the second model is a CSI recovery model.

[0423] In this embodiment, the inference value is implemented as a model output value output-CSI-UE of the CSI model.

[0424] In some embodiments, the first input value is an output value of the first model for the input sample. When the first model is implemented as a CSI compression model, the first input value is implemented as a binary bit stream.

[0425] In this embodiment, the performance monitoring method of the AI ​​model includes the following steps.

[0426] In the first step, the terminal device calculates the first data.

[0427] For the i-th subband channel, the terminal device inputs the input sample input-CSI-NW into the CSI compression model and outputs a binary bit stream as the first input value corresponding to the CSI recovery model. The CSI recovery model infers the first input value to obtain the inference value output-CSI-UE.

[0428] The terminal device represents e according to the feature corresponding to the i-th subband i and the estimated channel information v corresponding to the i-th subband i Calculate the square generalized cosine similarity (SGCS) K using formula 1 1,i .

[0429] When obtaining the square of cosine similarity K 1,i Then, the first data K1 is obtained according to the above formula 2.

[0430] It is worth noting that in the above method, the first data can be calculated based on the inferred value output-CSI-UE and the estimated channel information, or can be directly calculated based on the inferred value output-CSI-UE.

[0431] Optionally, the first data K1 is implemented as an intermediate KPI corresponding to at least two frequency domain units, or as an average value of the intermediate KPIs corresponding to at least two frequency domain units.

[0432] Where K1 is implemented as the average of the intermediate KPIs corresponding to at least two frequency domain units, the terminal device calculates the average value of K1 in the at least two frequency domain units to obtain K1' as the third data. Wherein, the at least two frequency domain unit data N are configured by the network or predefined by the communication protocol.

[0433] In the second step, the terminal device calculates the second data.

[0434] For the i-th subband channel, the terminal device calculates the second input value W according to the estimated channel information and the eType II codebook parameters configured by the network device through the above formula 3. In this embodiment, the second data is implemented as the precoding matrix indication PMI information W i.

[0435] Then, the terminal device calculates W according to the above formula 1 i and V i The square of the cosine similarity K 2,i , calculated according to the above formula 2 to get K 2,i The corresponding second data K2.

[0436] It is worth noting that the second data can be calculated based on the estimated channel information and the codebook parameters, or can be directly obtained based on the codebook parameters, or can be directly obtained based on the estimated channel information.

[0437] Optionally, the second data K2 is implemented as an intermediate KPI corresponding to at least two frequency domain units, or as an average value of the intermediate KPIs corresponding to at least two frequency domain units.

[0438] Where K2 is implemented as the average of the intermediate KPIs corresponding to at least two frequency domain units, the terminal device calculates the average value of K2 in the at least two frequency domain units to obtain K2' as the fourth data. Wherein, the at least two frequency domain unit data N are configured by the network or predefined by the communication protocol.

[0439] Optionally, the frequency domain unit selected for the third data and the frequency domain unit selected for the fourth data are the same or different.

[0440] In the third step, the terminal device reports the performance monitoring value / related information of the performance monitoring value to the network device.

[0441] In one feasible case, the terminal device first calculates the difference between K1 and K2 and reports it to the network device as a performance monitoring value; or, the terminal device first calculates the ratio of K1 and K2 and reports it to the network device as a performance monitoring value.

[0442] In another feasible case, the terminal device reports K1 and K2 directly to the network device.

[0443] In another feasible case, the terminal device reports the relevant value of the first data K1 to the network device, including at least one of estimated channel information, inference value output-CSI-UE, and binary bit stream.

[0444] In another feasible case, the terminal device reports a relevant value of the second data K2 to the network device, including at least one of input-out-NW, estimated channel information, and PMI information.

[0445] In the fourth step, the network device performs performance monitoring on the AI ​​model based on the performance monitoring value / related information of the performance monitoring value.

[0446] Optionally, the network device directly performs performance monitoring on the AI ​​model after receiving the performance monitoring value sent by the terminal device.

[0447] In some embodiments, management operations of the AI ​​model are performed based on the performance monitoring value of the second model; the management operations of the AI ​​model include: switching operations of the AI ​​model, activation operations of the AI ​​model, deactivation operations of the AI ​​model, and falling back to traditional communication operations that do not use the AI ​​model.

[0448] In this embodiment, the network device presets a first threshold value. If the difference / ratio between the first data and the second data, or the difference / ratio between the third data and the fourth data, reaches the first threshold value, the AI ​​model switching operation is implemented. If the first threshold value is not reached, the AI ​​model is not switched.

[0449] In this embodiment, the network device presets a second threshold value. If the difference / ratio between the first data and the second data, or the difference / ratio between the third data and the fourth data, is less than the second threshold value, at least one of the following operations is implemented: activation operation of the AI ​​model / deactivation operation of the AI ​​model / falling back to a traditional communication operation that does not use the AI ​​model.

[0450] The second type is that the first model is deployed on the terminal device and the second model is deployed on the network device.

[0451] In the case where the second model is not deployed on the terminal device, the following two cases are also included.

[0452] 2-1. The network device sends the inference value to the terminal device.

[0453] The inferred value is inferred by a second model in the network device.

[0454] Optionally, when the first model is not deployed in the network device, the terminal device inputs the input sample into the deployed first model, outputs a binary bit stream as the first input value, and sends the first input value to the network device. The network device infers the first input value through the second model to obtain an inference value output-CSI-UE corresponding to the first input value and sends it to the terminal device.

[0455] Optionally, when the first model is deployed in the network device, the network device inputs the input sample into the deployed first model, outputs a binary bit stream as the first input value, inputs the first input value into the second model, infers the first input value to obtain the inference value output-CSI-UE corresponding to the first input value, and sends it to the terminal device.

[0456] Schematically, after the terminal device receives the inference value, the performance monitoring method of the AI ​​model is implemented according to the following steps.

[0457] In the first step, the terminal device calculates the first data.

[0458] The terminal device calculates the square generalized cosine similarity (SGCS) K according to the feature representation corresponding to the i-th subband and the estimated channel information corresponding to the i-th subband using the above formula 1 3,i , in obtaining K 3,i Then, the first data K3 is obtained according to the above formula 2.

[0459] It is worth noting that in the above method, the first data can be calculated based on the inferred value output-CSI-UE and the estimated channel information, or can be directly calculated based on the inferred value output-CSI-UE.

[0460] Optionally, the first data K3 is implemented as an intermediate KPI corresponding to at least two frequency domain units, or as an average value of the intermediate KPIs corresponding to at least two frequency domain units.

[0461] Where K3 is implemented as the average of the intermediate KPIs corresponding to at least two frequency domain units, the terminal device calculates the average value of K3 in the at least two frequency domain units to obtain K3' as the third data. Wherein, the at least two frequency domain unit data N are configured by the network or predefined by the communication protocol.

[0462] In the second step, the terminal device calculates the second data.

[0463] For the i-th subband channel, the terminal device calculates the second input value using the above formula three according to the estimated channel information and the eType II codebook parameters configured by the network device.

[0464] Then, the terminal device calculates the SCGS K corresponding to the codebook parameters and the estimated channel information according to the above formula 1 2,i , calculated according to the above formula 2 to get K 2,i The corresponding second data K2.

[0465] It is worth noting that the second data can be calculated based on the estimated channel information and the codebook parameters, or can be directly obtained based on the codebook parameters, or can be directly obtained based on the estimated channel information.

[0466] Optionally, the second data K2 is implemented as an intermediate KPI corresponding to at least two frequency domain units, or as an average value of the intermediate KPIs corresponding to at least two frequency domain units.

[0467] Where K2 is implemented as the average of the intermediate KPIs corresponding to at least two frequency domain units, the terminal device calculates the average value of K2 in the at least two frequency domain units to obtain K2' as the fourth data. Wherein, the at least two frequency domain unit data N are configured by the network or predefined by the communication protocol.

[0468] Optionally, the frequency domain unit selected for the third data and the frequency domain unit selected for the fourth data are the same or different.

[0469] In the third step, the terminal device reports relevant information of the performance monitoring value to the network device.

[0470] In another feasible case, the terminal device reports K3 and K2 directly to the network device.

[0471] In another feasible scenario, the terminal device reports relevant values ​​of the first data K3 to the network device, including at least one of estimated channel information, input-CSI-NW, inference value output-CSI-UE, and binary bit stream.

[0472] In another feasible case, the terminal device reports a relevant value of the second data K2 to the network device, including at least one of input-CSI-NW, estimated channel information, and PMI information.

[0473] In the fourth step, the network device performs performance monitoring on the AI ​​model based on the performance monitoring value / related information of the performance monitoring value.

[0474] Optionally, after receiving relevant information about the performance monitoring value sent by the terminal device, the network device calculates the performance monitoring value of the second model based on the relevant information, and performs performance monitoring on the AI ​​model according to the performance monitoring value.

[0475] In some embodiments, management operations of the AI ​​model are performed based on the performance monitoring value of the second model; the management operations of the AI ​​model include: switching operations of the AI ​​model, activation operations of the AI ​​model, deactivation operations of the AI ​​model, and falling back to traditional communication operations that do not use the AI ​​model.

[0476] In this embodiment, the network device presets a first threshold value. If the difference / ratio between the first data and the second data, or the difference / ratio between the third data and the fourth data, reaches the first threshold value, the AI ​​model switching operation is implemented. If the first threshold value is not reached, the AI ​​model is not switched.

[0477] In this embodiment, the network device presets a second threshold value. If the difference / ratio between the first data and the second data, or the difference / ratio between the third data and the fourth data, is less than the second threshold value, at least one of the following operations is implemented: activation operation of the AI ​​model / deactivation operation of the AI ​​model / falling back to a traditional communication operation that does not use the AI ​​model.

[0478] In some embodiments, the network device calculates the performance monitoring value of the second model based on the relevant information and then monitors the performance of the AI ​​model, wherein the calculation method includes at least one of the following steps:

[0479] Calculating the first data based on the correlation value used to calculate the first data;

[0480] Calculating the second data based on the correlation value used to calculate the second data;

[0481] Calculating third data based on the average value, maximum value, or minimum value of the first data corresponding to at least two frequency domain units;

[0482] Calculating fourth data based on an average value, a maximum value, or a minimum value of the second data corresponding to at least two frequency domain units;

[0483] Calculating a performance monitoring value based on the first data and the second data;

[0484] For example: calculating the ratio of the first data and the second data; calculating the difference between the first data and the second data;

[0485] Calculating a performance monitoring value based on the third data and the fourth data.

[0486] For example: calculating the difference between the third data and the fourth data; calculating the ratio of the third data to the fourth data.

[0487] 2-2. Receive input samples sent by the terminal device.

[0488] Optionally, when the first model is not deployed in the network device, the terminal device sends the input sample input-CSI-NW and the first input value (binary bit stream) to the network device.

[0489] Optionally, when the first model is deployed in the network device, the terminal device sends the input sample input-CSI-NW to the network device.

[0490] Schematically, the performance monitoring method of the AI ​​model is implemented according to the following steps.

[0491] In the first step, the network device receives input samples.

[0492] Optionally, the terminal device sends the input sample input-CSI-NW and the second data K2 or third data K2′ calculated in the above steps to the network device simultaneously.

[0493] Optionally, the terminal device simultaneously sends the input sample input-CSI-NW and the second input value (PMI information) to the network device. In this case, the network device calculates the second data K2 using the above formula 2 based on the PMI information and the input sample input-CSI-NW, or calculates the fourth data K2' based on the second data in at least two frequency domain units.

[0494] In the second step, the network device calculates the first data.

[0495] The network device calculates first data K4 using Formula 1 based on the received input sample input-CSI-NW and the inferred value output-CSI-UE corresponding to the second model deployed in the network device. Alternatively, after calculating the first data K4, the network device calculates third data K4' based on the average of the first data in at least two frequency domains. The at least two frequency domain unit data N are configured by the network or predefined by the communication protocol.

[0496] In the third step, the network device performs performance monitoring of the AI ​​model based on the performance monitoring value / related information of the performance monitoring value.

[0497] Optionally, the network device performs performance monitoring on the AI ​​model by calculating a performance monitoring value, wherein the performance monitoring value includes at least one of a difference between the first data and the second data, a ratio of the first data and the second data, a difference between the third data and the fourth data, or a ratio of the third data to the fourth data.

[0498] In some embodiments, management operations of the AI ​​model are performed based on the performance monitoring value of the second model; the management operations of the AI ​​model include: switching operations of the AI ​​model, activation operations of the AI ​​model, deactivation operations of the AI ​​model, and falling back to traditional communication operations that do not use the AI ​​model.

[0499] In this embodiment, the network device presets a first threshold value. If the difference / ratio between the first data and the second data, or the difference / ratio between the third data and the fourth data, reaches the first threshold value, the AI ​​model switching operation is implemented. If the first threshold value is not reached, the AI ​​model is not switched.

[0500] In this embodiment, the network device presets a second threshold value. If the difference / ratio between the first data and the second data, or the difference / ratio between the third data and the fourth data, is less than the second threshold value, at least one of the following operations is implemented: activation operation of the AI ​​model / deactivation operation of the AI ​​model / falling back to a traditional communication operation that does not use the AI ​​model.

[0501] It is worth noting that in the embodiments of the present application, K1, K2, K3, K4, K1', K2', K3' and K4' are used to represent the first data / second data / third data / fourth data calculated under different circumstances.

[0502] In some embodiments, indication information is sent to the terminal device.

[0503] In this embodiment, when the second model is only deployed in the network device, the relevant information used to calculate the performance monitoring value of the second model needs to be reported when the reporting indication information sent by the network device is received or the target event is triggered in the terminal device.

[0504] The target event is determined based on the data distribution corresponding to the input data of the first model; or the target event is determined based on the data distribution corresponding to the output data of the first model.

[0505] Illustratively, the reporting indication information sent by the network device is used to instruct the terminal device to report the performance monitoring value of the second model within a specified time range, or to report relevant information used to calculate the performance monitoring value.

[0506] Schematically, the data distribution of the input data of the first model is used to indicate the correlation between multiple input data. The correlation between the multiple input data is determined by calculating the cosine similarity between the multiple input data. When the correlation is higher, it indicates that the difference between the input data is smaller.

[0507] Optionally, a third threshold is set, and when the data distribution of the input data of the first model reaches the third threshold, the performance monitoring value of the second model is reported or relevant information used to calculate the performance monitoring value is reported.

[0508] Schematically, the data distribution of the output data of the first model is used to indicate the correlation between multiple output data. The correlation between the multiple output data is determined by calculating the cosine similarity between the multiple output data. When the correlation is higher, it indicates that the difference in the output data is smaller.

[0509] Optionally, a fourth threshold is set, and when the data distribution of the output data of the first model reaches the fourth threshold, the performance monitoring value of the second model or the relevant information used to calculate the performance monitoring value is reported.

[0510] In summary, the technical solution provided in this embodiment provides a solution for AI model performance monitoring. When a terminal device deploys a first model and a second model, the terminal device reports the performance monitoring value of the second model in the AI ​​model, reports related information on the calculated performance monitoring value, or reports performance monitoring indication information, enabling the network device or terminal device to monitor the performance of the AI ​​model to ensure the model performance of the AI ​​model.

[0511] Figure 8 is a flowchart of the interaction between a terminal and a network device provided by an embodiment of the present disclosure. This embodiment uses the interaction process as an example of the terminal and network device in the communication system shown in Figure 2. As shown in Figure 8, the interaction process includes the following steps:

[0512] Step 801: The terminal device reports the performance monitoring value of the second model to the network device, or reports relevant information used to calculate the performance monitoring value, or reports performance monitoring indication information.

[0513] Schematically, the AI ​​model includes a first model and a second model, the first model and the second model are deployed on the terminal device, and the second model is deployed on the network device.

[0514] In some embodiments, the first model and the second model constitute a bilateral model, wherein the first model is used to process the input sample and output an intermediate value, which is input into the second model, and the intermediate value is processed by the second model to output an inference value, wherein the inference value is as identical or similar to the input sample as possible, and the input sample is data or information related to the mobile communication system, such as channel information.

[0515] Schematically, the performance monitoring value is used to indicate the quality of the model performance of the AI ​​model.

[0516] Optionally, the performance monitoring value is used to perform performance monitoring on the first model; or, used to perform performance monitoring on the second model; or, used to perform performance monitoring on both the first model and the second model.

[0517] In some embodiments, the performance monitoring value is determined based on the first data and the second data, or the performance monitoring value is determined based on the third data and the fourth data.

[0518] Among them, the first data is determined based on the inference value of the first input value of the second model, and the second data is determined based on the output value corresponding to the second input value when the second model is not used. The first input value and the second input value correspond to the same input sample, the input sample is the input of the first model, and the first input value is the output value of the first model for the input sample.

[0519] Illustratively, the first input value is realized as the output result of the first model, that is, the intermediate value. The first input value is input into the second model, and the output result is the inference value corresponding to the first input value.

[0520] Illustratively, the second input value refers to a result obtained by measuring the input sample using a specified configuration without using the second model.

[0521] The third data is determined based on an average value, a maximum value, or a minimum value of the first data corresponding to at least two frequency domain units, and the fourth data is determined based on an average value, a maximum value, or a minimum value of the second data corresponding to at least two frequency domain units;

[0522] Illustratively, the third data is obtained by an average value / maximum value / minimum value of a plurality of first data obtained within a historical time range corresponding to at least two frequency domain units.

[0523] Illustratively, the fourth data is obtained by an average value / maximum value / minimum value of a plurality of second data obtained within a historical time range corresponding to at least two frequency domain units.

[0524] In some embodiments, the at least two frequency domain unit data N are configured by a network device; or, are predefined by a communication protocol.

[0525] Optionally, the performance monitoring value includes at least one of the following monitoring value types:

[0526] 1) The difference between the first data and the second data;

[0527] 2) the ratio of the first data to the second data;

[0528] 3) the difference between the third data and the fourth data;

[0529] 4) a ratio of the third data to the fourth data;

[0530] 5) a weighted average of the first data and the second data;

[0531] 6) the average of the first data and the second data;

[0532] 7) a weighted average of the third data and the fourth data;

[0533] 8) The average value of the third data and the fourth data.

[0534] Optionally, the relevant information of the performance monitoring value includes at least one of the following information:

[0535] 1) first data;

[0536] 2) second data;

[0537] 3) third data;

[0538] 4) fourth data;

[0539] 5) used to calculate the correlation value of the first data;

[0540] The method for calculating the correlation value of the first data includes at least one of an input sample and a first input value.

[0541] 6) Used to calculate the correlation value of the second data.

[0542] The method for calculating the correlation value of the second data includes at least one of an input sample and a second input value.

[0543] Illustratively, the performance monitoring indication information indicates the quality of the performance parameters of the current first model and / or second model.

[0544] In one example, bit information is used as performance monitoring indication information. When the bit information takes a value of 1, it indicates that the performance parameters of the first model and / or the second model are better. When the bit information takes a value of 0, it indicates that the performance parameters of the first model and / or the second model are worse. Or, when the bit information takes a value of 0, it indicates that the performance parameters of the first model are better. When the bit information takes a value of 1, it indicates that the performance parameters of the second model are worse.

[0545] In a feasible situation, the terminal device reports the performance monitoring value of the second model to the network device. The network device directly determines the model performance of the second model in the current frequency domain unit based on the performance monitoring value and performs management operations on the AI ​​model.

[0546] In another feasible situation, the terminal device reports relevant information used to calculate the performance monitoring value of the second model to the network device. After the network device calculates the performance monitoring value of the second model based on the relevant information, it determines the model performance of the first model and / or the second model in the current frequency domain unit based on the performance monitoring indication information, and performs management operations on the AI ​​model.

[0547] In another feasible situation, the terminal device reports performance monitoring indication information to the network device, and the network device determines the model performance of the first model and / or the second model in the current frequency domain unit based on the performance monitoring indication information, and performs management operations on the AI ​​model.

[0548] The management operations on AI models include at least one of the following types of operations:

[0549] Switching operations of AI models;

[0550] Activation of AI models;

[0551] Deactivation of AI models;

[0552] Update the model parameters of the AI ​​model;

[0553] Falling back to traditional communication operations that do not use AI models.

[0554] In an illustrative manner, the network device presets a first threshold value. If the difference / ratio between the first data and the second data, or the difference / ratio between the third data and the fourth data, reaches the first threshold value, the switching operation of the AI ​​model is implemented. If the first threshold value is not reached, the AI ​​model is not switched.

[0555] In an illustrative manner, the network device presets a second threshold value. If the difference / ratio between the first data and the second data, or the difference / ratio between the third data and the fourth data, is less than the second threshold value, at least one of the following operations is implemented: activation of the AI ​​model, deactivation of the AI ​​model, and fallback to a traditional communication operation that does not use the AI ​​model.

[0556] In summary, the technical solution provided in this embodiment provides a solution for AI model performance monitoring. When a terminal device deploys a first model and a second model, the terminal device reports the performance monitoring value of the second model in the AI ​​model, reports related information on the calculated performance monitoring value, or reports performance monitoring indication information, enabling the network device or terminal device to monitor the performance of the AI ​​model to ensure the model performance of the AI ​​model.

[0557] Figure 9 is a flowchart of the interaction between a terminal and a network device provided by an embodiment of the present disclosure. This embodiment uses the interaction process as an example to illustrate the application of the interaction process to the terminal and network device in the communication system shown in Figure 2. As shown in Figure 9, the interaction process includes the following steps:

[0558] Step 901: The terminal device reports relevant information used to calculate the performance monitoring value of the second model to the network device.

[0559] Schematically, the AI ​​model includes a first model and a second model, the first model is deployed on the terminal device, and the second model is deployed on the network device.

[0560] In some embodiments, the first model and the second model constitute a bilateral model, wherein the first model is used to process the input sample and output an intermediate value, which is input into the second model, and the intermediate value is processed by the second model to output an inference value, wherein the inference value is as identical or similar to the input sample as possible, and the input sample is data or information related to the mobile communication system, such as channel information.

[0561] Illustratively, the relevant information of the performance monitoring value is used to calculate the performance monitoring value of the second model.

[0562] Optionally, the relevant information of the performance monitoring value includes at least one of the following information:

[0563] 1) first data;

[0564] 2) second data;

[0565] 3) third data;

[0566] 4) fourth data;

[0567] 5) used to calculate the correlation value of the first data;

[0568] The method for calculating the correlation value of the first data includes at least one of an input sample and a first input value.

[0569] 6) Used to calculate the correlation value of the second data.

[0570] The method for calculating the correlation value of the second data includes at least one of an input sample and a second input value.

[0571] Among them, the first data is determined based on the inference value of the first input value of the second model, and the second data is determined based on the output value corresponding to the second input value when the second model is not used. The first input value and the second input value correspond to the same input sample, the input sample is the input of the first model, and the first input value is the output value of the first model for the input sample.

[0572] Illustratively, the first input value is realized as the output result of the first model, that is, the intermediate value. The first input value is input into the second model, and the output result is the inference value corresponding to the first input value.

[0573] Illustratively, the second input value refers to a result obtained by measuring the input sample using a specified configuration without using the second model.

[0574] The third data is determined based on an average value, a maximum value, or a minimum value of the first data corresponding to at least two frequency domain units, and the fourth data is determined based on an average value, a maximum value, or a minimum value of the second data corresponding to at least two frequency domain units;

[0575] Illustratively, the third data is obtained by an average value / maximum value / minimum value of a plurality of first data obtained within a historical time range corresponding to at least two frequency domain units.

[0576] Illustratively, the fourth data is obtained by an average value / maximum value / minimum value of a plurality of second data obtained within a historical time range corresponding to at least two frequency domain units.

[0577] In some embodiments, the at least two frequency domain unit data N are configured by a network device; or, are predefined by a communication protocol.

[0578] In another feasible situation, the terminal device reports relevant information used to calculate the performance monitoring value of the second model to the network device. After the network device calculates the performance monitoring value of the second model based on the relevant information, it determines the model performance of the first model and / or the second model in the current frequency domain unit based on the performance monitoring indication information, and performs management operations on the AI ​​model.

[0579] The management operations on AI models include at least one of the following types of operations:

[0580] Switching operations of AI models;

[0581] Activation of AI models;

[0582] Deactivation of AI models;

[0583] Update the model parameters of the AI ​​model;

[0584] Falling back to traditional communication operations that do not use AI models.

[0585] In an illustrative manner, the network device presets a first threshold value. If the difference / ratio between the first data and the second data, or the difference / ratio between the third data and the fourth data, reaches the first threshold value, the switching operation of the AI ​​model is implemented. If the first threshold value is not reached, the AI ​​model is not switched.

[0586] In an illustrative manner, the network device presets a second threshold value. If the difference / ratio between the first data and the second data, or the difference / ratio between the third data and the fourth data, is less than the second threshold value, at least one of the following operations is implemented: activation of the AI ​​model, deactivation of the AI ​​model, and fallback to a traditional communication operation that does not use the AI ​​model.

[0587] In one implementation, the terminal device reports the relevant value of the performance monitoring value of the second model to the network device. The network device first calculates the performance monitoring value through the relevant value of the performance monitoring value, and then performs the management operations mentioned above on the AI ​​model based on the performance monitoring value.

[0588] In summary, the technical solution provided in this embodiment provides a solution for AI model performance monitoring. When a terminal device deploys a first model and a second model, the terminal device reports the performance monitoring value of the second model in the AI ​​model, reports related information on the calculated performance monitoring value, or reports performance monitoring indication information, enabling the network device or terminal device to monitor the performance of the AI ​​model to ensure the model performance of the AI ​​model.

[0589] The following are embodiments of the apparatus disclosed herein, which can be used to implement the method embodiments disclosed herein. For details not disclosed in the apparatus embodiments disclosed herein, please refer to the method embodiments disclosed herein.

[0590] Figure 10 shows a block diagram of a performance monitoring device for an AI model provided by an embodiment of the present disclosure. The device has the function of implementing the above-mentioned terminal-side method example, and the function can be implemented by hardware or by executing corresponding software through hardware. As shown in Figure 10, the AI ​​model includes a first model and a second model, the first model and the second model are deployed in the device, and the second model is deployed in the network device. The device includes:

[0591] The reporting module 1010 is configured to report the performance monitoring value of the second model to the network device, or to report relevant information used to calculate the performance monitoring value, or to report performance monitoring indication information.

[0592] In some embodiments, the performance monitoring value of the second model includes at least one of the following:

[0593] The performance monitoring value is determined based on first data and second data, the first data is determined based on an inference value of a first input value by the second model, and the second data is determined based on an output value corresponding to the second input value when the second model is not used, the first input value and the second input value correspond to the same input sample, the input sample is input to the first model, and the first input value is an output value of the first model for the input sample;

[0594] The performance monitoring value is determined based on third data and fourth data, the third data is determined based on the average value, maximum value or minimum value of the first data corresponding to at least two frequency domain units, and the fourth data is determined based on the average value, maximum value or minimum value of the second data corresponding to at least two frequency domain units.

[0595] In some embodiments, the relevant information of the performance monitoring value includes at least one of the following:

[0596] first data, where the first data is determined based on an inference value of a first input value by the second model, where the first input value is an output value of the first model for an input sample, and the input sample is an input of the first model;

[0597] second data, the second data being determined based on an output value corresponding to a second input value when the second model is not used, the first input value and the second input value corresponding to the same input sample;

[0598] third data, the third data being determined based on an average value, a maximum value, or a minimum value of the first data corresponding to at least two frequency domain units;

[0599] fourth data, where the fourth data is determined based on an average value, a maximum value, or a minimum value of the second data corresponding to at least two frequency domain units;

[0600] used to calculate the correlation value of the first data;

[0601] Used to calculate the correlation value of the second data.

[0602] In some embodiments, the inferred value is inferred by the second model in the terminal device.

[0603] In some embodiments, the correlation value used to calculate the first data includes: the input sample, and at least one of the first input value; the correlation value used to calculate the second data includes: the input sample, and at least one of the second input value.

[0604] In some embodiments, the first model is a CSI compression model and the second model is a CSI recovery model.

[0605] In some embodiments, the at least two frequency domain unit data N are configured by a network device; or, are predefined by a communication protocol.

[0606] Figure 11 shows a block diagram of a performance monitoring device for an AI model provided by an embodiment of the present disclosure. The device has the function of implementing the above-mentioned terminal-side method example, and the function can be implemented by hardware or by hardware executing corresponding software. As shown in Figure 11, the AI ​​model includes a first model and a second model, the first model is deployed in the device, and the second model is deployed in the network device. The device includes:

[0607] The reporting module 1110 is configured to report relevant information used to calculate the performance monitoring value of the second model to the network device.

[0608] In some embodiments, the relevant information of the performance monitoring value includes at least one of the following:

[0609] first data, where the first data is determined based on an inference value of a first input value by the second model, the inference value being inferred by the second model in the network device, the first input value being an output value of the first model for an input sample, and the input sample being an input of the first model;

[0610] second data, the second data being determined based on an output value corresponding to a second input value when the second model is not used, the first input value and the second input value corresponding to the same input sample;

[0611] third data, the third data being determined based on an average value, a maximum value, or a minimum value of the first data corresponding to at least two frequency domain units;

[0612] fourth data, where the fourth data is determined based on an average value, a maximum value, or a minimum value of the second data corresponding to at least two frequency domain units;

[0613] used to calculate the correlation value of the first data;

[0614] Used to calculate the correlation value of the second data.

[0615] In some embodiments, the correlation value used to calculate the first data includes: the input sample, and at least one of the first input value; the correlation value used to calculate the second data includes: the input sample, and at least one of the second input value.

[0616] In some embodiments, the first model is a CSI compression model and the second model is a CSI recovery model.

[0617] In some embodiments, the at least two frequency domain unit data N are configured by a network device; or, are predefined by a communication protocol.

[0618] In some embodiments, the reporting module 1110 includes:

[0619] The reporting unit 1111 is configured to report relevant information used to calculate the performance monitoring value of the second model to the network device when receiving the reporting instruction information sent by the network device;

[0620] Alternatively, the reporting unit 1111 is further configured to report relevant information used to calculate the performance monitoring value of the second model to the network device when a target event is triggered.

[0621] In some embodiments, the target event is determined based on a data distribution corresponding to input data of the first model; or, the target event is determined based on a data distribution corresponding to output data of the first model.

[0622] Figure 12 shows a block diagram of a performance monitoring device for an AI model provided by an embodiment of the present disclosure. The device has the function of implementing the above-mentioned network-side method example, and the function can be implemented by hardware or by hardware executing corresponding software. As shown in Figure 12, the AI ​​model includes a first model and a second model, the first model and the second model are deployed on the terminal device, and the second model is deployed on the device, which includes:

[0623] The receiving module 1210 is configured to receive the performance monitoring value of the second model reported by the terminal device, or

[0624] The receiving module 1210 is further configured to receive relevant information for calculating the performance monitoring value; calculate the performance monitoring value of the second model based on the relevant information; or

[0625] The receiving module 1210 is further configured to receive performance monitoring indication information.

[0626] In some embodiments, the performance monitoring value of the second model includes at least one of the following:

[0627] The performance monitoring value is determined based on first data and second data, the first data is determined based on an inference value of a first input value by the second model, and the second data is determined based on an output value corresponding to the second input value when the second model is not used, the first input value and the second input value correspond to the same input sample, the input sample is input to the first model, and the first input value is an output value of the first model for the input sample;

[0628] The performance monitoring value is determined based on third data and fourth data, the third data is determined based on the average value, maximum value or minimum value of the first data corresponding to at least two frequency domain units, and the fourth data is determined based on the average value, maximum value or minimum value of the second data corresponding to at least two frequency domain units.

[0629] In some embodiments, the relevant information of the performance monitoring value includes at least one of the following:

[0630] first data, where the first data is determined based on an inference value of a first input value by the second model, where the first input value is an output value of the first model for an input sample, and the input sample is an input of the first model;

[0631] second data, the second data being determined based on an output value corresponding to a second input value when the second model is not used, the first input value and the second input value corresponding to the same input sample;

[0632] third data, the third data being determined based on an average value, a maximum value, or a minimum value of the first data corresponding to at least two frequency domain units;

[0633] fourth data, where the fourth data is determined based on an average value, a maximum value, or a minimum value of the second data corresponding to at least two frequency domain units;

[0634] used to calculate the correlation value of the first data;

[0635] Used to calculate the correlation value of the second data.

[0636] In some embodiments, the correlation value used to calculate the first data includes: the input sample, and at least one of the first input value; the correlation value used to calculate the second data includes: the input sample, and at least one of the second input value.

[0637] In some embodiments, the first model is a CSI compression model and the second model is a CSI recovery model.

[0638] In some embodiments, the at least two frequency domain unit data N are configured by a network device; or, are predefined by the network device.

[0639] In some embodiments, the apparatus further comprises:

[0640] an operation module 1220, configured to perform a management operation of the AI ​​model based on the performance monitoring value of the second model, or to perform a management operation of the AI ​​model based on the performance monitoring indication information;

[0641] The management operations of the AI ​​model include: switching operations of the AI ​​model, activation operations of the AI ​​model, deactivation operations of the AI ​​model, and falling back to traditional communication operations that do not use the AI ​​model.

[0642] FIG13 shows a block diagram of a performance monitoring device for an AI model provided by an embodiment of the present disclosure. The device has the function of implementing the above-mentioned network-side method example, and the function can be implemented by hardware or by hardware executing corresponding software. As shown in FIG12 , the AI ​​model includes a first model and a second model, the first model is deployed on a terminal device, and the second model is deployed on a device, and the device includes:

[0643] A receiving module 1310 is configured to receive relevant information reported by the terminal device for calculating the performance monitoring value of the second model;

[0644] The calculation module 1320 is configured to calculate the performance monitoring value of the second model based on the relevant information.

[0645] In some embodiments, the relevant information of the performance monitoring value includes at least one of the following:

[0646] first data, where the first data is determined based on an inference value of a first input value by the second model, the inference value being inferred by the second model in the network device, the first input value being an output value of the first model for an input sample, and the input sample being an input of the first model;

[0647] second data, the second data being determined based on an output value corresponding to a second input value when the second model is not used, the first input value and the second input value corresponding to the same input sample;

[0648] third data, the third data being determined based on an average value, a maximum value, or a minimum value of the first data corresponding to at least two frequency domain units;

[0649] fourth data, where the fourth data is determined based on an average value, a maximum value, or a minimum value of the second data corresponding to at least two frequency domain units;

[0650] used to calculate the correlation value of the first data;

[0651] Used to calculate the correlation value of the second data.

[0652] In some embodiments, the method for calculating the correlation value of the first data includes: the input sample, and at least one of the first input value;

[0653] The correlation value used to calculate the second data includes: the input sample, and at least one of the second input value.

[0654] In some embodiments, the first model is a CSI compression model and the second model is a CSI recovery model.

[0655] In some embodiments, the at least two frequency domain unit data N are configured by a network device; or, are predefined by the network device.

[0656] In some embodiments, the apparatus further comprises:

[0657] The sending module 1330 is used to send reporting indication information to the terminal device.

[0658] An operation module 1340 is configured to perform a management operation of the AI ​​model based on the performance monitoring value of the second model, or to perform a management operation of the AI ​​model based on the performance monitoring indication information;

[0659] The management operations of the AI ​​model include: switching operations of the AI ​​model, activation operations of the AI ​​model, deactivation operations of the AI ​​model, and falling back to traditional communication operations that do not use the AI ​​model.

[0660] It should be noted that the device provided in the above embodiment only uses the division of the above-mentioned functional modules as an example to implement its functions. In actual applications, the above-mentioned functions can be assigned to different functional modules according to actual needs, that is, the content structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0661] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0662] Please refer to Figure 14, which shows a schematic diagram of the structure of a communication device (terminal device or network device) provided by an embodiment of the present disclosure. The communication device may include: a processor 1401, a receiver 1402, a transmitter 1403, a memory 1404 and a bus 1405.

[0663] The processor 1401 includes one or more processing cores. The processor 1401 executes various functional applications and performs GNSS measurements by running software programs and modules.

[0664] The receiver 1402 and the transmitter 1403 may be implemented as a transceiver 1406 , which may be a communication chip.

[0665] The memory 1404 is connected to the processor 1401 via a bus 1405 .

[0666] The memory 1404 may be used to store a computer program, and the processor 1401 may be used to execute the computer program to implement the various steps performed by the communication device in the above method embodiment.

[0667] In addition, the memory 1404 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, including but not limited to: RAM (Random-Access Memory) and ROM (Read-Only Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory or other solid-state storage technology, CD-ROM (Compact Disc Read-Only Memory), DVD (Digital Video Disc) or other optical storage, tape cassettes, magnetic tape, disk storage or other magnetic storage devices.

[0668] In a possible implementation, when the communication device is implemented as a terminal device,

[0669] The transceiver is used to report the performance monitoring value of the second model, or report relevant information used to calculate the performance monitoring value.

[0670] In a possible implementation, when the communication device is implemented as a network device,

[0671] The transceiver is configured to receive the performance monitoring value of the second model, or receive relevant information used to calculate the performance monitoring value; and calculate the performance monitoring value of the second model based on the relevant information.

[0672] An embodiment of the present disclosure also provides a computer-readable storage medium, which stores a computer program. The computer program is used to be executed by the transceiver of the core network device to implement the performance monitoring method of the AI ​​model on the terminal side.

[0673] Optionally, the computer-readable storage medium may include: a read-only memory, a random access memory, an SSD (Solid State Drives), or an optical disk, etc. Among them, the random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0674] An embodiment of the present disclosure also provides a chip, which includes a programmable logic circuit and / or program instructions. When the chip runs on a terminal device, it is used to implement the performance monitoring method of the AI ​​model on the terminal side.

[0675] An embodiment of the present disclosure also provides a computer program product or computer program, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the terminal device reads and executes the computer instructions from the computer-readable storage medium to implement the performance monitoring method of the AI ​​model on the above-mentioned terminal device side.

[0676] It should be understood that the "indication" mentioned in the embodiments of the present disclosure can be a direct indication, an indirect indication, or an indication of an association. For example, "A indicates B" can mean that A directly indicates B, for example, B can be obtained through A; it can also mean that A indirectly indicates B, for example, A indicates C, and B can be obtained through C; it can also mean that there is an association between A and B.

[0677] In the description of the embodiments of the present disclosure, the term "correspondence" may indicate a direct or indirect correspondence between the two, or an association relationship between the two, or a relationship between indication and indication, configuration and configuration, etc.

[0678] In this document, "plurality" refers to two or more. "And / or" describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates an "or" relationship between the associated objects.

[0679] In addition, the step numbers described in this document only illustrate a possible execution order between the steps. In some other embodiments, the above steps may also be executed in a non-numbered order, such as two steps with different numbers are executed simultaneously, or two steps with different numbers are executed in the opposite order to that shown in the figure. The embodiments of the present disclosure are not limited to this.

[0680] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the embodiments of the present disclosure can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0681] The above description is merely an exemplary embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure shall be included in the scope of protection of the present disclosure.

Claims

1. A performance monitoring method for an AI model, characterized in that: The method is executed by a terminal device, the AI ​​model includes a first model and a second model, the first model and the second model are deployed on the terminal device, and the second model is deployed on a network device, and the method includes: Report the performance monitoring value of the second model to the network device, or report relevant information used to calculate the performance monitoring value, or report performance monitoring indication information.

2. The method according to claim 1, characterized in that The performance monitoring value of the second model includes at least one of the following: The performance monitoring value is determined based on first data and second data, the first data is determined based on an inference value of a first input value by the second model, the second data is determined based on an output value corresponding to a second input value when the second model is not used, the first input value and the second input value correspond to the same input sample, the input sample is an input of the first model, and the first input value is an output value of the first model for the input sample; The performance monitoring value is determined based on third data and fourth data, the third data is determined based on the average value, maximum value or minimum value of the first data corresponding to at least two frequency domain units, and the fourth data is determined based on the average value, maximum value or minimum value of the second data corresponding to at least two frequency domain units.

3. The method according to claim 1, characterized in that The relevant information of the performance monitoring value includes at least one of the following: first data, where the first data is determined based on an inference value of a first input value by the second model, where the first input value is an output value of the first model for an input sample, and the input sample is an input of the first model; second data, the second data being determined based on an output value corresponding to a second input value when the second model is not used, the first input value and the second input value corresponding to the same input sample; third data, where the third data is determined based on an average value, a maximum value, or a minimum value of the first data corresponding to at least two frequency domain units; fourth data, where the fourth data is determined based on an average value, a maximum value, or a minimum value of the second data corresponding to at least two frequency domain units; Used to calculate the correlation value of the first data; Used to calculate the correlation value of the second data.

4. The method according to claim 2, characterized in that: The inferred value is inferred by the second model in the terminal device.

5. The method according to any one of claims 2 to 4, characterized in that: The correlation value for calculating the first data includes: the input sample, and at least one of the first input value; The correlation value used to calculate the second data includes: the input sample, and at least one of the second input value.

6. The method according to any one of claims 2 to 4, characterized in that: The first model is a CSI compression model, and the second model is a CSI recovery model.

7. The method according to claim 2, characterized in that The at least two frequency domain unit data N are configured by a network device; Alternatively, it is predefined by the communication protocol.

8. A performance monitoring method for an AI model, characterized in that: The method is executed by a terminal device, the AI ​​model includes a first model and a second model, the first model is deployed on the terminal device, and the second model is deployed on a network device, and the method includes: Report relevant information used to calculate the performance monitoring value of the second model to the network device.

9. The method according to claim 8, characterized in that The relevant information of the performance monitoring value includes at least one of the following: first data, where the first data is determined based on an inference value of a first input value by the second model, the inference value is inferred by the second model in the network device, the first input value is an output value of the first model for an input sample, and the input sample is an input of the first model; second data, the second data being determined based on an output value corresponding to a second input value when the second model is not used, the first input value and the second input value corresponding to the same input sample; The third data is based on the average value or maximum value of the first data corresponding to at least two frequency domain units. or minimum value determination; fourth data, where the fourth data is determined based on an average value, a maximum value, or a minimum value of the second data corresponding to at least two frequency domain units; Used to calculate the correlation value of the first data; Used to calculate the correlation value of the second data.

10. The method according to claim 8 or 9, characterized in that: The correlation value for calculating the first data includes: the input sample, and at least one of the first input value; The correlation value used to calculate the second data includes: the input sample, and at least one of the second input value.

11. The method according to claim 8 or 9, characterized in that: The first model is a CSI compression model, and the second model is a CSI recovery model.

12. The method according to claim 9, characterized in that The at least two frequency domain unit data N are configured by a network device; Alternatively, it is predefined by the communication protocol.

13. The method according to claim 8 or 9, characterized in that: The reporting of relevant information for calculating the performance monitoring value of the second model to the network device includes: Upon receiving the reporting indication information sent by the network device, reporting relevant information used to calculate the performance monitoring value of the second model to the network device; Alternatively, when a target event is triggered, relevant information used to calculate the performance monitoring value of the second model is reported to the network device.

14. The method according to claim 13, characterized in that The target event is determined according to the data distribution corresponding to the input data of the first model; or, The target event is determined according to data distribution corresponding to the output data of the first model.

15. A performance monitoring method for an AI model, characterized in that: The method is performed by a network device, the AI ​​model includes a first model and a second model, the first model and the second model are deployed on a terminal device, and the second model is deployed on the network device, and the method includes: receiving a performance monitoring value of the second model reported by the terminal device, or, receiving relevant information for calculating the performance monitoring value; calculating the performance monitoring value of the second model based on the relevant information; or, Receive performance monitoring indication information.

16. The method according to claim 15, characterized in that The performance monitoring value of the second model includes at least one of the following: The performance monitoring value is determined based on first data and second data, the first data is determined based on an inference value of a first input value by the second model, the second data is determined based on an output value corresponding to a second input value when the second model is not used, the first input value and the second input value correspond to the same input sample, the input sample is an input of the first model, and the first input value is an output value of the first model for the input sample; The performance monitoring value is determined based on third data and fourth data, the third data is determined based on the average value, maximum value or minimum value of the first data corresponding to at least two frequency domain units, and the fourth data is determined based on the average value, maximum value or minimum value of the second data corresponding to at least two frequency domain units.

17. The method according to claim 15, characterized in that The relevant information of the performance monitoring value includes at least one of the following: first data, where the first data is determined based on an inference value of a first input value by the second model, where the first input value is an output value of the first model for an input sample, and the input sample is an input of the first model; second data, the second data being determined based on an output value corresponding to a second input value when the second model is not used, the first input value and the second input value corresponding to the same input sample; third data, where the third data is determined based on an average value, a maximum value, or a minimum value of the first data corresponding to at least two frequency domain units; fourth data, where the fourth data is determined based on an average value, a maximum value, or a minimum value of the second data corresponding to at least two frequency domain units; Used to calculate the correlation value of the first data; Used to calculate the correlation value of the second data.

18. The method according to any one of claims 15 to 17, characterized in that: The correlation value for calculating the first data includes: the input sample, and at least one of the first input value; The correlation value used to calculate the second data includes: the input sample, and at least one of the second input value.

19. The method according to any one of claims 15 to 17, characterized in that: The first model is a CSI compression model, and the second model is a CSI recovery model.

20. The method according to claim 16 or 17, characterized in that The at least two frequency domain unit data N are configured by a network device; Alternatively, it is predefined by the network device.

21. The method according to any one of claims 15 to 17, characterized in that: The method further comprises: Performing a management operation of the AI ​​model based on the performance monitoring value of the second model, or performing a management operation of the AI ​​model based on the performance monitoring indication information; The management operations of the AI ​​model include: switching operations of the AI ​​model, activation operations of the AI ​​model, deactivation operations of the AI ​​model, and falling back to traditional communication operations that do not use the AI ​​model.

22. A performance monitoring method for an AI model, characterized in that: The method is performed by a network device, the AI ​​model includes a first model and a second model, the first model is deployed on a terminal device, and the second model is deployed on the network device, and the method includes: Receiving relevant information reported by the terminal device for calculating the performance monitoring value of the second model; The performance monitoring value of the second model is calculated based on the relevant information.

23. The method according to claim 22, characterized in that The relevant information of the performance monitoring value includes at least one of the following: first data, the first data is determined based on an inference value of the first input value by the second model, the inference value being The second model inference in the network device, the first input value is an output value of the first model for an input sample, and the input sample is an input of the first model; second data, the second data being determined based on an output value corresponding to a second input value when the second model is not used, the first input value and the second input value corresponding to the same input sample; third data, where the third data is determined based on an average value, a maximum value, or a minimum value of the first data corresponding to at least two frequency domain units; fourth data, where the fourth data is determined based on an average value, a maximum value, or a minimum value of the second data corresponding to at least two frequency domain units; Used to calculate the correlation value of the first data; Used to calculate the correlation value of the second data.

24. The method according to claim 22 or 23, characterized in that The correlation value for calculating the first data includes: the input sample, and at least one of the first input value; The correlation value used to calculate the second data includes: the input sample, and at least one of the second input value.

25. The method according to claim 22 or 23, characterized in that The first model is a CSI compression model, and the second model is a CSI recovery model.

26. The method according to claim 23, characterized in that The at least two frequency domain unit data N are configured by a network device; Alternatively, it is predefined by the network device.

27. The method according to claim 22 or 23, characterized in that Before receiving the relevant information reported by the terminal device for calculating the performance monitoring value of the second model, the method further includes: Sending reporting indication information to the terminal device.

28. The method according to claim 22 or 23, characterized in that The method further comprises: Based on the performance monitoring value of the second model, perform management operations on the AI ​​model, or based on the performance monitoring value of the second model, perform management operations on the AI ​​model. Measure indication information and perform management operations of the AI ​​model; The management operations of the AI ​​model include: switching operations of the AI ​​model, activation operations of the AI ​​model, deactivation operations of the AI ​​model, and falling back to traditional communication operations that do not use the AI ​​model.

29. A performance monitoring device for an AI model, characterized in that: The AI ​​model includes a first model and a second model, the first model and the second model are deployed in the device, the second model is deployed in a network device, and the device includes: A reporting module is used to report the performance monitoring value of the second model to the network device, or to report relevant information used to calculate the performance monitoring value, or to report performance monitoring indication information.

30. A performance monitoring device for an AI model, characterized in that: The AI ​​model includes a first model and a second model, the first model is deployed in the device, the second model is deployed in a network device, and the device includes: A reporting module is used to report relevant information used to calculate the performance monitoring value of the second model to the network device.

31. A performance monitoring device for an AI model, characterized in that: The AI ​​model includes a first model and a second model, the first model and the second model are deployed in a terminal device, the second model is deployed in the device, and the device includes: A receiving module, configured to receive a performance monitoring value of the second model reported by the terminal device, or The receiving module is further configured to receive relevant information for calculating the performance monitoring value; calculate the performance monitoring value of the second model based on the relevant information; or The receiving module is also used to receive performance monitoring indication information.

32. A performance monitoring device for an AI model, characterized in that: The AI ​​model includes a first model and a second model, the first model is deployed in a terminal device, the second model is deployed in the device, and the device includes: A receiving module, configured to receive relevant information reported by the terminal device for calculating the performance monitoring value of the second model; A calculation module is used to calculate the performance monitoring value of the second model based on the relevant information.

33. A terminal device, characterized in that: The terminal device comprises: processor; a transceiver connected to the processor; The processor is configured to load and execute executable instructions to implement the performance monitoring method of the AI ​​model as described in any one of claims 1 to 14.

34. A network device, characterized in that: The terminal device comprises: processor; a transceiver connected to the processor; The processor is configured to load and execute executable instructions to implement the performance monitoring method of the AI ​​model as described in any one of claims 15 to 28.

35. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one program, and the at least one program is loaded and executed by the processor to implement the performance monitoring method of the AI ​​model as described in any one of claims 1 to 28.

36. A chip, characterized in that: The chip includes a programmable logic circuit and / or program instructions, which, when running, are used to implement the performance monitoring method of the AI ​​model as described in any one of claims 1 to 28.

Images