Model performance monitoring method and apparatus, and device

Abstract

The present application relates to the field of communications, and discloses a model performance monitoring method and apparatus and a device. The method in embodiments of the present application comprises: a first device sends at least one of first information and second information to a second device, wherein the first information comprises related information of a first AI model or is used for determining the validity of the first AI model, the second information is used for determining the validity of a second model, and the second model is used for assisting in determining the related information of the first AI model.

Description

Model performance supervision method, apparatus and device

[0001] Cross-reference to Related Applications

[0002] The present application is based on the Chinese patent application No. 202411034118.6, filed on July 30, 2024, and claims the priority of the Chinese patent application No. 202411034118.6, the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0003] The present application relates to the field of communication technology, in particular to a model performance supervision method, apparatus and device. BACKGROUND

[0004] In a new radio (NR) system, an artificial intelligence (AI) model is introduced to improve system performance. For example, an AI model is introduced for positioning, beam management, channel state information (CSI) prediction, mobility management, CSI compression, etc. However, when the wireless propagation environment changes, the performance of the AI model may not be guaranteed. How to supervise the performance of the AI model is a problem to be solved. SUMMARY

[0005] Embodiments of the present application provide a model performance supervision method, apparatus and device, which can improve the accuracy of performance supervision of a first AI model by supervising the effectiveness of a second model.

[0006] In a first aspect, a model performance supervision method is provided, comprising:

[0007] The first device sends at least one of first information and second information to the second device; the first information comprises related information of a first AI model or is used to determine the effectiveness of the first AI model; the second information is used to determine the effectiveness of a second model;

[0008] The second model is used to assist in determining the related information of the first AI model

[0009] In a second aspect, a model performance supervision method is provided, comprising:

[0010] The second device receives at least one of first information and second information from the first device; the first information comprises related information of a first AI model or is used to determine the effectiveness of the first AI model; the second information is used to determine the effectiveness of a second model; wherein the second model is used to assist in determining the related information of the first AI model.

[0011] In a third aspect, a model performance supervision apparatus is provided, comprising:

[0012] a transceiving unit configured to send at least one of first information and second information to a second device, wherein the first information comprises information related to a first AI model or is used to determine validity of the first AI model, and the second information is used to determine validity of a second model;

[0013] wherein the second model is used to assist in determining the information related to the first AI model.

[0014] In a fourth aspect, a model performance supervision apparatus is provided, comprising:

[0015] a transceiving unit configured to receive at least one of first information and second information from a first device, wherein the first information comprises information related to a first AI model or is used to determine validity of the first AI model, and the second information is used to determine validity of a second model, and wherein the second model is used to assist in determining the information related to the first AI model.

[0016] In a fifth aspect, a terminal is provided, comprising a transceiver, a processor and a memory, wherein the memory stores programs or instructions executable on the processor, and the programs or instructions, when executed by the processor, implement steps of the method according to the first aspect.

[0017] In a sixth aspect, a terminal is provided, comprising a processor and a communication interface;

[0018] the communication interface is configured to send at least one of first information and second information to a second device, wherein the first information comprises information related to a first AI model or is used to determine validity of the first AI model, and the second information is used to determine validity of a second model;

[0019] wherein the second model is used to assist in determining the information related to the first AI model.

[0020] In a seventh aspect, a network-side device is provided, comprising a transceiver, a processor and a memory, wherein the memory stores programs or instructions executable on the processor, and the programs or instructions, when executed by the processor, implement steps of the method according to the second aspect.

[0021] In an eighth aspect, a network-side device is provided, comprising a processor and a communication interface;

[0022] The communication interface is configured to receive at least one of first information and second information from the first device, the first information includes related information of the first AI model or is used to determine the validity of the first AI model, and the second information is used to determine the validity of a second model, and the second model is used to assist in determining the related information of the first AI model.

[0023] In a ninth aspect, a readable storage medium is provided, and the readable storage medium stores a program or instructions, and the program or instructions are executed by a processor to implement the steps of the method in the first aspect or the steps of the method in the second aspect.

[0024] In a tenth aspect, a wireless communication system is provided, and the wireless communication system includes a terminal and a network side device, the terminal is configured to execute the steps of the method in the first aspect, and the network side device is configured to execute the steps of the method in the second aspect.

[0025] In an eleventh aspect, a chip is provided, and the chip includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is configured to run a program or instructions to implement the method in the first aspect or the method in the second aspect.

[0026] In a twelfth aspect, a computer program / program product is provided, and the computer program / program product is stored in a storage medium, and the program / program product is executed by at least one processor to implement the steps of the AI model-based CSI compression method in the first aspect or the steps of the AI model-based CSI decompression method in the second aspect.

[0027] In the embodiments of the present application, the first device sends at least one of first information and second information to the second device, the first information includes related information of the first AI model or is used to determine the validity of the first AI model, and the second information is used to determine the validity of a second model used to assist in determining the related information of the first AI model, so that the validity of at least one of the first AI model and the second model can be determined according to at least one of the first information and the second information, and therefore the accuracy of performance supervision of the first AI model can be improved by supervision of the validity of the second model. BRIEF DESCRIPTION OF DRAWINGS

[0028] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the description of the embodiments of the present application will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0029] FIG. 1 is a schematic diagram of a communication system architecture according to an embodiment of the present application;

[0030] FIG. 2 is a schematic diagram of a neural network according to an embodiment of the present application;

[0031] FIG. 3 is a schematic diagram of a neuron according to an embodiment of the present application;

[0032] FIG. 4A is a schematic diagram of AI model based CSI compression according to an embodiment of the present application;

[0033] FIG. 4B is a schematic diagram of AI model based CSI compression according to an embodiment of the present application;

[0034] FIG. 5 is a schematic flowchart of a model performance supervision method according to an embodiment of the present application;

[0035] FIG. 6 is a schematic diagram of a network architecture for model performance supervision according to an embodiment of the present application;

[0036] FIG. 7A is a schematic diagram of another model performance supervision method according to an embodiment of the present application;

[0037] FIG. 7B is a schematic diagram of another model performance supervision method according to an embodiment of the present application;

[0038] FIG. 8 is a schematic block diagram of a model performance supervision apparatus according to an embodiment of the present application;

[0039] FIG. 9 is a schematic block diagram of a model performance supervision apparatus according to an embodiment of the present application;

[0040] FIG. 10 is a schematic block diagram of a communication device according to an embodiment of the present application;

[0041] FIG. 11 is a schematic diagram of a hardware structure of a terminal according to an embodiment of the present application;

[0042] FIG. 12 is a schematic block diagram of a network side device according to an embodiment of the present application. DETAILED DESCRIPTION

[0043] The technical solutions in the embodiments of the present application will be clearly described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art belong to the scope of protection of the present application.

[0044] The terms "first", "second", and the like in the specification and claims of this application are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the terms so used are interchangeable under appropriate circumstances such that the embodiments of the present application described herein are, for example, capable of orderly or chronological progression. It is to be understood that the terms "comprises", "comprising", "includes", "including" and the like are not intended to exclude the presence of other elements or additional steps. It is to be understood that the terminology used herein is for the purpose of describing the embodiments only and is not intended to be limiting. The use of the singular is not intended to exclude the plural, and the use of the word "a" or "an" is not intended to exclude the concept of "plurality" or "one or more". It is to be understood that the terms "preferably", "preferred", "favorably", "favorable", "desirably", "desirable", "advantageously" and the like are not intended to convey an absolute necessarily desirable preferred characteristic, but merely that a feature described as preferred is more desirable than a feature not described as preferred. It is to be understood that the term "invention" encompasses both the features and the embodiments described herein, and all equivalents thereof. The application encompasses all possible combinations of the individual features described herein. It is to be understood that the terms "including", "comprising", "consisting" and "consisting essentially of" are not intended to exclude further, additional, or other elements or steps. It is to be understood that the terms "and / or" and "one or more of the preceding features" are intended to encompass the respective features individually or in combination. It is to be understood that the term "consisting essentially of" is intended to encompass the features recited in the claim and any additional feature or step, provided that the additional feature or step does not materially alter the basic and novel characteristics of the claimed application. It is to be understood that the term "consisting of" is intended to encompass only the features recited in the claim, excluding any additional feature or step. It is to be understood that the term "comprising" is intended to encompass the features recited in the claim, excluding any additional feature or step, but including the possibility of additional features or steps.

[0045] The term "indicate" in the present application can be a direct indication (or explicit indication) or an indirect indication (or implicit indication). The direct indication can be understood as that the sender explicitly informs the receiver of specific information, operation to be performed or requested result, etc. in the sent indication. The indirect indication can be understood as that the receiver determines the corresponding information according to the indication sent by the sender, or judges and determines the operation to be performed or the requested result according to the judgment result.

[0046] It is worth noting that the technology described in embodiments of the present application is not limited to Ambient Internet of Things (IoT) systems, and can be applied to other wireless communication systems, such as Long Term Evolution (LTE) / LTE-Advanced (LTE-A) systems, Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal Frequency Division Multiple Access (OFDMA), Single-carrier Frequency-Division Multiple Access (SC-FDMA), Wireless Local Area Networks (WLAN), Wireless Fidelity (WiFi), Bluetooth systems, or other systems. The terms "system" and "network" are often used interchangeably in embodiments of the present application, and the technology described can be applied to the above-mentioned systems and radio technologies, as well as other systems and radio technologies. The following description describes a New Radio (NR) system for the purpose of example, and NR terminology is used in most of the following description, but the technology can also be applied to systems other than NR systems, such as 6th Generation (6G) communication systems. th

[0047] ​FIG. 1 shows a block diagram of a wireless communication system to which embodiments of the present application can be applied. The wireless communication system includes a terminal 11 and a network-side device 12. The terminal 11 can also be referred to as a user equipment (UE). The terminal 11 can be a terminal-side device such as a mobile phone, a tablet personal computer, a laptop computer, a notebook computer, a personal digital assistant (PDA), a palmtop computer, a netbook, an ultra-mobile personal computer (UMPC), a mobile Internet device (MID), an augmented reality (AR) device, a virtual reality (VR) device, a robot, a wearable device, a flight vehicle, a vehicle user equipment (VUE), a shipboard terminal, a pedestrian user equipment (PUE), a smart home (a home device with a wireless communication function such as a refrigerator, a television, a washing machine, or furniture), a game console, a personal computer (PC), a kiosk, or a self-service machine. The wearable device includes a smart watch, a smart bracelet, a smart earphone, smart glasses, smart jewelry (a smart bracelet, a smart necklace, a smart ring, a smart necklace, a smart anklet, a smart necklace, and the like), a smart wristband, smart clothes, and the like. The vehicle-mounted device can also be referred to as a vehicle-mounted terminal, a vehicle-mounted controller, a vehicle-mounted module, a vehicle-mounted component, a vehicle-mounted chip, or a vehicle-mounted unit. It should be noted that the specific type of the terminal 11 is not limited in the embodiments of the present application.

[0048] The network-side device 12 can include an access network device or a core network device.

[0049] The access network device can also be referred to as a radio access network (RAN) device, a radio access network function or a radio access network unit. The access network device can include a base station, a wireless local area network (WLAN) access point (AP) or a wireless fidelity (WiFi) node, etc. Among them, the base station can be referred to as a node B (NB), an evolved node B (eNB), a next generation node B (gNB), a new radio node B (NR node B), an access point, a relay base station (RBS), a serving base station (SBS), a base transceiver station (BTS), a radio base station, a radio transceiver, a basic service set (BSS), an extended service set (ESS), a home node B (HNB), a home evolved node B, a transmission reception point (TRP), or some other suitable term in the art, as long as the same technical effect is achieved. The base station is not limited to a specific technical term, and it should be noted that in the embodiments of the present application, only the base station in the NR system is taken as an example for introduction, and the specific type of the base station is not limited.

[0050] The core network device can include, but is not limited to, at least one of the following: a core network node, a core network function, a mobility management entity (MME), an access and mobility management function (AMF), a session management function (SMF), a user plane function (UPF), a policy control function (PCF), a policy and charging rules function (PCRF), an edge application server discovery function (EASDF), a unified data management (UDM), a unified data repository (UDR), a home subscriber server (HSS), a centralized network configuration (CNC), a network repository function (NRF), a network exposure function (NEF), a local NEF (L-NEF), a binding support function (BSF), an application function (AF), a network data analytics function (NWDAF), a location management function (LMF), and the like. It should be noted that only the core network device in the NR system is taken as an example for introduction in the embodiments of the present application, and the specific type of the core network device is not limited.

[0051] For better understanding of the embodiments of the present application, artificial intelligence (AI) is described.

[0052] Artificial intelligence (AI) is currently widely used in various fields. It is an important task for future wireless communication networks to integrate AI into wireless communication networks to significantly improve technical indicators such as throughput, latency, and user capacity. There are various ways to implement an AI module, such as neural networks, decision trees, support vector machines, and Bayesian classifiers. The present application takes neural networks as an example for illustration, but does not limit the specific type of AI module.

[0053] A schematic diagram of a neural network can be shown in FIG. 2. The neural network is composed of neurons, and a schematic diagram of a neuron is shown in FIG. 3. In the figure, a1, a2, … aK are inputs, w is a weight (multiplicative coefficient), b is a bias (additive coefficient), and σ(.) is an activation function. Common activation functions include Sigmoid, tanh, and Rectified Linear Unit (ReLU).

[0054] The parameters of a neural network are optimized by a gradient optimization algorithm. Gradient optimization algorithms are a class of algorithms that minimize or maximize an objective function (sometimes also called a loss function), which is often a mathematical combination of model parameters and data. For example, given data X and its corresponding label Y, we construct a neural network model f(.). With the model, we can get the predicted output f(x) according to the input x, and we can calculate the difference between the predicted value and the true value (f(x)-Y), which is the loss function. Our goal is to find the appropriate W, b to minimize the value of the above loss function. The smaller the loss value, the closer our model is to the true situation.

[0055] The common optimization algorithm at present is basically based on the error back propagation (BP) algorithm. The basic idea of the BP algorithm is that the learning process consists of two processes: forward propagation of signals and backward propagation of errors. During forward propagation, input samples are transmitted from the input layer, processed layer by layer through each hidden layer, and transmitted to the output layer. If the actual output of the output layer does not match the expected output, the backward propagation of errors is entered. Error back propagation is to transmit the output error to the input layer through the hidden layer in a certain form, and allocate the error to all units in each layer, so as to obtain the error signal of each unit in each layer, which is used as the basis for correcting the weights of each unit. The process of adjusting the weights of each layer through forward propagation of signals and backward propagation of errors is repeated. The process of continuously adjusting the weights is the learning and training process of the network. This process continues until the error of the network output is reduced to an acceptable level, or until the pre-set number of learning times is reached.

[0056] Common optimization algorithms include Gradient Descent, Stochastic Gradient Descent (SGD), mini-batch gradient descent, Momentum, Nesterov (specifically Stochastic Gradient Descent with Momentum), ADAptive GRADient descent (Adagrad), Adadelta, root mean square prop (RMSprop), Adaptive Moment Estimation (Adam), and the like. When error backpropagation is performed, these optimization algorithms obtain errors / losses from a loss function, derive the gradient / derivative of the current neuron with respect to the loss, add the learning rate, the previous gradient / derivative, and the like, to obtain the gradient, and pass the gradient to the previous layer.

[0057] To better understand the embodiments of the present application, the CSI compression of a non-AI model is described.

[0058] The CSI compression of a non-AI model includes type I (type I) CSI compression, type II CSI compression, and enhanced type II (etype2) CSI compression.

[0059] 1. Type I CSI compression

[0060] Type I CSI compression reports a precoding matrix indicator (PMI) of a wideband or a subband in the case where there is no way to report a complete channel or a precoder, i.e., a two-dimensional Discrete Fourier Transform (DFT) vector and a phase rotation amount on a wideband or a subband. Type I mainly needs to report an index of a two-dimensional DFT vector and a phase rotation amount.

[0061] The reporting format of the type I CSI compression described above is as follows:

[0062] Wideband CSI: Rank Indicator (RI)-PMI-Channel Quality Indicator (CQI), where RI>4 indicates that two CQIs need to be reported for two transport blocks (TBs), and otherwise one CQI is reported.

[0063] Subband CSI:

[0064] Part 1 CSI (RI + CQI of the first TB);

[0065] Part 2 CSI (wideband CQI - wideband PMI - CQI + PMI of even subbands - CQI + PMI of odd subbands); omission principle: omit part 2 based on priority, i.e., odd subbands can be omitted first.

[0066] 2、type II CSI compression

[0067] Type II CSI compression is relative to a simple two-dimensional DFT vector and its phase rotation. The precoding vector PMI is expressed as a linear weighting of a set of basis vectors. Type 2 needs to report the index of the basis vector and the projection (amplitude and phase) on the basis vector.

[0068] The type II CSI compression reporting format is as follows:

[0069] Part 1: RI - CQI - number of non-zero wideband amplitude coefficients per layer {encoded separately};

[0070] Part 2: wideband PMI {L vector} - PMI - layer indicator (LI) {i 1,4,l (wideband amplitude 1) i 2,1,l (phase)

[0071] i 2，2，l (subband amplitude 2), the L vector is a basis vector.

[0072] Among them, amplitude 1: 3 bits (scalar); amplitude 2: 1 bit.

[0073] 3、etype II CSI compression

[0074] Since the overhead of type II CSI compression is several hundred or even several thousand bits, etype II CSI compression is a further compression of type II CSI compression, i.e., the vector composed of weighting coefficients on different subbands is further compressed into a vector composed of a set of frequency domain basis vectors.

[0075] The etype II CSI compression reporting format is as follows:

[0076] Part 1: RI - CQI - number of non-zero wideband amplitude coefficients per layer {encoded separately};

[0077] Part2: wideband PMI{vector}-PMI: i 2,4,l amplitude i 2,5,l phase and i 1,7,l , {reported bitmap}; Pri(l,i,f) = 2·L·υ·π(f) + υ·i + l,

[0078] where π(f)f is the frequency domain basis vector, π(0) = 0, π(N3-1) = 1, π(1) = 2;

[0079] where, l = 1, 2, …, υ, i = 0, 1, …, 2L-1, and f = 0, 1, …, M υ -1.

[0080] The above Part2 adopts a feedback mode of unified compression of all subbands, wherein:

[0081] 0: L spatial domain basis vectors i 1,1 , 1,2 and the strongest coefficient information of each layer {log2 2L bit} i 1,8,l (l = 1, …, υ);

[0082] 1: M frequency domain basis vectors (i 1,5 (if reported), i 1,6,l (if reported)), reference amplitude information i 2,3,l , the highest priority v2LM-[KNZ / 2] bit i 1,7,l in the position of the non-zero coefficient, v is the rank, v2LM-[KNZ / 2] coefficients with the highest priority {i 2,4,l , i 2,5,l};

[0083] 2: the coefficients with the lowest priority [KNZ / 2] in the position of the non-zero coefficient.

[0084] For better understanding of the embodiments of the present application, the CSI compression based on the AI model is described.

[0085] The necessity and feasibility of CSI monitoring options in Rel-19 are further studied in RAN-117. The monitoring on the NW-side needs to consider the overhead, delay, complexity, monitoring accuracy and UE capability.

[0086] Case 1, the target CSI reported by the UE based on the traditional eTypeII (eT2) codebook or the eT2 high-resolution codebook can be based on.

[0087] The monitoring at the UE side needs to consider the overhead, delay, complexity, monitoring accuracy and UE capability.

[0088] Case 2-1, based on the output of the CSI reconstruction model at the UE. Wherein, the CSI reconstruction model at the UE side can be the same as the actual CSI reconstruction model used by the NW side, the reference model provided by the NW or the proxy model deployed at the UE side.

[0089] Case 2-2, the intermediate Key Performance Indicator (KPI) can be estimated directly, such as Spectral Graph Convolutional Networks (SGCS), without the need to reconstruct the target CSI. Or, the monitoring output can be estimated in addition to the intermediate KPI, without the need to reconstruct the target CSI. Or, the precoding reference signal (Reference Signal, RS) based on the output of the CSI reconstruction model sent by the NW side, such as CSI-RS, DeModulation Reference Signal (DMRS). Or, it can also be based on the output of the CSI reconstruction model indicated by the NW through the traditional eT2 codebook or eT2 high-resolution codebook.

[0090] As shown in FIG. 4A, the terminal side (UE side) can encode and compress the target CSI (target CSI) through the encoding model (Encoder), such as encoding to obtain the AI-based CSI feedback value, and then report it to the network side device. The network side device performs decoding and decompression through the decoding model (Decoder), so as to obtain the reconstructed CSI. Wherein, the first AI model includes the encoding model, or the first AI model unit includes the encoding model and the decoding model.

[0091] Further, in FIG. 4A, the second model is used to monitor the function of the first AI model unit to estimate the KPI, which is used to represent the performance of the first AI model.

[0092] As shown in FIG. 4B, the terminal side can encode the target CSI (which can be represented as V) through the encoding model to obtain the AI-based CSI feedback (which can be represented as z), and then report it to the network side device. The network side device decodes and decompresses z through the decoding model, so as to obtain the reconstructed CSI (which can be represented as ). The terminal side can send the target CSI V to the network side, so that the network side can obtain the actual KPI, such as the actual SGCS, according to the reconstructed CSI The first AI model includes an encoding model, or the first AI model includes an encoding model and a decoding model.

[0093] Further, in FIG. 4B, the encoding model can also be monitored by an SGCS estimator, and an estimated KPI, such as an estimated SGCS, is output to represent the performance of the first AI model estimated by the terminal side. The SGCS estimator is an example of the second model.

[0094] Optionally, the second model can also represent the life cycle management (LCM) and AI / ML functionality of the first AI model unit. The life cycle includes at least one of model training, model deployment, model inference, model monitoring, and model updating.

[0095] The technical solutions of the present application are described in detail below through specific embodiments. The above related technologies can be combined with the technical solutions of the embodiments of the present application as optional solutions, which all belong to the protection scope of the embodiments of the present application. The embodiments of the present application include at least part of the following contents.

[0096] FIG. 5 is a schematic flowchart of a model performance supervision method 200 according to an embodiment of the present application. As shown in FIG. 5, the model performance supervision method 200 can include at least part of the following contents:

[0097] S210, the first device sends at least one of the first information and the second information to the second device; wherein the first information includes related information of the first AI model or is used to determine the validity of the first AI model; the second information is used to determine the validity of the second model; and the second model is used to assist in determining the related information of the first AI model.

[0098] S220, the second device receives at least one of the first information and the second information.

[0099] Optionally, the second device can determine the validity of at least one of the first AI model and the second model according to at least one of the first information and the second information.

[0100] It should be appreciated that FIG. 5 illustrates steps or operations of the model performance supervision method 200, but these steps or operations are examples only, and the present application can perform other operations or variations of the individual operations in FIG. 5.

[0101] In the embodiments of the present application, the first device sends at least one of the first information and the second information to the second device, the first information includes related information of the first AI model or is used to determine the effectiveness of the first AI model, and the second information is used to determine the effectiveness of a second model used to assist in determining the effectiveness of the first AI model, so that the effectiveness of at least one of the first AI model and the second model can be determined according to at least one of the first information and the second information, and therefore the embodiments of the present application can help to improve the accuracy of performance supervision of the first AI model through the supervision of the effectiveness of the second model.

[0102] In the embodiments of the present application, the first device sends at least one of the first information and the second information to the second device, the first information includes related information of the first AI model or is used to determine the effectiveness of the first AI model, and the second information is used to determine the effectiveness of a second model used to assist in determining the effectiveness of the first AI model, so that the effectiveness of at least one of the first AI model and the second model can be determined according to at least one of the first information and the second information, and therefore the embodiments of the present application can help to improve the accuracy of performance supervision of the first AI model through the supervision of the effectiveness of the second model.

[0103] It should be appreciated that the supervision result of the effectiveness of the first AI model obtained based on the embodiments of the present application can be combined with the supervision result of the effectiveness of the first AI model obtained by other model supervision methods, so as to obtain the final conclusion of the effectiveness of the first AI model.

[0104] It should be appreciated that the related information of the first AI model obtained based on the embodiments of the present application, such as the effectiveness of the first AI model, the SGCS, or the KPI of the first AI model, can be combined with the supervision method on the network side to determine the accuracy of the related information of the first AI model, so as to obtain the final conclusion of the effectiveness of the first AI model and / or the conclusion of the effectiveness of the second AI model.

[0105] In some embodiments, the first AI model can be used to implement one of the following functions: positioning, beam management, channel state information (CSI) prediction, mobility management, and CSI compression. Of course, the first AI model can also be used to implement other functions, which are not limited by the present application.

[0106] Specifically, the model performance supervision method 200 can be used to implement at least two different functions, that is, the model performance supervision method 200 can be applied to a common AI model supervision framework, so that different performance supervision schemes can be designed for different functions.

[0107] It should also be understood that, for ease of understanding, the embodiments of the present application are described by taking the function of the first AI model as CSI compression as an example, but this does not constitute a limitation on the embodiments of the present application. When the function of the first AI model is other, the performance supervision of the first AI model can still be implemented according to the scheme provided by the embodiments of the present application.

[0108] The first AI model described in the embodiments of the present application includes at least one of the following: a first AI unit used by a communication device (a first device, a second device, a network side device, a terminal, etc.); a reference AI unit of a second AI unit used by a communication device (a first device, a second device, a network side device, a terminal, etc.); a third AI unit used by a communication device (a first device, a second device, a network side device, a terminal, etc.) in testing; and a reference unit of a fourth AI unit used by a communication device (a first device, a second device, a network side device, a terminal, etc.) in testing.

[0109] The first AI model described in the embodiments of the present application can also be referred to as a first AI unit, a first AI model / first AI unit, a first machine learning (ML) model, a first ML unit, a first AI structure, a first AI function, a first AI feature, a first neural network, a first neural network function, a first neural network function, etc. Alternatively, the first AI model described in the present application can refer to a first processing unit capable of implementing a specific algorithm, formula, processing flow, capability, etc. related to AI, or the first AI model described in the present application can be a processing method, algorithm, function, module or unit for a first specific data set, or the first AI model described in the present application can be a first processing method, algorithm, function, module or unit running on an AI / ML related hardware such as a graphics processing unit (GPU), a neural processing unit (NPU), a tensor processing unit (TPU), an application specific integrated circuit (ASIC), etc. The present application does not make specific limitations. Optionally, the first specific data set includes the input or output of the first AI model.

[0110] The identifier of the first AI model in the embodiments of the present application can be an identifier of a first AI unit, an identifier of a first AI structure, an identifier of a first AI algorithm, or an identifier of a first specific data set associated with the first AI model in the present application, or a first identifier of a specific scene, environment, channel feature, or device related to the first AI model in the present application, or a first identifier of a function, property, capability, or module related to the first AI model in the present application, which is not specifically limited in the present application.

[0111] The second model in the embodiments of the present application can also be referred to as a second unit, a second model / second unit, a second machine learning (ML) model, a second ML unit, a second structure, a second function, a second property, a second neural network, a second neural network function, a second neural network function, etc., or the second model in the present application can also refer to a second processing unit capable of implementing a specific algorithm, formula, processing flow, capability, etc., or the second model in the present application can be a processing method, algorithm, function, module or unit for a second specific data set, or the second model in the present application can be a second processing method, algorithm, function, module or unit running on a GPU, NPU, TPU, ASIC, etc. AI / ML related hardware, which is not specifically limited in the present application. Optionally, the second specific data set includes input or output of the second model.

[0112] The identifier of the second model in the embodiments of the present application can be an identifier of a second unit, an identifier of a second structure, an identifier of a second algorithm, or an identifier of a second specific data set associated with the second model in the present application, or a second identifier of a specific scene, environment, channel feature, or device related to the second model in the present application, or a second identifier of a function, property, capability, or module related to the second model in the present application, which is not specifically limited in the present application.

[0113] It should be noted that the generalization ability of the first AI model is limited. A model trained based on data of one scene may not work when applied to another scene. Even a model trained based on data of the same scene may not work over time. The failure refers to a decrease in inference accuracy of the first AI model that cannot meet the target requirements. Therefore, the performance of the first AI model needs to be supervised.

[0114] In some embodiments, the first AI model can be a model in an active state, or the first AI model can be a model in an inactive state. Specifically, in the case where the first AI model is a model in an inactive state, after determining the effectiveness of the first AI model, the AI model selection can prioritize the selection of the effective first AI model from the at least two AI models, thereby benefiting the selection of the first AI model.

[0115] In some embodiments, the second model can be a model in an active state, or the second model can be a model in an inactive state. Specifically, in the case where the second model is a model in an inactive state, after determining the effectiveness of the second model, the model selection can prioritize the selection of the effective second model from the at least two models, thereby benefiting the selection of the second model.

[0116] In some embodiments, the first AI model can be sent by the second device to the first device, and optionally, the first AI model can include structure information, parameter information, and / or data information of the first AI model, and the data information can be one or more data used for training or testing the first AI model. Optionally, the data information can include at least one of a data set, type information of the data set, identification information of the data set, number information of the data, and / or specific data elements.

[0117] In some embodiments, the second AI model can be sent by the second device to the first device, and optionally, the second AI model can include structure information, parameter information, and / or data information of the second AI model, and the data information can be one or more data used for training or testing the second AI model. Optionally, the data information can include at least one of a data set, type information of the data set, identification information of the data set, number information of the data, and / or specific data elements.

[0118] In some embodiments, the first AI model and the second model can be sent by the second device to the first device, and optionally, the first AI model and / or the second model can include structure information, parameter information, and / or data information of the first AI model and / or the second model, and the data information can be one or more data used for training or testing the first AI model and / or the second model. Optionally, the structure information, parameter information, and / or data information of the first AI model and the second model can have an association relationship, such as the same type, the same data information, or the structure information, parameter information, and / or data information of the second model being a subset of the structure information, parameter information, and / or data information of the first AI model.

[0119] In some embodiments, the related information of the first AI model includes at least one of the following:

[0120] Input information of the first AI model;

[0121] Output information of the first AI model;

[0122] First validity information of the first AI model;

[0123] First validity degree of the first AI model;

[0124] Whether the input information and / or the output information associated with the first AI model is deviated;

[0125] Data deviation degree of the input information and / or the output information associated with the first AI model;

[0126] Second key performance indicator (KPI), which is used to represent a performance standard of the first AI model estimated by the first device.

[0127] Optionally, the first information can be relevance information of the first AI model, that is, the relevance information of the first AI model can be referred to as the first information.

[0128] Optionally, the first information can also be referred to as monitoring information of the first AI model, for example, the first validity information of the first AI model, the first validity degree of the first AI model, or the second key performance indicator KPI can all be understood as a kind of monitoring information. It can be understood that when the first information is the first validity information of the first AI model, the first validity degree of the first AI model, or the second key performance indicator KPI, the first device is used for model monitoring, such as terminal-side monitoring.

[0129] Optionally, the first information can also be referred to as monitoring information of the first AI model, for example, the input information of the first AI model, and when the first information is understood as the input information of the first AI model, the second device-based monitoring, such as network-side monitoring, is used.

[0130] Optionally, the first information includes whether the input information and / or the output information associated with the first AI model is deviated and / or the data deviation degree of the input information and / or the output information associated with the first AI model, which can be understood as the first information including data deviation related information, and the information can also be used to assist in monitoring the first AI model. Or the auxiliary judgment of the reason for the failure of the first AI model.

[0131] Optionally, the input information of the first AI model can include target CSI.

[0132] As an example, the target CSI can be input information of the first AI model, or input information of the first AI model can be obtained according to the target CSI. For example, the target CSI is processed to be input information of the first AI model.

[0133] Optionally, when the related information of the first AI model includes the target CSI, the first device and / or the second device can determine the effectiveness information of the first AI model and / or the second model based on the target CSI.

[0134] For example, the input information of the first AI model can include at least one of the target CSI, the quantized target CSI, the ground-truth CSI information, the target CSI information according to eType II quantization, and the like, without limitation. Optionally, the target CSI can include the ground-truth CSI.

[0135] Optionally, when the related information of the first AI model includes at least one of the quantized target CSI, the ground-truth CSI information, the target CSI information according to eType II quantization, and the like, the first device and / or the second device can determine the effectiveness information of the first AI model and / or the second model based on at least one of the quantized target CSI, the ground-truth CSI information, the target CSI information according to eType II quantization, and the like.

[0136] Optionally, the first AI model is used to determine CSI reporting information.

[0137] As an example, the CSI reporting information is CSI information reported by the first device to the second device.

[0138] As an example, the CSI reporting information can be output information of the first AI model, or the CSI reporting information can be obtained according to the output information of the first AI model. For example, the output information of the first AI model is processed to be the CSI reporting information.

[0139] Optionally, the output information of the first AI model includes feedback CSI or reconstructed CSI. That is, the CSI reporting information includes feedback CSI or reconstructed CSI.

[0140] Exemplarily, when the first AI model comprises an encoding model, the CSI reporting information comprises feedback CSI. That is, the feedback CSI is obtained according to the encoding model, for example, the feedback CSI can be output information of the encoding model, or is obtained by processing the output information of the encoding model. The processing can be at least one of quantization, truncation and reporting processing. The reporting processing can be understood as that the CSI reporting information comprises the feedback CSI and other information, such as Rank indication information, CQI information, etc.

[0141] Exemplarily, when the first AI model comprises a decoding model, the CSI reporting information comprises reconstructed CSI. That is, the reconstructed CSI is obtained according to the decoding model, for example, the reconstructed CSI can be output information of the decoding model, or is obtained by processing the output information of the decoding model.

[0142] Optionally, the first AI model is used to determine the CSI reporting information, comprising at least one of

[0143] obtaining first CSI information according to the target CSI and the first AI model;

[0144] obtaining second CSI information according to the Ground-truth CSI and the first AI model;

[0145] obtaining third CSI information according to the quantized target CSI information and the first AI model;

[0146] obtaining fourth CSI information according to the eType II quantized target CSI information and the first AI model.

[0147] For example, the target CSI can be input into the first AI model, and first CSI information is output, and the first CSI information is included in the CSI reporting information.

[0148] For another example, the Ground-truth CSI can be input into the first AI model, and second CSI information is output, and the second CSI information is included in the CSI reporting information.

[0149] For another example, the quantized target CSI information can be input into the first AI model, and third CSI information is output, and the third CSI information is included in the CSI reporting information.

[0150] For another example, the eType II quantized target CSI information can be input into the first AI model, and fourth CSI information is output, and the fourth CSI information is included in the CSI reporting information.

[0151] Optionally, the first validity information of the first AI model is used to indicate whether the first AI model is valid. For example, the first validity information includes 1 bit, and the bit is 1 to indicate that the first AI model is valid, and the bit is 0 to indicate that the first AI model is invalid. Or vice versa.

[0152] Optionally, the first validity degree of the first AI model is used to indicate the validity degree of the first AI model, such as a probability value of the validity of the first AI model, or a trust probability of the first AI model, without limitation.

[0153] Optionally, whether the input information and / or the output information associated with the first AI model is shifted, i.e., whether the input information and / or the output information associated with the first AI model is shifted relative to the first data set of the first AI model. Wherein the first AI model is obtained by training a machine learning model according to the first data set.

[0154] Exemplarily, the first data set can include N1 data, and each data can include at least two of the following information: {target CSI, CSI feedback, reconstructed CSI}.

[0155] Exemplarily, whether the input information and / or the output information associated with the first AI model is shifted can include at least one of the following:

[0156] Whether the input information associated with the first AI model is shifted relative to the target CSI in the first data set;

[0157] Whether the output information associated with the first AI model is shifted relative to the CSI feedback or the reconstructed CSI in the first data set;

[0158] Whether the input information associated with the first AI model is shifted relative to the target CSI in the first data set, and whether the output information associated with the first AI model is shifted relative to the CSI feedback or the reconstructed CSI in the first data set;

[0159] Whether the mapping relationship of the input information and the output information associated with the first AI model is shifted relative to the mapping relationship in the first data set.

[0160] Optionally, the data shift degree of the input information and / or the output information associated with the first AI model, i.e., the data shift degree of the input information and / or the output information associated with the first AI model relative to the first data set. The shift degree can be quantified as Z degrees, such as degree 0, 1, …, degree Z-1.

[0161] Optionally, the second KPI is used to represent the performance standard of the first AI model estimated by the second device using the second model.

[0162] The second model can be trained according to a second data set. The second data set can be the same as or different from the first data set.

[0163] For example, the second data set can include N2 data, each of which can include at least two of the following information: {target CSI, CSI feedback, reconstructed CSI, first KPI}.

[0164] The first KPI can represent a performance standard of the first AI model. For example, the first KPI can be an actual performance standard of the first AI model.

[0165] For example, the performance standard of the first AI model can be determined according to output information and input information of the first AI model, for example, a similarity between reconstructed CSI and target CSI of the first AI model.

[0166] For example, the similarity can be represented as SGCS (Squared Generalized Cosine Similarity).

[0167] In some embodiments, the related information of the first AI model is information determined according to the second model.

[0168] For example, at least one of the following information can be estimated according to the second model: first validity information of the first AI model, first validity degree, whether the input information and / or output information associated with the first AI model is offset, data offset degree of the input information and / or output information associated with the first AI model, and second KPI.

[0169] Therefore, according to the related information of the first AI model, such as at least one of the input information, output information, first validity information, first validity degree, whether the input information and / or output information associated with the first AI model is offset, data offset degree, and second KPI of the first AI model, the validity of the first AI model can be determined.

[0170] Optionally, when the related information of the first AI model includes the input information, output information, whether the input information and / or output information is offset, and the data offset degree of the input information and / or output information, the first validity information or the first validity degree of the first AI model can be determined according to the related information of the first AI model. Optionally, the validity of the first AI model can be determined according to the related information of the first AI model.

[0171] Optionally, when the related information of the first AI model includes whether the input information and / or the output information is shifted and the data shift degree of the input information and / or the output information, the reason for the invalidation of the first AI model can be determined according to whether the input information and / or the output information is shifted and the data shift degree of the input information and / or the output information.

[0172] Optionally, the first device or the second device can determine the validity of the first AI model according to the related information of the first AI model, which is not limited in the embodiments of the present application.

[0173] In some embodiments, the second information includes at least one of the following:

[0174] The first KPI, used to represent the performance standard of the first AI model;

[0175] The second KPI, used to represent the performance standard of the first AI model estimated by the first device;

[0176] The second validity information of the second model;

[0177] The second effective degree of the second model.

[0178] Optionally, the second information can also be referred to as the monitoring information of the second model, which is not limited.

[0179] The first KPI and the second KPI can refer to the related description in the foregoing, and will not be repeated here.

[0180] Optionally, the second validity information of the second model is used to indicate whether the second model is valid. For example, the second validity information includes 1 bit, and the bit is 1 when indicating that the second model is valid, and the bit is 0 when indicating that the second model is invalid. Or vice versa.

[0181] Optionally, the second effective degree of the second model is used to indicate the validity degree of the second model, such as a probability value of the validity of the second model, or a credibility probability of the second model, etc., which is not limited.

[0182] For example, the second information can be determined according to the related information (such as input information, output information) of the first AI model and / or the second model.

[0183] In some embodiments, the second information can also include difference information of the first KPI and the second KPI.

[0184] Therefore, according to the second information, such as at least one of the first KPI, the second KPI, the second validity information, and the second effective degree, the validity of the second model can be determined.

[0185] Optionally, when the second information includes the first KPI and the second KPI, the second effectiveness information or the second effectiveness degree of the second model can be determined according to the first KPI and the second KPI. Optionally, the effectiveness of the second model can be determined according to the second information. Optionally, the first device or the second device can determine the effectiveness of the first AI model according to the second information, and the embodiments of the present application do not limit this.

[0186] It should be understood that the first information and the second information can be reported in one message or in different messages, and the present application does not limit this.

[0187] It should be understood that the first information can be reported by the terminal to the network side device, and the second information can be sent by the network side device to the terminal.

[0188] It should be understood that the first information and the second information can be reported by the terminal to the network side device, or the first information and the second information can be sent by the network side device to the terminal.

[0189] In one embodiment, the first information can be reported again when the second information determines that the second model is effective.

[0190] In one embodiment, the first information can be reported again when the second model is effective.

[0191] In some embodiments, based on the effectiveness of the second model, the first information can be at least one of the first effectiveness information of the first AI model, the first effectiveness degree of the first AI model, whether the input information and / or the output information associated with the first AI model is offset, the data offset degree of the input information and / or the output information associated with the first AI model, and a second key performance indicator KPI.

[0192] In other embodiments, if the second model is invalid, the first information can be the input information of the first AI model and / or the output information of the first AI model.

[0193] It can be understood that the content of the first information can be different when the effectiveness state of the second model is different.

[0194] Alternatively, at the time corresponding to the period T1, the first information can be at least one of the first validity information of the first AI model, the first validity degree of the first AI model, whether the input information and / or the output information associated with the first AI model is offset, the data offset degree of the input information and / or the output information associated with the first AI model, and the second key performance indicator (KPI). At the time corresponding to the period T2, the first information can be the input information of the first AI model and / or the output information of the first AI model. Wherein T1 is less than or equal to T2, or T2 is an integer multiple of T1.

[0195] It should also be understood that the first information and the second information can be of the same type, such as both being validity information or validity degree of the model, or both being information for determining the validity of the model, such as KPI related information. Alternatively, the first information and the second information can be of different types, such as the first information being information for determining the validity of the model, such as KPI related information, and the second information being validity information or validity degree of the model.

[0196] In some embodiments, the first information further comprises at least one of:

[0197] first indication information of the first AI model;

[0198] second indication information of the second model;

[0199] a report configuration ID for indicating or configuring a CSI report, wherein the CSI report is associated with the first AI model, or the CSI report is CSI information obtained according to the first AI model;

[0200] a monitoring ID for indicating or configuring a monitoring configuration of the CSI report, wherein the monitoring configuration of the CSI report is associated with the first AI model, and / or the above-mentioned report configuration ID;

[0201] first time information, which is time information associated with the CSI report information, or time information associated with the first information.

[0202] Optionally, the first indication information of the first AI model, which can also be referred to as related information of the first AI model, such as model ID, first dataset ID, etc., is used to indicate the first AI model.

[0203] Optionally, the second indication information of the second model, which can also be referred to as related information of the second model, such as model ID, second dataset ID, etc., is used to indicate the second model.

[0204] Optionally, the Report config ID is used to indicate or configure CSI reporting associated with the first AI model, or to indicate or configure CSI reporting information obtained according to the first AI model. For example, the Report config ID included in the first information is used to indicate that the first information is associated with the CSI report indicated or configured by the Report config ID, the CSI report is associated with the first AI model, or the CSI information is obtained according to the first AI model.

[0205] Optionally, the Monitoring ID is used to indicate or configure monitoring configuration of CSI reporting associated with the first AI model. Optionally, the monitoring configuration can also be associated with the above-mentioned Report config ID. For example, the Monitoring ID included in the first information is used to indicate that the first information is associated with the monitoring configuration of the CSI reporting indicated or configured by the Monitoring ID, the CSI reporting is associated with the first AI model, or the CSI information is obtained according to the first AI model. Optionally, the CSI reporting is also associated with the Report config ID in the first information.

[0206] By carrying the report identifier, the monitoring identifier, the first indication information, the second indication information, etc. in the first information, the CSI report corresponding to the report identifier, the monitoring configuration or monitoring information corresponding to the monitoring identifier, the first AI model indicated by the first indication information, and / or the second model corresponding to the second indication information, etc. can be associated with the first information.

[0207] Optionally, the time information associated with the CSI reporting information can refer to the time related to obtaining the CSI reporting information. The time information associated with the first information can refer to the time related to obtaining the first information.

[0208] In the specific implementation mode, if the first information is associated with multiple periods of CSI, the first time information can be at least one of the start time and the end time of the multiple periods of CSI. Optionally, the first time information can also include the number of the multiple periods.

[0209] Therefore, by including at least one of the first indication information of the first AI model, the second indication information of the second model, the Report config ID, the Monitoring ID, and the first time information in the first information, embodiments of the present application can be used to assist in determining the effectiveness of a specific first AI model, or the effectiveness of the first AI model at a specific time.

[0210] In some embodiments, the second information further includes at least one of:

[0211] the above-mentioned first information;

[0212] the first indication information of the first AI model;

[0213] the second indication information of the second model;

[0214] a report identifier, the report identifier being used to indicate or configure a CSI report, the CSI report being associated with the first AI model and / or the second model, or the CSI report being CSI information obtained according to the first AI model;

[0215] a monitoring identifier, the monitoring identifier being used to indicate or configure a monitoring configuration of the CSI report, the monitoring configuration of the CSI report being associated with the first AI model and / or the second model, and / or the report identifier;

[0216] second time information, the second time information being time information associated with the second information.

[0217] Optionally, when the second information includes the first information, the related information of the first AI model or the information used to determine the validity of the first AI model can be carried in the second information, that is, as part of the second information, and is reported to the second device. It can be understood that, since the first information can be used to determine the validity of the first AI model, and can also be used to determine the validity of the second model in combination with the related information of the second model, the first information can be part of the second information. At the same time, by taking the first information as part of the second information, the first information and the second information can be associated.

[0218] Wherein, the first indication information of the AI model, the second indication information of the second model, etc. can refer to the description in the foregoing, and will not be described again.

[0219] Optionally, the Report config ID is used to indicate or configure the CSI report associated with the first AI model and / or the second model, or to indicate or configure the CSI report information obtained according to the first AI model. For example, the Report config ID included in the second information is used to indicate that the second information is associated with the CSI report indicated or configured by the Report config ID, the CSI report is associated with the first AI model and / or the second model, or is the CSI information obtained according to the first AI model.

[0220] Optionally, the Monitoring ID is used to indicate or configure the monitoring configuration of the CSI reporting associated with the first AI model. Optionally, the monitoring configuration can also be associated with the Report config ID. For example, the Monitoring ID included in the first information is used to indicate that the first information is associated with the monitoring configuration of the CSI reporting indicated or configured by the Monitoring ID, and the CSI reporting is associated with the first AI model or the CSI information obtained according to the first AI model. Optionally, the CSI reporting is also associated with the Report config ID in the first information.

[0221] In one embodiment, for the same CSI reporting information associated with the first AI model, the associated first information and the Report config ID in the second information are the same. For the same CSI reporting information associated with the first AI model, the associated first information and the Monitoring ID in the second information are the same.

[0222] By carrying the reporting identifier, the monitoring identifier, the first indication information, the second indication information, etc. in the second information, the CSI report corresponding to the reporting identifier, the monitoring configuration corresponding to the monitoring identifier, the first AI model indicated by the first indication information, the second model corresponding to the second indication information, etc. can be associated with the second information.

[0223] Optionally, the time information associated with the second information can refer to the time related to obtaining the second information.

[0224] Optionally, the second information can also be associated with the time information of the CSI reporting information. For example, the time related to obtaining the CSI reporting information.

[0225] In a specific implementation manner, if the second information is associated with multiple periodic CSIs, the second time information can be at least one of the start time and the end time of the multiple periodic CSIs. Optionally, the second time information can include the number of the multiple periods.

[0226] Therefore, by including at least one of the first information, the first indication information of the first AI model, the second indication information of the second model, the Report config ID, the Monitoring ID and the second time information in the second information, the embodiments of the present application can be used to assist in determining the validity of the second model.

[0227] In some embodiments, the second device can send third information to the first device. Correspondingly, the first device can also receive the third information from the second device. The third information includes at least one of expected CSI feedback information obtained by the second device, reconstructed CSI information, and a first KPI of the first AI model. The first device can determine the second information according to the third information.

[0228] In some embodiments, the second device can send third information to the first device. Correspondingly, the first device can also receive the third information from the second device. The third information includes at least one of expected CSI feedback information obtained by the second device, reconstructed CSI information, and a first KPI of the first AI model. The first device can determine the accuracy of the first information according to the third information.

[0229] The expected CSI feedback information, i.e., the CSI feedback information expected to be sent by the first device based on the model of the network side, can also be understood as the CSI feedback information obtained by the network side based on the model of the network side. The reconstructed CSI information, i.e., the CSI information recovered by the second device according to the received CSI feedback information. The first KPI, i.e., the performance standard of the first AI model determined by the second device.

[0230] For example, referring to FIG. 6, the first device (such as a UE) side can deploy an encoder (Encoder) 61. Optionally, the encoder 61 can be obtained according to the indication of the second device. Optionally, the first device (such as a UE) side can optimize the encoder obtained from the second device to obtain the actual used encoder 61. The second device (such as a network device) side can deploy the same or similar encoder 62 as that transmitted to the first device side, and deploy a decoder 63. The second device can also deploy a monitoring module 64 to obtain the effectiveness of the encoder 61 (or the encoder 61 and the decoder 63).

[0231] Continuing to refer to FIG. 6, the first device can input the target CSI into the encoder 61 to obtain the CSI feedback (an example of the CSI feedback information). Optionally, the first device can also send the target CSI to the second device.

[0232] The second device receives the CSI feedback and the target CSI. Then, the second device can input the target CSI into the encoder 62 to obtain the expected CSI (i.e., the expected CSI feedback). The second device can also input the CSI feedback received from the first device into the decoder 63 to obtain the reconstructed CSI #1. Optionally, the second device can also input the expected CSI into the decoder 63 to obtain the reconstructed CSI #2.

[0233] Optionally, the second device can further input the target CSI, the reconstructed CSI#1 and the reconstructed CSI#2 into the monitoring module 64 to obtain related information of the first KPI and / or the third KPI (calculated according to the CSI#2 obtained by the network side encoder 62 and the decoder 63) such as at least one of the SGCS1 and the SGCS3. For example, the monitoring module 64 can calculate the similarity between the reconstructed CSI#1 and the target CSI to obtain the SGCS1 (an example of the first KPI). For another example, the monitoring module 64 can calculate the similarity between the reconstructed CSI#2 and the target CSI to obtain the SGCS3.

[0234] As an example, the second device can send at least one of the expected CSI, the reconstructed CSI#1, the reconstructed CSI#2, the SGCS1 and the SGCS3 to the first device as the third information, so that the first device can determine the second information according to the received third information.

[0235] As an example, the second device can send the difference information of the reconstructed CSI#1 and the reconstructed CSI#2 to the first device as the third information, so that the first device can determine the second information according to the received third information.

[0236] As an example, the second device can send the difference information of the SGCS1 and the SGCS3 to the first device as the third information, so that the first device can determine the second information according to the received third information.

[0237] As an example, the second device can send the difference information of the expected CSI and the CSI feedback to the first device as the third information, so that the first device can determine the second information according to the received third information.

[0238] For example, the first device can obtain the first KPI such as the SGCS1 based on the target CSI of the local side and the reconstructed CSI#1 from the second device. For another example, the first device can directly receive the first KPI such as the SGCS1 from the network side.

[0239] Therefore, by sending the third information, i.e., at least one of the expected CSI reporting information, the reconstructed CSI information and the first KPI of the first AI model obtained by the second device to the first device, the second device can make the first device determine the second information based on the third information, thereby improving the effectiveness of the second model determined based on the second information.

[0240] In some embodiments, the third information further includes at least one of the following:

[0241] a report identifier, used for indicating or configuring CSI reporting, wherein the CSI reporting is associated with the first AI model and / or the second model, or the CSI reporting is CSI information obtained according to the first AI model;

[0242] a monitoring identifier, used for indicating or configuring monitoring configuration of the CSI reporting, wherein the monitoring configuration of the CSI reporting is associated with the first AI model and / or the second model, and / or the report identifier;

[0243] first indication information of the first AI model;

[0244] second indication information of the second model;

[0245] third information associated with third time information.

[0246] By carrying the report identifier, the monitoring identifier, the first indication information, the second indication information, etc. in the third information, the report identifier corresponding CSI report, the monitoring identifier corresponding monitoring configuration, the first indication information indicating the first AI model, the second indication information corresponding second model, etc. can be associated with the third information.

[0247] Wherein, the report identifier, the monitoring identifier, the first indication information, the second indication information and the corresponding information in the second information can be similar, which can be referred to the related description in the foregoing, and will not be repeated here.

[0248] Wherein, the third time information can be time information of obtaining the third information.

[0249] Optionally, the third time information can also be time information associated with the target CSI, such as time information of obtaining the target CSI.

[0250] Therefore, by including at least one of the identifier, the monitoring identifier, the first indication information of the first AI model, the second indication information of the second model, and the third time information in the third information, the embodiments of the present application can be used to assist in determining the second information.

[0251] In some embodiments, the first device determines the second information, i.e. determines the monitoring information of the second model, which can include determining the accuracy of the performance standard (such as the second KPI) of the first AI model output by the second model.

[0252] Optionally, the first device determines the second information according to the third information, which can include at least one of the following:

[0253] The first device determines the effectiveness of the first AI model according to the third information;

[0254] The first device compares the second KPI output by the second model with the first KPI, and determines the second information according to the comparison result.

[0255] The first device determines the effectiveness of the first AI model according to the third information, which can include that the first device determines the first KPI according to the third information. For example, when the third information includes expected CSI reporting information or reconstructed CSI information, the first device can obtain the effectiveness of the first AI model, such as the difference information between the expected CSI and the CSI feedback, or the first KPI (such as the first SGCS), based on the expected CSI reporting information or the reconstructed information, and the CSI feedback or target CSI saved at the first device. For another example, when the third information includes the first KPI or the first SGCS, the first device can directly obtain the first KPI or the first SGCS from the third information as the effectiveness of the first AI model.

[0256] Since the second KPI is a performance standard of the first AI model estimated by using the second model, and the first KPI is a standard representing the actual performance of the first AI model, by comparing the second KPI with the first KPI, the effectiveness of the second model can be determined based on the comparison result, that is, the second information is determined.

[0257] For example, when the second KPI is close to or differs little from or less than the first KPI, it is considered that the second KPI output by the second model is relatively accurate, that is, the second model is effective, or the effectiveness of the second model is high.

[0258] For another example, when the second KPI differs greatly from or more than the first KPI, it is considered that the second KPI output by the second model is not accurate enough, that is, the second model is ineffective, or the effectiveness of the second model is low.

[0259] In some embodiments, if the second KPI differs less than a first preset threshold from the first KPI, it is determined that the second model is effective; and if the second KPI differs more than a second preset threshold from the first KPI, it is determined that the second model is ineffective.

[0260] For example, if |second KPI-first KPI| is less than the first preset threshold, the second information is used to determine that the second model is effective, or it can be determined that the second model is effective.

[0261] For another example, if |second KPI-first KPI| is more than the second preset threshold, the second information is used to determine that the second model is ineffective, or it can be determined that the second model is ineffective.

[0262] The first preset threshold and the second preset threshold can be the same or different, and are not limited.

[0263] Optionally, the first device can receive at least one of the first preset threshold and the second preset threshold from the second device.

[0264] Optionally, the first device can acquire at least one of the first preset threshold and the second preset threshold according to a protocol.

[0265] In one embodiment, if the differences between the N consecutive second KPIs and the first KPI are all less than the first preset threshold, the second information is used to determine that the second model is valid, or to determine that the second model is valid. If the differences between the N consecutive second KPIs and the first KPI are all greater than the second preset threshold, the second information is used to determine that the second model is invalid, or to determine that the second model is invalid.

[0266] In one embodiment, if K1 of the N second KPIs and the first KPI have differences all less than the first preset threshold, the second information is used to determine that the second model is valid, or to determine that the second model is valid. Or, if K2 of the N second KPIs and the first KPI have differences all greater than the second preset threshold, the second information is used to determine that the second model is invalid, or to determine that the second model is invalid. Optionally, K1 and K2 can be the same; or optionally, K1 and K2 can be different.

[0267] In one embodiment, the first device determining the second information can include determining the second information in a second time window.

[0268] For example, the first device can determine the second information according to the plurality of second KPIs in the second time window, the first KPI, and the preset threshold. For another example, the first device can determine the second information according to a statistical value or an average value of the plurality of second KPIs in the second time window, a statistical value or an average value of the plurality of first KPIs, and the preset threshold.

[0269] In some embodiments, in the case of determining that the second model is valid, the first information can be acquired according to the second model, for example, a second KPI output by the second model.

[0270] For example, when the difference information between the current CSI and the CSI feedback is close or not large or less than a preset threshold, it is considered that the first AI model is valid.

[0271] For another example, when the difference information between the current CSI and the CSI feedback is large or greater than a preset threshold, it is considered that the first AI model is invalid or has a low validity.

[0272] In one embodiment, if K1 of the N difference information between the current CSI and the CSI feedback has a difference less than the first preset threshold, the second information is used to determine that the second model is valid, or to determine that the second model is valid. Or, if K2 of the N difference information between the current CSI and the CSI feedback has a difference greater than the second preset threshold, the second information is used to determine that the second model is invalid, or to determine that the second model is invalid. Optionally, K1 and K2 can be the same; or optionally, K1 and K2 can be different.

[0273] In some embodiments, if it is determined that the second model is invalid, the method 200 can further include at least one of the following:

[0274] The first device sends CSI reporting information to the second device;

[0275] The first device sends target CSI to the second device.

[0276] The first device receives at least one of reconstructed CSI, expected CSI reporting information, and the first KPI from the second device.

[0277] That is, in the case where the second model is invalid, the first information can be determined by the first device sending CSI reporting information and / or target CSI to the second device, or the first device receiving at least one of reconstructed CSI, expected CSI reporting information, and the first KPI from the second device. At this time, the first information does not need to be obtained according to the second model.

[0278] It can be understood that the CSI reporting information sent by the first device to the second device can be output information of the first AI model on the first device side, or information determined according to the output information of the first AI model on the first device side, without limitation.

[0279] The target CSI sent by the first device to the second device can be input information of the first AI model on the first device side, or the target CSI can be input information of the first AI model after processing (such as quantization), without limitation.

[0280] In some embodiments, if the second KPI satisfies the first condition, the method 200 can further include at least one of the following:

[0281] The first device sends CSI reporting information to the second device;

[0282] The first device sends target CSI to the second device.

[0283] In some embodiments, if the second KPI satisfies the first condition, the method 200 can further include:

[0284] The first device receives at least one of reconstructed CSI, expected CSI reporting information, and the first KPI from the second device.

[0285] Wherein, the second KPI satisfying the first condition can be at least one of the second KPI being less than a preset threshold, or the second KPI taking a value jump, or reporting once on consecutive C second KPIs. C is a positive integer.

[0286] Therefore, by sending at least one of the CSI reporting information and the target CSI from the first device to the second device, or receiving at least one of the reconstructed CSI, the expected CSI reporting information, and the first KPI from the second device in the case where the second model is invalid, the effectiveness of the first AI model can be determined by the second device when the second model is invalid.

[0287] In some embodiments, the first device or the second device can analyze the effectiveness of the first AI model unit and the second model according to the first KPI and the second KPI. Hereinafter, the effectiveness analysis process of the first AI model and the second model is described by taking the first KPI including the first SGCS and the second KPI including the second SGCS as an example.

[0288] Case 1: If the first SGCS is greater than the second threshold value, and the second SGCS is greater than the third threshold value, it is determined that the first AI model is effective, and the second model is effective.

[0289] Specifically, the first SGCS, i.e., the similarity between the reconstructed CSI of the first AI model and the target CSI, when it is greater than the second threshold value, indicates that the actual first AI model is effective. The second SGCS, i.e., the estimated SGCS output by the second model, when it is greater than the third threshold value, indicates that the first AI model estimated by the second model is effective. Here, the first AI model is actually effective, and the second model estimates that the first AI model is effective, so it can be understood that the second model is effective.

[0290] Optionally, in case 1, if the first SCGS and the second SGCS differ by less than the first threshold value, i.e., |first SGCS-second SGCS|<first threshold value, it indicates that the actual SGCS of the first AI model and the estimated SGCS output by the second model do not differ much, so it can be understood that the second model is effective.

[0291] Optionally, the third threshold value can be determined according to the second threshold value. For example, the third threshold value can be the same as the second threshold value, or the third threshold value is the difference between the second threshold value and the first range value, without limitation.

[0292] Case 2: If the first SGCS is greater than the second threshold value, and the second SGCS is less than the fifth threshold value, it is determined that the second model is invalid.

[0293] Specifically, the first SGCS, i.e., the similarity of the reconstructed CSI of the first AI model and the target CSI, when it is greater than the second threshold, indicates that the actual first AI model is valid. The second SGCS, i.e., the estimated SGCS output by the second model, when it is less than the sixth threshold, indicates that the first AI model estimated by the second model is invalid. Here, the first AI model is actually valid, but the second model estimates that the first AI model is invalid, so it can be understood that the validity of the first AI model is misjudged, i.e., the second model is invalid.

[0294] Case 3: If the first SGCS is greater than the second threshold, the second SGCS is less than the sixth threshold, and the difference between the first SCGS and the second SGCS is not greater than the first threshold, it is determined that the first AI model is valid, and the second model is valid.

[0295] Specifically, the first SGCS, i.e., the similarity of the reconstructed CSI of the first AI model and the target CSI, when it is greater than the second threshold, indicates that the actual first AI model is valid. The second SGCS, i.e., the estimated SGCS output by the second model, when it is less than the sixth threshold, indicates that the first AI model estimated by the second model is invalid. Here, the first AI model is actually valid, but the second model estimates that the first AI model is invalid, so it can be understood that the validity of the first AI model is misjudged. In this case, although the validity of the first AI model is misjudged by the second model, since |first SGCS-second SGCS|<first threshold, the second model is valid.

[0296] Optionally, for case 3, it can be considered that the second model misjudges, and an event report needs to be triggered. For example, the first device can report an event to the second device to indicate that the second model output misjudges.

[0297] Optionally, when the first AI model is continuously judged N times according to the second model, if there are M times of misjudgment (i.e., in accordance with the above case 3), it is considered that the second model is invalid.

[0298] Case 4, if the first SGCS is less than the fourth threshold, and the second SGCS is less than the seventh threshold, it is determined that the first AI model is invalid;

[0299] Specifically, the first SGCS, i.e., the similarity of the reconstructed CSI of the first AI model and the target CSI, when it is less than the fourth threshold, indicates that the actual first AI model is invalid. The second SGCS, i.e., the estimated SGCS output by the second model, when it is less than the seventh threshold, indicates that the first AI model estimated by the second model is invalid. At this time, it can be considered that the first AI model is invalid.

[0300] Case 5: If the first SGCS is less than the fourth threshold value, the second SGCS is less than the eighth threshold value, and the first SGCS and the second SGCS differ by no more than the first threshold value, it is determined that the first AI model is invalid and the second model is valid.

[0301] Specifically, the first SGCS, i.e., the similarity of the reconstructed CSI of the first AI model to the target CSI, when it is less than the fourth threshold value, indicates that the actual first AI model is invalid. The second SGCS, i.e., the estimated SGCS output by the second model, when it is less than the eighth threshold value, indicates that the first AI model estimated by the second model is invalid. Here, the first AI model is actually invalid, and the second model estimates that the first AI model is invalid, so it can be understood that the second model is valid. Further, in this case, since |first SGCS-second SGCS|<first threshold value, the second model is valid, and the first AI model is invalid.

[0302] Optionally, when the first AI model is continuously judged N times according to the second model whether it is valid, if M times of the estimated first AI model unit are invalid, it is considered that the first AI model is invalid.

[0303] Case 6: If the first SGCS is less than the fourth threshold value, and the second SGCS is greater than the third threshold value, the first AI model is invalid, and the second model is invalid.

[0304] Specifically, the first SGCS, i.e., the similarity of the reconstructed CSI of the first AI model to the target CSI, when it is less than the fourth threshold value, indicates that the actual first AI model is invalid. The second SGCS, i.e., the estimated SGCS output by the second model, when it is greater than the third threshold value, indicates that the first AI model estimated by the second model is valid. Here, the first AI model is actually invalid, but the second model estimates that the first AI model is valid, so it can be understood that the first AI model and the second model are both invalid.

[0305] Optionally, the fifth threshold value can be determined according to the fourth threshold value. For example, the fifth threshold value can be the same as the fourth threshold value, or the fifth threshold value is the difference between the fourth threshold value and the second range value, without limitation.

[0306] Optionally, the sixth threshold value can be determined according to the fourth threshold value. For example, the sixth threshold value can be the same as the fourth threshold value, or the sixth threshold value is the difference between the fourth threshold value and the third range value, without limitation.

[0307] Optionally, the seventh threshold value can be determined according to the fourth threshold value. For example, the seventh threshold value can be the same as the fourth threshold value, or the seventh threshold value is the difference between the fourth threshold value and the fourth range value, without limitation.

[0308] Optionally, the eighth threshold value can be determined according to the fourth threshold value. For example, the eighth threshold value can be the same as the fourth threshold value, or the eighth threshold value is the difference between the fourth threshold value and the fifth range value, without limitation.

[0309] Optionally, at least two of the fifth threshold value, the sixth threshold value, the seventh threshold value, and the eighth threshold value can be the same.

[0310] Optionally, the first threshold value to the eighth threshold value described above can be indicated by the second device to the first device, or defined by a protocol, which is not limited in the present application.

[0311] For example, the above-mentioned cases 1 to 6 can refer to the description in Table 1.

[0312] Table 1

[0313] Therefore, the embodiments of the present application can analyze the effectiveness of the first AI model and the second model according to the first KPI and the second KPI, so as to realize the supervision of the effectiveness of the first AI model by the second model, and realize the supervision of the effectiveness of the second model, and improve the accuracy of performance supervision of the first AI model.

[0314] Optionally, the threshold value can be calculated based on two parameters of threshold value and tolerance value. As described below, the tolerance value is a hysteresis parameter for determining whether the first AI model is effective or ineffective, and optionally, the tolerance value is a hysteresis parameter for determining whether the second model is effective or ineffective.

[0315] Case 1: If the first SGCS is greater than the second threshold value, and the second SGCS is greater than the third threshold value and the first tolerance value, it is determined that the first AI model is effective, and the second model is effective. The first tolerance value is a hysteresis parameter for determining whether the first AI model is effective or ineffective, and optionally, the first tolerance value is a hysteresis parameter for determining whether the second model is effective or ineffective.

[0316] Specifically, the first SGCS, i.e. the similarity of the reconstructed CSI of the first AI model and the target CSI, when it is greater than the second threshold value and the first tolerance value, indicates that the actual first AI model is effective. The second SGCS, i.e. the estimated SGCS output by the second model, when it is greater than the third threshold value and the first tolerance value, indicates that the first AI model estimated by the second model is effective. Here, the first AI model is actually effective, and the second model estimates that the first AI model is effective, so it can be understood that the second model is effective.

[0317] Optionally, in case 1, if the first SCGS and the second SGCS differ by less than the first threshold, i.e., |first SGCS-second SGCS|<first threshold±first tolerance value, it means that the actual SGCS of the first AI model does not differ much from the estimated SGCS output by the second model, and it can be understood that the second model is valid.

[0318] Optionally, the third threshold can be determined according to the second threshold. For example, the third threshold can be the same as the second threshold, or the third threshold is the difference between the second threshold and the first range value, without limitation.

[0319] Case 2: If the first SGCS is greater than the second threshold, and the second SGCS is less than the fifth threshold and the second tolerance value, it is determined that the second model is invalid. The second tolerance value is a hysteresis parameter for determining whether the first AI model is valid or invalid, which can be the same as the first tolerance value or 0-first tolerance value, or can be a parameter independent of the first tolerance value. Optionally, the second tolerance value is a hysteresis parameter for determining whether the second model is valid or invalid.

[0320] Specifically, the first SGCS, i.e., the similarity of the reconstructed CSI of the first AI model and the target CSI, when it is greater than the second threshold and the first tolerance value, indicates that the actual first AI model is valid. The second SGCS, i.e., the estimated SGCS output by the second model, when it is less than the fifth threshold and the second tolerance value, indicates that the first AI model estimated by the second model is invalid. Here, the first AI model is actually valid, but the second model estimates that the first AI model is invalid, so it can be understood that the validity of the first AI model is misjudged, i.e., the second model is invalid.

[0321] Case 3: If the first SGCS is greater than the second threshold, the second SGCS is less than the sixth threshold and the second tolerance value, and the first SCGS and the second SGCS differ by no more than the first threshold, it is determined that the first AI model is valid, and the second model is valid.

[0322] Specifically, the first SGCS, i.e., the similarity of the reconstructed CSI of the first AI model and the target CSI, when it is greater than the second threshold and the first tolerance value, indicates that the actual first AI model is valid. The second SGCS, i.e., the estimated SGCS output by the second model, when it is less than the sixth threshold and the second tolerance value, indicates that the first AI model estimated by the second model is invalid. Here, the first AI model is actually valid, but the second model estimates that the first AI model is invalid, so it can be understood that the validity of the first AI model is misjudged. In this case, although the validity of the first AI model is misjudged by the second model, since |first SGCS-second SGCS|<first threshold±third tolerance value, the second model is valid. The third tolerance value is a hysteresis parameter (such as Hysteresis) for determining whether the second model is valid or invalid.

[0323] Optionally, for case 3, it can be considered that the second model exists misjudgment, and the event reporting needs to be triggered. For example, the first device can report the event to the second device, indicating that the second model output exists misjudgment.

[0324] Optionally, when judging whether the first AI model is valid N times in succession according to the second model, if there are M times of misjudgment (i.e., in line with the above case 3), it is considered that the second model is invalid.

[0325] Case 4, if the first SGCS is less than the fourth threshold value and the fourth tolerance value, and the second SGCS is less than the seventh threshold value and the fifth tolerance value, it is determined that the first AI model is invalid; wherein the fourth tolerance value and the fifth tolerance value are hysteresis parameters (such as Hysteresis) for determining whether the first AI model is valid or invalid.

[0326] Specifically, the first SGCS, i.e., the similarity of the reconstructed CSI of the first AI model and the target CSI, when it is less than the fourth threshold value and the fourth tolerance value, indicates that the actual first AI model is invalid. The second SGCS, i.e., the estimated SGCS output by the second model, when it is less than the seventh threshold value and the fifth tolerance value, indicates that the first AI model estimated by the second model is invalid. At this time, it can be considered that the first AI model is invalid.

[0327] Case 5: if the first SGCS is less than the fourth threshold value and the fourth tolerance value, the second SGCS is less than the eighth threshold value and the fifth tolerance value, and the first SCGS and the second SGCS differ by no more than the first threshold value and the first tolerance value, it is determined that the first AI model is invalid and the second model is valid.

[0328] Specifically, the first SGCS, i.e., the similarity of the reconstructed CSI of the first AI model and the target CSI, when it is less than the fourth threshold value and the fourth tolerance value, indicates that the actual first AI model is invalid. The second SGCS, i.e., the estimated SGCS output by the second model, when it is less than the eighth threshold value and the fifth tolerance value, indicates that the first AI model estimated by the second model is invalid. Here, the first AI model is actually invalid, and the second model estimates that the first AI model is invalid, so it can be understood that the second model is valid. Further, in this case, since |first SGCS-second SGCS|<first threshold±first tolerance value, the second model is valid, and the first AI model is invalid.

[0329] Optionally, when judging whether the first AI model is valid N times in succession according to the second model, if there are M times of estimated first AI model units that are invalid, it is considered that the first AI model is invalid.

[0330] Case 6: If the first SGCS is less than the fourth threshold value and the fourth tolerance value, and the second SGCS is greater than the third threshold value and the first tolerance value, the first AI model is invalid, and the second model is invalid.

[0331] Specifically, the first SGCS, i.e., the similarity of the reconstructed CSI of the first AI model and the target CSI, when it is less than the fourth threshold value and the fourth tolerance value, indicates that the actual first AI model is invalid. The second SGCS, i.e., the estimated SGCS output by the second model, when it is greater than the third threshold value and the first tolerance value, indicates that the second model estimates that the first AI model is valid. Here, the first AI model is actually invalid, but the second model estimates that the first AI model is valid, so it can be understood that both the first AI model and the second model are invalid.

[0332] Optionally, the fifth threshold value can be determined according to the fourth threshold value. For example, the fifth threshold value can be the same as the fourth threshold value, or the fifth threshold value is the difference between the fourth threshold value and the second range value, without limitation.

[0333] Optionally, the sixth threshold value can be determined according to the fourth threshold value. For example, the sixth threshold value can be the same as the fourth threshold value, or the sixth threshold value is the difference between the fourth threshold value and the third range value, without limitation.

[0334] Optionally, the seventh threshold value can be determined according to the fourth threshold value. For example, the seventh threshold value can be the same as the fourth threshold value, or the seventh threshold value is the difference between the fourth threshold value and the fourth range value, without limitation.

[0335] Optionally, the eighth threshold value can be determined according to the fourth threshold value. For example, the eighth threshold value can be the same as the fourth threshold value, or the eighth threshold value is the difference between the fourth threshold value and the fifth range value, without limitation.

[0336] Optionally, at least two of the fifth threshold value, the sixth threshold value, the seventh threshold value, and the eighth threshold value can be the same.

[0337] Optionally, all of the fifth threshold value, the sixth threshold value, the seventh threshold value, and the eighth threshold value can be the same.

[0338] Optionally, at least two of the first tolerance value, the second tolerance value, the third tolerance value, and the fourth tolerance value can be the same.

[0339] Optionally, the tolerance values for determining the difference of the SGCS are all the same, such as the first tolerance value and the third tolerance value being the same.

[0340] It should be noted that the above describes the effectiveness process of the first AI model and the second model by taking KPI including SGCS as an example, but the embodiments of the present application are not limited thereto. For example, SGCS can also be replaced by other performance standards. It should be understood that after SGCS is replaced by other performance standards, the effectiveness process of the first AI model and the second model is similar to the above analysis process, and can refer to the related description in the above, or be appropriately transformed, which are all within the protection scope of the embodiments of the present application.

[0341] In some embodiments, the first device can further receive third indication information from the second device, the third indication information being used to indicate at least one of the following:

[0342] the effectiveness of the second model;

[0343] activation or deactivation of the second model;

[0344] switching of the monitoring scheme of the first AI model;

[0345] reduction of the reporting period of the target CSI.

[0346] For example, after the second device receives at least one of the first information and the second information, the second device can determine the above third indication information according to at least one of the first information and the second information, and send the third indication information to the first device.

[0347] For example, the third indication information can indicate whether the second model is effective, such as bit 1 indicating that the second model is effective and bit 0 indicating that the second model is ineffective. Or vice versa.

[0348] For another example, the third indication information can deactivate the second model when the second model is ineffective or has poor performance.

[0349] It should be understood that if the first AI model is effective and the second model is ineffective (i.e., only the second model is ineffective), only the second model is deactivated.

[0350] Optionally, if the network side deactivates the first AI model, the second model is also deactivated.

[0351] For another example, the third indication information can switch the monitoring scheme of the first AI model when the second model is ineffective or has poor performance, such as switching to not using the second model to determine the related information of the first AI model.

[0352] For example, the third indication information can reduce (or encrypt) the reporting period of the target CSI when the second model is invalid or has poor performance. It can be understood that the target CSI is associated with the input information of the first AI model. When the second model is invalid or has poor performance, the reporting period of the target CSI is reduced or encrypted, more target CSI is reported to the second device, the target CSI can be used by the second device to determine the related information or effectiveness of the first AI model, thereby improving the reliability of CSI reporting.

[0353] Therefore, the embodiments of the present application can indicate the effectiveness of the second model by the second device when the second model is invalid or has poor performance, or activate or deactivate the second model, or switch the monitoring scheme of the first AI model, or reduce the reporting period of the target CSI, so that the first device can use a more reliable performance supervision scheme for the first AI model when the second model is invalid or has poor performance, thereby improving the accuracy of performance supervision of the first AI model.

[0354] In some embodiments, the first device can also receive fourth information from the second device, and the fourth information includes at least one of the following:

[0355] The monitoring type of the first AI model;

[0356] The monitoring reporting period of the first AI model;

[0357] The first time window for obtaining the input information of the first AI model;

[0358] The reporting identifier related to the output information of the first AI model associated with the fourth information;

[0359] The monitoring identifier related to the output information of the first AI model associated with the fourth information;

[0360] The first indication information of the first AI model;

[0361] The second time window for obtaining the input information of the second model;

[0362] The second indication information of the second model.

[0363] For example, the second device can send fourth information to the first device for configuring the monitoring information of the first AI model. Optionally, the fourth information can also be referred to as monitoring configuration information. Correspondingly, the first device receives the fourth information and supervises the performance of the first AI model according to the fourth information.

[0364] Optionally, the monitoring type of the first AI model can include the following three types:

[0365] Type 1: reporting target CSI. The target CSI is reported by the first device to the second device, and the second device can obtain the related information or effectiveness of the first AI model, such as the first information, according to the target CSI and reconstructed CSI on the second device side.

[0366] Type 2: based on a second model. That is, the related information or effectiveness of the first AI model, such as the first information, is determined with the assistance of the second model. For example, the second model can estimate whether the first AI model is effective.

[0367] Type 3: receiving reconstructed CSI. The reconstructed CSI is received by the first device from the second device, and the first device can obtain the related information or effectiveness of the first AI model, such as the first information, according to the reconstructed CSI and the target CSI on the first device side.

[0368] It should be understood that the monitoring type of the first AI model can also include other types in addition to the above three types, and the embodiments of the present application do not limit this.

[0369] Optionally, the monitoring reporting period can include the reporting period of the first information and / or the second information.

[0370] Optionally, the first time window can be a time window for obtaining the target CSI. Alternatively, the related information or effectiveness of the first AI model, such as the first information, can be obtained based on multiple target CSIs in the first time window.

[0371] Optionally, the report identifier (Report config ID) is related to the output information of the first AI model associated with the fourth information, that is, the report identifier is used to indicate which CSI reporting information the fourth information is for.

[0372] Optionally, the monitoring identifier (Monitoring ID) is related to the output information of the first AI model associated with the fourth information, that is, the monitoring identifier is used to indicate which monitoring information the fourth information is associated with.

[0373] Optionally, the second time window can be a time window used by the second information. For example, when the monitoring type of the first AI model is the above type 2, the effectiveness of the second model, that is, the second information, can be obtained based on the second time window.

[0374] Optionally, the duration of the second time window is greater than that of the first time window, so that the performance of the first AI model can be supervised within a complete period of obtaining CSI reporting information by the first AI model according to the target CSI.

[0375] Optionally, the first indication information and the second indication information can refer to the related description in the foregoing, and will not be described herein.

[0376] Therefore, by receiving the fourth information by the first device, the monitoring type, the monitoring reporting period, the monitoring time window, etc. of the first AI model can be configured, so as to realize the supervision of the performance of the first AI model based on the configuration.

[0377] Optionally, in the embodiments of the present application, the time window involved can include at least one of period information and window length information. Optionally, the window length information can be an integer multiple of the period of the reference signal for obtaining the CSI related information.

[0378] In some embodiments, the first device can further obtain fifth information including at least one of input information and output information of the first AI model. The first device obtains the first information by using the fifth information and the second model.

[0379] For example, the first device can input the fifth information into the second model to obtain the first information according to the output of the second model. That is, the second device can map the fifth information to obtain the related information or effectiveness of the first AI model.

[0380] For example, the fifth information can include at least one of the following:

[0381] One or more input information of the first AI model, such as one or more target CSI, or one or more information obtained according to the target CSI;

[0382] One or more output information of the first AI model, such as one or more CSI feedback information (or CSI reporting information), or one or more information obtained according to the CSI feedback information (or CSI reporting information).

[0383] For example, the fifth information can include at least one of the following:

[0384] One or more input information of one or more specified layers of the first AI model, such as one or more target CSI of one or more layers, or one or more information obtained according to the target CSI of one or more layers;

[0385] One or more output information of one or more layers of the first AI model, such as one or more CSI feedback information (or CSI reporting information) of one or more layers, or one or more information obtained according to the CSI feedback information (or CSI reporting information) of one or more layers.

[0386] Optionally, the terminal obtains the fifth information by measuring the reference signal to obtain one or more input information of the first AI model, and / or obtaining one or more output information according to the first AI model.

[0387] Therefore, by using at least one of the input information and the output information of the first AI model and the second model to obtain the first information, the effectiveness of the first AI model can be supervised by the second model, so as to improve the accuracy of performance supervision of the first AI model.

[0388] In some embodiments, the first device uses the fifth information and the second model to obtain the first information, including at least one of:

[0389] At least one of the input information and the output information of the first AI model is input into the second model to obtain the first information output by the second model;

[0390] At least one of the input information and the output information of the first AI model at K time points and the output information at the K time points is input into the second model to obtain the first information output by the second model at the K time points; K is a positive integer;

[0391] At least one of the input information and the output information associated with the first time of the first AI model and the first information obtained before the first time is input into the second model to obtain the first information associated with the first time output by the second model; wherein the first information obtained before the first time is associated with the first information at the K time points.

[0392] For example, the second model can learn the mapping relationship between at least one of the input information and the output information of the first AI model and the related information or effectiveness of the first AI model, so that in the deployment stage of the second model, by inputting at least one of the input information and the output information of the first AI model, the related information or effectiveness of the first AI model, i.e. the first information, can be output.

[0393] When the first AI model is a time spatial frequency (TSF) compression type, at least one of the input information and the output information at the K time points or at least one of the input information and the output information associated with the first time can be used to obtain the first information according to the second model.

[0394] For example, when the target CSI at the K time points and / or the CSI feedback at the K time points of the first AI model are input into the second model, the monitoring information of the first AI model associated with the target CSI at the K time points can be obtained.

[0395] For another example, the target CSI and / or the CSI feedback associated with time N (an example of the first time) and the first information (such as the first KPI or the second KPI) obtained before time N can be input into the second model to obtain the monitoring information of the first AI model associated with time N.

[0396] Optionally, the monitoring information (e.g., the first KPI or the second KPI) obtained at time N is associated with the target CSI of K times.

[0397] For example, the monitoring information of the first AI model associated with the target CSI of K times at time N can be the monitoring information of the first AI model at time N, i.e., the monitoring information of the first AI model at time N has the relevant information of K times, or the monitoring information of the first AI model associated with the target CSI of K times is the comprehensive or statistical monitoring information of the K times.

[0398] Optionally, in training the second model, each data in the second data set needs to include K1 target CSI information of K1 times. Optionally, the K1 target CSI information is associated with one KPI information (e.g., the first KPI). Wherein, K1 is a positive integer. Optionally, the K1 target CSI information is associated with M1 KPI information (e.g., the first KPI). Wherein, M1 is a positive integer.

[0399] As an example, M1 is equal to K1, i.e., M1 = K1; or M1 is equal to K1 minus one, i.e., M1 = K1-1; or M1 is less than K1, i.e., M1 < K1.

[0400] Therefore, by inputting the time series features of the first AI model into the second model, the embodiments of the present application can predict the monitoring information of the first AI model, and can convert the time series features into statistical features through the second model, thereby supervising the effectiveness of the first AI model, and can be beneficial to improve the accuracy of performance supervision of the first AI model.

[0401] In some embodiments, if the rank of the input information of the first AI model or the rank of the target CSI information is greater than 1, the first information includes at least one of the following;

[0402] The second model outputs Z second KPIs, wherein Z is determined according to the rank of the input information or the target CSI, and Z is a positive integer greater than 1;

[0403] The second model outputs one second KPI, wherein the second KPI is obtained in combination with the KPIs of Z layers.

[0404] For example, if the first AI model is a high-rank (e.g., rank greater than 1) model, the rank of the input information of the first AI model or the target CSI is greater than 1, and at this time, the first information obtained according to the second model satisfies the above description.

[0405] For example, when the input information or target CSI of the first AI model is input into the second model, since the rank of the input information or target CSI is greater than 1, the second model can output Z second KPIs. Wherein, the value of Z is determined according to the rank of the input information or target CSI, for example, Z can be equal to the value of the rank. The Z second KPIs can be used as the first information.

[0406] For another example, when the input information or target CSI of the first AI model is input into the second model, since the rank of the input information or target CSI is greater than 1, the second model can output 1 second KPI at this time, and the second KPI is obtained according to the KPIs of Z layers. Optionally, the second KPI can be represented by synthesizing the KPIs of the Z layers. Optionally, the second KPI can be the average of the KPIs of the Z layers. For example, the second model outputs Z KPIs, and the first device can convert the Z KPIs into one KPI information, i.e., the final second KPI, through processing.

[0407] Therefore, by inputting the high-rank input information or target CSI of the first AI model into the second model, the embodiments of the present application can realize the prediction of the monitoring information of the first AI model by the second model according to the high-dimensional input information or target CSI, thereby supervising the effectiveness of the first AI model, and can be beneficial to improve the accuracy of performance supervision of the first AI model.

[0408] In some embodiments, the first device can further perform at least one of the following:

[0409] obtain the second model according to the second data set;

[0410] obtain the second model according to the first AI model;

[0411] obtain the second model according to the second data set and the first AI model;

[0412] obtain the second model according to at least one of the parameters and the structure of the second model from the second device.

[0413] For example, the first device can obtain the second data set used to obtain the second model, and further obtain the second model according to the second data set. Optionally, the first device can further obtain the first data set used to obtain the first AI model. Wherein, the first data set is the same as or different from the second data set.

[0414] For example, the first data set can include N1 data, and each data can include at least two of the following information: {target CSI, CSI feedback, reconstructed CSI}.

[0415] Exemplarily, the second data set can include N2 data, each of which can include at least two of the following information: {target CSI, CSI feedback, reconstructed CSI, first KPI}.

[0416] Optionally, the data in the second data set can be processed to obtain training samples, and the second model can be trained.

[0417] For example, when the second data set includes target CSI and reconstructed CSI, but does not include first KPI, the first KPI can be obtained according to the target CSI and the reconstructed CSI when constructing the training samples based on the second data set, and the second model can be trained as the training samples.

[0418] Optionally, the second data set includes at least one of the input information of the first AI model, the reconstructed information of the output information of the first AI model, and the first KPI of the first AI model.

[0419] Exemplarily, the input information of the first AI model can be target CSI or information obtained according to target CSI. The reconstructed information of the output information of the second AI model, such as reconstructed CSI or information obtained according to reconstructed CSI.

[0420] Exemplarily, the second model can be obtained from the first AI model through model conversion, transfer learning, model optimization, etc. For example, when the first AI model and the second model are trained based on different machine learning frameworks, the parameters and structures of the first AI model can be read, and the second model can be reconstructed in a new framework and loaded with parameters. For another example, most parameters of the first AI model can be kept unchanged, and only the last few layers of the model can be fine-tuned to obtain the second model. For another example, the second model can be obtained by adjusting the hyperparameters of the first AI model, such as learning rate, batch size, regularization coefficient, etc.

[0421] Exemplarily, the first device can obtain the second model according to the second data set in combination with the first AI model. For example, at least one of the machine learning framework, parameters, structure, etc. of the first AI model can be used, and the second model can be obtained by training using the second data set.

[0422] Optionally, the second device can send at least one of the parameters and structure of the second model to the first device, so that the first device obtains the second model according to at least one of the parameters and structure of the second model from the second device.

[0423] In an implementation manner, the second device can send the parameters of the second model to the first device, and the first device can obtain the second model according to the structure of the first AI model or the AI model agreed by the protocol.

[0424] Optionally, the second device can send the first device the related information of the first AI model and the second model. For example, at least one of the model parameters and the model structure of the second model. Correspondingly, the first device receives the related information of the first AI model and the second model.

[0425] Optionally, the second device can send the first device the second model and the association relationship between the second model and the first AI model, which can be used to indicate that the second model is used to determine the related information or effectiveness of the first AI model. That is, the second information is used to indicate that the second model is used to monitor the associated first AI model.

[0426] In some embodiments, the first device can obtain the first AI model based on the first data set. For example, the first device receives the first data set from the second device, and trains the first AI model according to the first data set.

[0427] In some embodiments, the first device can obtain the first AI model based on the first reference model. For example, the first device receives the related information of the first reference model, such as at least one of the model parameters and the model structure, from the second device, and obtains the first AI model according to the first reference model. For example, the first AI model can be the same as the first reference model.

[0428] In some embodiments, the first AI model can be selected from a first model candidate list. For example, the first model candidate list can include the model structure and / or model parameters of at least one candidate model. Optionally, the second device can indicate to the first device to select a first candidate model in the first model candidate list as the first AI model.

[0429] In some embodiments, the second model can be selected from a second model candidate list. For example, the second model candidate list can include the model structure and / or model parameters of at least one candidate model. Optionally, the second device can indicate to the first device to select a second candidate model in the second model candidate list as the second model.

[0430] Optionally, the second model has fewer parameters (such as hyperparameters) than the first AI model.

[0431] Optionally, the second model is smaller than the first model, for example, the structure is smaller than the first AI model.

[0432] In some embodiments, if the first AI model is updated, the second model also needs to be updated. Or, if the network finds that the second KPI (such as the second SGCS) decreases, the second model service can be updated.

[0433] In some embodiments, the relationship between the first AI model and the second model is as follows:

[0434] 1) The input information of the second model is the input information of the plurality of first AI models (the model B input is multiple model A input); wherein, the model B is the second model, and the model A is the first model.

[0435] 2) The output information of the second model is at least one of the following: whether the first AI model is valid, and the intermediate KPI of the first AI model.

[0436] That is, the second model can be used to detect whether the first AI model is valid, or to detect the intermediate KPI (such as the second KPI) of the first AI model.

[0437] In some embodiments, for the monitoring information of the second model, the first device can receive the monitoring result from the second device, cross-check the monitoring of the second model on the first device side, and reduce the reporting amount of the ground-truth CSI.

[0438] In summary, in the embodiments of the present application, the first device sends at least one of the first information and the second information to the second device, the first information includes related information of the first AI model or is used to determine the validity of the first AI model, and the second information is used to determine the validity of the second model for assisting in determining the related information of the first AI model, so that the validity of at least one of the first AI model and the second model can be determined according to at least one of the first information and the second information. Therefore, the embodiments of the present application can improve the accuracy of performance supervision of the first AI model by supervising the validity of the second model.

[0439] FIG. 7A is a schematic flowchart of another model performance supervision method according to an embodiment of the present application. Wherein, the terminal can be an example of the first device described above, and the network can be an example of the second device described above. As shown in FIG. 7A, the model performance supervision method can include at least part of the following contents:

[0440] S1, the network sends second dataset information to the terminal for obtaining a second model.

[0441] For example, the second dataset information can include the ID of the second dataset.

[0442] Optionally, the network can also send first dataset information to the terminal for obtaining a first AI model.

[0443] For example, the first dataset information can include the ID of the first dataset.

[0444] Optionally, the second data set can be the same as or different from the first data set used to obtain the first AI model, and is not limited.

[0445] Specifically, the first data set and the second data set can refer to the related description in the foregoing, which will not be described here again.

[0446] S2, the terminal obtains the second model and the first AI model.

[0447] For example, the terminal can train the machine learning model according to the first data set to obtain the first AI model, and train the machine learning model according to the second data set to obtain the second AI model. Optionally, the terminal can obtain the second model according to the first AI model and the second data set.

[0448] Specifically, the process of obtaining the second model and the first AI model can refer to the related description in the foregoing, which will not be described here again.

[0449] S3, the terminal reports the second model related information to the network.

[0450] For example, the second model related information can include at least one of the following: input information of the second model, output information of the second model, validity information of the second model, validity degree of the second model, whether the input information and / or output information associated with the second model is offset, and data offset degree of the input information and / or output information associated with the second model.

[0451] S4, the network configures or triggers the terminal to use the first AI model and / or the second model.

[0452] For example, the network can send monitoring configuration information (such as the fourth information in the foregoing) to the terminal to configure or trigger the terminal to use the first AI model and / or the second model.

[0453] For example, when the monitoring configuration information configures the monitoring type as type 2: based on the second model, the terminal assists the first AI model through the related information or validity information of the second model, that is, the second model can estimate whether the first model is valid.

[0454] S5, the terminal obtains first reporting information according to the first AI model.

[0455] For example, the terminal can input the target CSI or information obtained according to the target CSI into the first AI model, and obtain the CSI feedback as the first reporting information according to the output information of the first AI model. Optionally, the first reporting information further includes at least one of rank information, indication information of the first AI model, or CQI information.

[0456] S6a, the terminal reports the first reporting information.

[0457] S7, the terminal obtains the first information according to the second model.

[0458] For example, the terminal can determine the related information of the first AI model or the effectiveness of the first AI model (i.e., the first information) with the aid of the second model. Specifically, the first information can refer to the related description above, which will not be repeated here.

[0459] S6b, the terminal reports the first information.

[0460] Specifically, the terminal reports the first information to the network. Optionally, the first information can be regarded as monitoring information, and optionally, the terminal can determine the monitoring information of the first AI model.

[0461] S6c, the terminal reports the second report information.

[0462] Here, the terminal can send the input information of the first AI model or the associated information of the input information, such as the target CSI or the information obtained according to the target CSI, to the network as the second report information. Optionally, the second report information can be regarded as monitoring information.

[0463] S9a, the network determines the monitoring information of the first AI model and / or the second model.

[0464] For example, the network can determine the monitoring information of the first AI model and / or the second model according to at least one of the first information and the second report information.

[0465] For example, when the first information includes the second KPI estimated by the second model, the network can determine the monitoring information (i.e., the effectiveness) of the first AI model according to the second KPI. The network can obtain the first KPI according to the similarity between the target CSI and the reconstructed CSI, and then determine the monitoring information (i.e., the effectiveness) of the second model according to the difference between the second KPI and the first KPI.

[0466] It should be understood that through the above steps S6c and S9a, the terminal reports the second report information to the network, so that the network can determine the monitoring information of the first AI model and / or the second AI model according to the first information and the second report information.

[0467] S8, the network sends the third information.

[0468] For example, the third information can include the first KPI calculated by the network or the reconstructed CSI. For example, the network can obtain the first KPI according to the similarity between the reconstructed CSI and the target CSI.

[0469] S9b, the terminal determines the monitoring information of the first AI model and / or the second model.

[0470] For example, the terminal can determine the monitoring information of the first AI model and / or the second model based on at least one of the first information and the third information.

[0471] For example, the terminal can determine the monitoring information (i.e., effectiveness) of the first AI model based on the second KPI estimated by the second model. When the third information includes the first KPI, the terminal can determine the monitoring information (i.e., effectiveness) of the second model based on the difference between the first KPI and the second KPI. When the third information includes the reconstructed CSI, the terminal can determine the first KPI based on the similarity between the reconstructed CSI and the target CSI, and then determine the monitoring information (i.e., effectiveness) of the second model based on the difference between the first KPI and the second KPI.

[0472] S10, the terminal reports the second information.

[0473] For example, when the terminal determines the monitoring information of the second model (i.e., performs the above step S9b), the terminal reports the second information, so that the network can directly determine the validity of the second model based on the second information.

[0474] It should be understood that through the above steps S8, S9b and S10, the network sends third information to the terminal, so that the terminal can determine the monitoring information of the first AI model and / or the second AI model based on the third information and the first information.

[0475] Therefore, in this embodiment, the terminal sends at least one of first information and second information to the network. The first information includes relevant information about the first AI model or information used to determine the validity of the first AI model. The second information is used to determine the validity of a second model used to assist in determining the relevant information of the first AI model. Thus, the terminal or network can determine the validity of at least one of the first AI model and the second model based on at least one of the first and second information. Therefore, by supervising the validity of the second model, this embodiment can improve the accuracy of performance supervision of the first AI model.

[0476] Figure 7B is a schematic flowchart of another model performance supervision method according to an embodiment of this application. The terminal can be an example of the first device described above, and the network can be an example of the second device described above. As shown in Figure 7B, the model performance supervision method may include at least some of the following:

[0477] X1, the terminal reports the first reported information, which includes the CSI feedback obtained by the terminal based on the target CSI and the first AI model.

[0478] Exemplarily, the terminal can input the target CSI or information obtained according to the target CSI into the first AI model, and obtain the CSI feedback as the first reported information according to the output information of the first AI model. Optionally, the first reported information further includes at least one of rank information, indication information of the first AI model, or CQI information

[0479] X2, the terminal sends first information to the network side device, wherein the first information includes related information of the first AI model.

[0480] Optionally, the first information includes at least one of:

[0481] Information related to the target CSI:

[0482] Quantized target CSI information;

[0483] Ground-truth CSI information;

[0484] Target CSI information according to eType II quantization.

[0485] Optionally, the first information includes at least one of second CSI information, third CSI information or fourth CSI information determined according to the above-mentioned information related to the target CSI, such as:

[0486] Second CSI information obtained according to the Ground-truth CSI information and the first AI model;

[0487] Third CSI information obtained according to the quantized target CSI information and the first AI model;

[0488] Fourth CSI information obtained according to the target CSI information based on eType II quantization and the first AI model.

[0489] X3, the network side device determines the related information of the first AI model according to the above-mentioned first information and the first AI model of the network side.

[0490] Exemplarily, the network side device determines the related information of the first AI model according to the above-mentioned first information and the first AI model of the network side. Optionally, the first AI model of the network side includes an encoder and a decoder, respectively determining expected CSI feedback information and reconstructed CSI information #1.

[0491] Optionally, the network side device determines reconstructed CSI information #2 according to at least one of the second CSI information, the third CSI information or the fourth CSI information of the above-mentioned first information and the decoder of the network side. The effectiveness of the UE side model is determined by comparing the reconstructed CSI information #1 and the reconstructed CSI information #2.

[0492] Optionally, the network-side device determines reconstructed CSI information #3 according to the first reported information and a decoder of the network side, and the CSI information in the first information determines the effectiveness of the first AI model.

[0493] Therefore, the embodiments of the present application send the first reported information and the first information to the network by the terminal, and the first information includes the related information of the first AI model, such as target CSI, CSI reported information, etc., so that the network can determine the related information of the first AI model, such as expected CSI feedback, reconstructed CSI, or the effectiveness of the first AI model, according to at least one of the first reported information and the first information. Therefore, the embodiments of the present application can be beneficial to improving the accuracy of performance supervision of the first AI model.

[0494] FIG. 8 shows a schematic block diagram of the apparatus 300 according to an embodiment of the present application. As shown in FIG. 8, the model performance supervision apparatus 300 includes:

[0495] The transceiver unit 310 is configured to send at least one of the first information and the second information to the second device; the first information includes the related information of the first AI model or is used to determine the effectiveness of the first AI model; and the second information is used to determine the effectiveness of the second model.

[0496] The second model is used to assist in determining the related information of the first AI model.

[0497] Optionally, the related information of the first AI model includes at least one of the following:

[0498] The input information of the first AI model;

[0499] The output information of the first AI model;

[0500] The first effectiveness information of the first AI model;

[0501] The first effectiveness degree of the first AI model;

[0502] Whether the input information and / or the output information associated with the first AI model is offset;

[0503] The data offset degree of the input information and / or the output information associated with the first AI model;

[0504] The second key performance indicator (KPI), which is used to represent the performance standard of the first AI model estimated by the first device.

[0505] Optionally, the related information of the first AI model is information determined according to the second model.

[0506] Optionally, the first AI model is used to determine first CSI reporting information.

[0507] Optionally, the second information comprises at least one of:

[0508] a first KPI, the first KPI being used to represent a performance criterion of the first AI model;

[0509] a second KPI, the second KPI being used to represent a performance criterion of the first AI model estimated by the first device;

[0510] second validity information of the second model;

[0511] a second validity degree of the second model.

[0512] Optionally, the first information further comprises at least one of:

[0513] first indication information of the first AI model;

[0514] second indication information of the second model;

[0515] a reporting identifier, the reporting identifier being used to indicate or configure CSI reporting, wherein the CSI reporting is associated with the first AI model, or the CSI reporting is CSI information obtained according to the first AI model;

[0516] a monitoring identifier, the monitoring identifier being used to indicate or configure a monitoring configuration of CSI reporting, the monitoring configuration of CSI reporting being associated with the first AI model and / or the reporting identifier;

[0517] first time information, the first time information being time information associated with CSI reporting information, or time information associated with the first information.

[0518] Optionally, the second information further comprises at least one of:

[0519] the first information;

[0520] first indication information of the first AI model;

[0521] second indication information of the second model;

[0522] a reporting identifier, the reporting identifier being used to indicate or configure CSI reporting, wherein the CSI reporting is associated with the first AI model and / or the second model, or the CSI reporting is CSI information obtained according to the first AI model;

[0523] The monitoring identifier is used for indicating or configuring a monitoring configuration of CSI reporting, the monitoring configuration of CSI reporting is associated with the first AI model and / or the second model, and / or the report identifier.

[0524] The second time information is time information associated with the second information.

[0525] Optionally, the method further comprises:

[0526] The first device receives third information from the second device, the third information comprising at least one of expected CSI reporting information obtained by the second device, reconstructed CSI information, and a first KPI of the first AI model.

[0527] The first device determines the second information according to the third information.

[0528] Optionally, the third information further comprises at least one of:

[0529] The report identifier is used for indicating or configuring CSI reporting, wherein the CSI reporting is associated with the first AI model and / or the second model, or the CSI reporting is CSI information obtained according to the first AI model.

[0530] The monitoring identifier is used for indicating or configuring a monitoring configuration of CSI reporting, the monitoring configuration of CSI reporting is associated with the first AI model and / or the second model, and / or the report identifier.

[0531] The first indication information of the first AI model;

[0532] The second indication information of the second model;

[0533] The third time information associated with the third information.

[0534] Optionally, the first device determines the second information according to the third information, comprising at least one of:

[0535] The first device determines the effectiveness of the first AI model according to the third information.

[0536] The first device compares the second KPI output by the second model with the first KPI, and determines the second information according to a comparison result.

[0537] Optionally, the method further comprises a processing unit 320, configured to:

[0538] If the second KPI and the first KPI differ by less than a first preset threshold, it is determined that the second model is effective.

[0539] If the second KPI is different from the first KPI by more than a second preset threshold, it is determined that the second model is invalid.

[0540] Optionally, if the second information is used to determine that the second model is invalid, the transceiver 310 is further configured to perform at least one of the following:

[0541] send CSI reporting information to the second device;

[0542] send target CSI to the second device;

[0543] receive at least one of reconstructed CSI, expected CSI reporting information, and the first KPI from the second device.

[0544] Optionally, the first KPI includes a first SGCS, and the second KPI includes a second SGCS, and the processing unit 320 is configured to perform at least one of the following:

[0545] If the first SGCS is greater than a second threshold, and the second SGCS is greater than a third threshold, it is determined that the first AI model is valid, and the second model is valid.

[0546] If the first SGCS is greater than a second threshold, and the second SGCS is less than a fifth threshold, it is determined that the second model is invalid.

[0547] If the first SGCS is greater than a second threshold, the second SGCS is less than a sixth threshold, and the first SCGS and the second SGCS differ by less than a first threshold, it is determined that the first AI model is valid, and the second model is valid.

[0548] If the first SGCS is less than a fourth threshold, and the second SGCS is less than a seventh threshold, it is determined that the first AI model is invalid.

[0549] If the first SGCS is less than a fourth threshold, the second SGCS is less than an eighth threshold, and the first SCGS and the second SGCS differ by less than a first threshold, it is determined that the first AI model is invalid, and the second model is valid.

[0550] If the first SGCS is less than a fourth threshold, and the second SGCS is greater than a third threshold, the first AI model is invalid, and the second model is invalid.

[0551] Optionally, the transceiver 310 is further configured to:

[0552] receive third indication information from the second device, the third indication information being used to indicate at least one of the following:

[0553] validity of the second model;

[0554] activating or deactivating the second model;

[0555] switching a monitoring scheme of the first AI model;

[0556] reducing a reporting period of the target CSI.

[0557] Optionally, the transceiver 310 is further configured to:

[0558] receive fourth information from the second device, the fourth information comprising at least one of:

[0559] a monitoring type of the first AI model;

[0560] a monitoring reporting period of the first AI model;

[0561] a first time window for obtaining input information of the first AI model;

[0562] a reporting identifier related to output information of the first AI model associated with the fourth information;

[0563] a monitoring identifier related to output information of the first AI model associated with the fourth information;

[0564] first indication information of the first AI model;

[0565] a second time window for obtaining input information of the second model;

[0566] second indication information of the second model.

[0567] Optionally, the processing unit 320 is further configured to:

[0568] the first device obtains fifth information, the fifth information comprising at least one of input information and output information of the first AI model;

[0569] the first device obtains the first information by using the fifth information and the second model.

[0570] Optionally, the processing unit 320 obtains the first information by using the fifth information and the second model, comprising at least one of:

[0571] inputting at least one of input information and output information of the first AI model into the second model to obtain the first information output by the second model;

[0572] input at least one of input information and output information of the first AI model at K times and the first information obtained before the K times to the second model, and obtain the first information output by the second model at the K times; K is a positive integer;

[0573] input at least one of input information and output information of the first AI model at a first time and the first information obtained before the first time to the second model, and obtain the first information output by the second model at the first time; the first information obtained before the first time is associated with the first information at the K times.

[0574] Optionally, if the rank of the input information of the first AI model is greater than 1 or the rank of the target CSI information is greater than 1, the first information includes at least one of the following:

[0575] Z second KPIs output by the second model, wherein Z is determined according to the rank of the input information or the target CSI, and Z is a positive integer greater than 1;

[0576] one second KPI output by the second model, wherein the second KPI is obtained according to the KPIs of the Z layers.

[0577] Optionally, the processing unit 320 is specifically configured to perform at least one of the following:

[0578] obtain the second model according to a second data set;

[0579] obtain the second model according to the first AI model;

[0580] obtain the second model according to the second data set and the first AI model;

[0581] obtain the second model according to at least one of parameters and structures of the second model from the second device.

[0582] Optionally, the second data set includes at least one of the following: input information of the first AI model, reconstructed information of output information of the first AI model, and first KPIs of the first AI model.

[0583] Optionally, the input information of the first AI model includes K1 input information at K1 times, and the K1 input information at the K1 times is associated with at least one of the first KPIs.

[0584] Optionally, the second model has fewer parameters than the first AI model.

[0585] Optionally, the input information of the first AI model comprises target channel state information (CSI), and the output information of the first AI model comprises feedback CSI or reconstructed CSI.

[0586] Optionally, the input information of the first AI model comprises at least one of the following:

[0587] quantized target CSI information;

[0588] ground-truth CSI information;

[0589] target CSI information according to eType II quantization.

[0590] Optionally, the first AI model is used to determine first CSI reporting information, and the first CSI reporting information comprises at least one of the following:

[0591] first CSI information obtained according to the target CSI information and the first AI model;

[0592] second CSI information obtained according to the ground-truth CSI information and the first AI model;

[0593] third CSI information obtained according to the quantized target CSI information and the first AI model;

[0594] fourth CSI information obtained according to the target CSI information based on eType II quantization and the first AI model.

[0595] In some embodiments, the transceiver unit 310 can be a communication interface or a transceiver, or an input / output interface of a communication chip or a system on chip. The processing unit 320 can be embedded in a processor of the terminal in a hardware form or independent of the processor.

[0596] It should be understood that the model performance supervision apparatus 300 according to the embodiments of the present application can correspond to the terminal in the method embodiments of the present application, and each unit in the model performance supervision apparatus 300 is respectively used to implement the corresponding process of the terminal in the method 200 shown in FIG. 5. For the sake of brevity, details are not described herein.

[0597] FIG. 9 shows a schematic block diagram of a model performance supervision apparatus 400 according to an embodiment of the present application. As shown in FIG. 9, the model performance supervision apparatus 400 comprises:

[0598] The transceiver 410 is configured to receive at least one of first information and second information from the first device; the first information comprises related information of the first AI model or is used to determine the validity of the first AI model; the second information is used to determine the validity of a second model; and the second model is used to assist in determining the related information of the first AI model.

[0599] Optionally, the processing unit 420 is configured to determine the validity of the first AI model and / or the second model according to at least one of the first information and the second information.

[0600] Optionally, the related information of the first AI model comprises at least one of the following:

[0601] input information of the first AI model;

[0602] output information of the first AI model;

[0603] first validity information of the first AI model;

[0604] a first validity degree of the first AI model;

[0605] whether input information and / or output information associated with the first AI model has deviated;

[0606] a data deviation degree of input information and / or output information associated with the first AI model;

[0607] a second key performance indicator (KPI), the second KPI being used to represent a performance standard of the first AI model estimated by the first device.

[0608] Optionally, the related information of the first AI model is information determined according to the second model.

[0609] Optionally, the first AI model is used to determine first CSI reporting information.

[0610] Optionally, the second information comprises at least one of the following:

[0611] a first KPI, the first KPI being used to represent a performance standard of the first AI model;

[0612] a second KPI, the second KPI being used to represent a performance standard of the first AI model estimated by the first device;

[0613] second validity information of the second model;

[0614] a second validity degree of the second model.

[0615] Optionally, the first information further comprises at least one of:

[0616] first indication information of the first AI model;

[0617] second indication information of the second model;

[0618] reporting identification, the reporting identification being used for indicating or configuring CSI reporting, wherein the CSI reporting is associated with the first AI model, or the CSI reporting is CSI information obtained according to the first AI model;

[0619] monitoring identification, the monitoring identification being used for indicating or configuring monitoring configuration of CSI reporting, wherein the monitoring configuration of CSI reporting is associated with the first AI model, and / or the reporting identification;

[0620] first time information, the first time information being time information associated with CSI reporting information, or time information associated with the first information.

[0621] Optionally, the second information further comprises at least one of:

[0622] the first information;

[0623] first indication information of the first AI model;

[0624] second indication information of the second model;

[0625] reporting identification, the reporting identification being used for indicating or configuring CSI reporting, wherein the CSI reporting is associated with the first AI model and / or the second model, or the CSI reporting is CSI information obtained according to the first AI model;

[0626] monitoring identification, the monitoring identification being used for indicating or configuring monitoring configuration of CSI reporting, wherein the monitoring configuration of CSI reporting is associated with the first AI model and / or the second model, and / or the reporting identification;

[0627] second time information, the second time information being time information associated with the second information.

[0628] Optionally, the transceiver 410 is further configured to:

[0629] send third information to the first device, the third information comprising at least one of expected CSI reporting information obtained by the second device, reconstructed CSI information, and first KPI of the first AI model; the third information being used for determining the second information.

[0630] Optionally, the third information further comprises at least one of:

[0631] a report identifier, the report identifier being used for indicating or configuring CSI reporting, wherein the CSI reporting is associated with the first AI model and / or the second model, or the CSI reporting is CSI information obtained according to the first AI model;

[0632] a monitoring identifier, the monitoring identifier being used for indicating or configuring monitoring configuration of CSI reporting, wherein the monitoring configuration of CSI reporting is associated with the first AI model and / or the second model, and / or the report identifier;

[0633] first indication information of the first AI model;

[0634] second indication information of the second model;

[0635] third time information associated with the third information.

[0636] Optionally, the processing unit 320 is configured to:

[0637] if the second KPI is less than the first KPI by less than a first preset threshold, determining that the second model is valid;

[0638] if the second KPI is greater than the first KPI by more than a second preset threshold, determining that the second model is invalid.

[0639] Optionally, if the second information is used to determine that the second model is invalid, the transceiver 410 is further configured to at least one of:

[0640] receive CSI reporting information from the first device;

[0641] receive target CSI from the first device;

[0642] send at least one of reconstructed CSI, expected CSI reporting information, and the first KPI to the first device.

[0643] Optionally, the first KPI includes a first SGCS, and the second KPI includes a second SGCS, and the processing unit 420 is further configured to at least one of:

[0644] if the first SGCS is greater than a second threshold, and the second SGCS is greater than a third threshold, determining that the first AI model is valid, and the second model is valid;

[0645] if the first SGCS is greater than the second threshold, and the second SGCS is less than a fifth threshold, determining that the second model is invalid;

[0646] If the first SGCS is greater than a second threshold, the second SGCS is less than a sixth threshold, and the first SCGS and the second SGCS differ by less than a first threshold, it is determined that the first AI model is valid and the second model is valid.

[0647] If the first SGCS is less than a fourth threshold, and the second SGCS is less than a seventh threshold, it is determined that the first AI model is invalid.

[0648] If the first SGCS is less than a fourth threshold, the second SGCS is less than an eighth threshold, and the first SCGS and the second SGCS differ by less than a first threshold, it is determined that the first AI model is invalid and the second model is valid.

[0649] If the first SGCS is less than a fourth threshold, and the second SGCS is greater than a third threshold, the first AI model is invalid, and the second model is invalid.

[0650] Optionally, the transceiver 410 is further configured to:

[0651] The second device sends third indication information to the first device, and the third indication information is used to indicate at least one of the following:

[0652] The validity of the second model;

[0653] Activate or deactivate the second model;

[0654] Switch the monitoring scheme of the first AI model;

[0655] Reduce the reporting period of the target CSI.

[0656] Optionally, the transceiver 410 is further configured to:

[0657] The second device sends fourth information to the first device, and the fourth information includes at least one of the following:

[0658] The monitoring type of the first AI model;

[0659] The monitoring reporting period of the first AI model;

[0660] The first time window for obtaining the input information of the first AI model;

[0661] The reporting identifier related to the output information of the first AI model associated with the fourth information;

[0662] The monitoring identifier related to the output information of the first AI model associated with the fourth information;

[0663] The first indication information of the first AI model;

[0664] a second time window for obtaining input information of the second model;

[0665] second indication information of the second model.

[0666] Optionally, the transceiver 410 is further configured to:

[0667] send a second data set to the first device, the second data set being used to obtain the second model;

[0668] send at least one of a parameter and a structure of the second model to the first device, for obtaining the second model.

[0669] Optionally, the second data set comprises at least one of input information of the first AI model, and reconstruction information of output information of the first AI model and a first KPI of the first AI model.

[0670] Optionally, the input information of the first AI model comprises input information of K1 times, and the input information of the K1 times is associated with at least one of the first KPI.

[0671] Optionally, the second model has less parameters than the first AI model.

[0672] Optionally, the input information of the first AI model comprises target channel state information (CSI), and the output information of the first AI model comprises feedback CSI or reconstructed CSI.

[0673] Optionally, the input information of the first AI model comprises at least one of:

[0674] quantized target CSI information;

[0675] ground-truth CSI information;

[0676] target CSI information according to eType II quantization.

[0677] Optionally, the first AI model is used to determine first CSI reporting information, comprising at least one of

[0678] first CSI information obtained according to the target CSI information and the first AI model;

[0679] second CSI information obtained according to ground-truth CSI information and the first AI model;

[0680] third CSI information obtained according to quantized target CSI information and the first AI model;

[0681] The fourth CSI information is obtained according to the target CSI information based on the eType II quantization and the first AI model.

[0682] In some embodiments, the transceiver unit 410 described above can be a communication interface or a transceiver, or an input / output interface of a communication chip or a system on chip. The processing unit 420 can be embedded in a processor of the network side device in a hardware form or independent of the processor.

[0683] It should be understood that the model performance supervision apparatus 400 according to the embodiments of the present application can correspond to the network side device in the method embodiments of the present application, and each unit in the model performance supervision apparatus 400 is respectively used to implement the corresponding process of the network side device in the method 200 shown in FIG. 5, and for brevity, will not be described here.

[0684] The model performance supervision apparatus in the embodiments of the present application can be an electronic device, for example, an electronic device with an operating system, or a component in an electronic device, for example, an integrated circuit or a chip. The electronic device can be a terminal or a network side device, or other devices other than the terminal or the network side device. Illustratively, the terminal can include but is not limited to the types of the terminal 11 listed above, the network side device can include but is not limited to the types of the network side device 12 listed above, and the other devices can be servers, network attached storages (NAS), etc., and the embodiments of the present application are not limited specifically.

[0685] The model performance supervision apparatus provided in the embodiments of the present application can implement each process of the method embodiments shown in FIG. 5 or FIG. 7A and achieve the same technical effects, and for brevity, will not be described here.

[0686] As shown in FIG. 10, the embodiments of the present application further provide a communication device 500, which includes a processor 501 and a memory 502, and the memory 502 stores programs or instructions executable on the processor 501.

[0687] For example, when the communication device 500 is a terminal, the programs or instructions are executed by the processor 501 to implement each step of the terminal in the model performance supervision method embodiments described above, and the same technical effects can be achieved, and for brevity, will not be described here.

[0688] For another example, when the communication device 500 is a network side device, the programs or instructions are executed by the processor 501 to implement each step of the network side device in the model performance supervision method embodiments described above, and the same technical effects can be achieved, and for brevity, will not be described here.

[0689] The embodiment of the present application further provides a terminal, comprising a processor and a communication interface, the communication interface and the processor are coupled, the processor is used for running programs or instructions, and the steps performed by the terminal in the method embodiment shown in Fig. 5 are realized. The terminal embodiment corresponds to the terminal side method embodiment described above, and each implementation process and implementation mode of the above method embodiment can be applied to the terminal embodiment, and the same technical effects can be achieved.

[0690] Specifically, Fig. 11 is a schematic diagram of the hardware structure of a terminal for implementing the embodiment of the present application.

[0691] The terminal 600 includes, but is not limited to, at least part of the components such as a radio frequency unit 601, a network module 602, an audio output unit 603, an input unit 604, a sensor 605, a display unit 606, a user input unit 607, an interface unit 608, a memory 609, and a processor 610.

[0692] Those skilled in the art can understand that the terminal 600 can further include a power supply (such as a battery) for supplying power to each component, and the power supply can be logically connected to the processor 610 through a power management system, so as to realize the functions of power management, such as charging, discharging, and power consumption management, through the power management system. The terminal structure shown in Fig. 11 does not constitute a limitation on the terminal, and the terminal can include more or fewer components than the illustrated components, or combine certain components, or different component arrangements, which will not be described here.

[0693] It should be understood that in the embodiment of the present application, the input unit 604 can include a graphics processing unit (GPU) 6041 and a microphone 6042. The graphics processor 6041 processes image data of a still picture or a video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 606 can include a display panel 6061, which can be configured in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit 607 includes at least one of a touch panel 6071 and other input devices 6072. The touch panel 6071 is also called a touch screen. The touch panel 6071 can include a touch detection device and a touch controller. The other input devices 6072 can include, but are not limited to, a physical keyboard, function keys (such as volume control keys, on-off keys, etc.), trackballs, mice, joysticks, etc., which will not be described here.

[0694] In the embodiments of the present application, the radio frequency unit 601 can transmit the downlink data received from the network side device to the processor 610 for processing. In addition, the radio frequency unit 601 can send uplink data to the network side device. Generally, the radio frequency unit 601 includes, but is not limited to, an antenna, an amplifier, a transceiver, a coupler, a low noise amplifier, a duplexer, etc.

[0695] The memory 609 can be used to store software programs or instructions and various data. The memory 609 can mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area can store an operating system, application programs or instructions required by at least one function (such as a sound playing function, an image playing function, etc.), etc. In addition, the memory 609 can include a volatile memory or a non-volatile memory. The non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDR SDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synch link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 609 in the embodiments of the present application includes, but is not limited to, these and any other suitable types of memory.

[0696] The processor 610 can include at least one processing unit; optionally, the processor 610 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface and an application program, etc., and the modem processor mainly processes transmission signals in a multi-connection scenario, such as a baseband processor. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 610.

[0697] The radio frequency unit 601 is configured to send at least one of first information and second information from the first device to the second device. The first information includes related information of the first AI model or is used to determine the validity of the first AI model. The second information is used to determine the validity of a second model. The second model is used to assist in determining the related information of the first AI model.

[0698] It can be understood that the implementation processes of the implementation manners mentioned in the embodiments can refer to the related descriptions of the method embodiments and achieve the same or corresponding technical effects. To avoid repetition, details are not described herein again.

[0699] The embodiments of the present application also provide a network side device, which includes a processor and a communication interface. The communication interface is coupled with the processor. The processor is configured to run programs or instructions to implement the steps performed by the network side device in the method embodiments shown in FIG. 5. The network side device embodiments correspond to the network side device method embodiments described above. The implementation processes and implementation manners of the method embodiments described above can be applied to the network side device embodiments and achieve the same technical effects. To be brief, details are not described herein again.

[0700] Specifically, the embodiments of the present application also provide a network side device. As shown in FIG. 12, the network side device 700 includes an antenna 71, a radio frequency device 72, a baseband device 73, a processor 74 and a memory 75. The antenna 71 is connected with the radio frequency device 72. In the uplink direction, the radio frequency device 72 receives information through the antenna 71 and sends the received information to the baseband device 73 for processing. In the downlink direction, the baseband device 73 processes the information to be sent and sends it to the radio frequency device 72. The radio frequency device 72 processes the received information and sends it out through the antenna 71.

[0701] The method performed by the network side device in the above embodiments can be implemented in the baseband device 73, which includes a baseband processor.

[0702] The baseband device 73 may, for example, include at least one baseband board on which at least two chips are arranged, as shown in FIG. 12. One of the chips is, for example, a baseband processor which is connected with the memory 75 through a bus interface to call programs in the memory 75 and perform the network device operations shown in the above method embodiments.

[0703] The network side device may, for example, also include a network interface 76, which is, for example, a common public radio interface (CPRI).

[0704] Specifically, the network side device 700 of the embodiment of the present application further includes instructions or programs stored on the memory 75 and executable on the processor 74, the processor 74 invokes the instructions or programs in the memory 75 to execute the method performed by the units shown in FIG. 9, and achieves the same technical effects. To avoid repetition, details are not described here.

[0705] The embodiment of the present application also provides a readable storage medium, the readable storage medium stores programs or instructions, the programs or instructions are executed by a processor to implement various processes of the model performance supervision method embodiment, and the same technical effects can be achieved. To avoid repetition, details are not described here.

[0706] The processor is the processor in the terminal in the above embodiments. The readable storage medium includes a computer readable storage medium, such as a computer readable only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc. In some examples, the readable storage medium can be a non-transitory readable storage medium.

[0707] The embodiment of the present application further provides a chip, the chip includes a processor and a communication interface, the communication interface and the processor are coupled, the processor is used to run programs or instructions to implement various processes of the model performance supervision method embodiment, and the same technical effects can be achieved. To avoid repetition, details are not described here.

[0708] It should be understood that the chip mentioned in the embodiment of the present application can also be referred to as a system chip, a system chip, a chip system or a system on chip, etc.

[0709] The embodiment of the present application further provides a computer program / program product, the computer program / program product is stored in a storage medium, the computer program / program product is executed by at least one processor to implement various processes of the model performance supervision method embodiment, and the same technical effects can be achieved. To avoid repetition, details are not described here.

[0710] The embodiment of the present application further provides a communication system, including: a terminal and a network side device, wherein the terminal can be used to execute the steps executed by the terminal in the model performance supervision method, and the network side device can be used to execute the steps executed by the network side device in the model performance supervision method.

[0711] It should be noted that, in the present document, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the presence of additional identical elements in the process, method, article, or apparatus that comprises the element. Furthermore, it is to be understood that the method and apparatus of the present application can be carried out by specific hardware, software, or a combination thereof, and that the method and apparatus of the present application can be implemented in a computer program product that is executed by a computer. The computer program product can comprise a computer-readable storage medium having stored thereon instructions that, when executed by a computer, implement the method of the present application.

[0712] From the above description of the embodiments, it is clear that the above-mentioned method can be realized by means of a computer software product and a general hardware platform, of course, it can also be realized by hardware. The computer software product is stored in a storage medium (such as ROM, RAM, magnetic disc, optical disc, etc.), and the computer software product includes a plurality of instructions for making a terminal or a network side device execute the method described in each embodiment of the present application.

[0713] The embodiments of the present application are described above in combination with the drawings, but the present application is not limited to the above-mentioned specific embodiments, the above-mentioned specific embodiments are only illustrative, but not restrictive, and a person of ordinary skill in the art can make many forms of embodiments under the inspiration of the present application without departing from the scope of the present application and the scope protected by the claims.

Claims

A model performance supervision method, comprising: A first device sends at least one of first information and second information to a second device; the first information comprises relevant information of a first AI model or is used to determine validity of the first AI model; the second information is used to determine validity of a second model; The second model is used to assist in determining the relevant information of the first AI model. The method of claim 1, wherein, The relevant information of the first AI model comprises at least one of: Input information of the first AI model; Output information of the first AI model; First validity information of the first AI model; First validity degree of the first AI model; Whether the input information and / or the output information associated with the first AI model is offset; Data offset degree of the input information and / or the output information associated with the first AI model; Second key performance indicator (KPI), the second KPI is used to represent a performance standard of the first AI model estimated by the first device. The method according to claim 1 or 2, wherein The relevant information of the first AI model is information determined according to the second model. The method according to any one of claims 1 to 3, wherein The first AI model is used to determine first CSI reporting information. The method according to any one of claims 1 to 4, wherein The second information comprises at least one of: First KPI, the first KPI is used to represent a performance standard of the first AI model; Second KPI, the second KPI is used to represent a performance standard of the first AI model estimated by the first device; Second validity information of the second model; Second validity degree of the second model. The method of claim 2, wherein, The first information further comprises at least one of: First indication information of the first AI model; Second indication information of the second model; Reporting identifier, the reporting identifier is used to indicate or configure CSI reporting, wherein the CSI reporting is associated with the first AI model, or the CSI reporting is CSI information obtained according to the first AI model; Monitoring identifier, the monitoring identifier is used to indicate or configure monitoring configuration of CSI reporting, the monitoring configuration of CSI reporting is associated with the first AI model, and / or the reporting identifier; First time information, the first time information is time information associated with CSI reporting information, or the first information is associated with time information. The method of claim 5, wherein, The second information further comprises at least one of: The first information; First indication information of the first AI model; Second indication information of the second model; Reporting identifier, the reporting identifier is used to indicate or configure CSI reporting, wherein the CSI reporting is associated with the first AI model and / or the second model, or the CSI reporting is CSI information obtained according to the first AI model; Monitoring identifier, the monitoring identifier is used to indicate or configure monitoring configuration of CSI reporting, the monitoring configuration of CSI reporting is associated with the first AI model and / or the second model, and / or the reporting identifier; Second time information, the second time information is time information associated with the second information. The method according to any one of claims 1 to 7, wherein Further comprising: The first device receives third information from the second device, the third information comprising at least one of expected CSI reporting information obtained by the second device, reconstructed CSI information, and a first KPI of the first AI model; The first device determines the second information according to the third information. The method of claim 8, wherein, The third information further comprises at least one of: a reporting identifier, the reporting identifier being used to indicate or configure CSI reporting, wherein the CSI reporting is associated with the first AI model and / or a second model, or the CSI reporting is CSI information obtained according to the first AI model; a monitoring identifier, the monitoring identifier being used to indicate or configure monitoring configuration of CSI reporting, wherein the monitoring configuration of CSI reporting is associated with the first AI model and / or the second model, and / or the reporting identifier; first indication information of the first AI model; second indication information of the second model; third time information associated with the third information. The method according to claim 8 or 9, wherein The first device determines the second information according to the third information, comprising at least one of: The first device determines the validity of the first AI model according to the third information; The first device compares the second KPI output by the second model with the first KPI, and determines the second information according to the comparison result. The method of claim 10, wherein, Further comprising: if the second KPI and the first KPI differ by less than a first preset threshold, it is determined that the second model is valid; if the second KPI and the first KPI differ by more than a second preset threshold, it is determined that the second model is invalid. The method of claim 11, wherein, If the second information is used to determine that the second model is invalid, the method further comprises at least one of: The first device sends CSI reporting information to the second device; The first device sends target CSI to the second device; The first device receives at least one of reconstructed CSI, expected CSI reporting information, and a first KPI from the second device. The method according to any one of claims 10-12, wherein The first KPI comprises a first SGCS, and the second KPI comprises a second SGCS, and the method further comprises at least one of: if the first SGCS is greater than a second threshold, and the second SGCS is greater than a third threshold, it is determined that the first AI model is valid, and the second model is valid; if the first SGCS is greater than a second threshold, and the second SGCS is less than a fifth threshold, it is determined that the second model is invalid; if the first SGCS is greater than a second threshold, the second SGCS is less than a sixth threshold, and the first SCGS and the second SGCS differ by less than a first threshold, it is determined that the first AI model is valid, and the second model is valid; if the first SGCS is less than a fourth threshold, and the second SGCS is less than a seventh threshold, it is determined that the first AI model is invalid; if the first SGCS is less than a fourth threshold, the second SGCS is less than an eighth threshold, and the first SCGS and the second SGCS differ by less than a first threshold, it is determined that the first AI model is invalid, and the second model is valid; If the first SGCS is less than a fourth threshold value and the second SGCS is greater than a third threshold value, the first AI model is invalid and the second model is invalid. The method according to any one of claims 1 to 13, wherein Further comprising: receiving third indication information from the second device, the third indication information being used to indicate at least one of: validity of the second model; activation or deactivation of the second model; switching a monitoring scheme of the first AI model; reducing a reporting period of target CSI. The method according to any one of claims 1 to 14, wherein Further comprising: receiving fourth information from the second device, the fourth information including at least one of: a monitoring type of the first AI model; a monitoring reporting period of the first AI model; a first time window for obtaining input information of the first AI model; a reporting identifier related to output information of the first AI model associated with the fourth information; a monitoring identifier related to output information of the first AI model associated with the fourth information; first indication information of the first AI model; a second time window for obtaining input information of the second model; second indication information of the second model. The method according to any one of claims 1 to 15, wherein Further comprising: the first device obtaining fifth information, the fifth information including at least one of input information and output information of the first AI model; the first device obtaining the first information by using the fifth information and the second model. The method of claim 16, wherein, The first device obtaining the first information by using the fifth information and the second model includes at least one of: inputting at least one of input information and output information of the first AI model into the second model to obtain the first information output by the second model; inputting at least one of input information and output information of the first AI model at K time points and at the K time points into the second model to obtain the first information at the K time points output by the second model; K is a positive integer; inputting at least one of input information and output information associated with a first time of the first AI model and the first information obtained before the first time into the second model to obtain the first information associated with the first time output by the second model; wherein the first information obtained between the first time is associated with the first information at the K time points. The method of claim 16, wherein, If the rank of input information of the first AI model is greater than 1 or the rank of target CSI information is greater than 1, the first information includes at least one of: Z second KPIs output by the second model, wherein Z is determined according to the rank of the input information or the target CSI, and Z is a positive integer greater than 1; one second KPI output by the second model, wherein the second KPI is obtained according to the Z KPIs. The method of any one of claims 1-18, wherein, Further comprising at least one of: obtaining the second model according to a second data set; obtaining the second model according to the first AI model; obtaining the second model according to the second data set and the first AI model; obtaining the second model according to at least one of parameters and structures of the second model from the second device. The method of claim 19, wherein, The second dataset includes at least one of reconstructed information of output information of the first AI model and a first KPI of the first AI model. The method of claim 20, wherein, The input information of the first AI model includes input information of K1 times, and the input information of the K1 times is associated with at least one first KPI. The method of any one of claims 19-21, wherein, The second model has fewer parameters than the first AI model. The method of any one of claims 1-22, wherein, The input information of the first AI model includes target channel state information (CSI), and the output information of the first AI model includes feedback CSI or reconstructed CSI. The method of any one of claims 1-23, wherein The input information of the first AI model includes at least one of the following: quantized target CSI information; ground-truth CSI information; target CSI information quantized according to eType II. The method of any one of claims 1-23, wherein, The first AI model is used to determine first CSI reporting information, and includes at least one of the following: first CSI information obtained according to target CSI information and the first AI model; second CSI information obtained according to ground-truth CSI information and the first AI model; third CSI information obtained according to quantized target CSI information and the first AI model; fourth CSI information obtained according to target CSI information quantized based on eType II and the first AI model. A model performance supervision method includes: A second device receives at least one of first information and second information from a first device; the first information includes related information of a first AI model or is used to determine effectiveness of the first AI model; the second information is used to determine effectiveness of a second model; and the second model is used to assist in determining the related information of the first AI model. The method of claim 26, wherein, The related information of the first AI model includes at least one of the following: input information of the first AI model; output information of the first AI model; first effectiveness information of the first AI model; a first effectiveness degree of the first AI model; whether input information and / or output information associated with the first AI model is offset; a data offset degree of input information and / or output information associated with the first AI model; a second key performance indicator (KPI), which is used to represent a performance standard of the first AI model estimated by the first device. The method of claim 26 or 27, wherein, The related information of the first AI model is information determined according to the second model. The method of any one of claims 26-28, wherein, The first AI model is used to determine first CSI reporting information. The method of any one of claims 26-29, wherein, The second information includes at least one of the following: a first KPI, which is used to represent a performance standard of the first AI model; a second KPI, which is used to represent a performance standard of the first AI model estimated by the first device; second effectiveness information of the second model; a second effectiveness degree of the second model. The method of claim 27, wherein, The first information further includes at least one of the following: first indication information of the first AI model; second indication information of the second model; The report identifier is used for indicating or configuring CSI reporting, wherein the CSI reporting is associated with the first AI model, or the CSI reporting is CSI information obtained according to the first AI model. The monitoring identifier is used for indicating or configuring monitoring configuration of CSI reporting, wherein the monitoring configuration of CSI reporting is associated with the first AI model, and / or the report identifier. The first time information is time information associated with CSI reporting information, or the first information is associated with time information. The method of claim 30, wherein, The second information further includes at least one of the following: The first information; The first indication information of the first AI model; The second indication information of the second model; The report identifier is used for indicating or configuring CSI reporting, wherein the CSI reporting is associated with the first AI model and / or the second model, or the CSI reporting is CSI information obtained according to the first AI model. The monitoring identifier is used for indicating or configuring monitoring configuration of CSI reporting, wherein the monitoring configuration of CSI reporting is associated with the first AI model and / or the second model, and / or the report identifier. The second time information is time information associated with the second information. The method of any one of claims 26-32, wherein Further comprising: The second device sends third information to the first device, wherein the third information includes at least one of the following: expected CSI reporting information obtained by the second device, reconstructed CSI information, and the first KPI of the first AI model; and the third information is used for determining the second information. The method of claim 33, wherein, The third information further includes at least one of the following: The report identifier is used for indicating or configuring CSI reporting, wherein the CSI reporting is associated with the first AI model and / or the second model, or the CSI reporting is CSI information obtained according to the first AI model. The monitoring identifier is used for indicating or configuring monitoring configuration of CSI reporting, wherein the monitoring configuration of CSI reporting is associated with the first AI model and / or the second model, and / or the report identifier. The first indication information of the first AI model; The second indication information of the second model; The third time information associated with the third information. The method of claim 30, wherein, Further comprising: If the second KPI differs from the first KPI by less than a first preset threshold, it is determined that the second model is valid. If the second KPI differs from the first KPI by more than a second preset threshold, it is determined that the second model is invalid. The method of claim 35, wherein, If the second information is used to determine that the second model is invalid, the method further includes at least one of the following: The second device receives CSI reporting information from the first device. The second device receives target CSI from the first device. The second device sends at least one of the following to the first device: reconstructed CSI, expected CSI reporting information, and the first KPI. The method of claim 35, wherein, The first KPI includes a first SGCS, and the second KPI includes a second SGCS, and the method further includes at least one of the following: If the first SGCS is greater than a second threshold, and the second SGCS is greater than a third threshold, it is determined that the first AI model is valid, and the second model is valid. If the first SGCS is greater than a second threshold, and the second SGCS is less than a fifth threshold, it is determined that the second model is invalid. If the first SGCS is greater than a second threshold, the second SGCS is less than a sixth threshold, and the first SCGS and the second SGCS differ by less than a first threshold, it is determined that the first AI model is valid, and the second model is valid. If the first SGCS is less than a fourth threshold, and the second SGCS is less than a seventh threshold, it is determined that the first AI model is invalid. If the first SGCS is less than a fourth threshold, the second SGCS is less than an eighth threshold, and the first SCGS and the second SGCS differ by less than a first threshold, it is determined that the first AI model is invalid, and the second model is valid. If the first SGCS is less than a fourth threshold, and the second SGCS is greater than a third threshold, the first AI model is invalid, and the second model is invalid. The method of any one of claims 26-37, wherein, Further comprising: The second device sends third indication information to the first device, and the third indication information is used to indicate at least one of the following: Validity of the second model; Activation or deactivation of the second model; Switching the monitoring scheme of the first AI model; Reducing the reporting period of the target CSI. The method of any one of claims 26-38, wherein Further comprising: The second device sends fourth information to the first device, and the fourth information includes at least one of the following: Monitoring type of the first AI model; Monitoring reporting period of the first AI model; First time window for obtaining input information of the first AI model; Report identifier related to output information of the first AI model associated with the fourth information; Monitoring identifier related to output information of the first AI model associated with the fourth information; First indication information of the first AI model; Second time window for obtaining input information of the second model; Second indication information of the second model. The method of any one of claims 26-39, wherein Further comprising at least one of the following: Send a second data set to the first device, and the second data set is used to obtain the second model; Send at least one of the parameters and structure of the second model to the first device, which is used to obtain the second model. The method of claim 40, wherein, The second data set includes at least one of the input information of the first AI model, the reconstruction information of the output information of the first AI model, and the first KPI of the first AI model. The method of claim 41, wherein, The input information of the first AI model includes input information of K1 time, and the input information of K1 time is associated with at least one first KPI. The method of any one of claims 40-42, wherein, The number of parameters of the second model is less than that of the first AI model. The method of any one of claims 26-43, wherein, The input information of the first AI model includes target channel state information CSI, and the output information of the first AI model includes feedback CSI or reconstructed CSI. The method of any one of claims 26-44, wherein, The input information of the first AI model includes at least one of the following: Quantized target CSI information; Ground-truth CSI information; The target CSI information is quantized according to the eType II. The method of any one of claims 26-44, wherein, The first AI model is used to determine the first CSI reporting information, including at least one of The first CSI information obtained according to the target CSI information and the first AI model; The second CSI information obtained according to the Ground-truth CSI information and the first AI model; The third CSI information obtained according to the quantized target CSI information and the first AI model; The fourth CSI information obtained according to the target CSI information based on the eType II quantization and the first AI model. A model performance supervision device, comprising: A transceiver unit configured to send at least one of first information and second information to a second device; the first information includes related information of a first AI model or is used to determine the effectiveness of the first AI model; the second information is used to determine the effectiveness of a second model; The second model is used to assist in determining the related information of the first AI model. The apparatus of claim 47, wherein The related information of the first AI model includes at least one of: Input information of the first AI model; Output information of the first AI model; The first effectiveness information of the first AI model; The first effectiveness degree of the first AI model; Whether the input information and / or output information associated with the first AI model is offset; The data offset degree of the input information and / or output information associated with the first AI model; The second key performance indicator (KPI), which is used to represent the performance standard of the first AI model estimated by the first device. The apparatus of any one of claims 47-48, wherein The second information includes at least one of: The first KPI, which is used to represent the performance standard of the first AI model; The second KPI, which is used to represent the performance standard of the first AI model estimated by the first device; The second effectiveness information of the second model; The second effectiveness degree of the second model. The apparatus of any one of claims 47-49, wherein The transceiver unit is further configured to: Receive third information from the second device, the third information including at least one of expected CSI reporting information, reconstructed CSI information and the first KPI of the first AI model obtained by the second device; The device further comprises a processing unit configured to determine the second information according to the third information. The apparatus of claim 50, wherein, The processing unit is specifically configured to at least one of: The first device determines the effectiveness of the first AI model according to the third information; The first device compares the second KPI output by the second model with the first KPI, and determines the second information according to the comparison result. The apparatus of claim 51, wherein The processing unit is further configured to: If the difference between the second KPI and the first KPI is less than a first preset threshold, it is determined that the second model is effective; If the difference between the second KPI and the first KPI is greater than a second preset threshold, it is determined that the second model is invalid. The apparatus of claim 52, wherein, If the second information is used to determine that the second model is invalid, the transceiver unit is further configured to at least one of: Send CSI reporting information to the second device; Send target CSI to the second device; receiving at least one of reconstructed CSI, expected CSI reporting information, and a first KPI from the second device. The apparatus of any one of claims 51-53, wherein The first KPI includes a first SGCS, and the second KPI includes a second SGCS, and the processing unit is further configured to perform at least one of: if the first SGCS is greater than a second threshold value, and the second SGCS is greater than a third threshold value, determining that the first AI model is valid, and the second model is valid; if the first SGCS is greater than a second threshold value, and the second SGCS is less than a fifth threshold value, determining that the second model is invalid; if the first SGCS is greater than a second threshold value, the second SGCS is less than a sixth threshold value, and the first SCGS and the second SGCS differ by less than a first threshold value, determining that the first AI model is valid, and the second model is valid; if the first SGCS is less than a fourth threshold value, and the second SGCS is less than a seventh threshold value, determining that the first AI model is invalid; if the first SGCS is less than a fourth threshold value, the second SGCS is less than an eighth threshold value, and the first SCGS and the second SGCS differ by less than a first threshold value, determining that the first AI model is invalid, and the second model is valid; if the first SGCS is less than a fourth threshold value, and the second SGCS is greater than a third threshold value, the first AI model is invalid, and the second model is invalid. The apparatus of any one of claims 47-51, wherein The transceiver is further configured to: receive third indication information from the second device, the third indication information being used to indicate at least one of: validity of the second model; activation or deactivation of the second model; switching a monitoring scheme of the first AI model; reducing a reporting period of target CSI. The apparatus of any one of claims 47-55, wherein The transceiver is further configured to: receive fourth information from the second device, the fourth information including at least one of: a monitoring type of the first AI model; a monitoring reporting period of the first AI model; a first time window for obtaining input information of the first AI model; a report identifier related to output information of the first AI model associated with the fourth information; a monitoring identifier related to output information of the first AI model associated with the fourth information; first indication information of the first AI model; a second time window for obtaining input information of the second model; second indication information of the second model. The apparatus of any one of claims 47-56, wherein The processing unit is further configured to: the first device obtains fifth information, the fifth information including at least one of input information and output information of the first AI model; the first device obtains the first information using the fifth information and the second model. A model performance supervision apparatus includes: a transceiver configured to receive at least one of first information and second information from a first device; the first information includes related information of a first AI model or is used to determine validity of the first AI model; and the second information is used to determine validity of a second model; wherein the second model is used to assist in determining the related information of the first AI model. The apparatus of claim 58, wherein the related information of the first AI model includes at least one of: Input information of the first AI model; Output information of the first AI model; First validity information of the first AI model; First validity degree of the first AI model; Whether the input information and / or the output information associated with the first AI model deviates; Data deviation degree of the input information and / or the output information associated with the first AI model; A second key performance indicator (KPI), which is used to represent a performance standard of the first AI model estimated by the first device. The apparatus of any one of claims 58-59, wherein The second information includes at least one of: A first KPI, which is used to represent a performance standard of the first AI model; A second KPI, which is used to represent a performance standard of the first AI model estimated by the first device; Second validity information of the second model; Second validity degree of the second model. The apparatus of any one of claims 58-60, wherein The transceiver unit is further configured to: The second device sends third information to the first device, the third information including at least one of expected CSI reporting information, reconstructed CSI information and the first KPI of the first AI model obtained by the second device; and the third information is used to determine the second information. The apparatus of claim 60, wherein, The processing unit is further configured to: If the second KPI differs from the first KPI by less than a first preset threshold, it is determined that the second model is valid; If the second KPI differs from the first KPI by more than a second preset threshold, it is determined that the second model is invalid. The apparatus of claim 62, wherein If the second information is used to determine that the second model is invalid, the transceiver unit is further configured to at least one of: Receive CSI reporting information from the first device; Receive target CSI from the first device; Send at least one of reconstructed CSI, expected CSI reporting information and the first KPI to the first device. The apparatus of claim 62, wherein The first KPI includes a first SGCS, and the second KPI includes a second SGCS, and the processing unit is further configured to at least one of: If the first SGCS is greater than a second threshold, and the second SGCS is greater than a third threshold, it is determined that the first AI model is valid, and the second model is valid; If the first SGCS is greater than a second threshold, and the second SGCS is less than a fifth threshold, it is determined that the second model is invalid; If the first SGCS is greater than a second threshold, the second SGCS is less than a sixth threshold, and the first SGCS differs from the second SGCS by less than a first threshold, it is determined that the first AI model is valid, and the second model is valid; If the first SGCS is less than a fourth threshold, and the second SGCS is less than a seventh threshold, it is determined that the first AI model is invalid; If the first SGCS is less than a fourth threshold, the second SGCS is less than an eighth threshold, and the first SGCS differs from the second SGCS by less than a first threshold, it is determined that the first AI model is invalid, and the second model is valid; If the first SGCS is less than a fourth threshold, and the second SGCS is greater than a third threshold, the first AI model is invalid, and the second model is invalid. The apparatus of any one of claims 58-64, wherein The transceiver unit is further configured to: The second device sends third indication information to the first device, and the third indication information is used to indicate at least one of the following: validity of the second model; activation or deactivation of the second model; switching of the monitoring scheme of the first AI model; reduction of a reporting period of the target CSI. The apparatus of any one of claims 58-65, wherein The transceiver is further configured to: The second device sends fourth information to the first device, and the fourth information includes at least one of the following: monitoring type of the first AI model; monitoring reporting period of the first AI model; first time window for obtaining input information of the first AI model; reporting identifier related to output information of the first AI model associated with the fourth information; monitoring identifier related to output information of the first AI model associated with the fourth information; first indication information of the first AI model; second time window for obtaining input information of the second model; second indication information of the second model. A terminal comprising a transceiver, a processor and a memory, the memory storing programs or instructions executable on the processor, the programs or instructions being executed by the processor to implement the steps of the method according to any one of claims 1 to 25. A network-side device comprising a transceiver, a processor and a memory, the memory storing programs or instructions executable on the processor, the programs or instructions being executed by the processor to implement the steps of the method according to any one of claims 26 to 46. A readable storage medium, the readable storage medium storing programs or instructions, the programs or instructions being executed by a processor to implement the steps of the method according to any one of claims 1 to 25, or to implement the steps of the method according to any one of claims 26 to 46.

Images