Model performance supervision method and apparatus, and device

Abstract

The present application belongs to the field of communications. Disclosed are a model performance supervision method and apparatus, and a device, which can solve the problem of performance supervision for AI models. The model performance supervision method in the embodiments of the present application comprises: a first device acquiring first information, wherein the first information is used for representing the uncertainty or probability distribution of an output of a first AI model, and an input of the first AI model is a first feature; and the first device determining third information on the basis of the first information and the second information, wherein the second information is a label corresponding to the first feature, and the third information is used for determining the validity of the first AI model.

Description

Model performance supervision method, device and equipment

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed with the China Patent Office on November 3, 2023, with application number 202311467210.7 and invention name “Model Performance Supervision Method, Device and Equipment”, the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The present application relates to the field of communications, and more specifically, to a model performance supervision method, apparatus, and device. Background Art

[0004] Artificial Intelligence (AI) models are being introduced in New Radio (NR) systems to improve system performance. For example, AI models are used for positioning, beam management, channel state information (CSI) prediction, mobility management, and CSI compression. However, when the wireless propagation environment changes, the performance of AI models may be difficult to guarantee. Monitoring the performance of AI models is a problem that needs to be addressed.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide a model performance supervision method, device and equipment that can solve the performance supervision problem of AI models.

[0007] First, a model performance supervision method is provided, comprising:

[0008] The first device obtains first information, where the first information is used to characterize uncertainty or probability distribution of an output of a first AI model, and an input of the first AI model is a first feature;

[0009] The first device determines third information based on the first information and the second information, wherein the second information is a label corresponding to the first feature, and the third information is used to determine the validity of the first AI model.

[0010] Secondly, a model performance supervision method is provided, including:

[0011] The second device receives third information from the first device, where the third information is determined based on the first information and the second information, the first information is used to characterize the uncertainty or probability distribution of the output of the first AI model, the input of the first AI model is the first feature, and the second information is a label corresponding to the first feature;

[0012] The second device determines the validity of the first AI model based on the third information.

[0013] Thirdly, a model performance supervision method is provided, including:

[0014] The first device obtains fourth information, where the fourth information is used to characterize uncertainty or probability distribution of an output of a second AI model, where an input of the second AI model is the third feature;

[0015] The first device determines the validity of the second AI model based on the fourth information; or, the first device sends the fourth information to the second device.

[0016] Fourthly, a model performance supervision method is provided, including:

[0017] The second device receives fourth information from the first device, where the fourth information is used to characterize uncertainty or probability distribution of an output of a second AI model, where an input of the second AI model is the third feature;

[0018] The second device determines the validity of the second AI model based on the fourth information.

[0019] In a fifth aspect, a model performance monitoring device is provided, comprising:

[0020] an acquiring unit, configured to acquire first information, wherein the first information is used to characterize uncertainty or probability distribution of an output of a first AI model, and an input of the first AI model is a first feature;

[0021] A processing unit, configured to determine third information based on the first information and the second information, wherein the second information is a label corresponding to the first feature, and the third information is used to determine the validity of the first AI model.

[0022] In a sixth aspect, a model performance monitoring device is provided, comprising:

[0023] a transceiver unit, configured to receive third information from the first device, wherein the third information is determined based on the first information and second information, the first information is used to characterize the uncertainty or probability distribution of the output of the first AI model, the input of the first AI model is a first feature, and the second information is a label corresponding to the first feature;

[0024] A processing unit is configured to determine the validity of the first AI model based on the third information.

[0025] In a seventh aspect, a model performance monitoring device is provided, comprising:

[0026] an acquiring unit, configured to acquire fourth information, wherein the fourth information is used to characterize uncertainty or probability distribution of an output of a second AI model, where an input of the second AI model is the third feature;

[0027] A processing unit, configured to determine the validity of the second AI model based on the fourth information; or a transceiver unit, configured to send the fourth information to the second device.

[0028] In an eighth aspect, a model performance monitoring device is provided, comprising:

[0029] a transceiver unit, configured to receive fourth information from the first device, wherein the fourth information is used to characterize the uncertainty or probability distribution of the output of the second AI model, the input of the second AI model being the third feature;

[0030] A processing unit is configured to determine the validity of the second AI model based on the fourth information.

[0031] In the ninth aspect, a model performance monitoring device is provided, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0032] In the tenth aspect, a model performance supervision device is provided, comprising a processor and a communication interface; wherein the communication interface or the processor is used to obtain first information, wherein the first information is used to characterize the uncertainty or probability distribution of the output of a first AI model, and the input of the first AI model is a first feature; the processor is used to determine third information based on the first information and second information, wherein the second information is a label corresponding to the first feature, and the third information is used to determine the validity of the first AI model.

[0033] In the eleventh aspect, a model performance monitoring device is provided, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the second aspect are implemented.

[0034] In the twelfth aspect, a model performance supervision device is provided, comprising a processor and a communication interface; wherein the communication interface is used to receive third information from a first device, wherein the third information is determined based on the first information and the second information, the first information is used to characterize the uncertainty or probability distribution of the output of the first AI model, the input of the first AI model is a first feature, and the second information is a label corresponding to the first feature; the processor is used to determine the validity of the first AI model based on the third information.

[0035] In the thirteenth aspect, a model performance monitoring device is provided, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the third aspect are implemented.

[0036] In the fourteenth aspect, a model performance supervision device is provided, comprising a processor and a communication interface; wherein the communication interface or the processor is used to obtain fourth information, wherein the fourth information is used to characterize the uncertainty or probability distribution of the output of a second AI model, and the input of the second AI model is a third feature; the processor is used to determine the validity of the second AI model based on the fourth information; or, the communication interface is used to send the fourth information to the second device.

[0037] In the fifteenth aspect, a model performance monitoring device is provided, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the fourth aspect are implemented.

[0038] In the sixteenth aspect, a model performance supervision device is provided, comprising a processor and a communication interface; wherein the communication interface is used to receive fourth information from a first device, wherein the fourth information is used to characterize the uncertainty or probability distribution of the output of a second AI model, and the input of the second AI model is a third feature; the processor is used to determine the validity of the second AI model based on the fourth information.

[0039] In the seventeenth aspect, a readable storage medium is provided, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented, or the steps of the method described in the second aspect are implemented, or the steps of the method described in the third aspect are implemented, or the steps of the method described in the fourth aspect are implemented.

[0040] In aspect 18, a wireless communication system is provided, including: a first device and a second device, wherein the first device can be used to execute the steps of the method described in aspect 1 or aspect 3, and the second device can be used to execute the steps of the method described in aspect 2 or aspect 4.

[0041] In the nineteenth aspect, a chip is provided, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the method as described in the first aspect, or the method as described in the second aspect, or the method as described in the third aspect, or the method as described in the fourth aspect.

[0042] In the twentieth aspect, a computer program / program product is provided, which is stored in a storage medium and is executed by at least one processor to implement the steps of the model performance supervision method as described in at least one of the first to fourth aspects.

[0043] In the embodiments of the first aspect or the second aspect of the present application, third information can be determined based on the uncertainty or probability distribution of the output of the first AI model and the label corresponding to the input feature of the first AI model, and the validity of the first AI model can be determined based on the third information, thereby achieving performance supervision of the first AI model. Specifically, the uncertainty or probability distribution of the output of the first AI model can improve the robustness of the reasoning result of the first AI model. Since the reasoning result of the first AI model covers multiple possible results and their probability distribution, it is conducive to further processing and utilization of the reasoning result. For example, the reasoning result of the first AI model (including relevant information about uncertainty or probability distribution) is combined with other types of information to obtain a more accurate target result. Referring to the uncertainty or probability distribution of the output of the first AI model and the corresponding label in the performance supervision of the first AI model can improve the accuracy of the performance supervision of the first AI model.

[0044] In the embodiments of the third aspect or the fourth aspect of the present application, the validity of the second AI model can be determined based on the uncertainty or probability distribution of the output of the second AI model, thereby realizing the performance supervision of the second AI model. Specifically, the uncertainty or probability distribution of the output of the second AI model can improve the robustness of the reasoning result of the second AI model. Since the reasoning result of the second AI model covers multiple possible results and their probability distribution, it is conducive to further processing and utilization of the reasoning result. For example, the reasoning result of the second AI model (including relevant information of uncertainty or probability distribution) is combined with other types of information to obtain a more accurate target result. In the performance supervision of the second AI model, referring to the uncertainty or probability distribution of the output of the second AI model can judge the validity of the second AI model. Since no external information is required, it is also easier to implement. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0046] FIG1 is a schematic diagram of a communication system architecture provided in an embodiment of the present application.

[0047] FIG2 is a schematic diagram of a neural network provided by the present application.

[0048] FIG3 is a schematic diagram of a neuron provided in the present application.

[0049] FIG4 is a schematic flowchart of a model performance supervision method provided according to an embodiment of the present application.

[0050] FIG5 is a schematic diagram of improving positioning accuracy based on soft information according to an embodiment of the present application.

[0051] Figure 6 is a schematic flowchart of another model performance supervision method provided according to an embodiment of the present application.

[0052] Figure 7 is a schematic flowchart of another model performance supervision method provided according to an embodiment of the present application.

[0053] FIG8 is a schematic flowchart of another model performance supervision method provided according to an embodiment of the present application.

[0054] FIG9 is a schematic block diagram of a model performance monitoring device provided according to an embodiment of the present application.

[0055] FIG10 is a schematic block diagram of another model performance monitoring device provided according to an embodiment of the present application.

[0056] FIG11 is a schematic block diagram of another model performance monitoring device provided according to an embodiment of the present application.

[0057] FIG12 is a schematic block diagram of another model performance monitoring device provided according to an embodiment of the present application.

[0058] FIG13 is a schematic block diagram of a communication device provided according to an embodiment of the present application.

[0059] FIG14 is a schematic diagram of the hardware structure of a terminal provided according to an embodiment of the present application.

[0060] FIG15 is a schematic block diagram of a network-side device provided according to an embodiment of the present application.

[0061] Figure 16 is a schematic block diagram of another network-side device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0062] The following will be combined with the accompanying drawings in the embodiments of this application to clearly describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0063] The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same type, and do not limit the number of objects, for example, the first object can be one or more. In addition, "or" in this application represents at least one of the connected objects. For example, "A or B" covers three options, namely, Option 1: including A but not including B; Option 2: including B but not including A; Option 3: including both A and B. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0064] The term "indication" in this application can be either a direct indication (or explicit indication) or an indirect indication (or implicit indication). A direct indication can be understood as the sender explicitly informing the receiver of specific information, the operation to be performed, or the requested result, etc. in the instruction sent; an indirect indication can be understood as the receiver determining the corresponding information based on the instruction sent by the sender, or making a judgment and determining the operation to be performed or the requested result, etc. based on the judgment result.

[0065] It is worth noting that the technology described in the embodiments of the present application is not limited to the Internet of Things (IoT) system, but can also be used in other wireless communication systems, such as Long Term Evolution (LTE) / LTE-Advanced (LTE-A) systems, Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal Frequency Division Multiple Access (OFDMA), Single-carrier Frequency-Division Multiple Access (SC-FDMA), Wireless Local Area Networks (WLAN), Wireless Fidelity (WiFi), Bluetooth systems, or other systems. In the embodiments of the present application, the terms "system" and "network" are often used interchangeably, and the technology described can be used for the systems and radio technologies mentioned above, as well as for other systems and radio technologies. The following description describes a New Radio (NR) system for example purposes, and NR terminology is used in most of the following description, but these techniques can also be applied to systems other than NR systems, such as 6G (6 th Generation, 6G) communication system.

[0066] Figure 1 shows a block diagram of a wireless communication system applicable to embodiments of the present application. The wireless communication system includes a terminal 11 and a network-side device 12, wherein the terminal 11 can communicate with the network-side device 12 directly or through other network elements.

[0067] The terminal 11 may be a mobile phone, tablet computer, laptop computer, notebook computer, personal digital assistant (PDA), handheld computer, netbook, ultra-mobile personal computer (UMPC), mobile internet device (MID), augmented reality (AR), virtual reality (VR) device, robot, wearable device, flight vehicle, vehicle user equipment (VUE), shipborne equipment, pedestrian user equipment (PUE), smart home (home appliances with wireless communication functions, such as refrigerators, televisions, washing machines, or furniture), game console, personal computer (PC), ATM or self-service machine, etc. Wearable devices include: smart watches, smart bracelets, smart headphones, smart glasses, smart jewelry (smart bracelets, smart bracelets, smart rings, smart necklaces, smart anklets, smart anklets, etc.), smart wristbands, smart clothing, etc. The vehicle-mounted device may also be referred to as a vehicle-mounted terminal, a vehicle-mounted controller, a vehicle-mounted module, a vehicle-mounted component, a vehicle-mounted chip, or a vehicle-mounted unit, etc. It should be noted that the specific type of the terminal 11 is not limited in the embodiment of the present application.

[0068] The network side device 12 may include an access network device or a core network device.

[0069] Access network equipment may also be referred to as radio access network (RAN) equipment, radio access network functions, or radio access network units. Access network equipment may include base stations, wireless local area network (WLAN) access points (AS), or wireless fidelity (WiFi) nodes. Among them, the base station can be referred to as Node B (NB), Evolved Node B (eNB), the next generation Node B (gNB), New Radio Node B (NR Node B), access point, Relay Base Station (RBS), Serving Base Station (SBS), Base Transceiver Station (BTS), radio base station, radio transceiver, Basic Service Set (BSS), Extended Service Set (ESS), Home Node B (HNB), Home Evolved Node B (home evolved Node B), Transmission Reception Point (TRP) or other appropriate terms in the field. As long as the same technical effect is achieved, the base station is not limited to specific technical vocabulary. It should be noted that in the embodiment of the present application, only the base station in the NR system is used as an example for introduction, and the specific type of the base station is not limited.

[0070] Among them, the core network equipment may include but is not limited to at least one of the following: core network node, core network function, mobility management entity (MME), access mobility management function (AMF), session management function (SMF), user plane function (UPF), policy control function (PCF), policy and charging rules function unit (PCRF), edge application service discovery function (EASDF), unified data management (UDM), unified data repository (UDR), home user server (HSS), centralized network configuration (CNC), network storage function (NRF), network exposure function (NEF), local NEF (L-NEF), binding support function (BSF), application function ( Function, AF), Location Management Function (LMF), etc. It should be noted that in the embodiment of the present application, only the core network device in the NR system is introduced as an example, and the specific type of the core network device is not limited.

[0071] In order to facilitate a better understanding of the embodiments of the present application, the technologies related to the present application are explained.

[0072] Artificial intelligence (AI) is currently being widely applied in various fields. Integrating AI into wireless communication networks to significantly improve technical indicators such as throughput, latency, and user capacity is a key task for future wireless communication networks. AI modules can be implemented in a variety of ways, such as neural networks, decision trees, support vector machines, and Bayesian classifiers. This application uses neural networks as an example, but does not limit the specific type of AI module.

[0073] An exemplary neural network may be shown in FIG2 , wherein the neural network is composed of neurons, and the neurons may be shown in FIG3 , wherein α1, α2, … α K is the input, w is the weight (multiplicative coefficient), b is the bias (additive coefficient), and σ(.) is the activation function. Common activation functions include Sigmoid, Tanh, Rectified Linear Unit (ReLU), etc.

[0074] Neural network parameters are optimized using a gradient optimization algorithm. Gradient optimization algorithms are a class of algorithms that minimize or maximize an objective function (sometimes called a loss function), which is often a mathematical combination of model parameters and data. For example, given data X and its corresponding label Y, we construct a neural network model f(.). With this model, we can obtain the predicted output f(x) based on the input x, and calculate the difference between the predicted value and the true value (f(x) - Y). This is the loss function. The goal of model training is to find the appropriate w and b that minimize the loss function. The smaller the loss value, the closer the constructed neural network model is to the real world.

[0075] For example, in the training process of neural network models, common optimization algorithms are generally based on the error back propagation (BP) algorithm. The basic concept of the BP algorithm is that the learning process consists of two steps: forward signal propagation and error back propagation. During forward propagation, input samples are transmitted from the input layer, processed layer by layer by each hidden layer, and then transmitted to the output layer. If the actual output of the output layer does not match the expected output, the error back propagation phase begins. Error back propagation is to propagate the output error back through the hidden layers to the input layer layer by layer in some form, and distribute the error to all units in each layer, thereby obtaining an error signal for each unit in each layer. This error signal serves as the basis for correcting the weights of each unit. This process of adjusting the weights of each layer through forward signal propagation and error back propagation is repeated in a cycle. This process of continuous weight adjustment is the network learning and training process. This process continues until the error in the network output is reduced to an acceptable level, or until a pre-set number of learning cycles is reached.

[0076] Common optimization algorithms include gradient descent, stochastic gradient descent (SGD), mini-batch gradient descent, momentum method (Momentum), Nesterov (specifically stochastic gradient descent with momentum), adaptive gradient descent (Adagrad), Adadelta, root mean square error (RMSprop), adaptive momentum estimation (Adam), etc.

[0077] When these optimization algorithms are backpropagating errors, they can calculate the derivative / partial derivative of the neuron based on the error / loss obtained by the loss function, add the influence of the learning rate, the previous gradient / derivative / partial derivative, etc., obtain the gradient, and pass the gradient to the previous layer.

[0078] To facilitate understanding of the technical solutions of the embodiments of the present application, the technical solutions of the present application are described in detail below through specific embodiments. The above related technologies can be combined arbitrarily with the technical solutions of the embodiments of the present application as optional solutions, and all of them fall within the scope of protection of the embodiments of the present application. The embodiments of the present application include at least part of the following contents.

[0079] FIG4 is a schematic flow chart of a model performance supervision method 200 according to an embodiment of the present application. As shown in FIG4 , the model performance supervision method 200 may include at least part of the following contents:

[0080] S210: A first device obtains first information, where the first information is used to characterize uncertainty or probability distribution of an output of a first AI model, where an input of the first AI model is a first feature.

[0081] S220, the first device determines third information based on the first information and the second information, wherein the second information is a label corresponding to the first feature, and the third information is used to determine the validity of the first AI model.

[0082] It should be understood that FIG4 shows the steps or operations of the model performance supervision method 200, but these steps or operations are merely examples, and the embodiments of the present application may also perform other operations or variations of the operations in FIG4.

[0083] In an embodiment of the present application, third information can be determined based on the uncertainty or probability distribution of the output of the first AI model and the label corresponding to the input feature of the first AI model, and the validity of the first AI model can be determined based on the third information, thereby achieving performance supervision of the first AI model. Specifically, the uncertainty or probability distribution of the output of the first AI model can improve the robustness of the reasoning result of the first AI model. Since the reasoning result of the first AI model covers multiple possible results and their probability distribution, it is conducive to further processing and utilization of the reasoning result. For example, the reasoning result of the first AI model (including relevant information on uncertainty or probability distribution) is combined with other types of information to obtain a more accurate target result. Referring to the uncertainty or probability distribution of the output of the first AI model and the corresponding label in the performance supervision of the first AI model can improve the accuracy of the performance supervision of the first AI model.

[0084] Exemplarily, other types of information may be as follows: motion state information (such as speed, acceleration, etc.), signal quality measurement information (such as reference signal received power (RSRP), signal to interference plus noise ratio (SINR), reference signal received quality (RSRQ), etc.).

[0085] Exemplarily, the judgment result of the validity of the first AI model obtained based on the embodiment of the present application can also be combined with the judgment result of the validity of the first AI model obtained by other model supervision methods to obtain the final conclusion on the validity of the first AI model.

[0086] It should be noted that, in the embodiment of the present application, the higher the uncertainty of the output of the first AI model, the lower the accuracy of the output of the first AI model.

[0087] The label described in the embodiment of the present application is obtained in a certain way and is associated with the target task. This label can be obtained through measurement or other prior information; for example, in positioning based on an AI model, the input of the AI ​​model is channel state information, and the output of the AI ​​model is the uncertainty or probability distribution of the position. Then this label is the position information corresponding to the channel state information. The position information can be obtained through GPS and other positioning methods, or obtained from a positioning reference unit with a known position.

[0088] In an embodiment of the present application, the first information may also be referred to as soft information (such as probability distribution, confidence level, confidence interval, etc.), wherein the soft information gives the probability distribution or confidence level of the possible results. Optionally, the first AI model may be a soft information AI model or may not be a soft information AI model. Among them, the soft information AI model refers to a type of AI model that outputs soft information (such as probability distribution, confidence level, confidence interval, etc.), including both classic probability models and AI models based on neural networks; the soft information AI model measures the possibility of different prediction results and gives the probability distribution or confidence level of each possible result.

[0089] It should be noted that compared to the hard information (hard value) obtained by AI model reasoning (such as Time of Arrival (TOA), Reference Signal Time Difference (RSTD), Angle of Arrival (AoA), Angle of Departure (AoD), Reference Signal Received Power (RSRP), Line of Sight (LOS) indication, Non-Line of Sight (NLOS) indication, etc.), the soft information obtained by AI model reasoning can significantly improve the reasoning accuracy and robustness. Specifically, soft information can better describe the uncertainty of the world, improve the robustness of model reasoning, and provide better security for some services that require relatively high reasoning reliability.

[0090] In some embodiments, the dimension of the first feature is D1, and correspondingly, the first information is D2 groups of soft information of the second feature, where the D2 groups of soft information describe the range of the output of the first AI model. If there is only one group of soft information (i.e., D2=1), then this group of soft information describes the uncertainty of the output of the entire first AI model; if there are at least two groups of soft information (i.e., D2≥2), then each group of soft information describes the uncertainty of part of the features output by the first AI model.

[0091] For example, for prediction tasks: each prediction moment corresponds to a set of soft information, or each feature at each prediction moment corresponds to a set of soft information; for other tasks, each feature corresponds to a set of soft information, or all features correspond to a set of soft information. For example, if the output of the first AI model is 2-dimensional, then it corresponds to 2 sets of soft information.

[0092] For example, for a positioning task, the first AI model output is a 2-dimensional horizontal position coordinate, and the 2-dimensional horizontal position coordinate corresponds to a set of soft information, or each dimension of the 2-dimensional horizontal position coordinate corresponds to a set of soft information.

[0093] In some embodiments, the AI ​​model described in this application may also be referred to as an AI unit, an AI model / AI unit, a machine learning (ML) model, an ML unit, an AI structure, an AI function, an AI feature, a neural network, a neural network function, a neural network function, etc. Alternatively, the AI ​​model described in this application may also refer to a processing unit that can implement specific algorithms, formulas, processing procedures, capabilities, etc. related to AI, or the AI ​​model described in this application may be a processing method, algorithm, function, module or unit for a specific data set, or the AI ​​model described in this application may be a processing method, algorithm, function, module or unit running on AI / ML related hardware such as a graphics processing unit (GPU), a neural network processing unit (NPU), a tensor processing unit (TPU), an application specific integrated circuit (ASIC), etc., and this application does not specifically limit this. Optionally, the specific data set includes the input or output of the AI ​​model.

[0094] In some embodiments, the identifier of the AI ​​model described in this application may be an AI unit identifier, an AI structure identifier, an AI algorithm identifier, or an identifier of a specific data set associated with the AI ​​model described in this application, or an identifier of a specific scenario, environment, channel feature, or device related to the AI ​​model described in this application, or an identifier of a function, feature, capability, or module related to the AI ​​model described in this application. This application does not make any specific limitations on this.

[0095] It should be noted that the generalization ability of AI models is limited. A model trained based on data from one scenario may fail when applied to another scenario. Even a model trained based on data from the same scenario will fail over time. Failure refers to the reduction in the inference accuracy of the AI ​​model and its inability to meet the target requirements. Therefore, the performance of the AI ​​model needs to be supervised.

[0096] In some embodiments, the first AI model may be an active model, or the first AI model may be an inactive model. Specifically, if the first AI model is an inactive model, after determining the validity of the first AI model, the valid AI model may be preferentially selected from the at least two AI models during AI model selection, thereby facilitating AI model selection.

[0097] In some embodiments, the first information includes, but is not limited to, at least one of the following: a parameter of the probability density distribution of the second feature, a confidence interval of the second feature, a value of the second feature, a value of the second feature and its probability, and a value of the second feature and its confidence level. The embodiments of this application clarify the content of the first information, which is beneficial for implementing performance monitoring of AI models.

[0098] Exemplarily, the amount of information contained in the first information may be one or at least two, that is, the first information may include but is not limited to at least one of the following: parameters of the probability density distribution of one or at least two second features, confidence intervals of one or at least two second features, values ​​of one or at least two second features, values ​​of one or at least two second features and their probabilities, and values ​​of one or at least two second features and their confidence levels.

[0099] In some embodiments, the parameters of the probability density distribution include, but are not limited to, at least one of the following: a mean or expectation of the probability density distribution, a variance or standard deviation of the probability density distribution, and an indication of the type of the probability density distribution.

[0100] For example, the type of probability density distribution may include but is not limited to at least one of the following: Gaussian distribution, Poisson distribution.

[0101] For example, for a Gaussian distribution, there is a conversion relationship between the confidence interval, the confidence level, the mean, and the standard deviation, such as a typical Gaussian distribution with a mean of μ and a standard deviation of σ.

[0102] For example, the 90% confidence interval is: [μ-1.645σ, μ+1.645σ], which means that there is a 90% probability that the predicted target value is within the interval [μ-1.645σ, μ+1.645σ].

[0103] For example, the 95% confidence interval is: [μ-1.96σ, μ+1.96σ], which means that there is a 95% probability that the predicted target value is within the interval [μ-1.96σ, μ+1.96σ].

[0104] For example, there may be a conversion relationship between the confidence interval, the confidence level, the mean, and the standard deviation, where the mean is μ and the standard deviation is σ. The coefficient z can be obtained by the probability density distribution function or the type of probability density distribution. The confidence interval with a confidence level of p% can be as follows: [μ-zp％ σ,μ+z p％ σ].

[0105] In some embodiments, the first information is output information of the first AI model, or the first information is information determined based on the output information of the first AI model.

[0106] Exemplarily, the output information of the first AI model is at least one of the following: a parameter of the probability density distribution of the second feature, a confidence interval of the second feature, a value of the second feature, the value of the second feature and its probability, and the value of the second feature and its confidence level. That is, the first information is the output information of the first AI model.

[0107] Exemplarily, the output information of the first AI model is the value of the second feature, and the first information can be determined based on the value of the second feature. Optionally, the first information is determined based on the value of the second feature and other information (such as motion state information (speed, acceleration, etc.), signal quality measurement information (such as RSRP, SINR, RSRQ, etc.)). For example, the worse the signal quality, the higher the uncertainty of the second feature obtained based on the value of the second feature and other information; for another example, the faster the movement speed, the higher the uncertainty of the second feature obtained based on the value of the second feature and other information.

[0108] For example, if the output information of the first AI model is the value of a second feature, at least one of the following can be determined based on the value of the second feature: a parameter of the probability density distribution of the second feature, a confidence interval of the second feature, a probability of the value of the second feature, or a confidence level of the value of the second feature. In other words, the first information is information determined based on the output information of the first AI model.

[0109] In some embodiments, the first AI model can be used to implement one of the following functions: positioning, beam management, channel state information (CSI) prediction, mobility management, and CSI compression. Of course, the first AI model can also be used to implement other functions, which is not limited in this application.

[0110] Specifically, the model performance supervision method 200 described in this embodiment can be used to implement at least two different functions, that is, the model performance supervision method 200 described in this embodiment can be applicable to a public label-based AI model supervision framework, thereby avoiding the need to design performance supervision solutions for different functions separately.

[0111] In some embodiments, when the first AI model is used to implement a positioning function, the first feature may include, but is not limited to, at least one of the following: a time domain channel impulse response, RSRP (such as layer 1 RSRP or layer 3 RSRP), a frequency domain channel impulse response, and a time domain waveform of a received signal. Optionally, the time domain channel impulse response includes at least one of the following: time information, power information, and phase information. Optionally, the frequency domain channel impulse response includes at least one of the following: frequency information (subcarrier sequence number and spacing), power information, and phase information.

[0112] In some embodiments, when the first AI model is used to implement the positioning function, the second feature may include but is not limited to at least one of the following: line of sight TOA, RSTD, AoA, AoD, RSRP, LOS indication, NLOS indication.

[0113] For example, when the first AI model is used to implement the positioning function, the input of the first AI model (i.e., the first feature) is the time domain channel impulse response, and the output of the first AI model is the soft information of the intermediate feature quantity (i.e., the second feature) (such as parameters of the probability density distribution, confidence interval, confidence level, etc.). The intermediate feature quantity includes at least one of the following: line of sight TOA, RSTD, AoA, AoD, RSRP, LOS indication, NLOS indication, etc. The position coordinates can be further determined based on the soft information of the intermediate feature quantity; the output of the first AI model can also be the soft information of the position coordinates.

[0114] In some embodiments, when the first AI model is used to implement the beam management function, the first feature may include beam information at T1 historical moments, such as sequence number, angle, L1-RSRP, etc., and the second feature includes beam information at T2 future moments.

[0115] For example, when the first AI model is used to implement the beam management function, the input of the first AI model (i.e., the first feature) is the beam information at T1 historical moments, such as serial number, angle, L1-RSRP, etc., and the output of the first AI model is the soft information (such as parameters of probability density distribution, confidence interval, confidence level, etc.) of the beam information (i.e., the second feature) at T2 future moments. For example, the vertical beam and the horizontal beam at each moment correspond to a set of soft information, such as the confidence interval and probability density distribution of L1-RSRP, etc., and the beam information at each moment corresponds to a set of soft information.

[0116] In some embodiments, when the first AI model is used to implement the CSI prediction function, the first feature may include the CSI at T1 historical moments, and the second feature may include the CSI at T2 future moments.

[0117] For example, when the first AI model is used to implement the CSI prediction function, the input of the first AI model (i.e., the first feature) is the CSI at T1 historical moments, and the output of the first AI model is the soft information (such as parameters of the probability density distribution, confidence interval, confidence level, etc.) of the CSI at T2 future moments (i.e., the second feature), such as the CSI at each moment corresponds to a set of soft information, each dimension of the CSI at each moment corresponds to a set of soft information, or the CSI at each moment corresponds to a set of soft information.

[0118] In some embodiments, when the first AI model is used to implement mobility management, the first feature may include the layer 1 RSRP (L1-RSRP) or layer 3 RSRP (L3-RSRP) at a historical time T1, and the second feature may include the RSRP at a future time T2 or the decision on whether to perform a cell handover at a future time T2. It should be noted that L3-RSRP is obtained by filtering L1-RSRP.

[0119] For example, when the first AI model is used to implement the mobility management function, the input of the first AI model (i.e., the first feature) is the L1-RSRP or L3-RSRP at the historical T1 time points, and the output of the first AI model is the soft information (such as the parameters of the probability density distribution, the confidence interval, the confidence level, etc.) of the RSRP (i.e., the second feature) at the next T2 time points; or the output of the first AI model is the soft information (such as the parameters of the probability density distribution, the confidence interval, the confidence level, etc.) of the decision on whether a cell switching will occur (i.e., the second feature) at the next T2 time points, such as 1 for switching and 0 for no switching, and the soft information is a value between 0 and 1.

[0120] In some embodiments, when the first AI model is used to implement the CSI compression function, the first feature may include uncompressed CSI, and the second feature may include compressed CSI.

[0121] For example, when the first AI model is used to implement the CSI compression function, the input of the first AI model (i.e., the first feature) is the uncompressed CSI, and the output of the first AI model is the soft information of the compressed CSI (i.e., the second feature) (such as parameters of the probability density distribution, confidence interval, confidence level, etc.).

[0122] In some embodiments, the label corresponding to the first feature (ie, the second information) may be real, or may be obtained by measurement or estimation, which is not limited in this embodiment of the present application.

[0123] It should be noted that the type of label corresponding to the first feature is consistent with the type of output of the first AI model. For example, both are location information, both are TOA, both are CSI, both are RSRP, etc.; this depends on the specific task type.

[0124] In some embodiments, the label corresponding to the first feature (i.e., the second information) may be measured, estimated, or stored by the first device, or the label corresponding to the first feature (i.e., the second information) may be obtained by the first device from the second device or other devices.

[0125] In some embodiments, in the above S210, the first device obtains the first information, including one of the following:

[0126] The first device receives first information from the second device;

[0127] The first device obtains first information through output information of the first AI model;

[0128] The first device receives output information of the first AI model from the second device, and obtains first information based on the output information of the first AI model.

[0129] In some embodiments, when the first device obtains the first information through output information of the first AI model, the first AI model can be deployed on the first device side.

[0130] In some embodiments, when a first device receives first information from a second device, the first AI model can be deployed on the second device. For example, the second device obtains the first information through the output information of the first AI model, and the second device sends the first information to the first device.

[0131] In some embodiments, when the first device receives output information of the first AI model from the second device and obtains first information based on the output information of the first AI model, the first AI model can be deployed on the second device side.

[0132] In some embodiments, the first device determines the validity of the first AI model based on the third information. Specifically, after the first device determines the third information based on the first information and the second information, the first device determines the validity of the first AI model based on the third information. Optionally, the first device sends first indication information to the second device, where the first indication information is used to indicate the validity of the first AI model. For example, the first indication information occupies 1 bit; wherein a value of 0 indicates that the first AI model is valid, and a value of 1 indicates that the first AI model is invalid; or, a value of 1 indicates that the first AI model is valid, and a value of 0 indicates that the first AI model is invalid. Optionally, the first indication information includes an identifier of the first AI model or an identifier of a function associated with the first AI model.

[0133] In some embodiments, the first device sends third information to the second device. Further, the second device can determine the validity of the first AI model based on the third information. Optionally, the first device receives second indication information from the second device, wherein the second indication information is used to indicate the validity of the first AI model. For example, the second indication information occupies 1 bit; wherein a value of 0 indicates that the first AI model is valid, and a value of 1 indicates that the first AI model is invalid; or, a value of 1 indicates that the first AI model is valid, and a value of 0 indicates that the first AI model is invalid. Optionally, the second indication information includes an identifier of the first AI model or an identifier of a function associated with the first AI model.

[0134] Exemplarily, when the first AI model is deployed on the first device side, the first device obtains first information through the output information of the first AI model, then the first device obtains second information from a local or other device (such as a second device or a device other than the first device and the second device), and the first device determines third information based on the first information and the second information, then, the first device determines the validity of the first AI model based on the third information, and finally, the first device sends first indication information to the second device, wherein the first indication information is used to indicate the validity of the first AI model.

[0135] Exemplarily, when the first AI model is deployed on the first device side, the first device obtains the first information through the output information of the first AI model, then the first device obtains the second information from the local or other device (such as the second device or a device other than the first device and the second device), and the first device determines the third information based on the first information and the second information, then, the first device sends the third information to the second device, after which the second device determines the validity of the first AI model based on the third information, and finally, the second device sends the second indication information to the first device, wherein the second indication information is used to indicate the validity of the first AI model.

[0136] Exemplarily, when the first AI model is deployed on the second device side, the second device obtains the first information through the output information of the first AI model, and the second device sends the first information to the first device. Then, the first device obtains the second information from the local or other device (such as the second device or a device other than the first device and the second device), and the first device determines the third information based on the first information and the second information. Then, the first device determines the validity of the first AI model based on the third information. Finally, the first device sends the first indication information to the second device, wherein the first indication information is used to indicate the validity of the first AI model.

[0137] Exemplarily, when the first AI model is deployed on the second device side, the second device obtains the first information through the output information of the first AI model, and the second device sends the first information to the first device. Then, the first device obtains the second information from the local or other device (such as the second device or a device other than the first device and the second device), and the first device determines the third information based on the first information and the second information. Then, the first device sends the third information to the second device. Thereafter, the second device determines the validity of the first AI model based on the third information. Finally, the second device sends the second indication information to the first device, wherein the second indication information is used to indicate the validity of the first AI model.

[0138] Exemplarily, when the first AI model is deployed on the second device side, the first device receives the output information of the first AI model from the second device, and the first device obtains the first information based on the output information of the first AI model. Then, the first device obtains the second information from the local or other device (such as the second device or a device other than the first device and the second device), and the first device determines the third information based on the first information and the second information. Then, the first device determines the validity of the first AI model based on the third information. Finally, the first device sends the first indication information to the second device, wherein the first indication information is used to indicate the validity of the first AI model.

[0139] Exemplarily, when the first AI model is deployed on the second device side, the first device receives the output information of the first AI model from the second device, and the first device obtains the first information based on the output information of the first AI model. Then, the first device obtains the second information from the local or other device (such as the second device or a device other than the first device and the second device), and the first device determines the third information based on the first information and the second information. Then, the first device sends the third information to the second device, and then the second device determines the validity of the first AI model based on the third information. Finally, the second device sends the second indication information to the first device, wherein the second indication information is used to indicate the validity of the first AI model.

[0140] In some embodiments, the first device may be a terminal, a network-side device, or a third-party server, wherein the network-side device includes an access network device or a core network device.

[0141] In some embodiments, the second device may be a terminal, a network-side device, or a third-party server, wherein the network-side device includes an access network device or a core network device.

[0142] In some embodiments, when the first information includes parameters of the probability density distribution of the second feature, the third information is used to represent the mean of the first probabilities of N samples of the first feature, or the third information is used to represent the mean of the logarithms of the first probabilities of N samples of the first feature; wherein the first probability is the probability of the label corresponding to each sample in the N samples under the probability density distribution of the second feature, and N is a positive integer. Optionally, the N samples can also be replaced by N inference processes of the first AI model.

[0143] Exemplarily, when the first information includes parameters of the probability density distribution of the second feature, the third information may be a log-likelihood value.

[0144] In some embodiments, when the third information is used to represent the mean of the first probabilities of N samples of the first feature, the third information is determined based on the following formula 1:

[0145] Wherein, L1 represents the third information, i represents the i-th sample in the N samples, Represents the standard deviation or variance of the probability density distribution of the second feature corresponding to the i-th sample, represents the mean (such as statistical mean) of the probability density distribution of the second feature corresponding to the i-th sample, y i represents the label corresponding to the i-th sample, and g(·) represents the probability density distribution function obeyed by the output of the first AI model.

[0146] In some embodiments, when the third information is used to represent the mean of the logarithm of the first probability of N samples of the first feature, the third information is determined based on the following formula 2:

[0147] Wherein, L2 represents the third information, i represents the i-th sample in the N samples, represents the mean (such as the statistical mean) of the probability density distribution of the second feature corresponding to the i-th sample, Represents the standard deviation or variance of the probability density distribution of the second feature corresponding to the i-th sample, y i represents the label corresponding to the i-th sample, s is a positive integer, and g(·) represents the probability density distribution function obeyed by the output of the first AI model.

[0148] Optionally, s is agreed upon by a protocol, or s is determined by the first device, or s is configured or indicated by the second device.

[0149] In some embodiments, in the above formula 1 or formula 2, g(·) is related to the type of probability density distribution, for example, g(·) can be determined based on the type of probability density distribution.

[0150] For example, taking the probability density distribution type as Gaussian distribution as an example, the above formula 1 can be transformed into the following formula 3.

[0151] For example, taking the probability density distribution type as Gaussian distribution as an example, the above formula 2 can be transformed into the following formula 4.

[0152] In some embodiments, when the first information includes parameters of the probability density distribution of the second feature, the third information is used to determine the validity of the first AI model, including:

[0153] When the third information is greater than or equal to the first threshold, the first AI model is valid; or

[0154] When the third information is less than the first threshold, the first AI model fails.

[0155] Optionally, the first threshold is agreed upon by a protocol, or the first threshold is determined by the first device, or the first threshold is configured or indicated by the second device.

[0156] In some embodiments, when the first information includes the confidence interval of the second feature, the third information is used to represent the ratio of the number of samples of the first feature whose corresponding labels are within the confidence interval of the second feature to the total number of samples N, where N is a positive integer. Optionally, the N samples can also be replaced by N inference processes of the first AI model.

[0157] In some embodiments, when the first information includes a confidence interval of the second feature, the third information is determined based on the following formula 5:

[0158] Wherein, L3 represents the third information, i represents the i-th sample in the N samples, and x i Indicates the input information corresponding to the i-th sample, y i Indicates the label corresponding to the i-th sample. When y i The confidence interval f(x i ) within , otherwise

[0159] In some embodiments, when the first information includes a confidence interval of the second feature, the third information is used to determine the validity of the first AI model, including:

[0160] When the third information is greater than or equal to the second threshold, the first AI model is valid; or

[0161] When the third information is less than the second threshold, the first AI model fails.

[0162] Optionally, the second threshold is agreed upon by a protocol, or the second threshold is determined by the first device, or the second threshold is configured or indicated by the second device.

[0163] In some embodiments, when the first information includes the value of the second feature and its probability or confidence, the third information is used to represent the average of the weighted distances between the labels corresponding to N samples of the first feature and the value of the second feature, where the weight is determined based on the probability or confidence of the second feature, and N is a positive integer. Optionally, the N samples can also be replaced by N inference processes of the first AI model.

[0164] In some embodiments, when the first information includes the value of the second feature and its probability or confidence, the third information is determined based on the following formula 6:

[0165] Wherein, L4 represents the third information, i represents the i-th sample in the N samples, Represents the mean of the probability density distribution of the second feature corresponding to the i-th sample, y i Indicates the label corresponding to the i-th sample.

[0166] For example, in the above formula 6, Σ -1 The diagonal elements of are P powers of the probabilities or confidences, where P is greater than or equal to 0 and P is an integer.

[0167] In some embodiments, when the first information includes the value of the second feature and its probability or confidence, the third information is used to determine the validity of the first AI model, including:

[0168] When the third information is less than or equal to a third threshold, the first AI model is valid; or

[0169] When the third information is greater than a third threshold, the first AI model fails.

[0170] Optionally, the third threshold is agreed upon by a protocol, or the third threshold is determined by the first device, or the third threshold is configured or indicated by the second device.

[0171] It should be noted that in the above formula, y i 、 It can be a specific numerical value, or a vector or matrix, which is not limited in the embodiments of the present application.

[0172] In some embodiments, the N samples are acquired within M time units, where M is a positive integer.

[0173] Optionally, the time unit may include at least one of the following: an orthogonal frequency-division multiplexing (OFDM) symbol, a time slot, a subframe, a frame, microseconds, milliseconds, seconds, minutes, hours, days, weeks, and months.

[0174] Optionally, M is configured or indicated by the second device, or M is agreed upon by a protocol.

[0175] In some embodiments, the scene identifiers or data set identifiers associated with the N samples are the same.

[0176] In some embodiments, the first device may obtain at least one of the following parameters from the second device:

[0177] The number of samples N used for AI model performance supervision;

[0178] The minimum number of samples N used to monitor AI model performance;

[0179] Get the time range T of N samples;

[0180] The type and parameters of the probability density distribution; for example, in the case of a Gaussian mixture model, the number of Gaussian distributions involved must also be specified;

[0181] Parameter P;

[0182] Parameter M.

[0183] In some embodiments, the first device reports at least one of the following parameters to the second device:

[0184] The number of samples N used for AI model performance supervision;

[0185] The minimum number of samples N used to monitor AI model performance;

[0186] Get the time range T of N samples;

[0187] The type and parameters of the probability density distribution; for example, in the case of a Gaussian mixture model, the number of Gaussian distributions involved must also be specified;

[0188] Parameter P;

[0189] Parameter M.

[0190] In some embodiments, the third information may be positive incentive information or negative incentive information; for example, when the negative incentive information is greater than a certain threshold or the positive incentive information is less than or equal to a certain threshold, the first AI model is considered to be invalid.

[0191] For example, the positive motivation information may include but is not limited to at least one of the following:

[0192] Among the N samples of the first feature, or the sample size or proportion;

[0193] The number or proportion of samples whose labels are within the confidence interval among the N samples of the first feature;

[0194] The number or proportion of samples in the N samples of the first feature whose weighted distance between the label and the value of the second feature is less than or equal to t3.

[0195] Exemplarily, negative incentive information may include, but is not limited to, at least one of the following:

[0196] Among the N samples of the first feature, or the sample size or proportion;

[0197] The number or proportion of samples whose labels are outside the confidence interval among the N samples of the first feature;

[0198] The number or proportion of samples in which the weighted distance between the label and the value of the second feature is greater than t3 among the N samples of the first feature.

[0199] Specifically, t1 may be the first threshold, t2 may be the second threshold, and t3 may be the third threshold.

[0200] Therefore, in an embodiment of the present application, the third information can be determined based on the uncertainty or probability distribution of the output of the first AI model (i.e., the first information) and the label corresponding to the input feature of the first AI model (i.e., the second information), and the validity of the first AI model can be determined based on the third information, thereby achieving performance supervision of the first AI model. Specifically, the uncertainty or probability distribution of the output of the first AI model can improve the robustness of the reasoning results of the first AI model. Since the reasoning results of the first AI model cover multiple possible results and their probability distributions, it is conducive to further processing and utilization of the reasoning results. For example, the reasoning results of the first AI model (including relevant information on uncertainty or probability distribution) are combined with other types of information to obtain a more accurate target result. Referring to the uncertainty or probability distribution of the output of the first AI model and the corresponding label in the performance supervision of the first AI model can improve the accuracy of the performance supervision of the first AI model.

[0201] The following describes a specific example of an AI model positioning solution based on soft information. Soft information can improve the robustness of the AI ​​model's inference results because the inference results cover multiple possible outcomes and their probability distributions. This facilitates further processing and utilization of the AI ​​model's inference results. For example, soft information can be combined with other types of information (such as other information output by the AI ​​model used for positioning) to obtain more accurate target results. It can also provide better security for services that require high inference reliability.

[0202] Example 1

[0203] The input of the i-th AI model is the time-domain channel impulse response (CIR) of the i-th transmission reception point (TRP), which includes the time, power, and phase information of the multipath.

[0204] The output of the i-th AI model is the mean value μ of the line-of-sight TOA between the i-th TRP and the terminal. i and standard deviation σ i ;

[0205] The TOA estimate x between the i-th TRP and the terminal i Modeled as a Gaussian distribution:

[0206] For N TRPs, the likelihood function is modeled as:

[0207] Where x=[x1,...,x N ] T ;

[0208] The maximum likelihood estimation problem can be transformed into a weighted least squares problem:

[0209] Where μ=[μ1,...,μ N ] T ,Σ is an N*N matrix, whose i-th diagonal element is Its goal is to find a position The N TOA estimates obtained by combining this position with the coordinates of the N TRPs The weighted distance from the N TOA mean μ estimated by the AI ​​model is the smallest.

[0210] because and The relationship between is nonlinear, so it can be solved by linear approximation and greedy algorithm. The following takes the particle swarm optimization algorithm as an example to provide a specific implementation scheme and simulation results, as shown in Figure 5.

[0211] Among them, different optimization algorithms, TRP numbers, and corresponding positioning accuracy can be shown in Table 1 below.

[0212] Table 1

[0213] In addition, the framework can support mixed positioning of multiple types of soft information. The above example only gives the soft information μ,σ of the TOA of N TRPs, but it can also include soft information of angles, etc. Its likelihood function can be written as follows:

[0214] x, α, and z refer to different types of information, such as TOA, AOD, and AOA, respectively.

[0215] Secondly, the number of TRPs for different types of information can also be different. For example, the number of TRPs for x is N1, the number of TRPs for α is N2, and the number of TRPs for z is N3. Their likelihood functions can be written as follows:

[0216] The above, in combination with Figures 4 to 5, describes in detail the first device side embodiment of the present application. The following, in combination with Figure 6, describes in detail the second device side embodiment of the present application. It should be understood that the second device side embodiment corresponds to the first device side embodiment, and similar descriptions can refer to the first device side embodiment.

[0217] FIG6 is a schematic flow chart of a model performance supervision method 300 according to an embodiment of the present application. As shown in FIG6 , the model performance supervision method 300 may include at least part of the following contents:

[0218] S310: The second device receives third information from the first device, where the third information is determined based on the first information and second information, the first information is used to characterize the uncertainty or probability distribution of the output of the first AI model, the input of the first AI model is a first feature, and the second information is a label corresponding to the first feature;

[0219] S320: The second device determines the validity of the first AI model based on the third information.

[0220] It should be understood that FIG6 shows the steps or operations of the model performance supervision method 300, but these steps or operations are merely examples, and the embodiments of the present application may also perform other operations or variations of the operations in FIG6.

[0221] In some embodiments, the first information includes at least one of the following:

[0222] Parameters of the probability density distribution of the second feature, a confidence interval of the second feature, the value of the second feature, the value of the second feature and its probability, and the value of the second feature and its confidence level.

[0223] In some embodiments, the parameters of the probability density distribution include at least one of the following: a mean or expectation of the probability density distribution, a variance or standard deviation of the probability density distribution, and an indication of the type of the probability density distribution.

[0224] In some embodiments, when the first information includes a parameter of a probability density distribution of the second feature, the third information is used to represent a mean of the first probabilities of N samples of the first feature, or the third information is used to represent a mean of logarithms of the first probabilities of N samples of the first feature;

[0225] The first probability is the probability of the label corresponding to each of the N samples under the probability density distribution of the second feature, and N is a positive integer.

[0226] In some embodiments, when the third information is used to represent the mean of the first probabilities of N samples of the first feature, the third information is determined based on the following formula:

[0227] Wherein, L1 represents the third information, i represents the i-th sample in the N samples, Represents the standard deviation or variance of the probability density distribution of the second feature corresponding to the i-th sample, Represents the mean of the probability density distribution of the second feature corresponding to the i-th sample, y i represents the label corresponding to the i-th sample, and g(·) represents the probability density distribution function obeyed by the output of the first AI model.

[0228] In some embodiments, when the third information is used to represent the mean of the logarithm of the first probability of N samples of the first feature, the third information is determined based on the following formula:

[0229] Wherein, L2 represents the third information, i represents the i-th sample in the N samples, Represents the mean of the probability density distribution of the second feature corresponding to the i-th sample, Represents the standard deviation or variance of the probability density distribution of the second feature corresponding to the i-th sample, y i represents the label corresponding to the i-th sample, s is a positive integer, and g(·) represents the probability density distribution function obeyed by the output of the first AI model.

[0230] In some embodiments, when the first information includes parameters of the probability density distribution of the second feature, the above S320 may specifically include:

[0231] When the third information is greater than or equal to the first threshold, the second device determines that the first AI model is valid; or

[0232] When the third information is less than the first threshold, the second device determines that the first AI model is invalid.

[0233] In some embodiments, when the first information includes the confidence interval of the second feature, the third information is used to characterize the ratio of the number of samples whose corresponding labels are within the confidence interval of the second feature in the N samples of the first feature to the total number of samples N, where N is a positive integer.

[0234] In some embodiments, when the first information includes a confidence interval of the second feature, the third information is determined based on the following formula:

[0235] Wherein, L3 represents the third information, i represents the i-th sample in the N samples, and x i Indicates the input information corresponding to the i-th sample, y i Indicates the label corresponding to the i-th sample. When y i The confidence interval f(x i ) within , otherwise

[0236] In some embodiments, when the first information includes the confidence interval of the second feature, the above S320 may specifically include:

[0237] When the third information is greater than or equal to the second threshold, the second device determines that the first AI model is valid; or

[0238] When the third information is less than the second threshold, the second device determines that the first AI model is invalid.

[0239] In some embodiments, when the first information includes the value of the second feature and its probability or confidence, the third information is used to characterize the mean of the weighted distances between the labels corresponding to N samples of the first feature and the value of the second feature, where the weighted weight is determined based on the probability or confidence of the second feature, and N is a positive integer.

[0240] In some embodiments, when the first information includes the value of the second feature and its probability or confidence, the third information is determined based on the following formula:

[0241] Wherein, L4 represents the third information, i represents the i-th sample in the N samples, Represents the mean of the probability density distribution of the second feature corresponding to the i-th sample, y i Indicates the label corresponding to the i-th sample.

[0242] In some embodiments, when the first information includes the value of the second feature and its probability or confidence, the above S320 may specifically include:

[0243] When the third information is less than or equal to the third threshold, the second device determines that the first AI model is valid; or

[0244] When the third information is greater than a third threshold, the second device determines that the first AI model is invalid.

[0245] In some embodiments, before the second device receives the third information from the first device, the second device sends the first information to the first device.

[0246] In some embodiments, the second device sends second indication information to the first device, wherein the second indication information is used to indicate the validity of the first AI model. Optionally, the second indication information includes an identifier of the first AI model or an identifier of a function associated with the first AI model.

[0247] Therefore, in an embodiment of the present application, the third information can be determined based on the uncertainty or probability distribution of the output of the first AI model (i.e., the first information) and the label corresponding to the input feature of the first AI model (i.e., the second information), and the validity of the first AI model can be determined based on the third information, thereby achieving performance supervision of the first AI model. Specifically, the uncertainty or probability distribution of the output of the first AI model can improve the robustness of the reasoning results of the first AI model. Since the reasoning results of the first AI model cover multiple possible results and their probability distributions, it is conducive to further processing and utilization of the reasoning results. For example, the reasoning results of the first AI model (including relevant information on uncertainty or probability distribution) are combined with other types of information to obtain a more accurate target result. Referring to the uncertainty or probability distribution of the output of the first AI model and the corresponding label in the performance supervision of the first AI model can improve the accuracy of the performance supervision of the first AI model.

[0248] FIG7 is a schematic flow chart of a model performance supervision method 400 according to an embodiment of the present application. As shown in FIG7 , the model performance supervision method 400 may include at least part of the following contents:

[0249] S410: The first device obtains fourth information, where the fourth information is used to characterize uncertainty or probability distribution of an output of a second AI model, where the input of the second AI model is a third feature.

[0250] S420: The first device determines the validity of the second AI model based on the fourth information; or, the first device sends the fourth information to the second device.

[0251] It should be understood that FIG7 shows the steps or operations of the model performance supervision method 400, but these steps or operations are merely examples, and the embodiments of the present application may also perform other operations or variations of the operations in FIG7.

[0252] In an embodiment of the present application, the validity of the second AI model can be determined based on the uncertainty or probability distribution of the output of the second AI model, thereby achieving performance supervision of the second AI model. Specifically, the uncertainty or probability distribution of the output of the second AI model can improve the robustness of the reasoning results of the second AI model. Since the reasoning results of the second AI model cover multiple possible results and their probability distributions, it is conducive to further processing and utilization of the reasoning results. For example, the reasoning results of the second AI model (including relevant information about uncertainty or probability distribution) are combined with other types of information to obtain a more accurate target result. In the performance supervision of the second AI model, referring to the uncertainty or probability distribution of the output of the second AI model can determine the validity of the second AI model. Since no external information is required, it is also easier to implement.

[0253] Exemplarily, the judgment result of the validity of the second AI model obtained based on the embodiment of the present application can also be combined with the judgment result of the validity of the second AI model obtained by other model supervision methods to obtain the final conclusion on the validity of the second AI model.

[0254] It should be noted that in the embodiment of the present application, the higher the uncertainty of the output of the second AI model, the lower the accuracy of the output of the second AI model.

[0255] Illustratively, the second AI model described in the embodiment of the present application may be the same as or different from the first AI model described above, and the present application does not limit this.

[0256] In an embodiment of the present application, the fourth information may also be referred to as soft information (such as probability distribution, confidence level, confidence interval, etc.), wherein the soft information gives the probability distribution or confidence level of the possible results. Optionally, the second AI model may be a soft information AI model or may not be a soft information AI model. Among them, the soft information AI model refers to a type of AI model that outputs soft information (such as probability distribution, confidence level, confidence interval, etc.), including both classic probability models and AI models based on neural networks; the soft information AI model measures the possibility of different prediction results and gives the possibility, probability distribution or confidence level of each possible result.

[0257] It's important to note that compared to hard information (such as line-of-sight (TOA), RSTD, AoA, AoD, RSRP, LOS indication, NLOS indication, etc.) derived from AI model reasoning, soft information derived from AI model reasoning can significantly improve reasoning accuracy and robustness. Specifically, soft information can better describe the uncertainty of the world, improve the robustness of model reasoning, and provide better security for certain services that require relatively high reasoning reliability.

[0258] In some embodiments, the dimension of the third feature is D3, and correspondingly, the fourth information is D4 groups of soft information of the fourth feature, where D4 groups of soft information describe the range of the output of the second AI model. If there is only one group of soft information (i.e., D4=1), then this group of soft information describes the uncertainty of the entire second AI model output; if there are at least two groups of soft information (i.e., D4≥2), then each group of soft information describes the uncertainty of part of the features output by the second AI model.

[0259] For example, for prediction tasks: each prediction moment corresponds to a set of soft information, or each feature at each prediction moment corresponds to a set of soft information; for other tasks, each feature corresponds to a set of soft information, or all features correspond to a set of soft information.

[0260] For example, each feature corresponds to a set of soft information, and the output of the second AI model is 2-dimensional, corresponding to 2 sets of soft information.

[0261] For example, for the positioning task, the second AI model output is a 2-dimensional horizontal position coordinate, the 2-dimensional horizontal position coordinate corresponds to a set of soft information, or each dimension of the 2-dimensional horizontal position coordinate corresponds to a set of soft information.

[0262] In some embodiments, the AI ​​model described in this application may also be referred to as an AI unit, an AI model / AI unit, an ML model, an ML unit, an AI structure, an AI function, an AI feature, a neural network, a neural network function, a neural network function, etc. Alternatively, the AI ​​model described in this application may also refer to a processing unit that can implement specific algorithms, formulas, processing flows, capabilities, etc. related to AI, or the AI ​​model described in this application may be a processing method, algorithm, function, module or unit for a specific data set, or the AI ​​model described in this application may be a processing method, algorithm, function, module or unit running on AI / ML related hardware such as a GPU, NPU, TPU, ASIC, etc., which is not specifically limited in this application. Optionally, the specific data set includes the input or output of the AI ​​model.

[0263] In some embodiments, the identifier of the AI ​​model described in this application may be an AI unit identifier, an AI structure identifier, an AI algorithm identifier, or an identifier of a specific data set associated with the AI ​​model described in this application, or an identifier of a specific scenario, environment, channel feature, or device related to the AI ​​model described in this application, or an identifier of a function, feature, capability, or module related to the AI ​​model described in this application. This application does not make any specific limitations on this.

[0264] It should be noted that the generalization ability of AI models is limited. A model trained based on data from one scenario may fail when applied to another scenario. Even a model trained based on data from the same scenario will fail over time. Failure refers to the reduction in the inference accuracy of the AI ​​model and its inability to meet the target requirements. Therefore, the performance of the AI ​​model needs to be supervised.

[0265] In some embodiments, the second AI model may be an active model, or the second AI model may be an inactive model. Specifically, if the second AI model is an inactive model, after determining the validity of the second AI model, the valid AI model may be preferentially selected from the at least two AI models during AI model selection, thereby facilitating AI model selection.

[0266] In some embodiments, the fourth information includes, but is not limited to, at least one of the following: a parameter of the probability density distribution of the fourth feature, a confidence interval of the fourth feature, a value of the fourth feature, the value of the fourth feature and its probability, and the value of the fourth feature and its confidence. The embodiments of this application clarify the content of the fourth information, which is beneficial for implementing performance monitoring of AI models.

[0267] Exemplarily, the amount of information contained in the fourth information may be one or at least two, that is, the fourth information may include but is not limited to at least one of the following: parameters of the probability density distribution of one or at least two fourth features, confidence intervals of one or at least two fourth features, values ​​of one or at least two fourth features, values ​​of one or at least two fourth features and their probabilities, and values ​​of one or at least two fourth features and their confidence levels.

[0268] In some embodiments, the parameters of the probability density distribution include, but are not limited to, at least one of the following: a mean or expectation of the probability density distribution, a variance or standard deviation of the probability density distribution, and an indication of the type of the probability density distribution.

[0269] For example, the type of probability density distribution may include but is not limited to at least one of the following: Gaussian distribution, Poisson distribution.

[0270] For example, for a Gaussian distribution, there is a conversion relationship between the confidence interval, the confidence level, the mean, and the standard deviation, such as a typical Gaussian distribution with a mean of μ and a standard deviation of σ.

[0271] For example, the 90% confidence interval is: [μ-1.645σ, μ+1.645σ], which means that there is a 90% probability that the predicted target value is within the interval [μ-1.645σ, μ+1.645σ].

[0272] For example, the 95% confidence interval is: [μ-1.96σ, μ+1.96σ], which means that there is a 95% probability that the predicted target value is within the interval [μ-1.96σ, μ+1.96σ].

[0273] For example, there may be a conversion relationship between the confidence interval, the confidence level, the mean, and the standard deviation, where the mean is μ and the standard deviation is σ. The coefficient z can be obtained by the probability density distribution function or the type of probability density distribution. The confidence interval with a confidence level of p% can be as follows: [μ-z p％ σ,μ+z p％ σ].

[0274] In some embodiments, the fourth information is output information of the second AI model, or the fourth information is information determined based on the output information of the second AI model.

[0275] Exemplarily, the output information of the second AI model is at least one of the following: a parameter of the probability density distribution of the fourth feature, a confidence interval of the fourth feature, a value of the fourth feature, the value of the fourth feature and its probability, and the value of the fourth feature and its confidence level. That is, the fourth information is the output information of the second AI model.

[0276] Exemplarily, the output information of the second AI model is the value of the fourth feature, and the fourth information can be determined based on the value of the fourth feature, such as based on the value of the fourth feature and other information (such as motion state information (speed, acceleration, etc.), signal quality measurement information (such as RSRP, SINR, RSRQ, etc.)). For example, the worse the signal quality, the higher the uncertainty of the fourth feature obtained based on the value of the fourth feature and other information; for another example, the faster the motion speed, the higher the uncertainty of the fourth feature obtained based on the value of the fourth feature and other information.

[0277] For example, if the output information of the second AI model is the value of the fourth feature, at least one of the following can be determined based on the value of the fourth feature: a parameter of the probability density distribution of the fourth feature, a confidence interval of the fourth feature, a probability of the value of the fourth feature, and a confidence level of the value of the fourth feature. In other words, the fourth information is information determined based on the output information of the second AI model.

[0278] In some embodiments, the first AI model can be used to implement one of the following functions: positioning, beam management, CSI prediction, mobility management, and CSI compression. Of course, the first AI model can also be used to implement other functions, which is not limited in this application.

[0279] Specifically, the model performance supervision method 400 described in this embodiment can be used to implement at least two different functions, that is, the model performance supervision method 400 described in this embodiment can be applicable to a public AI model supervision framework based on Ground truth label, thereby avoiding the need to design performance supervision solutions for different functions.

[0280] In some embodiments, when the second AI model is used to implement the positioning function, the third feature may include, but is not limited to, at least one of the following: a time domain channel impulse response, RSRP (such as layer 1 RSRP or layer 3 RSRP), a frequency domain channel impulse response, and a time domain waveform of the received signal. Optionally, the time domain channel impulse response includes at least one of the following: time information, power information, and phase information. Optionally, the frequency domain channel impulse response includes at least one of the following: frequency information (subcarrier sequence number and spacing), power information, and phase information.

[0281] In some embodiments, when the second AI model is used to implement the positioning function, the fourth feature may include but is not limited to at least one of the following: line of sight TOA, RSTD, AoA, AoD, RSRP, LOS indication, NLOS indication.

[0282] For example, when the second AI model is used to implement the positioning function, the input of the second AI model (i.e., the third feature) is the time domain channel impulse response, and the output of the second AI model is the soft information of the intermediate feature quantity (i.e., the fourth feature) (such as parameters of the probability density distribution, confidence interval, confidence level, etc.). The intermediate feature quantity includes at least one of the following: line of sight TOA, RSTD, AoA, AoD, RSRP, LOS indication, NLOS indication, etc. The position coordinates can be further determined based on the soft information of the intermediate feature quantity; the output of the second AI model can also be the soft information of the position coordinates.

[0283] In some embodiments, when the second AI model is used to implement the beam management function, the third feature may include beam information at historical T1 moments, such as sequence number, angle, L1-RSRP, etc., and the fourth feature includes beam information at future T2 moments.

[0284] For example, when the second AI model is used to implement the beam management function, the input of the second AI model (i.e., the third feature) is the beam information at T1 historical moments, such as serial number, angle, L1-RSRP, etc., and the output of the second AI model is the soft information (such as parameters of probability density distribution, confidence interval, confidence level, etc.) of the beam information (i.e., the fourth feature) at T2 future moments. For example, the vertical beam and horizontal beam at each moment correspond to a set of soft information respectively, such as the confidence interval and probability density distribution of L1-RSRP, etc., and the beam information at each moment corresponds to a set of soft information.

[0285] In some embodiments, when the second AI model is used to implement the CSI prediction function, the third feature may include the CSI at T1 historical moments, and the fourth feature may include the CSI at T2 future moments.

[0286] For example, when the second AI model is used to implement the CSI prediction function, the input of the second AI model (i.e., the third feature) is the CSI at T1 historical moments, and the output of the second AI model is the soft information (such as parameters of the probability density distribution, confidence interval, confidence level, etc.) of the CSI at T2 future moments (i.e., the fourth feature), such as the CSI at each moment corresponds to a set of soft information, each dimension of the CSI at each moment corresponds to a set of soft information, or the CSI at each moment corresponds to a set of soft information.

[0287] In some embodiments, when the second AI model is used to implement mobility management, the third feature may include the layer 1 RSRP (L1-RSRP) or layer 3 RSRP (L3-RSRP) at the historical time T1, and the fourth feature may include the RSRP at the next time T2 or the decision on whether to perform a cell handover at the next time T2. It should be noted that L3-RSRP is obtained by filtering L1-RSRP.

[0288] For example, when the second AI model is used to implement the mobility management function, the input of the second AI model (i.e., the third feature) is the L1-RSRP or L3-RSRP at the historical T1 time moments, and the output of the second AI model is the soft information (such as the parameters of the probability density distribution, the confidence interval, the confidence level, etc.) of the RSRP (i.e., the fourth feature) at the next T2 time moments; or the output of the second AI model is the soft information (such as the parameters of the probability density distribution, the confidence interval, the confidence level, etc.) of the decision on whether a cell switching will occur (i.e., the fourth feature) at the next T2 time moments, such as 1 for switching and 0 for no switching, and the soft information is a value between 0 and 1.

[0289] In some embodiments, when the second AI model is used to implement the CSI compression function, the third feature may include uncompressed CSI, and the fourth feature may include compressed CSI.

[0290] For example, when the second AI model is used to implement the CSI compression function, the input of the second AI model (i.e., the third feature) is the uncompressed CSI, and the output of the second AI model is the soft information of the compressed CSI (i.e., the fourth feature) (such as parameters of the probability density distribution, confidence interval, confidence level, etc.).

[0291] In some embodiments, in the above S410, the first device obtains the fourth information, including one of the following:

[0292] The first device obtains the fourth information from the second device;

[0293] The first device obtains the fourth information through the output information of the second AI model;

[0294] The first device receives the output information of the second AI model from the second device, and obtains the fourth information according to the output information of the second AI model.

[0295] In some embodiments, when the first device obtains the fourth information through the output information of the second AI model, the second AI model can be deployed on the first device side.

[0296] In some embodiments, when the first device obtains the fourth information from another device, the second AI model can be deployed on the other device, where the other device can be the second device or a device other than the first device and the second device.

[0297] In some embodiments, when the first device receives output information of the second AI model from the second device and obtains fourth information based on the output information of the second AI model, the second AI model can be deployed on the second device side.

[0298] In some embodiments, when the first device determines the validity of the second AI model based on the fourth information, the first device sends third indication information to the second device, where the third indication information is used to indicate the validity of the second AI model. For example, the third indication information occupies one bit; a value of 0 indicates that the second AI model is valid, and a value of 1 indicates that the second AI model is invalid; or a value of 1 indicates that the second AI model is valid, and a value of 0 indicates that the second AI model is invalid.

[0299] Optionally, the third indication information includes an identifier of the second AI model or an identifier of a function associated with the second AI model.

[0300] In some embodiments, when a first device sends fourth information to a second device, the first device receives fourth indication information from the second device, where the fourth indication information is used to indicate the validity of the second AI model. Specifically, after receiving the fourth information, the second device can determine the validity of the second AI model based on the fourth information. For example, the fourth indication information occupies one bit; a value of 0 indicates that the second AI model is valid, and a value of 1 indicates that the second AI model is invalid; or a value of 1 indicates that the second AI model is valid, and a value of 0 indicates that the second AI model is invalid.

[0301] Optionally, the fourth indication information includes an identifier of the second AI model or an identifier of a function associated with the second AI model.

[0302] Exemplarily, when the second AI model is deployed on the first device side, the first device obtains the fourth information through the output information of the second AI model. Then, the first device determines the validity of the second AI model based on the fourth information. Finally, the first device sends third indication information to the second device, where the third indication information is used to indicate the validity of the second AI model.

[0303] Exemplarily, when the second AI model is deployed on the first device side, the first device obtains the fourth information through the output information of the second AI model. Then, the first device sends the fourth information to the second device. Thereafter, the second device determines the validity of the second AI model based on the fourth information. Finally, the second device sends fourth indication information to the first device, wherein the fourth indication information is used to indicate the validity of the second AI model.

[0304] Exemplarily, when the second AI model is deployed on the second device side, the second device obtains the fourth information through the output information of the second AI model, and the second device sends the fourth information to the first device. Then, the first device determines the validity of the second AI model based on the fourth information. Finally, the first device sends third indication information to the second device, wherein the third indication information is used to indicate the validity of the second AI model.

[0305] Exemplarily, when the second AI model is deployed on the third device side, the third device obtains the fourth information through the output information of the second AI model, and the third device sends the fourth information to the first device. Then, the first device sends the fourth information to the second device. Thereafter, the second device determines the validity of the second AI model based on the fourth information. Finally, the second device sends fourth indication information to the first device, wherein the fourth indication information is used to indicate the validity of the second AI model.

[0306] Exemplarily, when the second AI model is deployed on the second device side, the first device receives the output information of the second AI model from the second device, and the first device obtains fourth information based on the output information of the second AI model. Then, the first device determines the validity of the second AI model based on the fourth information. Finally, the first device sends third indication information to the second device, where the third indication information is used to indicate the validity of the second AI model.

[0307] Exemplarily, when the second AI model is deployed on the second device side, the first device receives the output information of the second AI model from the second device, and the first device obtains fourth information based on the output information of the second AI model. Then, the first device sends the fourth information to the second device. Thereafter, the second device determines the validity of the second AI model based on the fourth information. Finally, the second device sends fourth indication information to the first device, wherein the fourth indication information is used to indicate the validity of the second AI model.

[0308] In some embodiments, the first device may be a terminal, a network-side device, or a third-party server.

[0309] In some embodiments, the second device may be a terminal, a network-side device, or a third-party server.

[0310] In some embodiments, when the fourth information includes parameters of the probability density distribution of the fourth feature, the first device determines the validity of the second AI model based on the fourth information, including:

[0311] When the variance or standard deviation of the probability density distribution of the fourth feature is greater than or equal to the first threshold, the first device determines that the second AI model is invalid; or

[0312] When the mean of the variance or standard deviation of the probability density distribution of the fourth feature corresponding to the N samples of the third feature is greater than or equal to the second threshold, the first device determines that the second AI model is invalid; or

[0313] If, among the N samples of the third feature, the proportion of the number of samples for which the variance or standard deviation of the probability density distribution of the corresponding fourth feature is greater than or equal to the first threshold to the total number of samples N is greater than or equal to a third threshold, the first device determines that the second AI model has failed;

[0314] Wherein, N is a positive integer.

[0315] Optionally, the N samples may also be replaced by N inference processes of the second AI model.

[0316] In some embodiments, the first threshold is agreed upon by a protocol, or the first threshold is determined by the first device, or the first threshold is configured or indicated by the second device.

[0317] In some embodiments, the second threshold is agreed upon by a protocol, or the second threshold is determined by the first device, or the second threshold is configured or indicated by the second device.

[0318] In some embodiments, the third threshold is agreed upon by a protocol, or the third threshold is determined by the first device, or the third threshold is configured or indicated by the second device.

[0319] In some embodiments, when the fourth information includes a confidence interval of the fourth feature, the first device determines the validity of the second AI model based on the fourth information, including:

[0320] When the width of the confidence interval of the fourth feature is greater than or equal to a fourth threshold, the first device determines that the second AI model is invalid; or

[0321] When the average width of the confidence interval of the fourth feature corresponding to the N samples of the third feature is greater than or equal to a fifth threshold, the first device determines that the second AI model is invalid; or

[0322] If, among the N samples of the third feature, the number of samples for which the width of the confidence interval of the corresponding fourth feature is greater than or equal to the fourth threshold accounts for a proportion of the total number of samples N that is greater than or equal to a sixth threshold, the first device determines that the second AI model has failed;

[0323] Wherein, N is a positive integer.

[0324] In some embodiments, the fourth threshold is agreed upon by a protocol, or the fourth threshold is determined by the first device, or the fourth threshold is configured or indicated by the second device.

[0325] In some embodiments, the fifth threshold is agreed upon by a protocol, or the fifth threshold is determined by the first device, or the fifth threshold is configured or indicated by the second device.

[0326] In some embodiments, the sixth threshold is agreed upon by a protocol, or the sixth threshold is determined by the first device, or the sixth threshold is configured or indicated by the second device.

[0327] In some embodiments, when the fourth information includes the value of the fourth feature and its probability or confidence, the first device determines the validity of the second AI model based on the fourth information, including:

[0328] When the probability or confidence of the fourth feature is less than or equal to a seventh threshold, the first device determines that the second AI model is invalid; or

[0329] When the average of the probabilities or confidences of the fourth feature corresponding to the N samples of the third feature is less than or equal to an eighth threshold, the first device determines that the second AI model is invalid; or

[0330] If, among the N samples of the third feature, the proportion of the number of samples whose probability or confidence of the corresponding fourth feature is less than or equal to the seventh threshold to the total number of samples N is greater than or equal to a ninth threshold, the first device determines that the second AI model has failed;

[0331] Wherein, N is a positive integer.

[0332] In some embodiments, the seventh threshold is agreed upon by a protocol, or the seventh threshold is determined by the first device, or the seventh threshold is configured or indicated by the second device.

[0333] In some embodiments, the eighth threshold is agreed upon by a protocol, or the eighth threshold is determined by the first device, or the eighth threshold is configured or indicated by the second device.

[0334] In some embodiments, the ninth threshold is agreed upon by a protocol, or the ninth threshold is determined by the first device, or the ninth threshold is configured or indicated by the second device.

[0335] Therefore, in an embodiment of the present application, the validity of the second AI model can be determined based on the uncertainty or probability distribution of the output of the second AI model, thereby achieving performance supervision of the second AI model. Specifically, the uncertainty or probability distribution of the output of the second AI model can improve the robustness of the reasoning results of the second AI model. Since the reasoning results of the second AI model cover multiple possible results and their probability distributions, it is conducive to further processing and utilization of the reasoning results. For example, the reasoning results of the second AI model (including relevant information about uncertainty or probability distribution) are combined with other types of information to obtain a more accurate target result. In the performance supervision of the second AI model, referring to the uncertainty or probability distribution of the output of the second AI model can determine the validity of the second AI model. Since no external information is required, it is also easier to implement.

[0336] The above, in conjunction with Figure 7, describes in detail the first device side embodiment of the present application. The following, in conjunction with Figure 8, describes in detail the second device side embodiment of the present application. It should be understood that the second device side embodiment corresponds to the first device side embodiment, and similar descriptions can refer to the first device side embodiment.

[0337] FIG8 is a schematic flow chart of a model performance supervision method 500 according to an embodiment of the present application. As shown in FIG8 , the model performance supervision method 500 may include at least part of the following contents:

[0338] S510: The second device receives fourth information from the first device, where the fourth information is used to characterize uncertainty or probability distribution of an output of a second AI model, where the input of the second AI model is a third feature.

[0339] S520: The second device determines the validity of the second AI model according to the fourth information.

[0340] It should be understood that FIG8 shows the steps or operations of the model performance supervision method 500, but these steps or operations are merely examples, and the embodiments of the present application may also perform other operations or variations of the operations in FIG8.

[0341] In some embodiments, the fourth information includes at least one of the following: a parameter of a probability density distribution of the fourth feature, a confidence interval of the fourth feature, a value of the fourth feature, a value of the fourth feature and its probability, and a value of the fourth feature and its confidence.

[0342] In some embodiments, the parameters of the probability density distribution include at least one of the following: a mean or expectation of the probability density distribution, a variance or standard deviation of the probability density distribution, and an indication of the type of the probability density distribution.

[0343] In some embodiments, when the fourth information includes parameters of the probability density distribution of the fourth feature, the second device determines the validity of the second AI model based on the fourth information, including:

[0344] When the variance or standard deviation of the probability density distribution of the fourth feature is greater than or equal to the first threshold, the second device determines that the second AI model is invalid; or

[0345] When the mean of the variance or standard deviation of the probability density distribution of the fourth feature corresponding to the N samples of the third feature is greater than or equal to the second threshold, the second device determines that the second AI model is invalid; or

[0346] If, among the N samples of the third feature, the proportion of the number of samples for which the variance or standard deviation of the probability density distribution of the corresponding fourth feature is greater than or equal to the first threshold to the total number of samples N is greater than or equal to a third threshold, the second device determines that the second AI model has failed;

[0347] Wherein, N is a positive integer.

[0348] In some embodiments, when the fourth information includes a confidence interval of the fourth feature, the second device determines the validity of the second AI model based on the fourth information, including:

[0349] When the width of the confidence interval of the fourth feature is greater than or equal to a fourth threshold, the second device determines that the second AI model is invalid; or

[0350] When the average width of the confidence interval of the fourth feature corresponding to the N samples of the third feature is greater than or equal to a fifth threshold, the second device determines that the second AI model is invalid; or

[0351] If, among the N samples of the third feature, the proportion of the number of samples for which the width of the confidence interval of the corresponding fourth feature is greater than or equal to the fourth threshold to the total number of samples N is greater than or equal to a sixth threshold, the second device determines that the second AI model has failed;

[0352] Wherein, N is a positive integer.

[0353] In some embodiments, when the fourth information includes the value of the fourth feature and its probability or confidence, the second device determines the validity of the second AI model based on the fourth information, including:

[0354] When the probability or confidence of the fourth feature is less than or equal to a seventh threshold, the second device determines that the second AI model is invalid; or

[0355] When the average of the probability or confidence of the fourth feature corresponding to the N samples of the third feature is less than or equal to an eighth threshold, the second device determines that the second AI model is invalid; or

[0356] If, among the N samples of the third feature, the proportion of the number of samples whose probability or confidence of the corresponding fourth feature is less than or equal to the seventh threshold to the total number of samples N is greater than or equal to a ninth threshold, the second device determines that the second AI model has failed;

[0357] Wherein, N is a positive integer.

[0358] In some embodiments, the second device sends fourth indication information to the first device, where the fourth indication information is used to indicate the validity of the second AI model.

[0359] Optionally, the fourth indication information includes an identifier of the second AI model or an identifier of a function associated with the second AI model.

[0360] Therefore, in an embodiment of the present application, the validity of the second AI model can be determined based on the uncertainty or probability distribution of the output of the second AI model, thereby achieving performance supervision of the second AI model. Specifically, the uncertainty or probability distribution of the output of the second AI model can improve the robustness of the reasoning results of the second AI model. Since the reasoning results of the second AI model cover multiple possible results and their probability distributions, it is conducive to further processing and utilization of the reasoning results. For example, the reasoning results of the second AI model (including relevant information about uncertainty or probability distribution) are combined with other types of information to obtain a more accurate target result. In the performance supervision of the second AI model, referring to the uncertainty or probability distribution of the output of the second AI model can determine the validity of the second AI model. Since no external information is required, it is also easier to implement.

[0361] The model performance supervision method provided in the embodiment of the present application can be executed by a model performance supervision device or a processing unit in the model performance supervision device for executing the model performance supervision method. In the embodiment of the present application, the model performance supervision device is used as an example to illustrate the model performance supervision device provided in the embodiment of the present application.

[0362] FIG9 shows a schematic block diagram of a model performance monitoring device 600 according to an embodiment of the present application. As shown in FIG9 , the model performance monitoring device 600 includes:

[0363] An acquisition unit 610 is configured to acquire first information, wherein the first information is used to characterize uncertainty or probability distribution of an output of a first artificial intelligence (AI) model, where the input of the first AI model is a first feature;

[0364] Processing unit 620 is used to determine third information based on the first information and the second information, wherein the second information is a label corresponding to the first feature, and the third information is used to determine the validity of the first AI model.

[0365] In some embodiments, the acquiring unit 610 acquires the first information, including one of the following:

[0366] receiving the first information from a second device;

[0367] Obtaining the first information through output information of the first AI model;

[0368] Receive output information of the first AI model from a second device, and obtain the first information based on the output information of the first AI model.

[0369] In some embodiments, the model performance monitoring device 600 further includes: a transceiver unit 630;

[0370] The processing unit 620 is further configured to determine the validity of the first AI model according to the third information; or,

[0371] The transceiver unit 630 is configured to send the third information to the second device.

[0372] In some embodiments, when the model performance monitoring device 600 determines the validity of the first AI model based on the third information, the transceiver unit 630 is further configured to send first indication information to the second device, wherein the first indication information is used to indicate the validity of the first AI model; or

[0373] When the model performance monitoring device 600 sends the third information to the second device, the transceiver unit 630 is further used to receive second indication information from the second device, wherein the second indication information is used to indicate the validity of the first AI model.

[0374] In some embodiments, the first information includes at least one of the following:

[0375] Parameters of the probability density distribution of the second feature, a confidence interval of the second feature, the value of the second feature, the value of the second feature and its probability, and the value of the second feature and its confidence level.

[0376] In some embodiments, the parameters of the probability density distribution include at least one of the following: a mean or expectation of the probability density distribution, a variance or standard deviation of the probability density distribution, and an indication of the type of the probability density distribution.

[0377] In some embodiments, when the first information includes parameters of the probability density distribution of the second feature, the third information is used to represent the mean of the first probabilities of N samples of the first feature, or the third information is used to represent the mean of the logarithms of the first probabilities of N samples of the first feature;

[0378] The first probability is the probability of the label corresponding to each of the N samples under the probability density distribution of the second feature, and N is a positive integer.

[0379] In some embodiments, when the third information is used to represent the mean of the first probabilities of N samples of the first feature, the third information is determined based on the following formula:

[0380] Wherein, L1 represents the third information, i represents the i-th sample in the N samples, represents the standard deviation or variance of the probability density distribution of the second feature corresponding to the i-th sample, Represents the mean of the probability density distribution of the second feature corresponding to the i-th sample, y i represents the label corresponding to the i-th sample, and g(·) represents the probability density distribution function obeyed by the output of the first AI model.

[0381] In some embodiments, when the third information is used to represent the mean of the logarithm of the first probability of N samples of the first feature, the third information is determined based on the following formula:

[0382] Wherein, L2 represents the third information, i represents the i-th sample in the N samples, represents the mean of the probability density distribution of the second feature corresponding to the i-th sample, represents the standard deviation or variance of the probability density distribution of the second feature corresponding to the i-th sample, y i represents the label corresponding to the i-th sample, s is a positive integer, and g(·) represents the probability density distribution function obeyed by the output of the first AI model.

[0383] In some embodiments, the third information is used to determine the validity of the first AI model, including:

[0384] If the third information is greater than or equal to the first threshold, the first AI model is valid; or

[0385] When the third information is less than the first threshold, the first AI model fails.

[0386] In some embodiments, when the first information includes the confidence interval of the second feature, the third information is used to characterize the ratio of the number of samples whose corresponding labels are within the confidence interval of the second feature in the N samples of the first feature to the total number of samples N, where N is a positive integer.

[0387] In some embodiments, the third information is determined based on the following formula:

[0388] Wherein, L3 represents the third information, i represents the i-th sample in the N samples, and x i Represents the input information corresponding to the i-th sample, y i Indicates the label corresponding to the i-th sample, when y i The confidence interval f(x i ) within , otherwise

[0389] In some embodiments, the third information is used to determine the validity of the first AI model, including:

[0390] If the third information is greater than or equal to the second threshold, the first AI model is valid; or

[0391] When the third information is less than a second threshold, the first AI model fails.

[0392] In some embodiments, when the first information includes the value of the second feature and its probability or confidence, the third information is used to characterize the mean of the weighted distances between the labels corresponding to N samples of the first feature and the value of the second feature, where the weighted weight is determined based on the probability or confidence of the second feature, and N is a positive integer.

[0393] In some embodiments, the third information is determined based on the following formula:

[0394] Wherein, L4 represents the third information, i represents the i-th sample in the N samples, Represents the mean of the probability density distribution of the second feature corresponding to the i-th sample, y i Indicates the label corresponding to the i-th sample.

[0395] In some embodiments, the third information is used to determine the validity of the first AI model, including:

[0396] When the third information is less than or equal to a third threshold, the first AI model is valid; or

[0397] When the third information is greater than a third threshold, the first AI model fails.

[0398] In some embodiments, the transceiver unit may be a communication interface or a transceiver, or an input / output interface of a communication chip or a system on chip.

[0399] It should be understood that the model performance monitoring device 600 according to the embodiment of the present application may correspond to the first device in the method embodiment of the present application, and the above-mentioned and other operations or functions of each unit in the model performance monitoring device 600 are respectively for realizing the corresponding processes of the first device in the method 200 shown in Figure 4. For the sake of brevity, they will not be repeated here.

[0400] Therefore, in an embodiment of the present application, the third information can be determined based on the uncertainty or probability distribution of the output of the first AI model and the label corresponding to the input feature of the first AI model, and the validity of the first AI model can be determined based on the third information, thereby achieving performance supervision of the first AI model. Specifically, the uncertainty or probability distribution of the output of the first AI model can improve the robustness of the reasoning result of the first AI model. Since the reasoning result of the first AI model covers multiple possible results and their probability distribution, it is conducive to further processing and utilization of the reasoning result. For example, the reasoning result of the first AI model (including relevant information about uncertainty or probability distribution) is combined with other types of information to obtain a more accurate target result. Referring to the uncertainty or probability distribution of the output of the first AI model and the corresponding label in the performance supervision of the first AI model can improve the accuracy of the performance supervision of the first AI model.

[0401] FIG10 shows a schematic block diagram of a model performance monitoring device 700 according to an embodiment of the present application. As shown in FIG10 , the model performance monitoring device 700 includes:

[0402] The transceiver unit 710 is configured to receive third information from the first device, wherein the third information is determined based on the first information and second information, the first information being used to characterize the uncertainty or probability distribution of the output of a first artificial intelligence (AI) model, the input of the first AI model being a first feature, and the second information being a label corresponding to the first feature;

[0403] The processing unit 720 is configured to determine the validity of the first AI model based on the third information.

[0404] In some embodiments, the processing unit 720 is specifically configured to:

[0405] If the third information is less than or equal to a third threshold, determining that the first AI model is valid; or

[0406] When the third information is greater than a third threshold, it is determined that the first AI model is invalid.

[0407] In some embodiments, before the model performance monitoring apparatus 700 receives the third information from the first device, the transceiver unit 710 is further configured to send the first information to the first device.

[0408] In some embodiments, the transceiver unit 710 is further used to send second indication information to the first device, wherein the second indication information is used to indicate the validity of the first AI model.

[0409] In some embodiments, the first information includes at least one of the following:

[0410] Parameters of the probability density distribution of the second feature, a confidence interval of the second feature, the value of the second feature, the value of the second feature and its probability, and the value of the second feature and its confidence level.

[0411] In some embodiments, the parameters of the probability density distribution include at least one of the following: a mean or expectation of the probability density distribution, a variance or standard deviation of the probability density distribution, and an indication of the type of the probability density distribution.

[0412] In some embodiments, when the first information includes parameters of the probability density distribution of the second feature, the third information is used to represent the mean of the first probabilities of N samples of the first feature, or the third information is used to represent the mean of the logarithms of the first probabilities of N samples of the first feature;

[0413] The first probability is the probability of the label corresponding to each of the N samples under the probability density distribution of the second feature, and N is a positive integer.

[0414] In some embodiments, when the third information is used to represent the mean of the first probabilities of N samples of the first feature, the third information is determined based on the following formula:

[0415] Wherein, L1 represents the third information, i represents the i-th sample in the N samples, represents the standard deviation or variance of the probability density distribution of the second feature corresponding to the i-th sample, Represents the mean of the probability density distribution of the second feature corresponding to the i-th sample, y i represents the label corresponding to the i-th sample, and g(·) represents the probability density distribution function obeyed by the output of the first AI model.

[0416] In some embodiments, when the third information is used to represent the mean of the logarithm of the first probability of N samples of the first feature, the third information is determined based on the following formula:

[0417] Wherein, L2 represents the third information, i represents the i-th sample in the N samples, represents the mean of the probability density distribution of the second feature corresponding to the i-th sample, represents the standard deviation or variance of the probability density distribution of the second feature corresponding to the i-th sample, y i represents the label corresponding to the i-th sample, s is a positive integer, and g(·) represents the probability density distribution function obeyed by the output of the first AI model.

[0418] In some embodiments, the processing unit 720 is specifically configured to:

[0419] If the third information is greater than or equal to the first threshold, determining that the first AI model is valid; or,

[0420] When the third information is less than a first threshold, it is determined that the first AI model is invalid.

[0421] In some embodiments, when the first information includes the confidence interval of the second feature, the third information is used to characterize the ratio of the number of samples whose corresponding labels are within the confidence interval of the second feature in the N samples of the first feature to the total number of samples N, where N is a positive integer.

[0422] In some embodiments, the third information is determined based on the following formula:

[0423] Wherein, L3 represents the third information, i represents the i-th sample in the N samples, and x i Represents the input information corresponding to the i-th sample, y i Indicates the label corresponding to the i-th sample, when y i The confidence interval f(x i ) within , otherwise

[0424] In some embodiments, the processing unit 720 is specifically configured to:

[0425] If the third information is greater than or equal to a second threshold, determining that the first AI model is valid; or

[0426] When the third information is less than a second threshold, it is determined that the first AI model is invalid.

[0427] In some embodiments, when the first information includes the value of the second feature and its probability or confidence, the third information is used to characterize the mean of the weighted distances between the labels corresponding to N samples of the first feature and the value of the second feature, where the weighted weight is determined based on the probability or confidence of the second feature, and N is a positive integer.

[0428] In some embodiments, the third information is determined based on the following formula:

[0429] Wherein, L4 represents the third information, i represents the i-th sample in the N samples, Represents the mean of the probability density distribution of the second feature corresponding to the i-th sample, y i Indicates the label corresponding to the i-th sample.

[0430] In some embodiments, the transceiver unit may be a communication interface or a transceiver, or an input / output interface of a communication chip or a system on chip.

[0431] It should be understood that the model performance monitoring device 700 according to the embodiment of the present application may correspond to the second device in the method embodiment of the present application, and the above-mentioned and other operations or functions of each unit in the model performance monitoring device 700 are respectively for realizing the corresponding process of the second device in the method 300 shown in Figure 6. For the sake of brevity, they will not be repeated here.

[0432] Therefore, in an embodiment of the present application, the third information can be determined based on the uncertainty or probability distribution of the output of the first AI model and the label corresponding to the input feature of the first AI model, and the validity of the first AI model can be determined based on the third information, thereby achieving performance supervision of the first AI model. Specifically, the uncertainty or probability distribution of the output of the first AI model can improve the robustness of the reasoning result of the first AI model. Since the reasoning result of the first AI model covers multiple possible results and their probability distribution, it is conducive to further processing and utilization of the reasoning result. For example, the reasoning result of the first AI model (including relevant information about uncertainty or probability distribution) is combined with other types of information to obtain a more accurate target result. Referring to the uncertainty or probability distribution of the output of the first AI model and the corresponding label in the performance supervision of the first AI model can improve the accuracy of the performance supervision of the first AI model.

[0433] FIG11 shows a schematic block diagram of a model performance monitoring device 800 according to an embodiment of the present application. As shown in FIG11 , the model performance monitoring device 800 includes:

[0434] An acquiring unit 810 is configured to acquire fourth information, wherein the fourth information is used to characterize uncertainty or probability distribution of an output of a second artificial intelligence (AI) model, where the input of the second AI model is the third feature;

[0435] The processing unit 820 is used to determine the validity of the second AI model based on the fourth information; or the transceiver unit 830 is used to send the fourth information to the second device.

[0436] In some embodiments, the acquiring unit 810 acquires the fourth information, including one of the following:

[0437] Acquire the fourth information from other devices;

[0438] Obtaining the fourth information through the output information of the second AI model;

[0439] Receive output information of the second AI model from the second device, and obtain the fourth information based on the output information of the second AI model.

[0440] In some embodiments, when the model performance monitoring device 800 determines the validity of the second AI model based on the fourth information, the transceiver unit 830 is further configured to send third indication information to the second device, wherein the third indication information is used to indicate the validity of the second AI model; or

[0441] When the model performance monitoring device 800 sends the fourth information to the second device, the transceiver unit 830 is further used to receive fourth indication information from the second device, wherein the fourth indication information is used to indicate the validity of the second AI model.

[0442] In some embodiments, the fourth information includes at least one of the following: parameters of the probability density distribution of the fourth feature, the confidence interval of the fourth feature, the value of the fourth feature, the value of the fourth feature and its probability, and the value of the fourth feature and its confidence.

[0443] In some embodiments, the parameters of the probability density distribution include at least one of the following: a mean or expectation of the probability density distribution, a variance or standard deviation of the probability density distribution, and an indication of the type of the probability density distribution.

[0444] In some embodiments, when the fourth information includes parameters of the probability density distribution of the fourth feature, the processing unit 820 is specifically configured to:

[0445] When the variance or standard deviation of the probability density distribution of the fourth feature is greater than or equal to the first threshold, determining that the second AI model is invalid; or

[0446] If the mean of the variance or standard deviation of the probability density distribution of the fourth feature corresponding to the N samples of the third feature is greater than or equal to a second threshold, determining that the second AI model is invalid; or

[0447] If, among the N samples of the third feature, the proportion of the number of samples for which the variance or standard deviation of the probability density distribution of the corresponding fourth feature is greater than or equal to the first threshold to the total number of samples N is greater than or equal to a third threshold, the second AI model is determined to be invalid;

[0448] Wherein, N is a positive integer.

[0449] In some embodiments, when the fourth information includes a confidence interval of the fourth feature, the processing unit 820 is specifically configured to:

[0450] When the width of the confidence interval of the fourth feature is greater than or equal to a fourth threshold, determining that the second AI model is invalid; or

[0451] If the average width of the confidence interval of the fourth feature corresponding to the N samples of the third feature is greater than or equal to a fifth threshold, determining that the second AI model is invalid; or

[0452] If, among the N samples of the third feature, the proportion of the number of samples whose corresponding confidence interval for the fourth feature is greater than or equal to a fourth threshold to the total number of samples N is greater than or equal to a sixth threshold, the second AI model is determined to be invalid;

[0453] Wherein, N is a positive integer.

[0454] In some embodiments, when the fourth information includes the value of the fourth feature and its probability or confidence, the processing unit 820 is specifically configured to:

[0455] When the probability or confidence of the fourth feature is less than or equal to a seventh threshold, determining that the second AI model is invalid; or

[0456] If the average of the probabilities or confidences of the fourth feature corresponding to the N samples of the third feature is less than or equal to an eighth threshold, determining that the second AI model is invalid; or

[0457] If, among the N samples of the third feature, the proportion of the number of samples whose probability or confidence of the corresponding fourth feature is less than or equal to the seventh threshold to the total number of samples N is greater than or equal to a ninth threshold, it is determined that the second AI model has failed;

[0458] Wherein, N is a positive integer.

[0459] In some embodiments, the transceiver unit may be a communication interface or a transceiver, or an input / output interface of a communication chip or a system on chip.

[0460] It should be understood that the model performance monitoring device 800 according to the embodiment of the present application may correspond to the first device in the method embodiment of the present application, and the above-mentioned and other operations or functions of each unit in the model performance monitoring device 800 are respectively for realizing the corresponding processes of the first device in the method 400 shown in Figure 7. For the sake of brevity, they will not be repeated here.

[0461] Therefore, in an embodiment of the present application, the validity of the second AI model can be determined based on the uncertainty or probability distribution of the output of the second AI model, thereby achieving performance supervision of the second AI model. Specifically, the uncertainty or probability distribution of the output of the second AI model can improve the robustness of the reasoning results of the second AI model. Since the reasoning results of the second AI model cover multiple possible results and their probability distributions, it is conducive to further processing and utilization of the reasoning results. For example, the reasoning results of the second AI model (including relevant information about uncertainty or probability distribution) are combined with other types of information to obtain a more accurate target result. In the performance supervision of the second AI model, referring to the uncertainty or probability distribution of the output of the second AI model can determine the validity of the second AI model. Since no external information is required, it is also easier to implement.

[0462] FIG12 shows a schematic block diagram of a model performance monitoring device 900 according to an embodiment of the present application. As shown in FIG12 , the model performance monitoring device 900 includes:

[0463] The transceiver unit 910 is configured to receive fourth information from the first device, wherein the fourth information is used to characterize the uncertainty or probability distribution of the output of the second artificial intelligence (AI) model, the input of the second AI model being the third feature;

[0464] The processing unit 920 is configured to determine the validity of the second AI model based on the fourth information.

[0465] In some embodiments, the transceiver unit 910 is further used to send fourth indication information to the first device, wherein the fourth indication information is used to indicate the validity of the second AI model.

[0466] In some embodiments, the fourth information includes at least one of the following: parameters of the probability density distribution of the fourth feature, the confidence interval of the fourth feature, the value of the fourth feature, the value of the fourth feature and its probability, and the value of the fourth feature and its confidence.

[0467] In some embodiments, the parameters of the probability density distribution include at least one of the following: a mean or expectation of the probability density distribution, a variance or standard deviation of the probability density distribution, and an indication of the type of the probability density distribution.

[0468] In some embodiments, when the fourth information includes parameters of the probability density distribution of the fourth feature, the processing unit 920 is specifically configured to:

[0469] When the variance or standard deviation of the probability density distribution of the fourth feature is greater than or equal to the first threshold, determining that the second AI model is invalid; or

[0470] If the mean of the variance or standard deviation of the probability density distribution of the fourth feature corresponding to the N samples of the third feature is greater than or equal to a second threshold, determining that the second AI model is invalid; or

[0471] If, among the N samples of the third feature, the proportion of the number of samples for which the variance or standard deviation of the probability density distribution of the corresponding fourth feature is greater than or equal to the first threshold to the total number of samples N is greater than or equal to a third threshold, the second AI model is determined to be invalid;

[0472] Wherein, N is a positive integer.

[0473] In some embodiments, when the fourth information includes a confidence interval of the fourth feature, the processing unit 920 is specifically configured to:

[0474] When the width of the confidence interval of the fourth feature is greater than or equal to a fourth threshold, determining that the second AI model is invalid; or

[0475] If the average width of the confidence interval of the fourth feature corresponding to the N samples of the third feature is greater than or equal to a fifth threshold, determining that the second AI model is invalid; or

[0476] If, among the N samples of the third feature, the proportion of the number of samples whose corresponding confidence interval for the fourth feature is greater than or equal to a fourth threshold to the total number of samples N is greater than or equal to a sixth threshold, the second AI model is determined to be invalid;

[0477] Wherein, N is a positive integer.

[0478] In some embodiments, when the fourth information includes the value of the fourth feature and its probability or confidence, the processing unit 920 is specifically configured to:

[0479] When the probability or confidence of the fourth feature is less than or equal to a seventh threshold, determining that the second AI model is invalid; or

[0480] If the average of the probabilities or confidences of the fourth feature corresponding to the N samples of the third feature is less than or equal to an eighth threshold, determining that the second AI model is invalid; or

[0481] If, among the N samples of the third feature, the proportion of the number of samples whose probability or confidence of the corresponding fourth feature is less than or equal to the seventh threshold to the total number of samples N is greater than or equal to a ninth threshold, it is determined that the second AI model has failed;

[0482] Wherein, N is a positive integer.

[0483] In some embodiments, the transceiver unit may be a communication interface or a transceiver, or an input / output interface of a communication chip or a system on chip.

[0484] It should be understood that the model performance monitoring device 900 according to the embodiment of the present application may correspond to the second device in the method embodiment of the present application, and the above-mentioned and other operations or functions of each unit in the model performance monitoring device 900 are respectively for realizing the corresponding processes of the second device in the method 500 shown in Figure 8. For the sake of brevity, they will not be repeated here.

[0485] Therefore, in an embodiment of the present application, the validity of the second AI model can be determined based on the uncertainty or probability distribution of the output of the second AI model, thereby achieving performance supervision of the second AI model. Specifically, the uncertainty or probability distribution of the output of the second AI model can improve the robustness of the reasoning results of the second AI model. Since the reasoning results of the second AI model cover multiple possible results and their probability distributions, it is conducive to further processing and utilization of the reasoning results. For example, the reasoning results of the second AI model (including relevant information about uncertainty or probability distribution) are combined with other types of information to obtain a more accurate target result. In the performance supervision of the second AI model, referring to the uncertainty or probability distribution of the output of the second AI model can determine the validity of the second AI model. Since no external information is required, it is also easier to implement.

[0486] The model performance monitoring device in the embodiment of the present application can be an electronic device, such as an electronic device with an operating system, or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or a network-side device, or can be a device other than a terminal or a network-side device. For example, the terminal can include but is not limited to the types of the terminal 11 listed above, the network-side device can include but is not limited to the types of the network-side device 12 listed above, and other devices can be servers, network attached storage (NAS), etc., which are not specifically limited in the embodiment of the present application.

[0487] The model performance monitoring device provided in the embodiment of the present application can implement the various processes implemented in the method embodiments of Figures 4 to 8 and achieve the same technical effects. To avoid repetition, they will not be described here.

[0488] As shown in Figure 13, an embodiment of the present application further provides a communication device 1000, including a processor 1001 and a memory 1002, wherein the memory 1002 stores a program or instruction that can be run on the processor 1001. For example, when the communication device 1000 is a first device, the program or instruction is executed by the processor 1001 to implement the various steps of the above-mentioned model performance supervision method 200 or model performance supervision method 400 embodiment, and can achieve the same technical effect. When the communication device 1000 is a second device, the program or instruction is executed by the processor 1001 to implement the various steps of the above-mentioned model performance supervision method 300 or model performance supervision method 500 embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0489] The present application also provides a terminal including a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is configured to execute a program or instruction to implement the steps performed by the first device or the second device in the method embodiments shown in Figures 4 to 8. This terminal embodiment corresponds to the above-mentioned method embodiments on the first device or the second device side, and each implementation process and implementation method of the above-mentioned method embodiments are applicable to this terminal embodiment and can achieve the same technical effects. Specifically, Figure 14 is a schematic diagram of the hardware structure of a terminal implementing an embodiment of the present application.

[0490] The terminal 1100 includes but is not limited to: a radio frequency unit 1101, a network module 1102, an audio output unit 1103, an input unit 1104, a sensor 1105, a display unit 1106, a user input unit 1107, an interface unit 1108, a memory 1109 and at least some of the components of the processor 1110.

[0491] Those skilled in the art will appreciate that the terminal 1100 may also include a power supply (such as a battery) to power various components. The power supply may be logically connected to the processor 1110 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The terminal structure shown in FIG14 does not limit the terminal. The terminal may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be described in detail here.

[0492] It should be understood that in an embodiment of the present application, the input unit 1104 may include a graphics processing unit (GPU) 11041 and a microphone 11042, and the graphics processor 11041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 1106 may include a display panel 11061, and the display panel 11061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1107 includes a touch panel 11071 and at least one of other input devices 11072. The touch panel 11071 is also called a touch screen. The touch panel 11071 may include two parts: a touch detection device and a touch controller. Other input devices 11072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0493] In the embodiment of the present application, after receiving downlink data from a network-side device, the RF unit 1101 may transmit the data to the processor 1110 for processing. Furthermore, the RF unit 1101 may send uplink data to the network-side device. Typically, the RF unit 1101 includes, but is not limited to, an antenna, an amplifier, a transceiver, a coupler, a low-noise amplifier, a duplexer, and the like.

[0494] The memory 1109 can be used to store software programs or instructions and various data. The memory 1109 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, applications or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 1109 may include a volatile memory or a non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM). The memory 1109 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0495] Processor 1110 may include at least one processing unit. Optionally, processor 1110 integrates an application processor and a modem processor. The application processor primarily handles operations related to the operating system, user interface, and application programs, while the modem processor primarily processes wireless communication signals, such as a baseband processor. It is understood that the modem processor may not be integrated into processor 1110.

[0496] Exemplarily, the radio frequency unit 1101 is used to obtain first information, wherein the first information is used to characterize the uncertainty or probability distribution of the output of the first AI model, and the input of the first AI model is a first feature; the processor 1110 is used to determine third information based on the first information and the second information, wherein the second information is a label corresponding to the first feature, and the third information is used to determine the validity of the first AI model.

[0497] Exemplarily, the radio frequency unit 1101 is used to receive third information from the first device, wherein the third information is determined based on the first information and the second information, the first information is used to characterize the uncertainty or probability distribution of the output of the first artificial intelligence AI model, the input of the first AI model is a first feature, and the second information is a label corresponding to the first feature; the processor 1110 is used to determine the validity of the first AI model based on the third information.

[0498] Exemplarily, the RF unit 1101 is used to obtain fourth information, wherein the fourth information is used to characterize the uncertainty or probability distribution of the output of the second artificial intelligence AI model, and the input of the second AI model is the third feature; the processor 1110 is used to determine the validity of the first AI model based on the fourth information; or, the RF unit 1101 is used to send the fourth information to the second device.

[0499] Exemplarily, the radio frequency unit 1101 is used to receive fourth information from the first device, wherein the fourth information is used to characterize the uncertainty or probability distribution of the output of the second artificial intelligence AI model, and the input of the second AI model is the third feature; the processor 1110 is used to determine the validity of the first AI model based on the fourth information.

[0500] It can be understood that the implementation process of each implementation method mentioned in this embodiment can refer to the relevant description of the method embodiment and achieve the same or corresponding technical effects. To avoid repetition, it will not be described here.

[0501] The present application also provides a network-side device, including a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is configured to execute a program or instruction to implement the steps of the method embodiments shown in Figures 4 to 8. This network-side device embodiment corresponds to the first device or second device method embodiment described above, and each implementation process and implementation method of the above method embodiments are applicable to this network-side device embodiment and can achieve the same technical effects.

[0502] Specifically, embodiments of the present application also provide a network-side device. As shown in Figure 15, the network-side device 1200 includes an antenna 121, a radio frequency device 122, a baseband device 123, a processor 124, and a memory 125. The antenna 121 is connected to the radio frequency device 122. In the uplink direction, the radio frequency device 122 receives information via the antenna 121 and sends the received information to the baseband device 123 for processing. In the downlink direction, the baseband device 123 processes the information to be transmitted and sends it to the radio frequency device 122. The radio frequency device 122 processes the received information and then sends it through the antenna 121.

[0503] The method executed by the network-side device in the above embodiment may be implemented in the baseband device 123 , which includes a baseband processor.

[0504] The baseband device 123 may include, for example, at least one baseband board, on which at least two chips are arranged, as shown in Figure 15, one of the chips is, for example, a baseband processor, which is connected to the memory 125 through a bus interface to call the program in the memory 125 and execute the network device operations shown in the above method embodiment.

[0505] The network side device may further include a network interface 126 , which is, for example, a Common Public Radio Interface (CPRI).

[0506] Specifically, the network side device 1200 of the embodiment of the present application also includes: instructions or programs stored in the memory 125 and executable on the processor 124. The processor 124 calls the instructions or programs in the memory 125 to execute the method executed by each unit shown in any one of Figures 9 to 12, and achieves the same technical effect. To avoid repetition, it will not be described here.

[0507] Specifically, the embodiment of the present application further provides a network side device. As shown in FIG16 , the network side device 1300 includes: a processor 1301, a network interface 1302, and a memory 1303. The network interface 1302 is, for example, a common public radio interface (CPRI).

[0508] Specifically, the network side device 1300 of the embodiment of the present application also includes: instructions or programs stored in the memory 1303 and executable on the processor 1301. The processor 1301 calls the instructions or programs in the memory 1303 to execute the method executed by each unit shown in any one of Figures 9 to 12, and achieves the same technical effect. To avoid repetition, it will not be repeated here.

[0509] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned model performance supervision method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0510] The processor is the processor in the terminal described in the above embodiment. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk. In some examples, the readable storage medium may be a non-transitory readable storage medium.

[0511] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned model performance supervision method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0512] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0513] An embodiment of the present application further provides a computer program / program product, which is stored in a storage medium. The computer program / program product is executed by at least one processor to implement the various processes of the above-mentioned model performance supervision method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0514] An embodiment of the present application also provides a communication system, including: a first device and a second device, wherein the first device can be used to execute the steps performed by the first device in the model performance supervision method as described above, and the second device can be used to execute the steps performed by the second device in the model performance supervision method as described above.

[0515] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0516] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of a computer software product plus a necessary general-purpose hardware platform, or of course, by hardware. The computer software product is stored in a storage medium (such as ROM, RAM, magnetic disk, optical disk, etc.) and includes a number of instructions for enabling a terminal or network-side device to execute the methods described in each embodiment of the present application.

[0517] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms of implementation methods without departing from the purpose of this application and the scope of protection of the claims. These implementation methods are all within the protection of this application.

Claims

1. A model performance supervision method, comprising: The first device acquires first information, wherein the first information is used to characterize the uncertainty or probability distribution of the output of a first artificial intelligence (AI) model, and the input of the first AI model is a first feature; The first device determines third information based on the first information and the second information, wherein the second information is a label corresponding to the first feature, and the third information is used to determine the validity of the first AI model.

2. The method according to claim 1, wherein: The first device obtains first information, including one of the following: The first device receives the first information from the second device; The first device obtains the first information through output information of the first AI model; The first device receives output information of the first AI model from the second device, and obtains the first information according to the output information of the first AI model.

3. The method according to claim 1 or 2, further comprising: The first device determines the validity of the first AI model according to the third information; or, The first device sends the third information to the second device.

4. The method according to claim 3, wherein: The method further comprises: When the first device determines the validity of the first AI model according to the third information, the first device sends first indication information to the second device, wherein the first indication information is used to indicate the validity of the first AI model; or When the first device sends the third information to the second device, the first device receives second indication information from the second device, wherein the second indication information is used to indicate the validity of the first AI model.

5. The method according to any one of claims 1 to 4, wherein: The first information includes at least one of the following: Parameters of the probability density distribution of the second feature, a confidence interval of the second feature, a value of the second feature, a value of the second feature and its probability, and a value of the second feature and its confidence level.

6. The method according to claim 5, wherein: The parameters of the probability density distribution include at least one of the following: a mean or expectation of the probability density distribution, a variance or standard deviation of the probability density distribution, and an indication of the type of the probability density distribution.

7. The method according to claim 5 or 6, wherein: In the case where the first information includes parameters of the probability density distribution of the second feature, the third information is used to characterize a mean of the first probabilities of N samples of the first feature, or the third information is used to characterize a mean of the logarithms of the first probabilities of the N samples of the first feature; The first probability is the probability of the label corresponding to each sample in the N samples under the probability density distribution of the second feature, and N is a positive integer.

8. The method according to claim 7, wherein: In the case where the third information is used to characterize the mean of the first probabilities of N samples of the first feature, the third information is determined based on the following formula: Wherein, L1 represents the third information, i represents the i-th sample among the N samples, represents the standard deviation or variance of the probability density distribution of the second feature corresponding to the i-th sample, represents the mean of the probability density distribution of the second feature corresponding to the i-th sample, y i represents the label corresponding to the i-th sample, and g(·) represents the probability density distribution function obeyed by the output of the first AI model.

9. The method according to claim 7, wherein: In the case where the third information is used to characterize the mean of the logarithm of the first probability of N samples of the first feature, the third information is determined based on the following formula: Wherein, L2 represents the third information, i represents the i-th sample among the N samples, represents the mean of the probability density distribution of the second feature corresponding to the i-th sample, represents the standard deviation or variance of the probability density distribution of the second feature corresponding to the i-th sample, y i represents the label corresponding to the i-th sample, s is a positive integer, and g(·) represents the probability density distribution function obeyed by the output of the first AI model.

10. The method according to any one of claims 7 to 9, wherein: The third information is used to determine the validity of the first AI model, including: When the third information is greater than or equal to the first threshold, the first AI model is valid; or, When the third information is less than the first threshold, the first AI model fails.

11. The method according to claim 5 or 6, wherein: In the case where the first information includes the confidence interval of the second feature, the third information is used to characterize the ratio of the number of samples whose corresponding labels are within the confidence interval of the second feature among the N samples of the first feature to the total number of samples N, where N is a positive integer.

12. The method according to claim 11, wherein: The third information is determined based on the following formula: Wherein, L3 represents the third information, i represents the i-th sample among the N samples, and x i Represents the input information corresponding to the i-th sample, y i Indicates the label corresponding to the i-th sample. When y i The confidence interval f(x i ) within , otherwise 13. The method according to claim 11 or 12, wherein: The third information is used to determine the validity of the first AI model, including: When the third information is greater than or equal to the second threshold, the first AI model is valid; or, When the third information is less than the second threshold, the first AI model fails.

14. The method according to claim 5 or 6, wherein: When the first information includes the value of the second feature and its probability or confidence, the third information is used to characterize the mean of the weighted distances between the labels corresponding to N samples of the first feature and the value of the second feature, wherein the weighted weight is determined based on the probability or confidence of the second feature, and N is a positive integer.

15. The method according to claim 14, wherein: The third information is determined based on the following formula: Wherein, L4 represents the third information, i represents the i-th sample among the N samples, represents the mean of the probability density distribution of the second feature corresponding to the i-th sample, y i Represents the label corresponding to the i-th sample.

16. The method according to claim 14 or 15, wherein: The third information is used to determine the validity of the first AI model, including: When the third information is less than or equal to a third threshold, the first AI model is valid; or, When the third information is greater than a third threshold, the first AI model fails.

17. A model performance supervision method comprising: The second device receives third information from the first device, wherein the third information is determined based on the first information and the second information, the first information is used to characterize the uncertainty or probability distribution of the output of the first artificial intelligence AI model, the input of the first AI model is a first feature, and the second information is a label corresponding to the first feature; The second device determines the validity of the first AI model based on the third information.

18. The method according to claim 17, wherein: Before the second device receives the third information from the first device, the method further includes: The second device sends the first information to the first device.

19. The method according to claim 17 or 18, wherein: The method further comprises: The second device sends second indication information to the first device, wherein the second indication information is used to indicate the validity of the first AI model.

20. The method according to any one of claims 17 to 19, wherein: The first information includes at least one of the following: Parameters of the probability density distribution of the second feature, a confidence interval of the second feature, a value of the second feature, a value of the second feature and its probability, and a value of the second feature and its confidence level.

21. The method according to claim 20, wherein: The parameters of the probability density distribution include at least one of the following: a mean or expectation of the probability density distribution, a variance or standard deviation of the probability density distribution, and an indication of the type of the probability density distribution.

22. The method according to claim 20 or 21, wherein: In the case where the first information includes parameters of the probability density distribution of the second feature, the third information is used to characterize a mean of the first probabilities of N samples of the first feature, or the third information is used to characterize a mean of the logarithms of the first probabilities of the N samples of the first feature; The first probability is the probability of the label corresponding to each sample in the N samples under the probability density distribution of the second feature, and N is a positive integer.

23. The method according to claim 22, wherein: In the case where the third information is used to characterize the mean of the first probabilities of N samples of the first feature, the third information is determined based on the following formula: Wherein, L1 represents the third information, i represents the i-th sample among the N samples, represents the standard deviation or variance of the probability density distribution of the second feature corresponding to the i-th sample, represents the mean of the probability density distribution of the second feature corresponding to the i-th sample, y i represents the label corresponding to the i-th sample, and g(·) represents the probability density distribution function obeyed by the output of the first AI model.

24. The method according to claim 22, wherein: In the case where the third information is used to characterize the mean of the logarithm of the first probability of N samples of the first feature, the third information is determined based on the following formula: Wherein, L2 represents the third information, i represents the i-th sample among the N samples, represents the mean of the probability density distribution of the second feature corresponding to the i-th sample, represents the standard deviation or variance of the probability density distribution of the second feature corresponding to the i-th sample, y i represents the label corresponding to the i-th sample, s is a positive integer, and g(·) represents the probability density distribution function obeyed by the output of the first AI model.

25. The method according to any one of claims 22 to 24, wherein: The second device determines the validity of the first AI model according to the third information, including: When the third information is greater than or equal to the first threshold, the second device determines that the first AI model is valid; or, When the third information is less than the first threshold, the second device determines that the first AI model is invalid.

26. The method according to claim 20 or 21, wherein: In the case where the first information includes the confidence interval of the second feature, the third information is used to characterize the ratio of the number of samples whose corresponding labels are within the confidence interval of the second feature among the N samples of the first feature to the total number of samples N, where N is a positive integer.

27. The method according to claim 26, wherein: The third information is determined based on the following formula: Wherein, L3 represents the third information, i represents the i-th sample among the N samples, and x i Represents the input information corresponding to the i-th sample, y i Indicates the label corresponding to the i-th sample. When y i The confidence interval f(x i ) within , otherwise 28. The method according to claim 26 or 27, wherein: The second device determines the validity of the first AI model according to the third information, including: When the third information is greater than or equal to a second threshold, the second device determines that the first AI model is valid; or, When the third information is less than a second threshold, the second device determines that the first AI model is invalid.

29. The method according to claim 20 or 21, wherein: When the first information includes the value of the second feature and its probability or confidence, the third information is used to characterize the mean of the weighted distances between the labels corresponding to N samples of the first feature and the value of the second feature, wherein the weighted weight is determined based on the probability or confidence of the second feature, and N is a positive integer.

30. The method of claim 29, wherein: The third information is determined based on the following formula: Wherein, L4 represents the third information, i represents the i-th sample among the N samples, represents the mean of the probability density distribution of the second feature corresponding to the i-th sample, y i Represents the label corresponding to the i-th sample.

31. The method according to claim 29 or 30, wherein: The second device determines the validity of the first AI model according to the third information, including: When the third information is less than or equal to a third threshold, the second device determines that the first AI model is valid; or, When the third information is greater than a third threshold, the second device determines that the first AI model is invalid.

32. A model performance supervision method comprising: The first device obtains fourth information, wherein the fourth information is used to characterize the uncertainty or probability distribution of the output of the second artificial intelligence AI model, and the input of the second AI model is the third feature; The first device determines the validity of the second AI model based on the fourth information; or, the first device sends the fourth information to the second device.

33. The method of claim 32, wherein: The first device obtains fourth information, including one of the following: The first device obtains the fourth information from other devices; The first device obtains the fourth information through the output information of the second AI model; The first device receives the output information of the second AI model from the second device, and obtains the fourth information according to the output information of the second AI model.

34. The method according to claim 32 or 33, further comprising: When the first device determines the validity of the second AI model according to the fourth information, the first device sends third indication information to the second device, wherein the third indication information is used to indicate the validity of the second AI model; or When the first device sends the fourth information to the second device, the first device receives fourth indication information from the second device, wherein the fourth indication information is used to indicate the validity of the second AI model.

35. The method according to any one of claims 32 to 34, wherein: The fourth information includes at least one of the following: a parameter of a probability density distribution of the fourth feature, a confidence interval of the fourth feature, a value of the fourth feature, a value of the fourth feature and its probability, and a value of the fourth feature and its confidence.

36. The method of claim 35, wherein: The parameters of the probability density distribution include at least one of the following: a mean or expectation of the probability density distribution, a variance or standard deviation of the probability density distribution, and an indication of the type of the probability density distribution.

37. The method according to claim 35 or 36, wherein: In a case where the fourth information includes a parameter of a probability density distribution of the fourth feature, the first device determines the validity of the second AI model according to the fourth information, including: When the variance or standard deviation of the probability density distribution of the fourth feature is greater than or equal to the first threshold, the first device determines that the second AI model fails; or, When the mean of the variance or standard deviation of the probability density distribution of the fourth feature corresponding to the N samples of the third feature is greater than or equal to the second threshold, the first device determines that the second AI model fails; or, When, among the N samples of the third feature, the proportion of the number of samples whose variance or standard deviation of the probability density distribution of the corresponding fourth feature is greater than or equal to the first threshold to the total number of samples N is greater than or equal to the third threshold, the first device determines that the second AI model has failed; Wherein, N is a positive integer.

38. The method according to claim 35 or 36, wherein: In a case where the fourth information includes a confidence interval of the fourth feature, the first device determines the validity of the second AI model according to the fourth information, including: When the width of the confidence interval of the fourth feature is greater than or equal to a fourth threshold, the first device determines that the second AI model is invalid; or, When the average width of the confidence interval of the fourth feature corresponding to the N samples of the third feature is greater than or equal to a fifth threshold, the first device determines that the second AI model is invalid; or, When, among the N samples of the third feature, the number of samples whose width of the confidence interval of the corresponding fourth feature is greater than or equal to the fourth threshold accounts for a proportion of the total number of samples N that is greater than or equal to a sixth threshold, the first device determines that the second AI model has failed; Wherein, N is a positive integer.

39. The method according to claim 35 or 36, wherein: In a case where the fourth information includes a value of the fourth feature and a probability or confidence level thereof, the first device determines the validity of the second AI model according to the fourth information, including: When the probability or confidence of the fourth feature is less than or equal to a seventh threshold, the first device determines that the second AI model fails; or, When the average of the probability or confidence of the fourth feature corresponding to the N samples of the third feature is less than or equal to an eighth threshold, the first device determines that the second AI model fails; or, Among the N samples of the third feature, the probability or confidence of the corresponding fourth feature is less than or equal to When the ratio of the number of samples of the seventh threshold to the total number of samples N is greater than or equal to the ninth threshold, the first device determines that the second AI model is invalid; Wherein, N is a positive integer.

40. A model performance supervision method comprising: The second device receives fourth information from the first device, wherein the fourth information is used to characterize the uncertainty or probability distribution of the output of the second artificial intelligence (AI) model, and the input of the second AI model is the third feature; The second device determines the validity of the second AI model based on the fourth information.

41. The method of claim 40, wherein: The method further comprises: The second device sends fourth indication information to the first device, wherein the fourth indication information is used to indicate the validity of the second AI model.

42. The method according to claim 40 or 41, wherein: The fourth information includes at least one of the following: a parameter of a probability density distribution of the fourth feature, a confidence interval of the fourth feature, a value of the fourth feature, a value of the fourth feature and its probability, and a value of the fourth feature and its confidence.

43. The method of claim 42, wherein: The parameters of the probability density distribution include at least one of the following: a mean or expectation of the probability density distribution, a variance or standard deviation of the probability density distribution, and an indication of the type of the probability density distribution.

44. The method according to claim 42 or 43, wherein: In a case where the fourth information includes a parameter of a probability density distribution of the fourth feature, the second device determining the validity of the second AI model according to the fourth information includes: When the variance or standard deviation of the probability density distribution of the fourth feature is greater than or equal to the first threshold, the second device determines that the second AI model fails; or, When the mean of the variance or standard deviation of the probability density distribution of the fourth feature corresponding to the N samples of the third feature is greater than or equal to the second threshold, the second device determines that the second AI model fails; or, When, among the N samples of the third feature, the proportion of the number of samples whose variance or standard deviation of the probability density distribution of the corresponding fourth feature is greater than or equal to the first threshold to the total number of samples N is greater than or equal to the third threshold, the second device determines that the second AI model has failed; Wherein, N is a positive integer.

45. The method according to claim 42 or 43, wherein: In a case where the fourth information includes a confidence interval of the fourth feature, the second device determines the validity of the second AI model according to the fourth information, including: When the width of the confidence interval of the fourth feature is greater than or equal to a fourth threshold, the second device determines that the second AI model is invalid; or, When the average width of the confidence interval of the fourth feature corresponding to the N samples of the third feature is greater than or equal to a fifth threshold, the second device determines that the second AI model is invalid; or, When, among the N samples of the third feature, the number of samples whose width of the confidence interval of the corresponding fourth feature is greater than or equal to the fourth threshold accounts for a proportion of the total number of samples N that is greater than or equal to a sixth threshold, the second device determines that the second AI model has failed; Wherein, N is a positive integer.

46. ​​The method of claim 42 or 43, wherein: In a case where the fourth information includes a value of the fourth feature and a probability or confidence level thereof, the second device determining the validity of the second AI model according to the fourth information includes: When the probability or confidence of the fourth feature is less than or equal to a seventh threshold, the second device determines that the second AI model fails; or, When the average of the probability or confidence of the fourth feature corresponding to the N samples of the third feature is less than or equal to an eighth threshold, the second device determines that the second AI model fails; or, Among the N samples of the third feature, the probability or confidence of the corresponding fourth feature is less than or equal to When the ratio of the number of samples of the seventh threshold to the total number of samples N is greater than or equal to the ninth threshold, the second device determines that the second AI model is invalid; Wherein, N is a positive integer.

47. A model performance monitoring device, comprising: An acquisition unit, configured to acquire first information, wherein the first information is used to characterize the uncertainty or probability distribution of an output of a first artificial intelligence (AI) model, and an input of the first AI model is a first feature; A processing unit, used to determine third information based on the first information and the second information, wherein the second information is a label corresponding to the first feature, and the third information is used to determine the validity of the first AI model.

48. A model performance monitoring device, comprising: a transceiver unit, configured to receive third information from a first device, wherein the third information is determined based on the first information and the second information, the first information is used to characterize the uncertainty or probability distribution of the output of the first artificial intelligence (AI) model, the input of the first AI model is a first feature, and the second information is a label corresponding to the first feature; A processing unit is used to determine the validity of the first AI model based on the third information.

49. A model performance monitoring device, comprising: an acquisition unit, configured to acquire fourth information, wherein the fourth information is used to characterize the uncertainty or probability distribution of the output of the second artificial intelligence (AI) model, the input of the second AI model being the third feature; A processing unit, used to determine the validity of the second AI model according to the fourth information; or a transceiver unit, used to send the fourth information to the second device.

50. A model performance monitoring device, comprising: A transceiver unit, configured to receive fourth information from the first device, wherein the fourth information is used to characterize the uncertainty or probability distribution of the output of the second artificial intelligence (AI) model, and the input of the second AI model is the third feature; A processing unit, configured to determine the validity of the second AI model based on the fourth information.

51. A model performance supervision device, wherein the model performance supervision device is a first device, and the model performance supervision device includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the model performance supervision method as described in any one of claims 1 to 16 are implemented, or, when the program or instructions are executed by the processor, the steps of the model performance supervision method as described in any one of claims 32 to 39 are implemented.

52. A model performance supervision device, which is a second device, and includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the model performance supervision method as described in any one of claims 17 to 31 are implemented, or when the program or instructions are executed by the processor, the steps of the model performance supervision method as described in any one of claims 40 to 46 are implemented.

53. A readable storage medium storing a program or instruction, wherein the program or instruction, when executed by a processor, implements the method of the model performance supervision method as described in any one of claims 1 to 16, or implements the steps of the model performance supervision method as described in any one of claims 17 to 31, or implements the steps of the model performance supervision method as described in any one of claims 32 to 39, or implements the steps of the model performance supervision method as described in any one of claims 40 to 46.

Images