Model performance supervision method, device and equipment

Abstract

The invention discloses a model performance supervision method, device and equipment, belongs to the field of communication, and can solve the performance supervision problem of an AI model. The model performance supervision method of the embodiment of the invention comprises the steps that first equipment acquires first information, the first information is used for representing uncertainty or probability distribution of output of a first AI model, and input of the first AI model is a first feature; the first equipment determines third information according to the first information and second information, the second information is a label corresponding to the first feature, and the third information is used for determining the validity of the first AI model.

Description

Technical Field

[0001] The present application relates to the field of communications, and more specifically, to a model performance supervision method, apparatus and device. Background Art

[0002] In the New Radio (NR) system, artificial intelligence (AI) models are introduced to improve system performance. For example, AI models are introduced for positioning, beam management, channel state information (CSI) prediction, mobility management, CSI compression, etc. However, when the wireless propagation environment changes, the performance of the AI ​​model may be difficult to guarantee. How to supervise the performance of the AI ​​model is a problem that needs to be solved. Summary of the invention

[0003] The embodiments of the present application provide a model performance supervision method, device and equipment, which can solve the performance supervision problem of AI models.

[0004] In the first aspect, a model performance supervision method is provided, comprising:

[0005] The first device acquires first information, wherein the first information is used to characterize uncertainty or probability distribution of an output of a first AI model, and an input of the first AI model is a first feature;

[0006] The first device determines third information based on the first information and the second information, wherein the second information is a label corresponding to the first feature, and the third information is used to determine the validity of the first AI model.

[0007] Secondly, a model performance supervision method is provided, including:

[0008] The second device receives third information from the first device, wherein the third information is determined based on the first information and the second information, the first information is used to characterize the uncertainty or probability distribution of the output of the first AI model, the input of the first AI model is the first feature, and the second information is a label corresponding to the first feature;

[0009] The second device determines the validity of the first AI model based on the third information.

[0010] Thirdly, a model performance supervision method is provided, including:

[0011] The first device obtains fourth information, where the fourth information is used to characterize uncertainty or probability distribution of an output of a second AI model, and an input of the second AI model is a third feature;

[0012] The first device determines the validity of the second AI model based on the fourth information; or, the first device sends the fourth information to the second device.

[0013] Fourthly, a model performance supervision method is provided, including:

[0014] The second device receives fourth information from the first device, wherein the fourth information is used to characterize uncertainty or probability distribution of an output of a second AI model, and an input of the second AI model is a third feature;

[0015] The second device determines the validity of the second AI model based on the fourth information.

[0016] In a fifth aspect, a model performance monitoring device is provided, comprising:

[0017] an acquisition unit, configured to acquire first information, wherein the first information is used to characterize uncertainty or probability distribution of an output of a first AI model, and an input of the first AI model is a first feature;

[0018] A processing unit, used to determine third information based on the first information and the second information, wherein the second information is a label corresponding to the first feature, and the third information is used to determine the validity of the first AI model.

[0019] In a sixth aspect, a model performance monitoring device is provided, comprising:

[0020] a transceiver unit, configured to receive third information from a first device, wherein the third information is determined based on the first information and the second information, the first information is used to characterize the uncertainty or probability distribution of the output of the first AI model, the input of the first AI model is a first feature, and the second information is a label corresponding to the first feature;

[0021] A processing unit is used to determine the validity of the first AI model based on the third information.

[0022] In a seventh aspect, a model performance monitoring device is provided, comprising:

[0023] an acquisition unit, configured to acquire fourth information, wherein the fourth information is used to characterize uncertainty or probability distribution of an output of a second AI model, and an input of the second AI model is a third feature;

[0024] A processing unit, used to determine the validity of the second AI model according to the fourth information; or a transceiver unit, used to send the fourth information to the second device.

[0025] In an eighth aspect, a model performance monitoring device is provided, comprising:

[0026] a transceiver unit, configured to receive fourth information from the first device, wherein the fourth information is used to characterize the uncertainty or probability distribution of the output of the second AI model, and the input of the second AI model is the third feature;

[0027] A processing unit, configured to determine the validity of the second AI model based on the fourth information.

[0028] In the ninth aspect, a model performance monitoring device is provided, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0029] In the tenth aspect, a model performance supervision device is provided, comprising a processor and a communication interface; wherein the communication interface or the processor is used to obtain first information, wherein the first information is used to characterize the uncertainty or probability distribution of the output of a first AI model, and the input of the first AI model is a first feature; the processor is used to determine third information based on the first information and the second information, wherein the second information is a label corresponding to the first feature, and the third information is used to determine the validity of the first AI model.

[0030] In the eleventh aspect, a model performance monitoring device is provided, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the second aspect are implemented.

[0031] In the twelfth aspect, a model performance supervision device is provided, comprising a processor and a communication interface; wherein the communication interface is used to receive third information from a first device, wherein the third information is determined based on the first information and the second information, the first information is used to characterize the uncertainty or probability distribution of the output of the first AI model, the input of the first AI model is a first feature, and the second information is a label corresponding to the first feature; the processor is used to determine the validity of the first AI model based on the third information.

[0032] In the thirteenth aspect, a model performance monitoring device is provided, which includes a processor and a memory, wherein the memory stores programs or instructions that can be executed on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the third aspect are implemented.

[0033] In the fourteenth aspect, a model performance supervision device is provided, comprising a processor and a communication interface; wherein the communication interface or the processor is used to obtain fourth information, wherein the fourth information is used to characterize the uncertainty or probability distribution of the output of a second AI model, and the input of the second AI model is a third feature; the processor is used to determine the validity of the second AI model based on the fourth information; or, the communication interface is used to send the fourth information to a second device.

[0034] In the fifteenth aspect, a model performance monitoring device is provided, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the fourth aspect are implemented.

[0035] In the sixteenth aspect, a model performance supervision device is provided, comprising a processor and a communication interface; wherein the communication interface is used to receive fourth information from a first device, wherein the fourth information is used to characterize the uncertainty or probability distribution of the output of a second AI model, and the input of the second AI model is a third feature; the processor is used to determine the validity of the second AI model based on the fourth information.

[0036] In the seventeenth aspect, a readable storage medium is provided, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented, or the steps of the method described in the second aspect are implemented, or the steps of the method described in the third aspect are implemented, or the steps of the method described in the fourth aspect are implemented.

[0037] In the eighteenth aspect, a wireless communication system is provided, including: a first device and a second device, wherein the first device can be used to execute the steps of the method described in the first aspect or the third aspect, and the second device can be used to execute the steps of the method described in the second aspect or the fourth aspect.

[0038] In the nineteenth aspect, a chip is provided, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instructions to implement the method as described in the first aspect, or the method as described in the second aspect, or the method as described in the third aspect, or the method as described in the fourth aspect.

[0039] In the twentieth aspect, a computer program / program product is provided, wherein the computer program / program product is stored in a storage medium, and the program / program product is executed by at least one processor to implement the steps of the model performance supervision method as described in at least one of the first to fourth aspects.

[0040] In the embodiments of the first aspect or the second aspect of the present application, the third information can be determined based on the uncertainty or probability distribution of the output of the first AI model and the label corresponding to the input feature of the first AI model, and the validity of the first AI model can be determined based on the third information, so as to achieve performance supervision of the first AI model. Specifically, the uncertainty or probability distribution of the output of the first AI model can improve the robustness of the reasoning results of the first AI model. Since the reasoning results of the first AI model cover multiple possible results and their probability distributions, it is conducive to further processing and utilization of the reasoning results. For example, the reasoning results of the first AI model (including relevant information on uncertainty or probability distribution) are combined with other types of information to obtain a more accurate target result. In the performance supervision of the first AI model, referring to the uncertainty or probability distribution of the output of the first AI model and the corresponding label can improve the accuracy of the performance supervision of the first AI model.

[0041] In the embodiments of the third aspect or the fourth aspect of the present application, the validity of the second AI model can be determined based on the uncertainty or probability distribution of the output of the second AI model, thereby achieving performance supervision of the second AI model. Specifically, the uncertainty or probability distribution of the output of the second AI model can improve the robustness of the reasoning results of the second AI model. Since the reasoning results of the second AI model cover multiple possible results and their probability distributions, it is conducive to further processing and utilization of the reasoning results. For example, the reasoning results of the second AI model (including relevant information about uncertainty or probability distribution) are combined with other types of information to obtain a more accurate target result. In the performance supervision of the second AI model, the uncertainty or probability distribution of the output of the second AI model is referred to, and the validity of the second AI model can be judged. Since no external information is required, it is also easier to implement. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0043] Figure 1 It is a schematic diagram of a communication system architecture provided in an embodiment of the present application.

[0044] Figure 2 It is a schematic diagram of a neural network provided in this application.

[0045] Figure 3 is a schematic diagram of a neuron provided in the present application.

[0046] Figure 4It is a schematic flowchart of a model performance supervision method provided according to an embodiment of the present application.

[0047] Figure 5 It is a schematic diagram of improving positioning accuracy based on soft information provided according to an embodiment of the present application.

[0048] Figure 6 It is a schematic flowchart of another model performance supervision method provided according to an embodiment of the present application.

[0049] Figure 7 It is a schematic flowchart of another model performance supervision method provided according to an embodiment of the present application.

[0050] Figure 8 It is a schematic flowchart of another model performance supervision method provided according to an embodiment of the present application.

[0051] Fig. 9 It is a schematic block diagram of a model performance monitoring device provided according to an embodiment of the present application.

[0052] Fig.10 It is a schematic block diagram of another model performance monitoring device provided according to an embodiment of the present application.

[0053] Fig.11 It is a schematic block diagram of another model performance monitoring device provided according to an embodiment of the present application.

[0054] Fig.12 It is a schematic block diagram of another model performance monitoring device provided according to an embodiment of the present application.

[0055] Fig.13 It is a schematic block diagram of a communication device provided according to an embodiment of the present application.

[0056] Fig.14 It is a schematic diagram of the hardware structure of a terminal provided according to an embodiment of the present application.

[0057] Fig.15 It is a schematic block diagram of a network side device provided according to an embodiment of the present application.

[0058] Fig.16 It is a schematic block diagram of another network side device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of this application.

[0060] The terms "first", "second", etc. of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of one type, and the number of objects is not limited, for example, the first object can be one or more. In addition, "or" in the present application represents at least one of the connected objects. For example, "A or B" covers three schemes, namely, Scheme 1: including A but not including B; Scheme 2: including B but not including A; Scheme 3: including both A and B. The character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0061] The term "indication" in this application can be a direct indication (or explicit indication) or an indirect indication (or implicit indication). A direct indication can be understood as the sender explicitly informing the receiver of specific information, operations to be performed, or request results in the sent indication; an indirect indication can be understood as the receiver determining the corresponding information according to the indication sent by the sender, or making a judgment and determining the operation to be performed or the request result according to the judgment result.

[0062] It is worth noting that the technology described in the embodiments of the present application is not limited to the Internet of Things (IoT) system, but can also be used in other wireless communication systems, such as Long Term Evolution (LTE) / LTE-Advanced (LTE-A) system, Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal Frequency Division Multiple Access (OFDMA), Single-carrier Frequency-Division Multiple Access (SC-FDMA), Wireless Local Area Networks (WLAN), Wireless Fidelity (WiFi), Bluetooth system, or other systems. The terms "system" and "network" in the embodiments of the present application are often used interchangeably, and the described technology can be used for the above-mentioned systems and radio technologies as well as other systems and radio technologies. The following description describes a New Radio (NR) system for example purposes, and NR terminology is used in most of the following description, but these techniques may also be applied to systems other than NR systems, such as 6th generation (6 th Generation, 6G) communication system.

[0063] Figure 1 A block diagram of a wireless communication system applicable to the embodiments of the present application is shown. The wireless communication system includes a terminal 11 and a network side device 12, wherein the terminal 11 can communicate with the network side device 12 directly or through other network elements.

[0064] Among them, the terminal 11 can be a mobile phone, a tablet personal computer, a laptop computer, a notebook computer, a personal digital assistant (PDA), a handheld computer, a netbook, an ultra-mobile personal computer (UMPC), a mobile Internet device (MID), an augmented reality (AR), a virtual reality (VR) device, a robot, a wearable device (Wearable Device), a flight vehicle, a vehicle user equipment (VUE), a shipborne equipment, a pedestrian terminal (Pedestrian User Equipment, PUE), a smart home (home appliances with wireless communication functions, such as refrigerators, televisions, washing machines or furniture, etc.), a game console, a personal computer (PC), a teller machine or a self-service machine and other terminal side devices. Wearable devices include: smart watches, smart bracelets, smart headphones, smart glasses, smart jewelry (smart bracelets, smart bracelets, smart rings, smart necklaces, smart anklets, smart anklets, etc.), smart wristbands, smart clothing, etc. Among them, the vehicle-mounted device can also be called a vehicle-mounted terminal, a vehicle-mounted controller, a vehicle-mounted module, a vehicle-mounted component, a vehicle-mounted chip or a vehicle-mounted unit, etc. It should be noted that the specific type of the terminal 11 is not limited in the embodiment of the present application.

[0065] The network side device 12 may include an access network device or a core network device.

[0066] Among them, the access network equipment can also be called Radio Access Network (RAN) equipment, Radio Access Network function or Radio Access Network unit. The access network equipment can include base stations, Wireless Local Area Network (WLAN) access points (AS) or Wireless Fidelity (WiFi) nodes, etc. Among them, the base station may be referred to as a Node B (NB), an evolved Node B (eNB), a next generation Node B (gNB), a New Radio Node B (NR Node B), an access point, a Relay Base Station (RBS), a Serving Base Station (SBS), a Base Transceiver Station (BTS), a radio base station, a radio transceiver, a Basic Service Set (BSS), an Extended Service Set (ESS), a Home Node B (HNB), a Home Evolved Node B (home evolved Node B), a Transmission Reception Point (TRP) or other appropriate terms in the field. As long as the same technical effect is achieved, the base station is not limited to specific technical vocabulary. It should be noted that in the embodiments of the present application, only the base station in the NR system is used as an example for introduction, and the specific type of the base station is not limited.

[0067] Among them, the core network equipment may include but is not limited to at least one of the following: core network node, core network function, mobility management entity (Mobility Management Entity, MME), access mobility management function (Access and Mobility Management Function, AMF), session management function (Session Management Function, SMF), user plane function (User Plane Function, UPF), policy control function (Policy Control Function, PCF), policy and charging rules function unit (Policy and Charging Rules Function, PCRF), edge application service discovery function (Edge Application Server Discovery Function, EASDF), unified data management (Unified Data Management, UDM), unified data storage (Unified Data Repository, UDR), home user server (Home Subscriber Server, HSS), centralized network configuration (CNC), network storage function (Network Repository Function, NRF), network exposure function (Network Exposure Function, NEF), local NEF (Local NEF, or L-NEF), binding support function (Binding Support Function, BSF), application function (Application Function, AF), location management function (Location Management Function, LMF), etc. It should be noted that in the embodiment of the present application, only the core network device in the NR system is introduced as an example, and the specific type of the core network device is not limited.

[0068] In order to facilitate a better understanding of the embodiments of the present application, the technologies related to the present application are explained.

[0069] Artificial intelligence (AI) has been widely used in various fields. Integrating artificial intelligence into wireless communication networks and significantly improving technical indicators such as throughput, latency, and user capacity are important tasks for future wireless communication networks. There are many ways to implement AI modules, such as neural networks, decision trees, support vector machines, Bayesian classifiers, etc. This application takes neural networks as an example for illustration, but does not limit the specific type of AI modules.

[0070] An exemplary neural network can be Figure 2 As shown, the neural network is composed of neurons, and the neurons can be Figure 3 As shown, where α1, α2, … α K is the input, w is the weight (multiplicative coefficient), b is the bias (additive coefficient), and σ(.) is the activation function. Common activation functions include Sigmoid, tanh, Rectified Linear Unit (ReLU), etc.

[0071] The parameters of the neural network are optimized using a gradient optimization algorithm. A gradient optimization algorithm is a type of algorithm that minimizes or maximizes an objective function (sometimes called a loss function), and the objective function is often a mathematical combination of model parameters and data. For example, given data X and its corresponding label Y, we build a neural network model f(.). With the model, we can get the predicted output f(x) based on the input x, and we can calculate the difference between the predicted value and the true value (f(x)-Y), which is the loss function. The purpose of model training is to find the appropriate w, b to minimize the value of the above loss function. The smaller the loss value, the closer the constructed neural network model is to the actual situation.

[0072] For example, in the training process of the neural network model, common optimization algorithms are basically based on the error back propagation (BP) algorithm. The basic idea of ​​the BP algorithm is that the learning process consists of two processes: the forward propagation of the signal and the back propagation of the error. During the forward propagation, the input sample is transmitted from the input layer, and after being processed layer by layer by each hidden layer, it is transmitted to the output layer. If the actual output of the output layer does not match the expected output, the error back propagation stage is entered. Error back propagation is to transmit the output error layer by layer through the hidden layer to the input layer in a certain form, and distribute the error to all units of each layer, so as to obtain the error signal of each layer unit, and this error signal is used as the basis for correcting the weights of each unit. This process of adjusting the weights of each layer of the signal forward propagation and error back propagation is repeated. The process of continuous adjustment of weights is the learning and training process of the network. This process continues until the error of the network output is reduced to an acceptable level, or until the preset number of learning times is reached.

[0073] Common optimization algorithms include gradient descent, stochastic gradient descent (SGD), mini-batch gradient descent, momentum method (Momentum), Nesterov (specifically stochastic gradient descent with momentum), adaptive gradient descent (Adagrad), Adadelta, root mean square prop (RMSprop), adaptive momentum estimation (Adam), etc.

[0074] When these optimization algorithms are backpropagating errors, they can calculate the derivative / partial derivative of neurons based on the error / loss obtained from the loss function, add the influence of the learning rate, the previous gradient / derivative / partial derivative, etc., get the gradient, and pass the gradient to the previous layer.

[0075] To facilitate understanding of the technical solutions of the embodiments of the present application, the technical solutions of the present application are described in detail below through specific embodiments. The above related technologies can be combined arbitrarily with the technical solutions of the embodiments of the present application as optional solutions, and they all belong to the protection scope of the embodiments of the present application. The embodiments of the present application include at least part of the following contents.

[0076] Figure 4 is a schematic flow chart of a model performance supervision method 200 according to an embodiment of the present application, such as Figure 4 As shown, the model performance supervision method 200 may include at least part of the following contents:

[0077] S210: The first device obtains first information, where the first information is used to characterize uncertainty or probability distribution of an output of a first AI model, and an input of the first AI model is a first feature;

[0078] S220, the first device determines third information based on the first information and the second information, wherein the second information is a label corresponding to the first feature, and the third information is used to determine the validity of the first AI model.

[0079] It should be understood that Figure 4 The steps or operations of the model performance monitoring method 200 are shown, but these steps or operations are only examples. The embodiment of the present application may also perform other operations or Figure 4 Variations of the various operations in .

[0080] In an embodiment of the present application, third information can be determined based on the uncertainty or probability distribution of the output of the first AI model and the label corresponding to the input feature of the first AI model, and the validity of the first AI model can be determined based on the third information, thereby achieving performance supervision of the first AI model. Specifically, the uncertainty or probability distribution of the output of the first AI model can improve the robustness of the reasoning results of the first AI model. Since the reasoning results of the first AI model cover multiple possible results and their probability distributions, it is conducive to further processing and utilization of the reasoning results. For example, the reasoning results of the first AI model (including relevant information on uncertainty or probability distribution) are combined with other types of information to obtain a more accurate target result. In the performance supervision of the first AI model, referring to the uncertainty or probability distribution of the output of the first AI model and the corresponding label can improve the accuracy of the performance supervision of the first AI model.

[0081] Exemplarily, other types of information may be as follows: motion state information (such as speed, acceleration, etc.), signal quality measurement information (such as reference signal received power (Reference Signal Received Power, RSRP), signal to interference plus noise ratio (Signal to Interference plus Noise Ratio, SINR), reference signal received quality (Reference Signal Received Quality, RSRQ), etc.).

[0082] Exemplarily, the judgment result of the effectiveness of the first AI model obtained based on the embodiment of the present application can also be combined with the judgment result of the effectiveness of the first AI model obtained by other model supervision methods to obtain the final conclusion on the effectiveness of the first AI model.

[0083] It should be noted that in the embodiment of the present application, the higher the uncertainty of the output of the first AI model, the lower the accuracy of the output of the first AI model.

[0084] The label described in the embodiment of the present application is obtained in some way and is associated with the target task. This label can be obtained through measurement or other prior information; for example, in positioning based on an AI model, the input of the AI ​​model is channel state information, and the output of the AI ​​model is the uncertainty or probability distribution of the position. Then this label is the position information corresponding to the channel state information. The position information can be obtained through GPS and other positioning methods, or obtained from a positioning reference unit with a known position.

[0085] In an embodiment of the present application, the first information may also be referred to as soft information (such as probability distribution, confidence level, confidence interval, etc.), wherein the soft information gives the probability distribution or confidence level of the possible results. Optionally, the first AI model may be a soft information AI model or may not be a soft information AI model. Among them, the soft information AI model refers to a type of AI model that outputs soft information (such as probability distribution, confidence level, confidence interval, etc.), including both classic probability models and AI models based on neural networks; the soft information AI model measures the possibility of different prediction results and gives the probability distribution or confidence level of each possible result.

[0086] It should be noted that compared with the hard information (hard value) obtained by AI model reasoning (such as Time of Arrival (TOA), Reference Signal Time Difference (RSTD), Angle of Arrival (AoA), Angle of Departure (AoD), Reference Signal Received Power (RSRP), Line of Sight (LOS) indication, Non Line of Sight (NLOS) indication, etc.), the soft information obtained by AI model reasoning can significantly improve the reasoning accuracy and robustness. Specifically, soft information can better describe the uncertainty of the world, improve the robustness of model reasoning, and provide better security for some services that require relatively high reasoning reliability.

[0087] In some embodiments, the dimension of the first feature is D1 dimension, and correspondingly, the first information is D2 group of soft information of the second feature, wherein the D2 group of soft information describes the range of the output of the first AI model. If there is only one group of soft information (ie, D2=1), then this group of soft information describes the uncertainty of the output of the entire first AI model; if there are at least two groups of soft information (ie, D2≥2), then each group of soft information describes the uncertainty of some features of the output of the first AI model.

[0088] For example, for prediction tasks: each prediction moment corresponds to a set of soft information, or each feature at each prediction moment corresponds to a set of soft information; for other tasks, each feature corresponds to a set of soft information, or all features correspond to a set of soft information. For example, if the output of the first AI model is 2-dimensional, then it corresponds to 2 sets of soft information.

[0089] For example, for a positioning task, the first AI model output is a 2-dimensional horizontal position coordinate, the 2-dimensional horizontal position coordinate corresponds to a set of soft information, or each dimension of the 2-dimensional horizontal position coordinate corresponds to a set of soft information.

[0090] In some embodiments, the AI ​​model described in this application may also be referred to as an AI unit, an AI model / AI unit, a machine learning (ML) model, an ML unit, an AI structure, an AI function, an AI feature, a neural network, a neural network function, a neural network function, etc., or the AI ​​model described in this application may also refer to a processing unit capable of implementing specific algorithms, formulas, processing procedures, capabilities, etc. related to AI, or the AI ​​model described in this application may be a processing method, algorithm, function, module or unit for a specific data set, or the AI ​​model described in this application may be a processing method, algorithm, function, module or unit running on AI / ML related hardware such as a graphics processing unit (GPU), a neural network processing unit (NPU), a tensor processing unit (TPU), an application specific integrated circuit (ASIC), etc., and this application does not specifically limit this. Optionally, the specific data set includes the input or output of the AI ​​model.

[0091] In some embodiments, the identifier of the AI ​​model described in this application may be an AI unit identifier, an AI structure identifier, an AI algorithm identifier, or an identifier of a specific data set associated with the AI ​​model described in this application, or an identifier of a specific scenario, environment, channel feature, or device related to the AI ​​model described in this application, or an identifier of a function, feature, capability, or module related to the AI ​​model described in this application. This application does not make any specific limitations on this.

[0092] It should be noted that the generalization ability of AI models is limited. A model trained based on data from one scenario may fail when applied to another scenario. Even a model trained based on data from the same scenario will fail over time. Failure refers to the reduction in reasoning accuracy of the AI ​​model to the point where it cannot meet the target requirements. Therefore, the performance of the AI ​​model needs to be supervised.

[0093] In some embodiments, the first AI model may be an activated model, or the first AI model may be an inactivated model. Specifically, when the first AI model is an inactivated model, after determining the validity of the first AI model, when performing AI model selection, an effective AI model may be preferentially selected from at least two AI models, thereby facilitating the selection of an AI model.

[0094] In some embodiments, the first information includes but is not limited to at least one of the following: a parameter of the probability density distribution of the second feature, a confidence interval of the second feature, a value of the second feature, a value of the second feature and its probability, and a value of the second feature and its confidence. The embodiment of the present application clarifies the content of the first information, which is beneficial to realizing the performance supervision of the AI ​​model.

[0095] Exemplarily, the amount of information contained in the first information may be one or at least two, that is, the first information may include but is not limited to at least one of the following: parameters of the probability density distribution of one or at least two second features, confidence intervals of one or at least two second features, values ​​of one or at least two second features, values ​​of one or at least two second features and their probabilities, and values ​​of one or at least two second features and their confidence levels.

[0096] In some embodiments, the parameters of the probability density distribution include, but are not limited to, at least one of the following: a mean or expectation of the probability density distribution, a variance or standard deviation of the probability density distribution, and an indication of the type of the probability density distribution.

[0097] For example, the type of probability density distribution may include but is not limited to at least one of the following: Gaussian distribution, Poisson distribution.

[0098] For example, for a Gaussian distribution, there is a conversion relationship between the confidence interval, the confidence level, the mean, and the standard deviation, such as a typical Gaussian distribution with a mean of μ and a standard deviation of σ.

[0099] For example, the 90% confidence interval is: [μ-1.645σ, μ+1.645σ], which means that there is a 90% probability that the predicted target value is within the interval [μ-1.645σ, μ+1.645σ].

[0100] For example, the 95% confidence interval is: [μ-1.96σ, μ+1.96σ], which means that there is a 95% probability that the predicted target value is within the interval [μ-1.96σ, μ+1.96σ].

[0101] Exemplarily, there may be a conversion relationship between the confidence interval, the confidence level, the mean, and the standard deviation, where the mean is μ, the standard deviation is σ, the coefficient z may be obtained by the probability density distribution function or the type of probability density distribution, and the confidence interval with a confidence level of p% may be as follows:

[0102] [μ-z p％ σ,μ+z p％ σ].

[0103] In some embodiments, the first information is output information of the first AI model, or the first information is information determined based on the output information of the first AI model.

[0104] Exemplarily, the output information of the first AI model is at least one of the following: a parameter of the probability density distribution of the second feature, a confidence interval of the second feature, a value of the second feature, a value of the second feature and its probability, and a value of the second feature and its confidence. That is, the first information is the output information of the first AI model.

[0105] Exemplarily, the output information of the first AI model is the value of the second feature, and the first information can be determined based on the value of the second feature. Optionally, the first information is determined based on the value of the second feature and other information (such as motion state information (speed, acceleration, etc.), signal quality measurement information (such as RSRP, SINR, RSRQ, etc.)). For example, the worse the signal quality, the higher the uncertainty of the second feature obtained based on the value of the second feature and other information; for another example, the faster the movement speed, the higher the uncertainty of the second feature obtained based on the value of the second feature and other information.

[0106] For example, if the output information of the first AI model is the value of the second feature, at least one of the following can be determined based on the value of the second feature: a parameter of the probability density distribution of the second feature, a confidence interval of the second feature, a probability of the value of the second feature, and a confidence level of the value of the second feature. That is, the first information is information determined based on the output information of the first AI model.

[0107] In some embodiments, the first AI model can be used to implement one of the following functions: positioning, beam management, channel state information (CSI) prediction, mobility management, and CSI compression. Of course, the first AI model can also be used to implement other functions, which is not limited in this application.

[0108] Specifically, the model performance supervision method 200 described in this embodiment can be used to implement at least two different functions, that is, the model performance supervision method 200 described in this embodiment can be applicable to a public label-based AI model supervision framework, thereby avoiding the need to design performance supervision solutions for different functions.

[0109] In some embodiments, when the first AI model is used to implement the positioning function, the first feature may include but is not limited to at least one of the following: time domain channel impulse response, RSRP (such as layer 1 RSRP or layer 3 RSRP), frequency domain channel impulse response, time domain waveform of the received signal. Optionally, the time domain channel impulse response includes at least one of the following: time information, power information, phase information. Optionally, the frequency domain channel impulse response includes at least one of the following: frequency information (subcarrier sequence number and interval), power information, phase information.

[0110] In some embodiments, when the first AI model is used to implement the positioning function, the second feature may include but is not limited to at least one of the following: line-of-sight TOA, RSTD, AoA, AoD, RSRP, LOS indication, NLOS indication.

[0111] For example, when the first AI model is used to realize the positioning function, the input of the first AI model (i.e., the first feature) is the time domain channel impulse response, and the output of the first AI model is the soft information of the intermediate feature quantity (i.e., the second feature) (such as parameters of the probability density distribution, confidence interval, confidence level, etc.), and the intermediate feature quantity includes at least one of the following: line of sight TOA, RSTD, AoA, AoD, RSRP, LOS indication, NLOS indication, etc. The position coordinates can be further determined based on the soft information of the intermediate feature quantity; the output of the first AI model may also be the soft information of the position coordinates.

[0112] In some embodiments, when the first AI model is used to implement the beam management function, the first feature may include beam information at T1 historical moments, such as sequence number, angle, L1-RSRP, etc., and the second feature includes beam information at T2 future moments.

[0113] For example, when the first AI model is used to implement the beam management function, the input of the first AI model (i.e., the first feature) is the beam information at T1 historical moments, such as serial number, angle, L1-RSRP, etc., and the output of the first AI model is the soft information (such as parameters of probability density distribution, confidence interval, confidence level, etc.) of the beam information (i.e., the second feature) at T2 future moments. For example, the vertical beam and the horizontal beam at each moment correspond to a set of soft information, such as the confidence interval and probability density distribution of L1-RSRP, etc., and the beam information at each moment corresponds to a set of soft information.

[0114] In some embodiments, when the first AI model is used to implement the CSI prediction function, the first feature may include the CSI at T1 historical moments, and the second feature may include the CSI at T2 future moments.

[0115] For example, when the first AI model is used to implement the CSI prediction function, the input of the first AI model (i.e., the first feature) is the CSI at T1 historical moments, and the output of the first AI model is the soft information (such as parameters of the probability density distribution, confidence interval, confidence level, etc.) of the CSI at T2 moments in the future (i.e., the second feature), such as the CSI at each moment corresponds to a set of soft information, such as each dimension of the CSI at each moment corresponds to a set of soft information, or the CSI at each moment corresponds to a set of soft information.

[0116] In some embodiments, when the first AI model is used to implement the mobility management function, the first feature may include the layer 1 RSRP (L1-RSRP) or layer 3 RSRP (L3-RSRP) at the historical T1 moment, and the second feature may include the RSRP at the future T2 moments or the decision of whether a cell switching occurs at the future T2 moments. It should be noted that L3-RSRP is obtained by filtering L1-RSRP.

[0117] For example, when the first AI model is used to implement the mobility management function, the input of the first AI model (i.e., the first feature) is the L1-RSRP or L3-RSRP at T1 historical moments, and the output of the first AI model is the soft information (such as parameters of probability density distribution, confidence interval, confidence level, etc.) of the RSRP (i.e., the second feature) at T2 moments in the future; or the output of the first AI model is the soft information (such as parameters of probability density distribution, confidence interval, confidence level, etc.) of the decision on whether a cell switching will occur (i.e., the second feature) at T2 moments in the future, such as 1 for switching and 0 for no switching, and the soft information is a value between 0 and 1.

[0118] In some embodiments, when the first AI model is used to implement the CSI compression function, the first feature may include uncompressed CSI, and the second feature may include compressed CSI.

[0119] For example, when the first AI model is used to implement the CSI compression function, the input of the first AI model (ie, the first feature) is the uncompressed CSI, and the output of the first AI model is the soft information of the compressed CSI (ie, the second feature) (such as parameters of the probability density distribution, confidence interval, confidence level, etc.).

[0120] In some embodiments, the label corresponding to the first feature (ie, the second information) may be real, or may be measured or estimated, which is not limited in this embodiment of the present application.

[0121] It should be noted that the type of label corresponding to the first feature is consistent with the type of output of the first AI model. For example, both are location information, both are TOA, both are CSI, both are RSRP, etc.; this depends on the specific task type.

[0122] In some embodiments, the label corresponding to the first feature (ie, the second information) may be measured, estimated, or stored by the first device, or the label corresponding to the first feature (ie, the second information) may be obtained by the first device from the second device or other devices.

[0123] In some embodiments, in the above S210, the first device obtains the first information, including one of the following:

[0124] The first device receives first information from the second device;

[0125] The first device obtains first information through output information of the first AI model;

[0126] The first device receives output information of the first AI model from the second device, and obtains first information according to the output information of the first AI model.

[0127] In some embodiments, when the first device obtains the first information through output information of the first AI model, the first AI model can be deployed on the first device side.

[0128] In some embodiments, when the first device receives the first information from the second device, the first AI model can be deployed on the second device side. For example, the second device obtains the first information through the output information of the first AI model, and the second device sends the first information to the first device.

[0129] In some embodiments, when the first device receives output information of the first AI model from the second device and obtains the first information based on the output information of the first AI model, the first AI model can be deployed on the second device side.

[0130] In some embodiments, the first device determines the validity of the first AI model based on the third information. Specifically, after the first device determines the third information based on the first information and the second information, the first device determines the validity of the first AI model based on the third information. Optionally, the first device sends first indication information to the second device, wherein the first indication information is used to indicate the validity of the first AI model. For example, the first indication information occupies 1 bit; wherein a value of 0 indicates that the first AI model is valid, and a value of 1 indicates that the first AI model is invalid; or, a value of 1 indicates that the first AI model is valid, and a value of 0 indicates that the first AI model is invalid. Optionally, the first indication information includes an identifier of the first AI model or an identifier of a function associated with the first AI model.

[0131] In some embodiments, the first device sends third information to the second device. Further, the second device may determine the validity of the first AI model based on the third information. Optionally, the first device receives second indication information from the second device, wherein the second indication information is used to indicate the validity of the first AI model. For example, the second indication information occupies 1 bit; wherein a value of 0 indicates that the first AI model is valid, and a value of 1 indicates that the first AI model is invalid; or, a value of 1 indicates that the first AI model is valid, and a value of 0 indicates that the first AI model is invalid. Optionally, the second indication information includes an identifier of the first AI model or an identifier of a function associated with the first AI model.

[0132] Exemplarily, when the first AI model is deployed on the first device side, the first device obtains first information through the output information of the first AI model, then the first device obtains second information from a local or other device (such as a second device or a device other than the first device and the second device), and the first device determines third information based on the first information and the second information, and then, the first device determines the validity of the first AI model based on the third information, and finally, the first device sends first indication information to the second device, wherein the first indication information is used to indicate the validity of the first AI model.

[0133] Exemplarily, when the first AI model is deployed on the first device side, the first device obtains the first information through the output information of the first AI model, then the first device obtains the second information from the local or other device (such as the second device or a device other than the first device and the second device), and the first device determines the third information based on the first information and the second information, and then the first device sends the third information to the second device, after which the second device determines the validity of the first AI model based on the third information, and finally, the second device sends the second indication information to the first device, wherein the second indication information is used to indicate the validity of the first AI model.

[0134] Exemplarily, when the first AI model is deployed on the second device side, the second device obtains the first information through the output information of the first AI model, and the second device sends the first information to the first device. Then, the first device obtains the second information from the local or other devices (such as the second device or a device other than the first device and the second device), and the first device determines the third information based on the first information and the second information. Then, the first device determines the validity of the first AI model based on the third information. Finally, the first device sends the first indication information to the second device, wherein the first indication information is used to indicate the validity of the first AI model.

[0135] Exemplarily, when the first AI model is deployed on the second device side, the second device obtains the first information through the output information of the first AI model, and the second device sends the first information to the first device. Then, the first device obtains the second information from the local or other devices (such as the second device or a device other than the first device and the second device), and the first device determines the third information based on the first information and the second information. Then, the first device sends the third information to the second device, and then, the second device determines the validity of the first AI model based on the third information. Finally, the second device sends second indication information to the first device, wherein the second indication information is used to indicate the validity of the first AI model.

[0136] Exemplarily, when the first AI model is deployed on the second device side, the first device receives the output information of the first AI model from the second device, and the first device obtains the first information based on the output information of the first AI model. Then, the first device obtains the second information from the local or other device (such as the second device or a device other than the first device and the second device), and the first device determines the third information based on the first information and the second information. Then, the first device determines the validity of the first AI model based on the third information. Finally, the first device sends the first indication information to the second device, wherein the first indication information is used to indicate the validity of the first AI model.

[0137] Exemplarily, when the first AI model is deployed on the second device side, the first device receives the output information of the first AI model from the second device, and the first device obtains the first information based on the output information of the first AI model. Then, the first device obtains the second information from the local or other device (such as the second device or a device other than the first device and the second device), and the first device determines the third information based on the first information and the second information. Then, the first device sends the third information to the second device, and then, the second device determines the validity of the first AI model based on the third information. Finally, the second device sends the second indication information to the first device, wherein the second indication information is used to indicate the validity of the first AI model.

[0138] In some embodiments, the first device may be a terminal, a network side device, or a third-party server, wherein the network side device includes an access network device or a core network device.

[0139] In some embodiments, the second device may be a terminal, a network side device, or a third-party server, wherein the network side device includes an access network device or a core network device.

[0140] In some embodiments, when the first information includes parameters of the probability density distribution of the second feature, the third information is used to characterize the mean of the first probability of N samples of the first feature, or the third information is used to characterize the mean of the logarithm of the first probability of N samples of the first feature; wherein the first probability is the probability of the label corresponding to each sample in the N samples under the probability density distribution of the second feature, and N is a positive integer. Optionally, the N samples can also be replaced by N inference processes of the first AI model.

[0141] Exemplarily, when the first information includes parameters of the probability density distribution of the second feature, the third information may be a log-likelihood value.

[0142] In some embodiments, when the third information is used to characterize the mean of the first probabilities of N samples of the first feature, the third information is determined based on the following formula 1:

[0143]

[0144] Wherein, L1 represents the third information, i represents the i-th sample among the N samples, Represents the standard deviation or variance of the probability density distribution of the second feature corresponding to the i-th sample, represents the mean (such as the statistical mean) of the probability density distribution of the second feature corresponding to the i-th sample, y i represents the label corresponding to the i-th sample, and g(·) represents the probability density distribution function obeyed by the output of the first AI model.

[0145] In some embodiments, when the third information is used to characterize the mean of the logarithm of the first probability of N samples of the first feature, the third information is determined based on the following formula 2:

[0146]

[0147] Wherein, L2 represents the third information, i represents the i-th sample among the N samples, represents the mean (such as the statistical mean) of the probability density distribution of the second feature corresponding to the i-th sample, Represents the standard deviation or variance of the probability density distribution of the second feature corresponding to the i-th sample, y i represents the label corresponding to the i-th sample, s is a positive integer, and g(·) represents the probability density distribution function obeyed by the output of the first AI model.

[0148] Optionally, s is agreed upon by a protocol, or s is determined by the first device, or s is configured or indicated by the second device.

[0149] In some embodiments, in the above formula 1 or formula 2, g(·) is related to the type of probability density distribution, for example, g(·) can be determined based on the type of probability density distribution.

[0150] For example, taking the Gaussian distribution as the type of probability density distribution, the above formula 1 can be transformed into the following formula 3.

[0151]

[0152] For example, taking the probability density distribution type as Gaussian distribution as an example, the above formula 2 can be transformed into the following formula 4.

[0153]

[0154] In some embodiments, when the first information includes parameters of the probability density distribution of the second feature, the third information is used to determine the validity of the first AI model, including:

[0155] When the third information is greater than or equal to the first threshold, the first AI model is valid; or,

[0156] When the third information is less than the first threshold, the first AI model fails.

[0157] Optionally, the first threshold is agreed upon by a protocol, or the first threshold is determined by the first device, or the first threshold is configured or indicated by the second device.

[0158] In some embodiments, when the first information includes the confidence interval of the second feature, the third information is used to characterize the ratio of the number of samples whose corresponding labels are within the confidence interval of the second feature among the N samples of the first feature to the total number of samples N, where N is a positive integer. Optionally, the N samples can also be replaced by N inference processes of the first AI model.

[0159] In some embodiments, when the first information includes a confidence interval of the second feature, the third information is determined based on the following formula 5:

[0160]

[0161] Wherein, L3 represents the third information, i represents the i-th sample in the N samples, and x i Represents the input information corresponding to the i-th sample, y i Indicates the label corresponding to the i-th sample. When y i The confidence interval f(x i ) within , otherwise

[0162] In some embodiments, when the first information includes a confidence interval of the second feature, the third information is used to determine the validity of the first AI model, including:

[0163] When the third information is greater than or equal to the second threshold, the first AI model is valid; or,

[0164] When the third information is less than the second threshold, the first AI model fails.

[0165] Optionally, the second threshold is agreed upon by a protocol, or the second threshold is determined by the first device, or the second threshold is configured or indicated by the second device.

[0166] In some embodiments, when the first information includes the value of the second feature and its probability or confidence, the third information is used to characterize the average of the weighted distances between the labels corresponding to the N samples of the first feature and the value of the second feature, where the weighted weight is determined based on the probability or confidence of the second feature, and N is a positive integer. Optionally, the N samples can also be replaced by N inference processes of the first AI model.

[0167] In some embodiments, when the first information includes the value of the second feature and its probability or confidence, the third information is determined based on the following formula 6:

[0168]

[0169] Wherein, L4 represents the third information, i represents the i-th sample among the N samples, Represents the mean of the probability density distribution of the second feature corresponding to the i-th sample, y i Indicates the label corresponding to the i-th sample.

[0170] For example, in the above formula 6, Σ -1 The diagonal elements of are P powers of probability or confidence, where P is greater than or equal to 0 and P is an integer.

[0171] In some embodiments, when the first information includes the value of the second feature and its probability or confidence, the third information is used to determine the validity of the first AI model, including:

[0172] When the third information is less than or equal to the third threshold, the first AI model is valid; or,

[0173] When the third information is greater than a third threshold, the first AI model fails.

[0174] Optionally, the third threshold is agreed upon by a protocol, or the third threshold is determined by the first device, or the third threshold is configured or indicated by the second device.

[0175] It should be noted that in the above formula, y i , It can be a specific numerical value, or a vector or a matrix, which is not limited in the embodiments of the present application.

[0176] In some embodiments, the N samples are acquired within M time units, where M is a positive integer.

[0177] Optionally, the time unit may include at least one of the following: an orthogonal frequency-division multiplexing (OFDM) symbol, a time slot, a subframe, a frame, microseconds, milliseconds, seconds, minutes, hours, days, weeks, and months.

[0178] Optionally, M is configured or indicated by the second device, or M is agreed upon by a protocol.

[0179] In some embodiments, the scene identifiers or data set identifiers associated with the N samples are the same.

[0180] In some embodiments, the first device may obtain at least one of the following parameters from the second device:

[0181] The number of samples N used for AI model performance supervision;

[0182] The minimum number of samples N used for AI model performance supervision;

[0183] Get the time range T of N samples;

[0184] The type and parameters of the probability density distribution; for example, in the case of a Gaussian mixture model, the number of Gaussian distributions involved needs to be indicated;

[0185] Parameter P;

[0186] Parameter M.

[0187] In some embodiments, the first device reports at least one of the following parameters to the second device:

[0188] The number of samples N used for AI model performance supervision;

[0189] The minimum number of samples N used for AI model performance supervision;

[0190] Get the time range T of N samples;

[0191] The type and parameters of the probability density distribution; for example, in the case of a Gaussian mixture model, the number of Gaussian distributions involved needs to be indicated;

[0192] Parameter P;

[0193] Parameter M.

[0194] In some embodiments, the third information may be positive incentive information or negative incentive information; for example, when the negative incentive information is greater than a certain threshold or the positive incentive information is less than or equal to a certain threshold, the first AI model is considered to be invalid.

[0195] Exemplarily, the positive incentive information may include but is not limited to at least one of the following:

[0196] Among the N samples of the first feature, or The sample size or proportion;

[0197] The number or proportion of samples whose labels are within the confidence interval among the N samples of the first feature;

[0198] The number or proportion of samples for which the weighted distance between the label and the value of the second feature is less than or equal to t3 among the N samples of the first feature.

[0199] Exemplarily, the negative incentive information may include but is not limited to at least one of the following:

[0200] Among the N samples of the first feature, or The sample size or proportion;

[0201] The number or proportion of samples whose labels are outside the confidence interval among the N samples of the first feature;

[0202] The number or proportion of samples for which the weighted distance between the label and the value of the second feature is greater than t3 among the N samples of the first feature.

[0203] Specifically, t1 may be the first threshold, t2 may be the second threshold, and t3 may be the third threshold.

[0204] Therefore, in an embodiment of the present application, the third information can be determined based on the uncertainty or probability distribution of the output of the first AI model (i.e., the first information) and the label corresponding to the input feature of the first AI model (i.e., the second information), and the validity of the first AI model can be determined based on the third information, thereby achieving performance supervision of the first AI model. Specifically, the uncertainty or probability distribution of the output of the first AI model can improve the robustness of the reasoning results of the first AI model. Since the reasoning results of the first AI model cover multiple possible results and their probability distributions, it is conducive to further processing and utilization of the reasoning results. For example, the reasoning results of the first AI model (including relevant information on uncertainty or probability distribution) are combined with other types of information to obtain a more accurate target result. In the performance supervision of the first AI model, referring to the uncertainty or probability distribution of the output of the first AI model and the corresponding label can improve the accuracy of the performance supervision of the first AI model.

[0205] The following describes a solution for AI model positioning based on soft information through a specific example. Soft information can improve the robustness of the reasoning results of the AI ​​model because the reasoning results cover multiple possible results and their probability distribution. It is conducive to further processing and utilization of the reasoning results of the AI ​​model, such as combining soft information with other types of information (such as information output by other AI models used to implement positioning functions) to obtain more accurate target results, and it can also provide better security for some businesses that require relatively high reasoning reliability.

[0206] Example 1

[0207] The input of the i-th AI model is the time domain channel impulse response (CIR) of the i-th transmission reception point (TRP), including the time, power and phase information of the multipath;

[0208] The output of the i-th AI model is the mean value μ of the line-of-sight TOA between the i-th TRP and the terminal. i and standard deviation σ i ;

[0209] The line-of-sight TOA estimate x between the i-th TRP and the terminal i Modeled as a Gaussian distribution:

[0210]

[0211] For N TRPs, the likelihood function is modeled as:

[0212] where x=[x1,...,x N ] T ;

[0213] The maximum likelihood estimation problem can be transformed into a weighted least squares problem:

[0214]

[0215] Where μ=[μ1,...,μ N ] T ,Σ is an N*N matrix, and its i-th diagonal element is Its goal is to find a location The N TOA estimates obtained by combining this position with the coordinates of the N TRPs are The weighted distance to the N TOA mean μ estimated by the AI ​​model is the smallest.

[0216] because and The relationship between is nonlinear, so it can be solved by linear approximation and greedy algorithm. The following takes the particle swarm optimization algorithm as an example to give a specific implementation scheme and simulation results. Figure 5 shown.

[0217] Among them, different optimization algorithms, TRP numbers, and corresponding positioning accuracy can be shown in Table 1 below.

[0218] Table 1

[0219] algorithm Is there soft information? TRP quantity Positioning accuracy @90%m Soft-PSO yes 18 0.90 PSO no 18 2.30 CHAN no 18 2.39 LS no 18 3.64 Soft-PSO yes 6 2.80 PSO no 6 4.16 CHAN no 6 4.07 LS no 6 6.16

[0220] In addition, the framework can support mixed positioning of multiple types of soft information. In the above example, only the soft information μ, σ of TOA of N TRPs is given. In addition, soft information of angles can also be included. Its likelihood function can be written as follows:

[0221]

[0222] x, α, and z refer to different types of information, such as TOA, AOD, and AOA, respectively.

[0223] Secondly, the number of TRPs for different types of information can also be different. For example, the number of TRPs for x is N1, the number of TRPs for α is N2, and the number of TRPs for z is N3. The likelihood function can be written as follows:

[0224]

[0225] Combination of the above Figures 4 to 5 , describes in detail the first device side embodiment of the present application, and the following is combined with Figure 6 , the second device side embodiment of the present application is described in detail. It should be understood that the second device side embodiment corresponds to the first device side embodiment, and similar descriptions can refer to the first device side embodiment.

[0226] Figure 6 is a schematic flow chart of a model performance supervision method 300 according to an embodiment of the present application, such as Figure 6 As shown, the model performance supervision method 300 may include at least part of the following contents:

[0227] S310, the second device receives third information from the first device, wherein the third information is determined based on the first information and the second information, the first information is used to characterize the uncertainty or probability distribution of the output of the first AI model, the input of the first AI model is the first feature, and the second information is a label corresponding to the first feature;

[0228] S320: The second device determines the validity of the first AI model based on the third information.

[0229] It should be understood that Figure 6 The steps or operations of the model performance supervision method 300 are shown, but these steps or operations are only examples. The embodiment of the present application may also perform other operations or Figure 6 Variations of the various operations in .

[0230] In some embodiments, the first information includes at least one of the following:

[0231] Parameters of the probability density distribution of the second feature, a confidence interval of the second feature, a value of the second feature, a value of the second feature and its probability, and a value of the second feature and its confidence level.

[0232] In some embodiments, the parameters of the probability density distribution include at least one of the following: a mean or expectation of the probability density distribution, a variance or standard deviation of the probability density distribution, and an indication of the type of the probability density distribution.

[0233] In some embodiments, when the first information includes a parameter of a probability density distribution of the second feature, the third information is used to characterize a mean of the first probabilities of N samples of the first feature, or the third information is used to characterize a mean of the logarithms of the first probabilities of N samples of the first feature;

[0234] The first probability is the probability of the label corresponding to each of the N samples under the probability density distribution of the second feature, and N is a positive integer.

[0235] In some embodiments, when the third information is used to characterize the mean of the first probabilities of N samples of the first feature, the third information is determined based on the following formula:

[0236]

[0237] Wherein, L1 represents the third information, i represents the i-th sample among the N samples, Represents the standard deviation or variance of the probability density distribution of the second feature corresponding to the i-th sample, Represents the mean of the probability density distribution of the second feature corresponding to the i-th sample, y i represents the label corresponding to the i-th sample, and g(·) represents the probability density distribution function obeyed by the output of the first AI model.

[0238] In some embodiments, when the third information is used to characterize the mean of the logarithm of the first probability of N samples of the first feature, the third information is determined based on the following formula:

[0239]

[0240] Wherein, L2 represents the third information, i represents the i-th sample among the N samples, represents the mean of the probability density distribution of the second feature corresponding to the i-th sample, Represents the standard deviation or variance of the probability density distribution of the second feature corresponding to the i-th sample, y i represents the label corresponding to the i-th sample, s is a positive integer, and g(·) represents the probability density distribution function obeyed by the output of the first AI model.

[0241] In some embodiments, when the first information includes parameters of the probability density distribution of the second feature, the above S320 may specifically include:

[0242] When the third information is greater than or equal to the first threshold, the second device determines that the first AI model is valid; or,

[0243] When the third information is less than the first threshold, the second device determines that the first AI model is invalid.

[0244] In some embodiments, when the first information includes the confidence interval of the second feature, the third information is used to characterize the ratio of the number of samples whose corresponding labels are within the confidence interval of the second feature among the N samples of the first feature to the total number of samples N, where N is a positive integer.

[0245] In some embodiments, when the first information includes a confidence interval of the second feature, the third information is determined based on the following formula:

[0246]

[0247] Wherein, L3 represents the third information, i represents the i-th sample in the N samples, and x i Represents the input information corresponding to the i-th sample, y i Indicates the label corresponding to the i-th sample. When y i The confidence interval f(x i ) within , otherwise

[0248] In some embodiments, when the first information includes the confidence interval of the second feature, the above S320 may specifically include:

[0249] When the third information is greater than or equal to the second threshold, the second device determines that the first AI model is valid; or,

[0250] When the third information is less than the second threshold, the second device determines that the first AI model is invalid.

[0251] In some embodiments, when the first information includes the value of the second feature and its probability or confidence, the third information is used to characterize the mean of the weighted distances between the labels corresponding to N samples of the first feature and the value of the second feature, wherein the weighted weight is determined based on the probability or confidence of the second feature, and N is a positive integer.

[0252] In some embodiments, when the first information includes the value of the second feature and its probability or confidence, the third information is determined based on the following formula:

[0253]

[0254] Wherein, L4 represents the third information, i represents the i-th sample among the N samples, Represents the mean of the probability density distribution of the second feature corresponding to the i-th sample, y i Indicates the label corresponding to the i-th sample.

[0255] In some embodiments, when the first information includes the value of the second feature and its probability or confidence, the above S320 may specifically include:

[0256] When the third information is less than or equal to the third threshold, the second device determines that the first AI model is valid; or,

[0257] When the third information is greater than the third threshold, the second device determines that the first AI model is invalid.

[0258] In some embodiments, before the second device receives the third information from the first device, the second device sends the first information to the first device.

[0259] In some embodiments, the second device sends second indication information to the first device, wherein the second indication information is used to indicate the validity of the first AI model. Optionally, the second indication information includes an identifier of the first AI model or an identifier of a function associated with the first AI model.

[0260] Therefore, in an embodiment of the present application, the third information can be determined based on the uncertainty or probability distribution of the output of the first AI model (i.e., the first information) and the label corresponding to the input feature of the first AI model (i.e., the second information), and the validity of the first AI model can be determined based on the third information, thereby achieving performance supervision of the first AI model. Specifically, the uncertainty or probability distribution of the output of the first AI model can improve the robustness of the reasoning results of the first AI model. Since the reasoning results of the first AI model cover multiple possible results and their probability distributions, it is conducive to further processing and utilization of the reasoning results. For example, the reasoning results of the first AI model (including relevant information on uncertainty or probability distribution) are combined with other types of information to obtain a more accurate target result. In the performance supervision of the first AI model, referring to the uncertainty or probability distribution of the output of the first AI model and the corresponding label can improve the accuracy of the performance supervision of the first AI model.

[0261] Figure 7 is a schematic flow chart of a model performance supervision method 400 according to an embodiment of the present application, such as Figure 7 As shown, the model performance supervision method 400 may include at least part of the following contents:

[0262] S410: The first device obtains fourth information, where the fourth information is used to characterize uncertainty or probability distribution of an output of a second AI model, and an input of the second AI model is a third feature;

[0263] S420, the first device determines the validity of the second AI model according to the fourth information; or, the first device sends the fourth information to the second device.

[0264] It should be understood that Figure 7 The steps or operations of the model performance monitoring method 400 are shown, but these steps or operations are only examples. The embodiment of the present application may also perform other operations or Figure 7 Variations of the various operations in .

[0265] In an embodiment of the present application, the validity of the second AI model can be determined based on the uncertainty or probability distribution of the output of the second AI model, thereby achieving performance supervision of the second AI model. Specifically, the uncertainty or probability distribution of the output of the second AI model can improve the robustness of the reasoning results of the second AI model. Since the reasoning results of the second AI model cover multiple possible results and their probability distributions, it is conducive to further processing and utilization of the reasoning results. For example, the reasoning results of the second AI model (including relevant information about uncertainty or probability distribution) are combined with other types of information to obtain a more accurate target result. In the performance supervision of the second AI model, the uncertainty or probability distribution of the output of the second AI model is referred to, and the validity of the second AI model can be judged. Since no external information is required, it is also easier to implement.

[0266] Exemplarily, the judgment result of the effectiveness of the second AI model obtained based on the embodiment of the present application can also be combined with the judgment result of the effectiveness of the second AI model obtained by other model supervision methods to obtain the final conclusion on the effectiveness of the second AI model.

[0267] It should be noted that in the embodiment of the present application, the higher the uncertainty of the output of the second AI model, the lower the accuracy of the output of the second AI model.

[0268] Illustratively, the second AI model described in the embodiment of the present application may be the same as or different from the first AI model described above, and the present application is not limited to this.

[0269] In an embodiment of the present application, the fourth information may also be referred to as soft information (such as probability distribution, confidence level, confidence interval, etc.), wherein the soft information gives the probability distribution or confidence level of the possible results. Optionally, the second AI model may be a soft information AI model or may not be a soft information AI model. Among them, the soft information AI model refers to a type of AI model that outputs soft information (such as probability distribution, confidence level, confidence interval, etc.), including both classic probability models and AI models based on neural networks; the soft information AI model measures the possibility of different prediction results and gives the possibility, probability distribution or confidence level of each possible result.

[0270] It should be noted that compared with the hard information (such as TOA, RSTD, AoA, AoD, RSRP, LOS indication, NLOS indication, etc.) obtained by AI model reasoning, the soft information obtained by AI model reasoning can significantly improve the reasoning accuracy and robustness. Specifically, soft information can better describe the uncertainty of the world, improve the robustness of model reasoning, and provide better security for some businesses that require relatively high reasoning reliability.

[0271] In some embodiments, the dimension of the third feature is D3, and correspondingly, the fourth information is D4 group of soft information of the fourth feature, wherein D4 group of soft information describes the range of the output of the second AI model. If there is only one group of soft information (ie, D4=1), then this group of soft information describes the uncertainty of the output of the entire second AI model; if there are at least two groups of soft information (ie, D4≥2), then each group of soft information describes the uncertainty of some features of the output of the second AI model.

[0272] For example, for prediction tasks: each prediction moment corresponds to a set of soft information, or each feature at each prediction moment corresponds to a set of soft information; for other tasks, each feature corresponds to a set of soft information, or all features correspond to a set of soft information.

[0273] For example, each feature corresponds to a set of soft information, and the output of the second AI model is 2-dimensional, corresponding to 2 sets of soft information.

[0274] For example, for the positioning task, the second AI model output is a 2-dimensional horizontal position coordinate, the 2-dimensional horizontal position coordinate corresponds to a set of soft information, or each dimension of the 2-dimensional horizontal position coordinate corresponds to a set of soft information.

[0275] In some embodiments, the AI ​​model described in this application may also be referred to as an AI unit, an AI model / AI unit, an ML model, an ML unit, an AI structure, an AI function, an AI feature, a neural network, a neural network function, a neural network function, etc., or the AI ​​model described in this application may also refer to a processing unit capable of implementing specific algorithms, formulas, processing procedures, capabilities, etc. related to AI, or the AI ​​model described in this application may be a processing method, algorithm, function, module or unit for a specific data set, or the AI ​​model described in this application may be a processing method, algorithm, function, module or unit running on AI / ML related hardware such as a GPU, NPU, TPU, ASIC, etc., which is not specifically limited in this application. Optionally, the specific data set includes the input or output of the AI ​​model.

[0276] In some embodiments, the identifier of the AI ​​model described in this application may be an AI unit identifier, an AI structure identifier, an AI algorithm identifier, or an identifier of a specific data set associated with the AI ​​model described in this application, or an identifier of a specific scenario, environment, channel feature, or device related to the AI ​​model described in this application, or an identifier of a function, feature, capability, or module related to the AI ​​model described in this application. This application does not make any specific limitations on this.

[0277] It should be noted that the generalization ability of AI models is limited. A model trained based on data from one scenario may fail when applied to another scenario. Even a model trained based on data from the same scenario will fail over time. Failure refers to the reduction in reasoning accuracy of the AI ​​model to the point where it cannot meet the target requirements. Therefore, the performance of the AI ​​model needs to be supervised.

[0278] In some embodiments, the second AI model may be an activated model, or the second AI model may be an inactivated model. Specifically, when the second AI model is an inactivated model, after determining the validity of the second AI model, when performing AI model selection, a valid AI model may be preferentially selected from at least two AI models, thereby facilitating the selection of an AI model.

[0279] In some embodiments, the fourth information includes but is not limited to at least one of the following: a parameter of the probability density distribution of the fourth feature, a confidence interval of the fourth feature, a value of the fourth feature, a value of the fourth feature and its probability, and a value of the fourth feature and its confidence. The embodiment of the present application clarifies the content of the fourth information, which is beneficial to realizing the performance supervision of the AI ​​model.

[0280] Exemplarily, the amount of information contained in the fourth information may be one or at least two, that is, the fourth information may include but is not limited to at least one of the following: parameters of the probability density distribution of one or at least two fourth features, confidence intervals of one or at least two fourth features, values ​​of one or at least two fourth features, values ​​of one or at least two fourth features and their probabilities, and values ​​of one or at least two fourth features and their confidence levels.

[0281] In some embodiments, the parameters of the probability density distribution include, but are not limited to, at least one of the following: a mean or expectation of the probability density distribution, a variance or standard deviation of the probability density distribution, and an indication of the type of the probability density distribution.

[0282] For example, the type of probability density distribution may include but is not limited to at least one of the following: Gaussian distribution, Poisson distribution.

[0283] For example, for a Gaussian distribution, there is a conversion relationship between the confidence interval, the confidence level, the mean, and the standard deviation, such as a typical Gaussian distribution with a mean of μ and a standard deviation of σ.

[0284] For example, the 90% confidence interval is: [μ-1.645σ, μ+1.645σ], which means that there is a 90% probability that the predicted target value is within the interval [μ-1.645σ, μ+1.645σ].

[0285] For example, the 95% confidence interval is: [μ-1.96σ, μ+1.96σ], which means that there is a 95% probability that the predicted target value is within the interval [μ-1.96σ, μ+1.96σ].

[0286] Exemplarily, there may be a conversion relationship between the confidence interval, the confidence level, the mean, and the standard deviation, where the mean is μ, the standard deviation is σ, the coefficient z may be obtained by the probability density distribution function or the type of probability density distribution, and the confidence interval with a confidence level of p% may be as follows:

[0287] [μ-z p％ σ,μ+z p％ σ].

[0288] In some embodiments, the fourth information is output information of the second AI model, or the fourth information is information determined based on the output information of the second AI model.

[0289] Exemplarily, the output information of the second AI model is at least one of the following: a parameter of the probability density distribution of the fourth feature, a confidence interval of the fourth feature, a value of the fourth feature, a value of the fourth feature and its probability, and a value of the fourth feature and its confidence. That is, the fourth information is the output information of the second AI model.

[0290] Exemplarily, the output information of the second AI model is the value of the fourth feature, and the fourth information can be determined based on the value of the fourth feature, such as based on the value of the fourth feature and other information (such as motion state information (speed, acceleration, etc.), signal quality measurement information (such as RSRP, SINR, RSRQ, etc.)). For example, the worse the signal quality, the higher the uncertainty of the fourth feature obtained based on the value of the fourth feature and other information; for another example, the faster the movement speed, the higher the uncertainty of the fourth feature obtained based on the value of the fourth feature and other information.

[0291] For example, if the output information of the second AI model is the value of the fourth feature, at least one of the following can be determined based on the value of the fourth feature: a parameter of the probability density distribution of the fourth feature, a confidence interval of the fourth feature, a probability of the value of the fourth feature, and a confidence level of the value of the fourth feature. That is, the fourth information is information determined based on the output information of the second AI model.

[0292] In some embodiments, the first AI model can be used to implement one of the following functions: positioning, beam management, CSI prediction, mobility management, CSI compression. Of course, the first AI model can also be used to implement other functions, which is not limited in this application.

[0293] Specifically, the model performance supervision method 400 described in this embodiment can be used to implement at least two different functions, that is, the model performance supervision method 400 described in this embodiment can be applicable to a public AI model supervision framework based on Ground truthlabel, thereby avoiding the need to design performance supervision solutions for different functions.

[0294] In some embodiments, when the second AI model is used to implement the positioning function, the third feature may include but is not limited to at least one of the following: time domain channel impulse response, RSRP (such as layer 1 RSRP or layer 3 RSRP), frequency domain channel impulse response, time domain waveform of the received signal. Optionally, the time domain channel impulse response includes at least one of the following: time information, power information, phase information. Optionally, the frequency domain channel impulse response includes at least one of the following: frequency information (subcarrier sequence number and interval), power information, phase information.

[0295] In some embodiments, when the second AI model is used to implement the positioning function, the fourth feature may include but is not limited to at least one of the following: line-of-sight TOA, RSTD, AoA, AoD, RSRP, LOS indication, NLOS indication.

[0296] For example, when the second AI model is used to realize the positioning function, the input of the second AI model (i.e., the third feature) is the time domain channel impulse response, and the output of the second AI model is the soft information of the intermediate feature quantity (i.e., the fourth feature) (such as parameters of probability density distribution, confidence interval, confidence level, etc.), and the intermediate feature quantity includes at least one of the following: line of sight TOA, RSTD, AoA, AoD, RSRP, LOS indication, NLOS indication, etc. The position coordinates can be further determined based on the soft information of the intermediate feature quantity; the output of the second AI model may also be the soft information of the position coordinates.

[0297] In some embodiments, when the second AI model is used to implement the beam management function, the third feature may include beam information at T1 historical moments, such as sequence number, angle, L1-RSRP, etc., and the fourth feature includes beam information at T2 future moments.

[0298] For example, when the second AI model is used to implement the beam management function, the input of the second AI model (i.e., the third feature) is the beam information at T1 historical moments, such as serial number, angle, L1-RSRP, etc., and the output of the second AI model is the soft information (such as parameters of probability density distribution, confidence interval, confidence level, etc.) of the beam information (i.e., the fourth feature) at T2 future moments. For example, the vertical beam and the horizontal beam at each moment correspond to a set of soft information, such as the confidence interval and probability density distribution of L1-RSRP, etc.; the beam information at each moment corresponds to a set of soft information.

[0299] In some embodiments, when the second AI model is used to implement the CSI prediction function, the third feature may include the CSI at T1 historical moments, and the fourth feature may include the CSI at T2 future moments.

[0300] For example, when the second AI model is used to implement the CSI prediction function, the input of the second AI model (i.e., the third feature) is the CSI at T1 historical moments, and the output of the second AI model is the soft information (such as parameters of the probability density distribution, confidence interval, confidence level, etc.) of the CSI at T2 moments in the future (i.e., the fourth feature), such as the CSI at each moment corresponds to a set of soft information, such as each dimension of the CSI at each moment corresponds to a set of soft information, or the CSI at each moment corresponds to a set of soft information.

[0301] In some embodiments, when the second AI model is used to implement the mobility management function, the third feature may include the layer 1 RSRP (L1-RSRP) or layer 3 RSRP (L3-RSRP) at the historical T1 moment, and the fourth feature may include the RSRP at the future T2 moments or the decision of whether a cell switching occurs at the future T2 moments. It should be noted that L3-RSRP is obtained by filtering L1-RSRP.

[0302] For example, when the second AI model is used to implement the mobility management function, the input of the second AI model (i.e., the third feature) is the L1-RSRP or L3-RSRP at T1 historical moments, and the output of the second AI model is the soft information of the RSRP (i.e., the fourth feature) at T2 moments in the future (such as parameters of the probability density distribution, confidence interval, confidence level, etc.); or the output of the second AI model is the soft information of the decision on whether a cell switching will occur (i.e., the fourth feature) at T2 moments in the future (such as parameters of the probability density distribution, confidence interval, confidence level, etc.), such as 1 for switching and 0 for no switching, and the soft information is a value between 0 and 1.

[0303] In some embodiments, when the second AI model is used to implement the CSI compression function, the third feature may include uncompressed CSI, and the fourth feature may include compressed CSI.

[0304] For example, when the second AI model is used to implement the CSI compression function, the input of the second AI model (ie, the third feature) is the uncompressed CSI, and the output of the second AI model is the soft information of the compressed CSI (ie, the fourth feature) (such as parameters of the probability density distribution, confidence interval, confidence level, etc.).

[0305] In some embodiments, in the above S410, the first device obtains the fourth information, including one of the following:

[0306] The first device obtains the fourth information from the second device;

[0307] The first device obtains the fourth information through the output information of the second AI model;

[0308] The first device receives the output information of the second AI model from the second device, and obtains the fourth information according to the output information of the second AI model.

[0309] In some embodiments, when the first device obtains the fourth information through the output information of the second AI model, the second AI model can be deployed on the first device side.

[0310] In some embodiments, when the first device obtains the fourth information from another device, the second AI model can be deployed on the other device side. The other device can be the second device or a device other than the first device and the second device.

[0311] In some embodiments, when the first device receives output information of the second AI model from the second device and obtains fourth information based on the output information of the second AI model, the second AI model can be deployed on the second device side.

[0312] In some embodiments, when the first device determines the validity of the second AI model according to the fourth information, the first device sends third indication information to the second device, wherein the third indication information is used to indicate the validity of the second AI model. For example, the third indication information occupies 1 bit; wherein a value of 0 indicates that the second AI model is valid, and a value of 1 indicates that the second AI model is invalid; or, a value of 1 indicates that the second AI model is valid, and a value of 0 indicates that the second AI model is invalid.

[0313] Optionally, the third indication information includes an identifier of the second AI model or an identifier of a function associated with the second AI model.

[0314] In some embodiments, when the first device sends the fourth information to the second device, the first device receives the fourth indication information from the second device, wherein the fourth indication information is used to indicate the validity of the second AI model. Specifically, after receiving the fourth information, the second device can determine the validity of the second AI model according to the fourth information. For example, the fourth indication information occupies 1 bit; wherein a value of 0 indicates that the second AI model is valid, and a value of 1 indicates that the second AI model is invalid; or, a value of 1 indicates that the second AI model is valid, and a value of 0 indicates that the second AI model is invalid.

[0315] Optionally, the fourth indication information includes an identifier of the second AI model or an identifier of a function associated with the second AI model.

[0316] Exemplarily, when the second AI model is deployed on the first device side, the first device obtains the fourth information through the output information of the second AI model. Then, the first device determines the validity of the second AI model based on the fourth information. Finally, the first device sends third indication information to the second device, wherein the third indication information is used to indicate the validity of the second AI model.

[0317] Exemplarily, when the second AI model is deployed on the first device side, the first device obtains the fourth information through the output information of the second AI model, then, the first device sends the fourth information to the second device, thereafter, the second device determines the validity of the second AI model based on the fourth information, and finally, the second device sends fourth indication information to the first device, wherein the fourth indication information is used to indicate the validity of the second AI model.

[0318] Exemplarily, when the second AI model is deployed on the second device side, the second device obtains the fourth information through the output information of the second AI model, and the second device sends the fourth information to the first device. Then, the first device determines the validity of the second AI model based on the fourth information. Finally, the first device sends third indication information to the second device, wherein the third indication information is used to indicate the validity of the second AI model.

[0319] Exemplarily, when the second AI model is deployed on the third device side, the third device obtains the fourth information through the output information of the second AI model, and the third device sends the fourth information to the first device, then the first device sends the fourth information to the second device, and then the second device determines the validity of the second AI model based on the fourth information, and finally, the second device sends fourth indication information to the first device, wherein the fourth indication information is used to indicate the validity of the second AI model.

[0320] Exemplarily, when the second AI model is deployed on the second device side, the first device receives the output information of the second AI model from the second device, and the first device obtains fourth information based on the output information of the second AI model. Then, the first device determines the validity of the second AI model based on the fourth information. Finally, the first device sends third indication information to the second device, wherein the third indication information is used to indicate the validity of the second AI model.

[0321] Exemplarily, when the second AI model is deployed on the second device side, the first device receives the output information of the second AI model from the second device, and the first device obtains fourth information based on the output information of the second AI model. Then, the first device sends the fourth information to the second device. Thereafter, the second device determines the validity of the second AI model based on the fourth information. Finally, the second device sends fourth indication information to the first device, wherein the fourth indication information is used to indicate the validity of the second AI model.

[0322] In some embodiments, the first device may be a terminal, a network-side device, or a third-party server.

[0323] In some embodiments, the second device may be a terminal, a network-side device, or a third-party server.

[0324] In some embodiments, when the fourth information includes a parameter of a probability density distribution of the fourth feature, the first device determines the validity of the second AI model according to the fourth information, including:

[0325] When the variance or standard deviation of the probability density distribution of the fourth feature is greater than or equal to the first threshold, the first device determines that the second AI model fails; or,

[0326] When the mean of the variance or standard deviation of the probability density distribution of the fourth feature corresponding to the N samples of the third feature is greater than or equal to the second threshold, the first device determines that the second AI model is invalid; or,

[0327] When, among the N samples of the third feature, the proportion of the number of samples whose variance or standard deviation of the probability density distribution of the corresponding fourth feature is greater than or equal to the first threshold to the total number of samples N is greater than or equal to the third threshold, the first device determines that the second AI model has failed;

[0328] Wherein, N is a positive integer.

[0329] Optionally, the N samples may also be replaced by N reasoning processes of the second AI model.

[0330] In some embodiments, the first threshold is agreed upon by a protocol, or the first threshold is determined by the first device, or the first threshold is configured or indicated by the second device.

[0331] In some embodiments, the second threshold is agreed upon by a protocol, or the second threshold is determined by the first device, or the second threshold is configured or indicated by the second device.

[0332] In some embodiments, the third threshold is agreed upon by a protocol, or the third threshold is determined by the first device, or the third threshold is configured or indicated by the second device.

[0333] In some embodiments, when the fourth information includes a confidence interval of the fourth feature, the first device determines the validity of the second AI model according to the fourth information, including:

[0334] When the width of the confidence interval of the fourth feature is greater than or equal to a fourth threshold, the first device determines that the second AI model is invalid; or,

[0335] When the average width of the confidence interval of the fourth feature corresponding to the N samples of the third feature is greater than or equal to the fifth threshold, the first device determines that the second AI model is invalid; or,

[0336] When, among the N samples of the third feature, the number of samples whose width of the confidence interval of the corresponding fourth feature is greater than or equal to the fourth threshold accounts for a proportion of the total number of samples N that is greater than or equal to a sixth threshold, the first device determines that the second AI model is invalid;

[0337] Wherein, N is a positive integer.

[0338] In some embodiments, the fourth threshold is agreed upon by a protocol, or the fourth threshold is determined by the first device, or the fourth threshold is configured or indicated by the second device.

[0339] In some embodiments, the fifth threshold is agreed upon by a protocol, or the fifth threshold is determined by the first device, or the fifth threshold is configured or indicated by the second device.

[0340] In some embodiments, the sixth threshold is agreed upon by a protocol, or the sixth threshold is determined by the first device, or the sixth threshold is configured or indicated by the second device.

[0341] In some embodiments, when the fourth information includes the value of the fourth feature and its probability or confidence, the first device determines the validity of the second AI model according to the fourth information, including:

[0342] When the probability or confidence of the fourth feature is less than or equal to the seventh threshold, the first device determines that the second AI model fails; or,

[0343] When the average of the probability or confidence of the fourth feature corresponding to the N samples of the third feature is less than or equal to the eighth threshold, the first device determines that the second AI model fails; or,

[0344] When, among the N samples of the third feature, the number of samples whose probability or confidence of the corresponding fourth feature is less than or equal to the seventh threshold accounts for a proportion of the total number of samples N that is greater than or equal to the ninth threshold, the first device determines that the second AI model has failed;

[0345] Wherein, N is a positive integer.

[0346] In some embodiments, the seventh threshold is agreed upon by a protocol, or the seventh threshold is determined by the first device, or the seventh threshold is configured or indicated by the second device.

[0347] In some embodiments, the eighth threshold is agreed upon by a protocol, or the eighth threshold is determined by the first device, or the eighth threshold is configured or indicated by the second device.

[0348] In some embodiments, the ninth threshold is agreed upon by a protocol, or the ninth threshold is determined by the first device, or the ninth threshold is configured or indicated by the second device.

[0349] Therefore, in an embodiment of the present application, the validity of the second AI model can be determined based on the uncertainty or probability distribution of the output of the second AI model, thereby achieving performance supervision of the second AI model. Specifically, the uncertainty or probability distribution of the output of the second AI model can improve the robustness of the reasoning results of the second AI model. Since the reasoning results of the second AI model cover multiple possible results and their probability distributions, it is conducive to further processing and utilization of the reasoning results. For example, the reasoning results of the second AI model (including relevant information about uncertainty or probability distribution) are combined with other types of information to obtain a more accurate target result. In the performance supervision of the second AI model, the uncertainty or probability distribution of the output of the second AI model is referred to, and the validity of the second AI model can be judged. Since no external information is required, it is also easier to implement.

[0350] Combination of the above Figure 7 , describes in detail the first device side embodiment of the present application, and the following is combined with Figure 8 , the second device side embodiment of the present application is described in detail. It should be understood that the second device side embodiment corresponds to the first device side embodiment, and similar descriptions can refer to the first device side embodiment.

[0351] Figure 8 is a schematic flow chart of a model performance supervision method 500 according to an embodiment of the present application, such as Figure 8 As shown, the model performance supervision method 500 may include at least part of the following contents:

[0352] S510: The second device receives fourth information from the first device, wherein the fourth information is used to characterize uncertainty or probability distribution of an output of a second AI model, and an input of the second AI model is a third feature;

[0353] S520: The second device determines the validity of the second AI model according to the fourth information.

[0354] It should be understood that Figure 8The steps or operations of the model performance supervision method 500 are shown, but these steps or operations are only examples. The embodiment of the present application may also perform other operations or Figure 8 Variations of the various operations in .

[0355] In some embodiments, the fourth information includes at least one of the following: a parameter of a probability density distribution of the fourth feature, a confidence interval of the fourth feature, a value of the fourth feature, a value of the fourth feature and its probability, and a value of the fourth feature and its confidence.

[0356] In some embodiments, the parameters of the probability density distribution include at least one of the following: a mean or expectation of the probability density distribution, a variance or standard deviation of the probability density distribution, and an indication of the type of the probability density distribution.

[0357] In some embodiments, when the fourth information includes a parameter of a probability density distribution of the fourth feature, the second device determines the validity of the second AI model according to the fourth information, including:

[0358] When the variance or standard deviation of the probability density distribution of the fourth feature is greater than or equal to the first threshold, the second device determines that the second AI model fails; or,

[0359] When the mean of the variance or standard deviation of the probability density distribution of the fourth feature corresponding to the N samples of the third feature is greater than or equal to the second threshold, the second device determines that the second AI model is invalid; or,

[0360] When, among the N samples of the third feature, the proportion of the number of samples whose variance or standard deviation of the probability density distribution of the corresponding fourth feature is greater than or equal to the first threshold to the total number of samples N is greater than or equal to the third threshold, the second device determines that the second AI model has failed;

[0361] Wherein, N is a positive integer.

[0362] In some embodiments, when the fourth information includes a confidence interval of the fourth feature, the second device determines the validity of the second AI model according to the fourth information, including:

[0363] When the width of the confidence interval of the fourth feature is greater than or equal to the fourth threshold, the second device determines that the second AI model is invalid; or,

[0364] When the average width of the confidence interval of the fourth feature corresponding to the N samples of the third feature is greater than or equal to the fifth threshold, the second device determines that the second AI model is invalid; or,

[0365] When, among the N samples of the third feature, the number of samples whose width of the confidence interval of the corresponding fourth feature is greater than or equal to the fourth threshold accounts for a proportion of the total number of samples N that is greater than or equal to a sixth threshold, the second device determines that the second AI model is invalid;

[0366] Wherein, N is a positive integer.

[0367] In some embodiments, when the fourth information includes the value of the fourth feature and its probability or confidence, the second device determines the validity of the second AI model according to the fourth information, including:

[0368] When the probability or confidence of the fourth feature is less than or equal to the seventh threshold, the second device determines that the second AI model fails; or,

[0369] When the average of the probability or confidence of the fourth feature corresponding to the N samples of the third feature is less than or equal to the eighth threshold, the second device determines that the second AI model fails; or,

[0370] When, among the N samples of the third feature, the proportion of the number of samples whose probability or confidence of the corresponding fourth feature is less than or equal to the seventh threshold to the total number of samples N is greater than or equal to the ninth threshold, the second device determines that the second AI model is invalid;

[0371] Wherein, N is a positive integer.

[0372] In some embodiments, the second device sends fourth indication information to the first device, wherein the fourth indication information is used to indicate the validity of the second AI model.

[0373] Optionally, the fourth indication information includes an identifier of the second AI model or an identifier of a function associated with the second AI model.

[0374] Therefore, in an embodiment of the present application, the validity of the second AI model can be determined based on the uncertainty or probability distribution of the output of the second AI model, thereby achieving performance supervision of the second AI model. Specifically, the uncertainty or probability distribution of the output of the second AI model can improve the robustness of the reasoning results of the second AI model. Since the reasoning results of the second AI model cover multiple possible results and their probability distributions, it is conducive to further processing and utilization of the reasoning results. For example, the reasoning results of the second AI model (including relevant information about uncertainty or probability distribution) are combined with other types of information to obtain a more accurate target result. In the performance supervision of the second AI model, the uncertainty or probability distribution of the output of the second AI model is referred to, and the validity of the second AI model can be judged. Since no external information is required, it is also easier to implement.

[0375] The model performance supervision method provided in the embodiment of the present application can be executed by a model performance supervision device or a processing unit in the model performance supervision device for executing the model performance supervision method. In the embodiment of the present application, the model performance supervision device executing the model performance supervision method is taken as an example to illustrate the model performance supervision device provided in the embodiment of the present application.

[0376] Fig. 9 FIG. 6 shows a schematic block diagram of a model performance monitoring device 600 according to an embodiment of the present application. Fig. 9 As shown, the model performance monitoring device 600 includes:

[0377] An acquisition unit 610 is used to acquire first information, wherein the first information is used to characterize the uncertainty or probability distribution of the output of a first artificial intelligence AI model, and the input of the first AI model is a first feature;

[0378] Processing unit 620 is used to determine third information based on the first information and the second information, wherein the second information is a label corresponding to the first feature, and the third information is used to determine the validity of the first AI model.

[0379] In some embodiments, the acquiring unit 610 acquires the first information, including one of the following:

[0380] receiving the first information from a second device;

[0381] Obtaining the first information through output information of the first AI model;

[0382] Receive output information of the first AI model from a second device, and obtain the first information according to the output information of the first AI model.

[0383] In some embodiments, the model performance monitoring device 600 further includes: a transceiver unit 630;

[0384] The processing unit 620 is further configured to determine the validity of the first AI model according to the third information; or,

[0385] The transceiver unit 630 is used to send the third information to the second device.

[0386] In some embodiments, when the model performance monitoring device 600 determines the validity of the first AI model according to the third information, the transceiver unit 630 is further used to send first indication information to the second device, wherein the first indication information is used to indicate the validity of the first AI model; or,

[0387] When the model performance monitoring device 600 sends the third information to the second device, the transceiver unit 630 is also used to receive second indication information from the second device, wherein the second indication information is used to indicate the validity of the first AI model.

[0388] In some embodiments, the first information includes at least one of the following:

[0389] Parameters of the probability density distribution of the second feature, a confidence interval of the second feature, a value of the second feature, a value of the second feature and its probability, and a value of the second feature and its confidence level.

[0390] In some embodiments, the parameters of the probability density distribution include at least one of the following: a mean or expectation of the probability density distribution, a variance or standard deviation of the probability density distribution, and an indication of the type of the probability density distribution.

[0391] In some embodiments, when the first information includes parameters of the probability density distribution of the second feature, the third information is used to characterize the mean of the first probabilities of N samples of the first feature, or the third information is used to characterize the mean of the logarithms of the first probabilities of N samples of the first feature;

[0392] The first probability is the probability of the label corresponding to each sample in the N samples under the probability density distribution of the second feature, and N is a positive integer.

[0393] In some embodiments, when the third information is used to characterize the mean of the first probabilities of N samples of the first feature, the third information is determined based on the following formula:

[0394]

[0395] Wherein, L1 represents the third information, i represents the i-th sample among the N samples, represents the standard deviation or variance of the probability density distribution of the second feature corresponding to the i-th sample, represents the mean of the probability density distribution of the second feature corresponding to the i-th sample, y i represents the label corresponding to the i-th sample, and g(·) represents the probability density distribution function obeyed by the output of the first AI model.

[0396] In some embodiments, when the third information is used to characterize the mean of the logarithm of the first probability of N samples of the first feature, the third information is determined based on the following formula:

[0397]

[0398] Wherein, L2 represents the third information, i represents the i-th sample among the N samples, represents the mean of the probability density distribution of the second feature corresponding to the i-th sample, represents the standard deviation or variance of the probability density distribution of the second feature corresponding to the i-th sample, y i represents the label corresponding to the i-th sample, s is a positive integer, and g(·) represents the probability density distribution function obeyed by the output of the first AI model.

[0399] In some embodiments, the third information is used to determine the validity of the first AI model, including:

[0400] When the third information is greater than or equal to the first threshold, the first AI model is valid; or,

[0401] When the third information is less than the first threshold, the first AI model fails.

[0402] In some embodiments, when the first information includes the confidence interval of the second feature, the third information is used to characterize the ratio of the number of samples whose corresponding labels are within the confidence interval of the second feature among the N samples of the first feature to the total number of samples N, where N is a positive integer.

[0403] In some embodiments, the third information is determined based on the following formula:

[0404]

[0405] Wherein, L3 represents the third information, i represents the i-th sample among the N samples, and x i Represents the input information corresponding to the i-th sample, y i Indicates the label corresponding to the i-th sample. When y i The confidence interval f(x i ) within , otherwise

[0406] In some embodiments, the third information is used to determine the validity of the first AI model, including:

[0407] When the third information is greater than or equal to the second threshold, the first AI model is valid; or,

[0408] When the third information is less than the second threshold, the first AI model fails.

[0409] In some embodiments, when the first information includes the value of the second feature and its probability or confidence, the third information is used to characterize the mean of the weighted distances between the labels corresponding to N samples of the first feature and the value of the second feature, wherein the weighted weight is determined based on the probability or confidence of the second feature, and N is a positive integer.

[0410] In some embodiments, the third information is determined based on the following formula:

[0411]

[0412] Wherein, L4 represents the third information, i represents the i-th sample among the N samples, represents the mean of the probability density distribution of the second feature corresponding to the i-th sample, y i Represents the label corresponding to the i-th sample.

[0413] In some embodiments, the third information is used to determine the validity of the first AI model, including:

[0414] When the third information is less than or equal to a third threshold, the first AI model is valid; or,

[0415] When the third information is greater than a third threshold, the first AI model fails.

[0416] In some embodiments, the transceiver unit may be a communication interface or a transceiver, or an input / output interface of a communication chip or a system on chip.

[0417] It should be understood that the model performance monitoring device 600 according to the embodiment of the present application may correspond to the first device in the method embodiment of the present application, and the above and other operations or functions of each unit in the model performance monitoring device 600 are respectively to achieve Figure 4 For the sake of brevity, the corresponding process of the first device in the method 200 is not repeated here.

[0418] Therefore, in an embodiment of the present application, the third information can be determined based on the uncertainty or probability distribution of the output of the first AI model and the label corresponding to the input feature of the first AI model, and the validity of the first AI model can be determined based on the third information, so as to achieve performance supervision of the first AI model. Specifically, the uncertainty or probability distribution of the output of the first AI model can improve the robustness of the reasoning results of the first AI model. Since the reasoning results of the first AI model cover multiple possible results and their probability distributions, it is conducive to further processing and utilization of the reasoning results. For example, the reasoning results of the first AI model (including relevant information on uncertainty or probability distribution) are combined with other types of information to obtain a more accurate target result. In the performance supervision of the first AI model, referring to the uncertainty or probability distribution of the output of the first AI model and the corresponding label can improve the accuracy of the performance supervision of the first AI model.

[0419] Fig.10 FIG. 8 is a schematic block diagram of a model performance monitoring device 700 according to an embodiment of the present application. Fig.10 As shown, the model performance monitoring device 700 includes:

[0420] The transceiver unit 710 is configured to receive third information from the first device, wherein the third information is determined based on the first information and the second information, the first information is used to characterize the uncertainty or probability distribution of the output of the first artificial intelligence AI model, the input of the first AI model is the first feature, and the second information is the label corresponding to the first feature;

[0421] The processing unit 720 is used to determine the validity of the first AI model according to the third information.

[0422] In some embodiments, the processing unit 720 is specifically configured to:

[0423] When the third information is less than or equal to a third threshold, determining that the first AI model is valid; or,

[0424] When the third information is greater than a third threshold, it is determined that the first AI model is invalid.

[0425] In some embodiments, before the model performance monitoring apparatus 700 receives the third information from the first device, the transceiver unit 710 is further configured to send the first information to the first device.

[0426] In some embodiments, the transceiver unit 710 is further used to send second indication information to the first device, wherein the second indication information is used to indicate the validity of the first AI model.

[0427] In some embodiments, the first information includes at least one of the following:

[0428] Parameters of the probability density distribution of the second feature, a confidence interval of the second feature, a value of the second feature, a value of the second feature and its probability, and a value of the second feature and its confidence level.

[0429] In some embodiments, the parameters of the probability density distribution include at least one of the following: a mean or expectation of the probability density distribution, a variance or standard deviation of the probability density distribution, and an indication of the type of the probability density distribution.

[0430] In some embodiments, when the first information includes parameters of the probability density distribution of the second feature, the third information is used to characterize the mean of the first probabilities of N samples of the first feature, or the third information is used to characterize the mean of the logarithms of the first probabilities of N samples of the first feature;

[0431] The first probability is the probability of the label corresponding to each sample in the N samples under the probability density distribution of the second feature, and N is a positive integer.

[0432] In some embodiments, when the third information is used to characterize the mean of the first probabilities of N samples of the first feature, the third information is determined based on the following formula:

[0433]

[0434] Wherein, L1 represents the third information, i represents the i-th sample among the N samples, represents the standard deviation or variance of the probability density distribution of the second feature corresponding to the i-th sample, represents the mean of the probability density distribution of the second feature corresponding to the i-th sample, y i represents the label corresponding to the i-th sample, and g(·) represents the probability density distribution function obeyed by the output of the first AI model.

[0435] In some embodiments, when the third information is used to characterize the mean of the logarithm of the first probability of N samples of the first feature, the third information is determined based on the following formula:

[0436]

[0437] Wherein, L2 represents the third information, i represents the i-th sample among the N samples, represents the mean of the probability density distribution of the second feature corresponding to the i-th sample, represents the standard deviation or variance of the probability density distribution of the second feature corresponding to the i-th sample, y irepresents the label corresponding to the i-th sample, s is a positive integer, and g(·) represents the probability density distribution function obeyed by the output of the first AI model.

[0438] In some embodiments, the processing unit 720 is specifically configured to:

[0439] When the third information is greater than or equal to the first threshold, determining that the first AI model is valid; or,

[0440] When the third information is less than the first threshold, it is determined that the first AI model is invalid.

[0441] In some embodiments, when the first information includes the confidence interval of the second feature, the third information is used to characterize the ratio of the number of samples whose corresponding labels are within the confidence interval of the second feature among the N samples of the first feature to the total number of samples N, where N is a positive integer.

[0442] In some embodiments, the third information is determined based on the following formula:

[0443]

[0444] Wherein, L3 represents the third information, i represents the i-th sample among the N samples, and x i Represents the input information corresponding to the i-th sample, y i Indicates the label corresponding to the i-th sample. When y i The confidence interval f(x) of the second feature i ) within , otherwise

[0445] In some embodiments, the processing unit 720 is specifically configured to:

[0446] When the third information is greater than or equal to the second threshold, determining that the first AI model is valid; or,

[0447] When the third information is less than a second threshold, it is determined that the first AI model is invalid.

[0448] In some embodiments, when the first information includes the value of the second feature and its probability or confidence, the third information is used to characterize the mean of the weighted distances between the labels corresponding to N samples of the first feature and the value of the second feature, wherein the weighted weight is determined based on the probability or confidence of the second feature, and N is a positive integer.

[0449] In some embodiments, the third information is determined based on the following formula:

[0450]

[0451] Wherein, L4 represents the third information, i represents the i-th sample among the N samples, represents the mean of the probability density distribution of the second feature corresponding to the i-th sample, y i Represents the label corresponding to the i-th sample.

[0452] In some embodiments, the transceiver unit may be a communication interface or a transceiver, or an input / output interface of a communication chip or a system on chip.

[0453] It should be understood that the model performance monitoring device 700 according to the embodiment of the present application may correspond to the second device in the method embodiment of the present application, and the above and other operations or functions of each unit in the model performance monitoring device 700 are respectively to achieve Figure 6 For the sake of brevity, the corresponding process of the second device in the method 300 is not repeated here.

[0454] Therefore, in an embodiment of the present application, the third information can be determined based on the uncertainty or probability distribution of the output of the first AI model and the label corresponding to the input feature of the first AI model, and the validity of the first AI model can be determined based on the third information, so as to achieve performance supervision of the first AI model. Specifically, the uncertainty or probability distribution of the output of the first AI model can improve the robustness of the reasoning results of the first AI model. Since the reasoning results of the first AI model cover multiple possible results and their probability distributions, it is conducive to further processing and utilization of the reasoning results. For example, the reasoning results of the first AI model (including relevant information on uncertainty or probability distribution) are combined with other types of information to obtain a more accurate target result. In the performance supervision of the first AI model, referring to the uncertainty or probability distribution of the output of the first AI model and the corresponding label can improve the accuracy of the performance supervision of the first AI model.

[0455] Fig.11 FIG. 8 is a schematic block diagram of a model performance monitoring device 800 according to an embodiment of the present application. Fig.11 As shown, the model performance monitoring device 800 includes:

[0456] An acquisition unit 810 is used to acquire fourth information, wherein the fourth information is used to characterize the uncertainty or probability distribution of the output of the second artificial intelligence AI model, and the input of the second AI model is the third feature;

[0457] The processing unit 820 is used to determine the validity of the second AI model according to the fourth information; or the transceiver unit 830 is used to send the fourth information to the second device.

[0458] In some embodiments, the acquiring unit 810 acquires the fourth information, including one of the following:

[0459] Acquire the fourth information from other devices;

[0460] Obtaining the fourth information through the output information of the second AI model;

[0461] Receive output information of the second AI model from the second device, and obtain the fourth information according to the output information of the second AI model.

[0462] In some embodiments, when the model performance monitoring device 800 determines the validity of the second AI model according to the fourth information, the transceiver unit 830 is further used to send third indication information to the second device, wherein the third indication information is used to indicate the validity of the second AI model; or,

[0463] When the model performance monitoring device 800 sends the fourth information to the second device, the transceiver unit 830 is also used to receive fourth indication information from the second device, wherein the fourth indication information is used to indicate the validity of the second AI model.

[0464] In some embodiments, the fourth information includes at least one of the following: a parameter of a probability density distribution of the fourth feature, a confidence interval of the fourth feature, a value of the fourth feature, a value of the fourth feature and its probability, and a value of the fourth feature and its confidence.

[0465] In some embodiments, the parameters of the probability density distribution include at least one of the following: a mean or expectation of the probability density distribution, a variance or standard deviation of the probability density distribution, and an indication of the type of the probability density distribution.

[0466] In some embodiments, when the fourth information includes parameters of the probability density distribution of the fourth feature, the processing unit 820 is specifically configured to:

[0467] When the variance or standard deviation of the probability density distribution of the fourth feature is greater than or equal to the first threshold, determining that the second AI model is invalid; or,

[0468] When the mean of the variance or standard deviation of the probability density distribution of the fourth feature corresponding to the N samples of the third feature is greater than or equal to the second threshold, determining that the second AI model is invalid; or,

[0469] When, among the N samples of the third feature, the number of samples whose variance or standard deviation of the probability density distribution of the corresponding fourth feature is greater than or equal to the first threshold accounts for a proportion of the total number of samples N that is greater than or equal to the third threshold, it is determined that the second AI model is invalid;

[0470] Wherein, N is a positive integer.

[0471] In some embodiments, when the fourth information includes a confidence interval of the fourth feature, the processing unit 820 is specifically configured to:

[0472] When the width of the confidence interval of the fourth feature is greater than or equal to a fourth threshold, determining that the second AI model is invalid; or,

[0473] When the average width of the confidence interval of the fourth feature corresponding to the N samples of the third feature is greater than or equal to a fifth threshold, determining that the second AI model is invalid; or,

[0474] When, among the N samples of the third feature, the number of samples whose width of the confidence interval of the corresponding fourth feature is greater than or equal to the fourth threshold accounts for a proportion of the total number of samples N that is greater than or equal to the sixth threshold, it is determined that the second AI model is invalid;

[0475] Wherein, N is a positive integer.

[0476] In some embodiments, when the fourth information includes the value of the fourth feature and its probability or confidence, the processing unit 820 is specifically configured to:

[0477] When the probability or confidence of the fourth feature is less than or equal to a seventh threshold, determining that the second AI model fails; or,

[0478] When the average of the probability or confidence of the fourth feature corresponding to the N samples of the third feature is less than or equal to an eighth threshold, determining that the second AI model is invalid; or,

[0479] When, among the N samples of the third feature, the number of samples whose probability or confidence of the corresponding fourth feature is less than or equal to the seventh threshold accounts for a proportion of the total number of samples N that is greater than or equal to the ninth threshold, it is determined that the second AI model has failed;

[0480] Wherein, N is a positive integer.

[0481] In some embodiments, the transceiver unit may be a communication interface or a transceiver, or an input / output interface of a communication chip or a system on chip.

[0482] It should be understood that the model performance monitoring device 800 according to the embodiment of the present application may correspond to the first device in the method embodiment of the present application, and the above and other operations or functions of each unit in the model performance monitoring device 800 are respectively to achieve Figure 7 For the sake of brevity, the corresponding process of the first device in the method 400 is not repeated here.

[0483] Therefore, in an embodiment of the present application, the validity of the second AI model can be determined based on the uncertainty or probability distribution of the output of the second AI model, thereby achieving performance supervision of the second AI model. Specifically, the uncertainty or probability distribution of the output of the second AI model can improve the robustness of the reasoning results of the second AI model. Since the reasoning results of the second AI model cover multiple possible results and their probability distributions, it is conducive to further processing and utilization of the reasoning results. For example, the reasoning results of the second AI model (including relevant information about uncertainty or probability distribution) are combined with other types of information to obtain a more accurate target result. In the performance supervision of the second AI model, the uncertainty or probability distribution of the output of the second AI model is referred to, and the validity of the second AI model can be judged. Since no external information is required, it is also easier to implement.

[0484] Fig.12 FIG. 8 is a schematic block diagram of a model performance monitoring device 900 according to an embodiment of the present application. Fig.12 As shown, the model performance monitoring device 900 includes:

[0485] The transceiver unit 910 is configured to receive fourth information from the first device, wherein the fourth information is used to characterize the uncertainty or probability distribution of the output of the second artificial intelligence AI model, and the input of the second AI model is the third feature;

[0486] The processing unit 920 is used to determine the validity of the second AI model according to the fourth information.

[0487] In some embodiments, the transceiver unit 910 is further used to send fourth indication information to the first device, wherein the fourth indication information is used to indicate the validity of the second AI model.

[0488] In some embodiments, the fourth information includes at least one of the following: a parameter of a probability density distribution of the fourth feature, a confidence interval of the fourth feature, a value of the fourth feature, a value of the fourth feature and its probability, and a value of the fourth feature and its confidence.

[0489] In some embodiments, the parameters of the probability density distribution include at least one of the following: a mean or expectation of the probability density distribution, a variance or standard deviation of the probability density distribution, and an indication of the type of the probability density distribution.

[0490] In some embodiments, when the fourth information includes parameters of the probability density distribution of the fourth feature, the processing unit 920 is specifically configured to:

[0491] When the variance or standard deviation of the probability density distribution of the fourth feature is greater than or equal to the first threshold, determining that the second AI model is invalid; or,

[0492] When the mean of the variance or standard deviation of the probability density distribution of the fourth feature corresponding to the N samples of the third feature is greater than or equal to the second threshold, determining that the second AI model is invalid; or,

[0493] When, among the N samples of the third feature, the number of samples whose variance or standard deviation of the probability density distribution of the corresponding fourth feature is greater than or equal to the first threshold accounts for a proportion of the total number of samples N that is greater than or equal to the third threshold, it is determined that the second AI model is invalid;

[0494] Wherein, N is a positive integer.

[0495] In some embodiments, when the fourth information includes a confidence interval of the fourth feature, the processing unit 920 is specifically configured to:

[0496] When the width of the confidence interval of the fourth feature is greater than or equal to a fourth threshold, determining that the second AI model is invalid; or,

[0497] When the average width of the confidence interval of the fourth feature corresponding to the N samples of the third feature is greater than or equal to a fifth threshold, determining that the second AI model is invalid; or,

[0498] When, among the N samples of the third feature, the number of samples whose width of the confidence interval of the corresponding fourth feature is greater than or equal to the fourth threshold accounts for a proportion of the total number of samples N that is greater than or equal to the sixth threshold, it is determined that the second AI model is invalid;

[0499] Wherein, N is a positive integer.

[0500] In some embodiments, when the fourth information includes the value of the fourth feature and its probability or confidence, the processing unit 920 is specifically configured to:

[0501] When the probability or confidence of the fourth feature is less than or equal to a seventh threshold, determining that the second AI model fails; or,

[0502] When the average of the probability or confidence of the fourth feature corresponding to the N samples of the third feature is less than or equal to an eighth threshold, determining that the second AI model is invalid; or,

[0503] When, among the N samples of the third feature, the number of samples whose probability or confidence of the corresponding fourth feature is less than or equal to the seventh threshold accounts for a proportion of the total number of samples N that is greater than or equal to the ninth threshold, it is determined that the second AI model has failed;

[0504] Wherein, N is a positive integer.

[0505] In some embodiments, the transceiver unit may be a communication interface or a transceiver, or an input / output interface of a communication chip or a system on chip.

[0506] It should be understood that the model performance monitoring device 900 according to the embodiment of the present application may correspond to the second device in the method embodiment of the present application, and the above and other operations or functions of each unit in the model performance monitoring device 900 are respectively to achieve Figure 8 For the sake of brevity, the corresponding process of the second device in the method 500 is not repeated here.

[0507] Therefore, in an embodiment of the present application, the validity of the second AI model can be determined based on the uncertainty or probability distribution of the output of the second AI model, thereby achieving performance supervision of the second AI model. Specifically, the uncertainty or probability distribution of the output of the second AI model can improve the robustness of the reasoning results of the second AI model. Since the reasoning results of the second AI model cover multiple possible results and their probability distributions, it is conducive to further processing and utilization of the reasoning results. For example, the reasoning results of the second AI model (including relevant information about uncertainty or probability distribution) are combined with other types of information to obtain a more accurate target result. In the performance supervision of the second AI model, the uncertainty or probability distribution of the output of the second AI model is referred to, and the validity of the second AI model can be judged. Since no external information is required, it is also easier to implement.

[0508] The model performance monitoring device in the embodiment of the present application can be an electronic device, such as an electronic device with an operating system, or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or a network side device, or can be other devices other than a terminal or a network side device. Exemplarily, the terminal can include but is not limited to the types of terminals 11 listed above, the network side device can include but is not limited to the types of network side devices 12 listed above, and other devices can be servers, network attached storage (NAS), etc., which are not specifically limited in the embodiment of the present application.

[0509] The model performance monitoring device provided in the embodiment of the present application can achieve Figures 4 to 8 The various processes implemented by the method embodiment and achieving the same technical effect are not described here to avoid repetition.

[0510] like Fig.13 As shown, the embodiment of the present application also provides a communication device 1000, including a processor 1001 and a memory 1002, and the memory 1002 stores a program or instruction that can be run on the processor 1001. For example, when the communication device 1000 is a first device, the program or instruction is executed by the processor 1001 to implement the various steps of the above-mentioned model performance supervision method 200 or model performance supervision method 400 embodiments, and can achieve the same technical effect. When the communication device 1000 is a second device, the program or instruction is executed by the processor 1001 to implement the various steps of the above-mentioned model performance supervision method 300 or model performance supervision method 500 embodiments, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0511] The embodiment of the present application also provides a terminal, including a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the following Figures 4 to 8 The steps performed by the first device or the second device in the method embodiment shown. This terminal embodiment corresponds to the first device or the second device side method embodiment described above, and each implementation process and implementation method of the above method embodiment can be applied to this terminal embodiment and can achieve the same technical effect. Specifically, Fig.14 A schematic diagram of the hardware structure of a terminal for implementing an embodiment of the present application.

[0512] The terminal 1100 includes but is not limited to: a radio frequency unit 1101, a network module 1102, an audio output unit 1103, an input unit 1104, a sensor 1105, a display unit 1106, a user input unit 1107, an interface unit 1108, a memory 1109 and at least some of the components of a processor 1110.

[0513] Those skilled in the art will appreciate that the terminal 1100 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1110 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.14 The terminal structure shown in the figure does not constitute a limitation on the terminal. The terminal may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0514] It should be understood that in the embodiment of the present application, the input unit 1104 may include a graphics processing unit (GPU) 11041 and a microphone 11042, and the graphics processor 11041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 1106 may include a display panel 11061, and the display panel 11061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1107 includes a touch panel 11071 and at least one of other input devices 11072. The touch panel 11071 is also called a touch screen. The touch panel 11071 may include two parts: a touch detection device and a touch controller. Other input devices 11072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0515] In the embodiment of the present application, after receiving downlink data from the network side device, the RF unit 1101 can transmit the data to the processor 1110 for processing; in addition, the RF unit 1101 can send uplink data to the network side device. Generally, the RF unit 1101 includes but is not limited to an antenna, an amplifier, a transceiver, a coupler, a low noise amplifier, a duplexer, etc.

[0516] The memory 1109 can be used to store software programs or instructions and various data. The memory 1109 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instruction required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1109 may include a volatile memory or a non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1109 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0517] The processor 1110 may include at least one processing unit; optionally, the processor 1110 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1110.

[0518] Exemplarily, the RF unit 1101 is used to obtain first information, wherein the first information is used to characterize the uncertainty or probability distribution of the output of a first AI model, and the input of the first AI model is a first feature; the processor 1110 is used to determine third information based on the first information and the second information, wherein the second information is a label corresponding to the first feature, and the third information is used to determine the validity of the first AI model.

[0519] Exemplarily, the radio frequency unit 1101 is used to receive third information from a first device, wherein the third information is determined based on the first information and the second information, the first information is used to characterize the uncertainty or probability distribution of the output of the first artificial intelligence AI model, the input of the first AI model is a first feature, and the second information is a label corresponding to the first feature; the processor 1110 is used to determine the validity of the first AI model based on the third information.

[0520] Exemplarily, the RF unit 1101 is used to obtain fourth information, wherein the fourth information is used to characterize the uncertainty or probability distribution of the output of the second artificial intelligence AI model, and the input of the second AI model is the third feature; the processor 1110 is used to determine the validity of the first AI model based on the fourth information; or, the RF unit 1101 is used to send the fourth information to the second device.

[0521] Exemplarily, the radio frequency unit 1101 is used to receive fourth information from the first device, wherein the fourth information is used to characterize the uncertainty or probability distribution of the output of the second artificial intelligence AI model, and the input of the second AI model is the third feature; the processor 1110 is used to determine the validity of the first AI model based on the fourth information.

[0522] It can be understood that the implementation process of each implementation method mentioned in this embodiment can refer to the relevant description of the method embodiment and achieve the same or corresponding technical effect. To avoid repetition, it will not be repeated here.

[0523] The embodiment of the present application also provides a network side device, including a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the following Figures 4 to 8 The steps of the method embodiment shown. The network side device embodiment corresponds to the first device or second device method embodiment described above, and each implementation process and implementation mode of the method embodiment described above can be applied to the network side device embodiment and can achieve the same technical effect.

[0524] Specifically, the embodiment of the present application also provides a network side device. Fig.15 As shown, the network side device 1200 includes: an antenna 121, a radio frequency device 122, a baseband device 123, a processor 124 and a memory 125. The antenna 121 is connected to the radio frequency device 122. In the uplink direction, the radio frequency device 122 receives information through the antenna 121 and sends the received information to the baseband device 123 for processing. In the downlink direction, the baseband device 123 processes the information to be sent and sends it to the radio frequency device 122. The radio frequency device 122 processes the received information and sends it out through the antenna 121.

[0525] The method executed by the network-side device in the above embodiment may be implemented in the baseband device 123, which includes a baseband processor.

[0526] The baseband device 123 may include, for example, at least one baseband board, on which at least two chips are arranged. Fig.15 As shown, one of the chips is, for example, a baseband processor, which is connected to the memory 125 through a bus interface to call the program in the memory 125 to execute the network device operations shown in the above method embodiment.

[0527] The network side device may further include a network interface 126, which is, for example, a Common Public Radio Interface (CPRI).

[0528] Specifically, the network side device 1200 of the embodiment of the present application further includes: instructions or programs stored in the memory 125 and executable on the processor 124, and the processor 124 calls the instructions or programs in the memory 125 to execute. Figures 9 to 12 The methods performed by the units shown in any of the items can achieve the same technical effects, so they will not be described here to avoid repetition.

[0529] Specifically, the embodiment of the present application also provides a network side device. Fig.16 As shown, the network side device 1300 includes: a processor 1301, a network interface 1302 and a memory 1303. The network interface 1302 is, for example, a common public radio interface (CPRI).

[0530] Specifically, the network side device 1300 of the embodiment of the present application further includes: an instruction or program stored in the memory 1303 and executable on the processor 1301, and the processor 1301 calls the instruction or program in the memory 1303 to execute Figure 13 to Figure 12 The methods performed by the units shown in any of the items can achieve the same technical effects, so they will not be described here to avoid repetition.

[0531] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned model performance supervision method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0532] The processor is the processor in the terminal described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk. In some examples, the readable storage medium may be a non-transient readable storage medium.

[0533] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned model performance supervision method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0534] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0535] An embodiment of the present application further provides a computer program / program product, which is stored in a storage medium. The computer program / program product is executed by at least one processor to implement the various processes of the above-mentioned model performance supervision method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0536] An embodiment of the present application also provides a communication system, including: a first device and a second device, wherein the first device can be used to execute the steps performed by the first device in the model performance supervision method as described above, and the second device can be used to execute the steps performed by the second device in the model performance supervision method as described above.

[0537] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises one..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiment of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0538] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of a computer software product plus a necessary general hardware platform, and of course, can also be implemented by hardware. The computer software product is stored in a storage medium (such as ROM, RAM, disk, CD, etc.), including several instructions to enable a terminal or a network-side device to execute the methods described in each embodiment of the present application.

[0539] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms of implementation methods without departing from the purpose of the present application and the scope of protection of the claims, and these implementation methods are all within the protection of the present application.

Claims

1. A model performance supervision method, characterized in that: include: The first device acquires first information, wherein the first information is used to characterize the uncertainty or probability distribution of the output of a first artificial intelligence (AI) model, and the input of the first AI model is a first feature; The first device determines third information based on the first information and the second information, wherein the second information is a label corresponding to the first feature, and the third information is used to determine the validity of the first AI model.

2. The method according to claim 1, characterized in that The first device obtains first information, including one of the following: The first device receives the first information from the second device; The first device obtains the first information through output information of the first AI model; The first device receives output information of the first AI model from the second device, and obtains the first information according to the output information of the first AI model.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: The first device determines the validity of the first AI model according to the third information; or, The first device sends the third information to the second device.

4. The method according to claim 3, characterized in that The method further comprises: When the first device determines the validity of the first AI model according to the third information, the first device sends first indication information to the second device, wherein the first indication information is used to indicate the validity of the first AI model; or When the first device sends the third information to the second device, the first device receives second indication information from the second device, wherein the second indication information is used to indicate the validity of the first AI model.

5. The method according to any one of claims 1 to 4, characterized in that The first information includes at least one of the following: Parameters of the probability density distribution of the second feature, a confidence interval of the second feature, a value of the second feature, a value of the second feature and its probability, and a value of the second feature and its confidence level.

6. The method according to claim 5, characterized in that The parameters of the probability density distribution include at least one of the following: a mean or expectation of the probability density distribution, a variance or standard deviation of the probability density distribution, and an indication of the type of the probability density distribution.

7. The method according to claim 5 or 6, characterized in that: In the case where the first information includes parameters of the probability density distribution of the second feature, the third information is used to characterize a mean of the first probabilities of N samples of the first feature, or the third information is used to characterize a mean of the logarithms of the first probabilities of the N samples of the first feature; The first probability is the probability of the label corresponding to each sample in the N samples under the probability density distribution of the second feature, and N is a positive integer.

8. The method according to claim 7, characterized in that In the case where the third information is used to characterize the mean of the first probabilities of N samples of the first feature, the third information is determined based on the following formula: Wherein, L1 represents the third information, i represents the i-th sample among the N samples, represents the standard deviation or variance of the probability density distribution of the second feature corresponding to the i-th sample, represents the mean of the probability density distribution of the second feature corresponding to the i-th sample, y i represents the label corresponding to the i-th sample, and g(·) represents the probability density distribution function obeyed by the output of the first AI model.

9. The method according to claim 7, characterized in that: In the case where the third information is used to characterize the mean of the logarithm of the first probability of N samples of the first feature, the third information is determined based on the following formula: Wherein, L2 represents the third information, i represents the i-th sample among the N samples, represents the mean of the probability density distribution of the second feature corresponding to the i-th sample, represents the standard deviation or variance of the probability density distribution of the second feature corresponding to the i-th sample, y i represents the label corresponding to the i-th sample, s is a positive integer, and g(·) represents the probability density distribution function obeyed by the output of the first AI model.

10. The method according to any one of claims 7 to 9, characterized in that The third information is used to determine the validity of the first AI model, including: When the third information is greater than or equal to the first threshold, the first AI model is valid; or, When the third information is less than the first threshold, the first AI model fails.

11. The method according to claim 5 or 6, characterized in that: In the case where the first information includes the confidence interval of the second feature, the third information is used to characterize the ratio of the number of samples whose corresponding labels are within the confidence interval of the second feature among the N samples of the first feature to the total number of samples N, where N is a positive integer.

12. The method according to claim 11, characterized in that The third information is determined based on the following formula: Wherein, L3 represents the third information, i represents the i-th sample among the N samples, and x i Represents the input information corresponding to the i-th sample, y i Indicates the label corresponding to the i-th sample. When y i The confidence interval f(x i ) within , otherwise 13. The method according to claim 11 or 12, characterized in that: The third information is used to determine the validity of the first AI model, including: When the third information is greater than or equal to the second threshold, the first AI model is valid; or, When the third information is less than the second threshold, the first AI model fails.

14. The method according to claim 5 or 6, characterized in that: When the first information includes the value of the second feature and its probability or confidence, the third information is used to characterize the mean of the weighted distances between the labels corresponding to N samples of the first feature and the value of the second feature, wherein the weighted weight is determined based on the probability or confidence of the second feature, and N is a positive integer.

15. The method according to claim 14, characterized in that The third information is determined based on the following formula: Wherein, L4 represents the third information, i represents the i-th sample among the N samples, represents the mean of the probability density distribution of the second feature corresponding to the i-th sample, y i Represents the label corresponding to the i-th sample.

16. The method according to claim 14 or 15, characterized in that The third information is used to determine the validity of the first AI model, including: When the third information is less than or equal to a third threshold, the first AI model is valid; or, When the third information is greater than a third threshold, the first AI model fails.

17. A model performance supervision method, characterized in that: include: The second device receives third information from the first device, wherein the third information is determined based on the first information and the second information, the first information is used to characterize the uncertainty or probability distribution of the output of the first artificial intelligence AI model, the input of the first AI model is a first feature, and the second information is a label corresponding to the first feature; The second device determines the validity of the first AI model based on the third information.

18. The method according to claim 17, characterized in that Before the second device receives the third information from the first device, the method further includes: The second device sends the first information to the first device.

19. The method according to claim 17 or 18, characterized in that The method further comprises: The second device sends second indication information to the first device, wherein the second indication information is used to indicate the validity of the first AI model.

20. The method according to any one of claims 17 to 19, characterized in that The first information includes at least one of the following: Parameters of the probability density distribution of the second feature, a confidence interval of the second feature, a value of the second feature, a value of the second feature and its probability, and a value of the second feature and its confidence level.

21. The method according to claim 20, characterized in that The parameters of the probability density distribution include at least one of the following: a mean or expectation of the probability density distribution, a variance or standard deviation of the probability density distribution, and an indication of the type of the probability density distribution.

22. The method according to claim 20 or 21, characterized in that In the case where the first information includes parameters of the probability density distribution of the second feature, the third information is used to characterize a mean of the first probabilities of N samples of the first feature, or the third information is used to characterize a mean of the logarithms of the first probabilities of the N samples of the first feature; The first probability is the probability of the label corresponding to each sample in the N samples under the probability density distribution of the second feature, and N is a positive integer.

23. The method according to claim 22, characterized in that In the case where the third information is used to characterize the mean of the first probabilities of N samples of the first feature, the third information is determined based on the following formula: Wherein, L1 represents the third information, i represents the i-th sample among the N samples, represents the standard deviation or variance of the probability density distribution of the second feature corresponding to the i-th sample, represents the mean of the probability density distribution of the second feature corresponding to the i-th sample, y i represents the label corresponding to the i-th sample, and g(·) represents the probability density distribution function obeyed by the output of the first AI model.

24. The method according to claim 22, characterized in that In the case where the third information is used to characterize the mean of the logarithm of the first probability of N samples of the first feature, the third information is determined based on the following formula: Wherein, L2 represents the third information, i represents the i-th sample among the N samples, represents the mean of the probability density distribution of the second feature corresponding to the i-th sample, represents the standard deviation or variance of the probability density distribution of the second feature corresponding to the i-th sample, y i represents the label corresponding to the i-th sample, s is a positive integer, and g(v) represents the probability density distribution function obeyed by the output of the first AI model.

25. The method according to any one of claims 22 to 24, characterized in that The second device determines the validity of the first AI model according to the third information, including: When the third information is greater than or equal to the first threshold, the second device determines that the first AI model is valid; or, When the third information is less than the first threshold, the second device determines that the first AI model is invalid.

26. The method according to claim 20 or 21, characterized in that In the case where the first information includes the confidence interval of the second feature, the third information is used to characterize the ratio of the number of samples whose corresponding labels are within the confidence interval of the second feature among the N samples of the first feature to the total number of samples N, where N is a positive integer.

27. The method according to claim 26, characterized in that The third information is determined based on the following formula: Wherein, L3 represents the third information, i represents the i-th sample among the N samples, and x i Represents the input information corresponding to the i-th sample, y i Indicates the label corresponding to the i-th sample. When y i The confidence interval f(x i ) within , otherwise 28. The method according to claim 26 or 27, characterized in that The second device determines the validity of the first AI model according to the third information, including: When the third information is greater than or equal to a second threshold, the second device determines that the first AI model is valid; or, When the third information is less than a second threshold, the second device determines that the first AI model is invalid.

29. The method according to claim 20 or 21, characterized in that When the first information includes the value of the second feature and its probability or confidence, the third information is used to characterize the mean of the weighted distances between the labels corresponding to N samples of the first feature and the value of the second feature, wherein the weighted weight is determined based on the probability or confidence of the second feature, and N is a positive integer.

30. The method according to claim 29, characterized in that The third information is determined based on the following formula: Wherein, L4 represents the third information, i represents the i-th sample among the N samples, represents the mean of the probability density distribution of the second feature corresponding to the i-th sample, y i Represents the label corresponding to the i-th sample.

31. The method according to claim 29 or 30, characterized in that The second device determines the validity of the first AI model according to the third information, including: When the third information is less than or equal to a third threshold, the second device determines that the first AI model is valid; or, When the third information is greater than a third threshold, the second device determines that the first AI model is invalid.

32. A model performance supervision method, characterized in that: include: The first device obtains fourth information, wherein the fourth information is used to characterize the uncertainty or probability distribution of the output of the second artificial intelligence AI model, and the input of the second AI model is the third feature; The first device determines the validity of the second AI model based on the fourth information; or, the first device sends the fourth information to the second device.

33. The method according to claim 32, characterized in that The first device obtains fourth information, including one of the following: The first device obtains the fourth information from other devices; The first device obtains the fourth information through the output information of the second AI model; The first device receives the output information of the second AI model from the second device, and obtains the fourth information according to the output information of the second AI model.

34. The method according to claim 32 or 33, characterized in that The method further comprises: When the first device determines the validity of the second AI model according to the fourth information, the first device sends third indication information to the second device, wherein the third indication information is used to indicate the validity of the second AI model; or When the first device sends the fourth information to the second device, the first device receives fourth indication information from the second device, wherein the fourth indication information is used to indicate the validity of the second AI model.

35. The method according to any one of claims 32 to 34, characterized in that The fourth information includes at least one of the following: a parameter of a probability density distribution of the fourth feature, a confidence interval of the fourth feature, a value of the fourth feature, a value of the fourth feature and its probability, and a value of the fourth feature and its confidence.

36. The method according to claim 35, characterized in that The parameters of the probability density distribution include at least one of the following: a mean or expectation of the probability density distribution, a variance or standard deviation of the probability density distribution, and an indication of the type of the probability density distribution.

37. The method according to claim 35 or 36, characterized in that In a case where the fourth information includes a parameter of a probability density distribution of the fourth feature, the first device determines the validity of the second AI model according to the fourth information, including: When the variance or standard deviation of the probability density distribution of the fourth feature is greater than or equal to the first threshold, the first device determines that the second AI model fails; or, When the mean of the variance or standard deviation of the probability density distribution of the fourth feature corresponding to the N samples of the third feature is greater than or equal to the second threshold, the first device determines that the second AI model fails; or, When, among the N samples of the third feature, the proportion of the number of samples whose variance or standard deviation of the probability density distribution of the corresponding fourth feature is greater than or equal to the first threshold to the total number of samples N is greater than or equal to the third threshold, the first device determines that the second AI model has failed; Wherein, N is a positive integer.

38. The method according to claim 35 or 36, characterized in that In a case where the fourth information includes a confidence interval of the fourth feature, the first device determines the validity of the second AI model according to the fourth information, including: When the width of the confidence interval of the fourth feature is greater than or equal to a fourth threshold, the first device determines that the second AI model is invalid; or, When the average width of the confidence interval of the fourth feature corresponding to the N samples of the third feature is greater than or equal to a fifth threshold, the first device determines that the second AI model is invalid; or, When, among the N samples of the third feature, the number of samples whose width of the confidence interval of the corresponding fourth feature is greater than or equal to the fourth threshold accounts for a proportion of the total number of samples N that is greater than or equal to a sixth threshold, the first device determines that the second AI model has failed; Wherein, N is a positive integer.

39. The method according to claim 35 or 36, characterized in that In a case where the fourth information includes a value of the fourth feature and a probability or confidence level thereof, the first device determines the validity of the second AI model according to the fourth information, including: When the probability or confidence of the fourth feature is less than or equal to a seventh threshold, the first device determines that the second AI model fails; or, When the average of the probability or confidence of the fourth feature corresponding to the N samples of the third feature is less than or equal to an eighth threshold, the first device determines that the second AI model fails; or, When, among the N samples of the third feature, the number of samples whose probability or confidence of the corresponding fourth feature is less than or equal to the seventh threshold accounts for a proportion of the total number of samples N that is greater than or equal to a ninth threshold, the first device determines that the second AI model has failed; Wherein, N is a positive integer.

40. A model performance supervision method, characterized in that: include: The second device receives fourth information from the first device, wherein the fourth information is used to characterize the uncertainty or probability distribution of the output of the second artificial intelligence (AI) model, and the input of the second AI model is the third feature; The second device determines the validity of the second AI model based on the fourth information.

41. The method according to claim 40, characterized in that The method further comprises: The second device sends fourth indication information to the first device, wherein the fourth indication information is used to indicate the validity of the second AI model.

42. The method according to claim 40 or 41, characterized in that The fourth information includes at least one of the following: a parameter of a probability density distribution of the fourth feature, a confidence interval of the fourth feature, a value of the fourth feature, a value of the fourth feature and its probability, and a value of the fourth feature and its confidence.

43. The method according to claim 42, characterized in that The parameters of the probability density distribution include at least one of the following: a mean or expectation of the probability density distribution, a variance or standard deviation of the probability density distribution, and an indication of the type of the probability density distribution.

44. The method according to claim 42 or 43, characterized in that In a case where the fourth information includes a parameter of a probability density distribution of the fourth feature, the second device determining the validity of the second AI model according to the fourth information includes: When the variance or standard deviation of the probability density distribution of the fourth feature is greater than or equal to the first threshold, the second device determines that the second AI model fails; or, When the mean of the variance or standard deviation of the probability density distribution of the fourth feature corresponding to the N samples of the third feature is greater than or equal to the second threshold, the second device determines that the second AI model fails; or, When, among the N samples of the third feature, the proportion of the number of samples whose variance or standard deviation of the probability density distribution of the corresponding fourth feature is greater than or equal to the first threshold to the total number of samples N is greater than or equal to the third threshold, the second device determines that the second AI model has failed; Wherein, N is a positive integer.

45. The method according to claim 42 or 43, characterized in that In a case where the fourth information includes a confidence interval of the fourth feature, the second device determines the validity of the second AI model according to the fourth information, including: When the width of the confidence interval of the fourth feature is greater than or equal to a fourth threshold, the second device determines that the second AI model is invalid; or, When the average width of the confidence interval of the fourth feature corresponding to the N samples of the third feature is greater than or equal to a fifth threshold, the second device determines that the second AI model is invalid; or, When, among the N samples of the third feature, the number of samples whose width of the confidence interval of the corresponding fourth feature is greater than or equal to the fourth threshold accounts for a proportion of the total number of samples N that is greater than or equal to a sixth threshold, the second device determines that the second AI model has failed; Wherein, N is a positive integer.

46. ​​The method according to claim 42 or 43, characterized in that In a case where the fourth information includes the value of the fourth feature and the probability or confidence thereof, the second device determines the validity of the second AI model according to the fourth information, including: When the probability or confidence of the fourth feature is less than or equal to a seventh threshold, the second device determines that the second AI model fails; or, When the average of the probability or confidence of the fourth feature corresponding to the N samples of the third feature is less than or equal to an eighth threshold, the second device determines that the second AI model fails; or, When, among the N samples of the third feature, the number of samples whose probability or confidence of the corresponding fourth feature is less than or equal to the seventh threshold accounts for a proportion of the total number of samples N that is greater than or equal to a ninth threshold, the second device determines that the second AI model has failed; Wherein, N is a positive integer.

47. A model performance monitoring device, characterized in that: include: An acquisition unit, configured to acquire first information, wherein the first information is used to characterize the uncertainty or probability distribution of an output of a first artificial intelligence (AI) model, and an input of the first AI model is a first feature; A processing unit, used to determine third information based on the first information and the second information, wherein the second information is a label corresponding to the first feature, and the third information is used to determine the validity of the first AI model.

48. A model performance monitoring device, characterized in that: include: a transceiver unit, configured to receive third information from a first device, wherein the third information is determined based on the first information and the second information, the first information is used to characterize the uncertainty or probability distribution of the output of the first artificial intelligence (AI) model, the input of the first AI model is a first feature, and the second information is a label corresponding to the first feature; A processing unit is used to determine the validity of the first AI model based on the third information.

49. A model performance monitoring device, characterized in that: include: an acquisition unit, configured to acquire fourth information, wherein the fourth information is used to characterize the uncertainty or probability distribution of the output of the second artificial intelligence (AI) model, the input of the second AI model being the third feature; A processing unit, used to determine the validity of the second AI model according to the fourth information; or a transceiver unit, used to send the fourth information to the second device.

50. A model performance monitoring device, characterized in that: include: A transceiver unit, configured to receive fourth information from the first device, wherein the fourth information is used to characterize the uncertainty or probability distribution of the output of the second artificial intelligence (AI) model, and the input of the second AI model is the third feature; A processing unit, configured to determine the validity of the second AI model based on the fourth information.

51. A model performance monitoring device, characterized in that: The model performance supervision device is a first device, and the model performance supervision device includes a processor and a memory, and the memory stores programs or instructions that can be run on the processor. When the program or instructions are executed by the processor, the steps of the model performance supervision method as described in any one of claims 1 to 16 are implemented, or, when the program or instructions are executed by the processor, the steps of the model performance supervision method as described in any one of claims 32 to 39 are implemented.

52. A model performance monitoring device, characterized in that: The model performance supervision device is a second device, and the model performance supervision device includes a processor and a memory, and the memory stores programs or instructions that can be run on the processor. When the program or instructions are executed by the processor, the steps of the model performance supervision method as described in any one of claims 17 to 31 are implemented, or when the program or instructions are executed by the processor, the steps of the model performance supervision method as described in any one of claims 40 to 46 are implemented.

53. A readable storage medium, characterized in that: The readable storage medium stores programs or instructions, and when the programs or instructions are executed by the processor, they implement the method of the model performance supervision method as described in any one of claims 1 to 16, or implement the steps of the model performance supervision method as described in any one of claims 17 to 31, or implement the steps of the model performance supervision method as described in any one of claims 32 to 39, or implement the steps of the model performance supervision method as described in any one of claims 40 to 46.

Images