Performance supervision methods and apparatuses for ai function, network-side device, and terminal device

Abstract

The present application belongs to the technical field of communications. Disclosed are performance supervision methods and apparatuses for an artificial intelligence (AI) function, a network-side device, and a terminal device. A performance supervision method for an AI function in the embodiments of the present application comprises: a terminal device performing performance supervision on a first AI function to determine first information, the first information being used for indicating the effectiveness of the first AI function and / or the accuracy of a reasoning result corresponding to the first AI function; and the terminal device sending the first information to a network-side device, wherein the first AI function is associated with at least one AI model, and the AI model comprises an activated AI model and an inactivated AI model.

Description

Method, device, network-side device, and terminal device for monitoring performance of AI functions

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed with the China Patent Office on December 14, 2023, with application number 202311735036.X and titled “Performance supervision method, device, network-side equipment and terminal equipment for AI functions”, the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The present application belongs to the field of communication technology, and specifically relates to a performance supervision method, apparatus, network-side equipment, and terminal equipment for AI functions. Background Art

[0004] To support the Life Cycle Management (LCM) of functionality identification, the terminal side needs to perform performance supervision at both the functionality level and the model level to ensure the performance of the terminal-side artificial intelligence (AI) model / function.

[0005] The current 3GPP discussions have not yet addressed the design of performance monitoring methods at the functional level. Summary of the Invention

[0006] The embodiments of the present application provide a performance monitoring method, apparatus, network-side equipment, and terminal equipment for AI functions, which can provide a function-level performance monitoring method.

[0007] First, a performance supervision method for AI functions is provided, including:

[0008] The terminal device performs performance monitoring on the first artificial intelligence (AI) function to determine first information; the first information is used to indicate the effectiveness of the first AI function and / or the accuracy of the reasoning result corresponding to the first AI function;

[0009] The terminal device sends the first information to the network side device;

[0010] The first AI function is associated with at least one AI model, and the AI ​​models include activated AI models and inactivated AI models.

[0011] Secondly, another performance supervision method for AI functions is provided, including:

[0012] The network-side device receives first information sent by the terminal device, where the first information is used to indicate the validity of the first AI function and / or the accuracy of an inference result of an AI model corresponding to the first AI function;

[0013] The first AI function is associated with at least one AI model, and the AI ​​models include activated AI models and inactivated AI models.

[0014] In a third aspect, a performance monitoring device for an AI function is provided, which is applied to a terminal device, and the device includes:

[0015] a performance monitoring module, configured to perform performance monitoring on a first artificial intelligence (AI) function and determine first information; the first information being used to indicate the effectiveness of the first AI function and / or the accuracy of a reasoning result corresponding to the first AI function;

[0016] A first sending module, configured to send the first information to a network-side device;

[0017] The first AI function is associated with at least one AI model, and the AI ​​models include activated AI models and inactivated AI models.

[0018] In a fourth aspect, another AI function performance monitoring device is provided, which is applied to a network-side device, and includes:

[0019] a first receiving module, configured to receive first information sent by a terminal device, where the first information is used to indicate the validity of a first AI function and / or the accuracy of an inference result of an AI model corresponding to the first AI function;

[0020] The first AI function is associated with at least one AI model, and the AI ​​models include activated AI models and inactivated AI models.

[0021] In a fifth aspect, a terminal device is provided, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0022] In a sixth aspect, a network side device is provided, comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the second aspect are implemented.

[0023] In the seventh aspect, a performance supervision system for an AI function is provided, comprising: a terminal device and a network side device, wherein the terminal device can be used to execute the steps of the performance supervision method for the AI ​​function as described in the first aspect above, and the network side device can be used to execute the steps of the performance supervision method for the AI ​​function as described in the second aspect above.

[0024] In an eighth aspect, a readable storage medium is provided, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented, or the steps of the method described in the second aspect are implemented.

[0025] In the ninth aspect, a chip is provided, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the method described in the first aspect, or to implement the method described in the second aspect.

[0026] In a tenth aspect, a computer program / program product is provided, wherein the computer program / program product is stored in a storage medium and is executed by at least one processor to implement the steps of the method described in the first aspect or the second aspect.

[0027] In the eleventh aspect, a performance supervision device / equipment for an AI function is provided, which includes the device / equipment (configured to) be used to execute the steps of the performance supervision method for the AI ​​function as described in the first aspect, or to execute the steps of the performance supervision method for the AI ​​function as described in the second aspect.

[0028] In an embodiment of the present application, the terminal device performs performance monitoring on the first AI function, determines first information, and sends the first information to the network side device, reporting the validity of the first AI function and / or the accuracy of the inference result corresponding to the first AI function to the network side device. While ensuring the inference performance of the first AI function in the terminal device, the model performance of the AI ​​model associated with the first AI function can be promptly notified to the network side device, so as to facilitate the network side device to perform corresponding configuration. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] FIG1 is a block diagram of a wireless communication system to which embodiments of the present application may be applied;

[0030] FIG2 is a flow chart of a method for supervising the performance of an AI function in an embodiment of the present application;

[0031] FIG3 is a schematic diagram of the structure of a neural network model in an embodiment of the present application;

[0032] FIG4 is a schematic diagram of a neuron in an embodiment of the present application;

[0033] FIG5 is a flowchart of another method for supervising the performance of an AI function in an embodiment of the present application;

[0034] FIG6 is a structural block diagram of a performance monitoring device for an AI function according to an embodiment of the present application;

[0035] FIG7 is a structural block diagram of another device for monitoring the performance of an AI function in an embodiment of the present application;

[0036] FIG8 is a structural block diagram of a communication device in an embodiment of the present application;

[0037] FIG9 is a block diagram of a terminal device according to an embodiment of the present application;

[0038] FIG10 is a structural block diagram of a network-side device in an embodiment of the present application;

[0039] FIG11 is a structural block diagram of another network-side device in an embodiment of the present application. DETAILED DESCRIPTION

[0040] The following will be combined with the accompanying drawings in the embodiments of this application to clearly describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0041] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first" and "second" are generally of the same type, and do not limit the number of objects. For example, the first object can be one or more. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0042] It is worth noting that the technology described in the embodiments of the present application is not limited to the Long Term Evolution (LTE) / LTE-Advanced (LTE-A) system, but can also be used in other wireless communication systems, such as Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal Frequency Division Multiple Access (OFDMA), Single-carrier Frequency Division Multiple Access (SC-FDMA) and other systems. The terms "system" and "network" in the embodiments of the present application are often used interchangeably, and the technology described can be used for the systems and radio technologies mentioned above, as well as for other systems and radio technologies. The following description describes a New Radio (NR) system for illustrative purposes, and the NR terminology is used in most of the following description, but these technologies can also be applied to applications other than NR system applications, such as 6th generation (6G) systems. th Generation, 6G) communication system.

[0043] FIG1 shows a block diagram of a wireless communication system applicable to embodiments of the present application. The wireless communication system includes a terminal device 11 and a network-side device 12 . The terminal device 11 may be a mobile phone, a tablet computer (Tablet Personal Computer), a laptop computer (Laptop Computer) or a notebook computer, a personal digital assistant (PDA), a handheld computer, a netbook, an ultra-mobile personal computer (UMPC), a mobile internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device (Wearable Device), a vehicle-mounted device (VUE), a pedestrian terminal (PUE), a smart home (home appliances with wireless communication functions, such as refrigerators, televisions, washing machines, or furniture, etc.), a game console, a personal computer (PC), an ATM or a self-service machine, and other terminal-side devices. Wearable devices include: smart watches, smart bracelets, smart headphones, smart glasses, smart jewelry (smart bracelets, smart bracelets, smart rings, smart necklaces, smart anklets, smart anklets, etc.), smart wristbands, smart clothing, etc. It should be noted that the specific type of the terminal device 11 is not limited in the embodiments of the present application. The network side device 12 may include an access network device or a core network device, wherein the access network device 12 may also be referred to as a radio access network device, a radio access network (RAN), a radio access network function, or a radio access network unit. The access network device 12 may include a base station, a WLAN access point, or a WiFi node, etc. The base station may be referred to as a node B, an evolved node B (eNB), an access point, a base transceiver station (BTS), a radio base station, a radio transceiver, a basic service set (BSS), an extended service set (ESS), a home node B, a home evolved node B, a transmitting and receiving point (TRP), or other appropriate terms in the field. As long as the same technical effect is achieved, the base station is not limited to a specific technical vocabulary. It should be noted that in the embodiment of the present application, only the base station in the NR system is used as an example for introduction, and the specific type of the base station is not limited.The core network equipment may include but is not limited to at least one of the following: core network node, core network function, mobility management entity (MME), access mobility management function (AMF), session management function (SMF), user plane function (UPF), policy control function (PCF), policy and charging rules function unit (PCRF), edge application service discovery function (EASDF), unified data management (UDM), unified data storage (UDR), home user server (HSS), centralized network configuration (CNC), network storage function (NRF), network exposure function (NEF), local NEF (L-NEF), binding support function (BSF), application function ( It should be noted that in the embodiments of the present application, only the core network device in the NR system is introduced as an example, and the specific type of the core network device is not limited.

[0044] The following describes in detail the model performance supervision method provided in the embodiments of the present application through some embodiments and their application scenarios in combination with the accompanying drawings.

[0045] The present application embodiment provides a method for monitoring the performance of an AI function. Referring to FIG2 , a flow chart of a method for monitoring the performance of an AI function provided by the present application embodiment is shown. The method is applied to a terminal device, as shown in FIG2 , and the method may specifically include:

[0046] Step 101: A terminal device monitors the performance of a first artificial intelligence (AI) function and determines first information; the first information is used to indicate the effectiveness of the first AI function and / or the accuracy of a reasoning result corresponding to the first AI function.

[0047] Step 102: The terminal device sends the first information to the network side device.

[0048] The first AI function is associated with at least one AI model, and the AI ​​models include activated AI models and inactivated AI models.

[0049] In an embodiment of the present invention, an AI function can be associated with one or more AI models. It should be noted that the AI ​​model in the embodiment of the present application can be a machine learning model or a neural network model, such as any one of a fully connected neural network, a convolutional neural network, a decision tree, a support vector machine, and a Bayesian classifier. Taking the neural network model as an example, its schematic diagram can be shown in Figure 3. As shown in Figure 3, the neural network may include one or more input layers, one or more hidden layers, and an output layer. The data to be processed [X1, X2…Xn] are input into the neural network from the corresponding input layer, and after processing by the input layer, hidden layer, and output layer, the output result Y is obtained. In addition, the neural network is composed of neurons, and a schematic diagram of the neurons is shown in Figure 4. In Figure 4, a1, a2,…aK represent inputs, w represents weights (i.e., multiplicative coefficients), b represents biases (i.e., additive coefficients), and σ(.) represents activation functions. Common activation functions include Sigmoid (mapping variables between 0 and 1), tanh (translation and contraction of Sigmoid), linear rectification function / rectified linear unit (Rectified Linear Unit, ReLU), etc.

[0050] Taking the neural network model as an example, the model training process is introduced as follows:

[0051] The parameters of a neural network can be optimized using a gradient optimization algorithm. A gradient optimization algorithm is a type of algorithm that minimizes or maximizes an objective function (sometimes also called a loss function), which is often a mathematical combination of model parameters and data. For example, given data X and its corresponding label Y, a neural network model f(.) can be constructed. Based on the input x, the predicted output f(x) can be obtained, and the difference between the predicted value and the true value (f(x)-Y) can be calculated. This is the loss function. The optimization goal of the gradient optimization algorithm is to find the appropriate w (i.e., weight) and b (i.e., bias) to minimize the value of the aforementioned loss function. The smaller the loss value, the closer the model is to the true situation.

[0052] Currently, most common optimization algorithms are based on the back propagation (BP) algorithm. The basic idea of ​​the BP algorithm is that the learning process consists of two steps: forward propagation of signals and back propagation of errors. During forward propagation, input samples are passed from the input layer, processed layer by layer through each hidden layer, and then transmitted to the output layer. If the actual output of the output layer does not match the expected output, the error begins back propagation. Back propagation involves propagating the output error back through the hidden layers to the input layer layer by layer in some form, distributing the error to all units in each layer. This error signal is then generated for each unit in each layer, and used as the basis for correcting the weights of each unit. This process of adjusting the weights of each layer, including forward propagation of signals and back propagation of errors, is repeated over and over again. This continuous adjustment of weights is the network's learning and training process. This process continues until the error in the network output is reduced to an acceptable level, or until a pre-set number of learning cycles has been completed.

[0053] In addition, common optimization algorithms include gradient descent, stochastic gradient descent (SGD), mini-batch gradient descent, momentum method (Momentum), Nesterov (the name of the inventor, specifically stochastic gradient descent with momentum), adaptive gradient descent (ADAptive GRADient descent, Adagrad), Adagrad's extended algorithm (Adadelta), root mean square error deceleration (root mean square prop, RMSprop), Adaptive Moment Estimation (Addam), etc.

[0054] When these optimization algorithms backpropagate errors, they all calculate the derivative / partial derivative of the current neuron based on the error / loss obtained by the loss function, add the influence of the learning rate, the previous gradient / derivative / partial derivative, etc., obtain the gradient, and pass the gradient to the previous layer.

[0055] The AI ​​function / AI model in the embodiments of the present application may also be referred to as an AI unit, an ML (machine learning) model, an ML unit, an AI structure, an AI unit, an AI characteristic, a neural network, a neural network function, a neural network function, etc., or the AI ​​function / AI model may also refer to a processing unit capable of implementing specific algorithms, formulas, processing flows, capabilities, etc. related to AI, or the AI ​​function / AI model may be a processing method, algorithm, function, module or unit for a specific data set, or the AI ​​function / AI model may be a processing method, algorithm, function, module or unit running on AI / ML related hardware such as a GPU, NPU, TPU, ASIC, etc., which is not specifically limited in the present invention. Optionally, the specific data set includes the input and / or output of the AI ​​function / AI model.

[0056] In this embodiment of the present invention, the terminal device performs performance monitoring on the first AI function, which may include monitoring the performance of one or more AI models associated with the first AI function to determine the effectiveness of the first AI function and / or the accuracy of the inference result corresponding to the first AI function. The monitoring result is then sent to the network device via the first information.

[0057] Performance supervision in the embodiments of the present application refers to supervision of the performance of the AI ​​function / AI model. The performance of the AI ​​function / AI model may include but is not limited to the effectiveness of the AI ​​function / AI model in the current environment, the accuracy of the reasoning results of the AI ​​function / AI model, etc.

[0058] In the lifecycle management of each AI use case, performance indicators for AI functions or AI models, or methods for monitoring model performance, can include at least one of the following:

[0059] Model performance supervision based on inference accuracy, using performance metrics related to intermediate KPIs;

[0060] Model performance monitoring based on system performance, using performance indicators related to system performance KPIs;

[0061] Other model performance supervision schemes.

[0062] Among them, other model performance supervision solutions may include at least two of the following options:

[0063] Model performance supervision based on data distribution. For example, input-based model performance supervision, such as supervising the input of an AI model to determine its effectiveness, can use methods such as out-of-distribution detection and input data drift detection; output-based model performance supervision, for example, output data drift detection.

[0064] Model performance supervision based on applicable conditions.

[0065] It should be noted that the performance indicator calculation can be completed on the network side device or the terminal device.

[0066] The model performance indication, model validity indication, or model reasoning accuracy indication obtained based on the above-mentioned model performance supervision method can be further used to deactivate, activate, select, or switch the AI ​​model / AI function on the UE side, or to deactivate, activate, select, or switch the UE part of the AI ​​model / AI function at both ends. Among them, the AI ​​model / AI function at both ends refers to an AI model / AI function that runs on a terminal device with one part and on a network-side device with the other part, and the models at both ends need to be paired for use.

[0067] In function-based life cycle management (LCM), network-side devices can indicate the activation / deactivation / fallback / switching of AI functions through 3GPP signaling, such as LTE Positioning Protocol (LPP), Radio Resource Control Protocol (RRC) signaling, Media Access Control (MAC) signaling, downlink control information (DCI), etc. An AI use case may include multiple AI functions. It should be noted that the UE may have one AI / ML model for the function, or the UE may have multiple AI / ML models for the function.

[0068] Optionally, the first information includes at least one of the following:

[0069] a function identifier of the first AI function;

[0070] a validity flag of the first AI function, where the validity flag is used to indicate whether the first AI function is valid;

[0071] Second information, where the second information is used to indicate resource information associated with the first AI function;

[0072] A first indication is used to indicate the accuracy of the inference result corresponding to the first AI function.

[0073] Among them, the function identifier of the first AI function can be an AI model identifier, an AI structure identifier, an AI algorithm identifier, or an identifier of a specific data set associated with the AI ​​unit / AI model, or an identifier of a specific scenario, environment, channel feature, or device related to the AI / ML, or an identifier of a function, feature, capability, or module related to the AI / ML. This embodiment of the present application does not specifically limit this.

[0074] The validity flag of the first AI function is used to indicate whether the first AI function is valid. It is understood that the validity of the first AI function can be determined based on the validity of the AI ​​model associated with the first AI function. For example, if the number of valid AI models among the AI ​​models associated with the first AI function is greater than a preset threshold, the first AI function can be considered valid. The validity of the AI ​​model can be determined based on a model performance supervision method. If an AI model is invalid, the AI ​​model can be deactivated; if an AI model is valid, the AI ​​model can be activated.

[0075] In addition, as an example, the validity identifier of the first AI function can be a number between 0 and 1 with an interval of 0.1. The closer the value is to 1, the better the performance of the first AI function and the higher the reliability; the larger the value of the validity identifier of an AI function, the higher the proportion of accurate results in the multiple inference results obtained using the AI ​​function. The validity identifier of the first AI function can be determined based on the loss value, number of training times, etc. of the AI ​​model associated with the first AI function during the training process. Before the training reaches the plateau, the more training times and the smaller the loss value, the larger the value of the validity identifier of the first AI function. It is understandable that the loss value of the AI ​​model will not decrease indefinitely with the increase in the number of training times. When the number of training times reaches a certain threshold, the training process reaches a plateau, the loss value of the AI ​​model will no longer change significantly, and the model performance has become relatively stable and cannot be further improved.

[0076] Alternatively, the validity flag of the first AI function can be a fixed number. For example, the validity flag can be "1," indicating that the first AI function is valid, or "0," indicating that the first AI function is invalid. It is understood that the validity flag can be specified by the protocol, configured by the network device or the terminal device, or configured collaboratively by the network device and the terminal device.

[0077] The first indication may be a numerical value representing the accuracy of a single inference result corresponding to the first AI function, such as 80% or 75.6%, or an identifier associated with the accuracy of the inference result corresponding to the first AI function, such as A, B, or C, where each identifier corresponds to a range of accuracy values. The accuracy of the inference result corresponding to the first AI function may be determined based on the similarity between the inference result corresponding to the first AI function and a reference result, or based on the loss value of the AI ​​model associated with the first AI function during training, etc.

[0078] Optionally, the second information includes at least one of the following:

[0079] the cell identifier associated with the first AI function;

[0080] an identifier of the sending / receiving point associated with the first AI function;

[0081] a reference signal identifier associated with the first AI function;

[0082] an identifier of a reference signal resource set associated with the first AI function;

[0083] The port number associated with the first AI function.

[0084] It is understandable that the cell identifier associated with the first AI function may be one or more, and the identifier of the transmission and reception point (TRP) associated with the first AI function may also be one or more.

[0085] The reference signals associated with the first AI function may include, but are not limited to, positioning reference signals, downlink channel sounding reference signals (CSI-RS), uplink sounding reference signals (SRS), synchronization signal blocks (SSB), and time-frequency tracking reference signals (TRS).

[0086] Optionally, the method further includes:

[0087] The terminal device determines that the first AI function is valid when the AI ​​model associated with the first AI function meets a first condition.

[0088] The first condition includes at least one of the following:

[0089] The number of valid AI models among the M AI models associated with the first AI function is greater than or equal to a first threshold; M is a positive integer;

[0090] A ratio of the number of valid AI models in the M AI models associated with the first AI to M is greater than or equal to a second threshold.

[0091] Optionally, the method further includes:

[0092] The terminal device determines that the first AI function is invalid when the AI ​​model associated with the first AI function meets a second condition.

[0093] The second condition includes at least one of the following:

[0094] The number of valid AI models among the M AI models associated with the first AI function is less than or equal to a third threshold; M is a positive integer;

[0095] A ratio of the number of valid AI models in the M AI models associated with the first AI to M is less than or equal to a fourth threshold.

[0096] It is understood that if the number of valid models among the M AI models associated with the first AI function is less than or equal to a third threshold, or if the ratio of valid models to M among the M AI models associated with the first AI function is less than or equal to a fourth threshold, it indicates that the majority of the AI ​​models associated with the first AI function are invalid AI models, and the first AI function can be considered invalid. For example, if the number of valid models among the M AI models associated with the first AI function is 0, the first AI function can be determined to be invalid.

[0097] Optionally, the method further includes:

[0098] The terminal device sends third information to the network side device.

[0099] The third information includes at least one of the following:

[0100] the number of valid AI models among the AI ​​models associated with the first AI function;

[0101] a model identifier of a valid AI model among the AI ​​models associated with the first AI function;

[0102] the number of invalid AI models among the AI ​​models associated with the first AI function;

[0103] a model identifier of an invalid AI model among the AI ​​models associated with the first AI function;

[0104] a first identifier associated with the number of valid AI models among the AI ​​models associated with the first AI function;

[0105] a second identifier associated with a number of invalid AI models among the AI ​​models associated with the first AI function;

[0106] Fourth information, the fourth information being used to indicate resource information associated with the valid AI model;

[0107] Fifth information, the fifth information is used to indicate resource information associated with the invalid AI model.

[0108] In an embodiment of the present application, the terminal device can report the validity of the first AI function to the network side device through the third information, such as the number of valid AI models in the AI ​​models associated with the first AI function, the model IDs of the valid AI models, the number of invalid AI models, the model IDs of the invalid AI models, the area information associated with the first AI function (such as the fourth information and the fifth information), etc. The area information can be the cell ID, tracking area (Tracking Area) ID, TRP ID, etc. associated with the first AI function. The network side device does not need to know the specific meaning of the model ID, but the network side device can know which AI models in the terminal device are valid in which areas through the fourth information, and know which AI models in the terminal device are invalid in which areas through the fifth information, which is beneficial to model-level LCM.

[0109] Among them, the first identifier is associated with the number of valid AI models in the AI ​​models associated with the first AI function. For example, the first identifier can be the ratio of the number of valid AI models associated with the first AI function to the total number of AI models, or the first identifier can be identifiers such as A, B, and C. Each identifier corresponds to a value range. The first identifier is the identifier corresponding to the value range in which the number of valid AI models associated with the first AI function falls.

[0110] Similarly, the second identifier can be the ratio of the number of invalid AI models associated with the first AI function to the total number of AI models, or identifiers such as A, B, and C. Each identifier corresponds to a value range. The second identifier is the identifier corresponding to the value range in which the number of invalid AI models associated with the first AI function falls.

[0111] It should be noted that the total number of models in the embodiment of the present application can be the number of models simultaneously supervised by the terminal device, or the number of AI models associated with the first AI function in the terminal device.

[0112] Optionally, the fourth information includes at least one of the following:

[0113] The cell identifier associated with the valid AI model;

[0114] The identifier of the sending and receiving point associated with the valid AI model;

[0115] A reference signal identifier associated with the valid AI model;

[0116] An identifier of a reference signal resource set associated with the valid AI model;

[0117] The port number associated with the valid AI model.

[0118] Optionally, the fifth information includes at least one of the following:

[0119] The cell identifier associated with the invalid AI model;

[0120] The identifier of the sending and receiving point associated with the invalid AI model;

[0121] A reference signal identifier associated with the invalid AI model;

[0122] an identifier of a reference signal resource set associated with the invalid AI model;

[0123] The port number associated with the invalid AI model.

[0124] Optionally, the method further includes:

[0125] The terminal device determines that the AI ​​model is a valid AI model when the AI ​​model satisfies the third condition.

[0126] The third condition includes at least one of the following:

[0127] The number of first instances for determining that the model is valid is greater than or equal to a fifth threshold within T consecutive time units; T is a positive integer;

[0128] Within T consecutive time units, the ratio of the first number of instances for determining that the model is valid to the total number of instances is greater than or equal to a sixth threshold;

[0129] In Y consecutive instances, the first number of instances in which the model is determined to be valid is greater than or equal to a seventh threshold; Y is a positive integer;

[0130] In Y consecutive instances, the ratio of the first number of instances in which the model is determined to be valid to the total number of instances is greater than or equal to an eighth threshold.

[0131] In an embodiment of the present application, the terminal device can determine whether an AI model is a valid AI model through the third condition. It should be noted that an instance can represent the task of an AI model to perform a model reasoning, and the number of instances can represent the number of times the AI ​​model performs the model reasoning task. An instance in which the model is determined to be valid means that the model reasoning result of the instance meets the performance requirements; conversely, an instance in which the model is determined to be invalid means that the model reasoning result of the instance does not meet the performance requirements, or it is impossible to determine whether the performance requirements are met. Performance requirements may include but are not limited to accuracy requirements, timeliness requirements, precision requirements, etc. of the model reasoning results.

[0132] Optionally, the method further includes:

[0133] When the AI ​​model satisfies a fourth condition, the terminal device determines that the AI ​​model is an invalid AI model.

[0134] The fourth condition includes at least one of the following:

[0135] The number of first instances of determining that the model is valid is less than or equal to a ninth threshold value within T consecutive time units; T is a positive integer;

[0136] Within T consecutive time units, the ratio of the first number of instances that determine the model to be valid to the total number of instances is less than or equal to the tenth threshold;

[0137] In Y consecutive instances, the first number of instances in which the model is determined to be valid is less than or equal to an eleventh threshold; Y is a positive integer and greater than 1;

[0138] In Y consecutive instances, the ratio of the first number of instances in which the model is determined to be valid to the total number of instances is less than or equal to a twelfth threshold.

[0139] In an embodiment of the present application, the length T and the number of instances Y of the performance monitoring time window of the terminal device can be configured by the network side device, or specified by the protocol, or determined by the terminal device itself.

[0140] Optionally, the method further includes:

[0141] Step S11: The terminal device receives a second indication sent by the network side device, where the second indication is used to indicate the length of the performance monitoring time window and the number of second instances;

[0142] Step S12: The terminal device determines the value of T according to the length of the performance monitoring time window, and determines the value of Y according to the second instance number.

[0143] The value of T is greater than or equal to the length of the performance monitoring time window, and the value of Y is greater than or equal to the second instance number.

[0144] In an embodiment of the present application, the network side device can indicate the minimum length of the performance supervision time window and the minimum number of instances (i.e., the second number of instances) to the terminal device through a second indication. The terminal device configures the length T and the number Y of instances of the performance supervision time window actually used based on the received second indication. The length of the performance supervision time window actually used by the terminal device cannot be less than the length of the performance supervision time window indicated by the second indication, and the number Y of instances actually used by the terminal device cannot be less than the second number of instances indicated by the second indication.

[0145] Optionally, the method further includes:

[0146] The terminal device sends sixth information to the network side device, where the sixth information carries the time window length T and the number of instances Y used by the terminal device to perform performance supervision on the first AI function.

[0147] In an embodiment of the present application, the terminal device can report to the network side device through the sixth information the time window length T and the number of instances Y actually used in the performance supervision process of the first AI function.

[0148] Optionally, the method further includes:

[0149] The terminal capability information sent by the terminal device to the network side device.

[0150] In an embodiment of the present application, the terminal device may also report terminal capability information related to model performance supervision to the network side device. The terminal capability information includes at least one of the following:

[0151] The AI ​​functional-level functional performance supervision capability of the terminal device is used to indicate whether the terminal device supports AI functional-level performance supervision;

[0152] The AI ​​model-level model performance supervision capability of the terminal device, wherein the model performance supervision capability is used to indicate whether the terminal device supports performing AI model-level performance supervision;

[0153] Model performance supervision method supported by the terminal device;

[0154] an indication of the reliability of the model performance supervision method supported by the terminal device;

[0155] The maximum time window length and maximum number of instances of model performance supervision supported by the terminal device;

[0156] The number of AI models supported by the terminal device to run simultaneously;

[0157] The number of AI models that the terminal device supports simultaneous supervision;

[0158] The delay of model activation and model deactivation of the terminal device;

[0159] The activation and deactivation delay of the first AI function of the terminal device;

[0160] The type of the validity indicator or accuracy indicator of the AI ​​function in the terminal device;

[0161] The granularity of the validity identifier or accuracy identifier of the AI ​​function in the terminal device, where the granularity is used to indicate the interval value between two consecutive numerical values ​​in the validity identifier or accuracy identifier.

[0162] Among them, the model performance supervision methods supported by the terminal device may include but are not limited to: methods based on model reasoning accuracy, methods based on system performance, methods based on data distribution, methods based on application conditions, etc.

[0163] Whether the AI ​​model is effective can be determined through the model performance supervision method, which depends on the specific algorithm adopted by the terminal device or is specified by the protocol.

[0164] The identification type of the validity identifier or accuracy identifier of the AI ​​function in the terminal device can include a variable value (soft value) and a fixed value (hard value). Among them, the variable value means that the value of the validity identifier or accuracy identifier can float within a value range. For example, the validity identifier can be a value between 0 and 1. The closer the value is to 1, the better the performance of the AI ​​function and the higher the reliability. The fixed value means that the value of the validity identifier or accuracy identifier is a fixed value. For example, the validity identifier can be "1", indicating that the AI ​​function is valid; the validity identifier can also be "0", indicating that the AI ​​function is invalid.

[0165] The granularity of the validity flag or accuracy flag of the AI ​​function in the terminal device is used to indicate the interval between two consecutive values ​​in the validity flag or accuracy flag. For example, if the validity flag type is 0 and the granularity is 0.1, then the validity flag value of the AI ​​function in the terminal device can be determined to be a number between 0 and 1 with an interval of 0.1.

[0166] In an embodiment of the present application, the terminal device can report information such as the identification type and granularity of the validity identifier or accuracy identifier of the AI ​​function in the terminal device to the network side device, so that the network side device can determine the validity of the first AI function based on the received validity identifier of the first AI function, and / or determine the accuracy of the inference result corresponding to the first AI function based on the accuracy identifier of the first AI function.

[0167] Optionally, the method further includes:

[0168] The terminal device sends seventh information to the network side device, where the seventh information is used to indicate whether the first model has been activated.

[0169] It should be noted that the first model in the embodiments of the present application is not limited to a specific AI model. In other words, the first model in the embodiments of the present application may include one or more AI models, or may be an AI function, and an AI function may be associated with one or more AI models. Activating the first model may mean simultaneously activating one or more AI models included in the first model, or simultaneously activating one or more AI functions referred to by the first model.

[0170] After activating the first model, the terminal device can report to the network side device through the seventh information that the first model has been activated; or, after deactivating the first model, the terminal device can report to the network side device through the seventh information that the first model has been deactivated.

[0171] Optionally, the method further includes:

[0172] Step S21: When the terminal device activates the first AI function, the terminal device sends eighth information to the network side device; or

[0173] Step S22: When the terminal device deactivates the first AI function, the terminal device sends ninth information to the network side device.

[0174] The eighth information includes at least one of the following:

[0175] a function identifier of the first AI function;

[0176] a third indication, where the third indication is used to indicate that the terminal device has activated the first AI function;

[0177] a fourth indication, where the fourth indication is used to indicate that the terminal device has activated the AI ​​model associated with the first AI function;

[0178] a model identifier of an activated AI model associated with the first AI function in the terminal device;

[0179] The ninth information includes at least one of the following:

[0180] a function identifier of the first AI function;

[0181] a fifth indication, where the fifth indication is used to instruct the terminal device to deactivate the first AI function;

[0182] a sixth indication, where the sixth indication is used to instruct the terminal device to deactivate the AI ​​model associated with the first AI function;

[0183] A model identifier of a deactivated AI model associated with the first AI function in the terminal device.

[0184] It should be noted that, in the embodiment of the present application, performing an activation operation on the first AI function includes activating the first AI function and / or activating at least one AI model among the AI ​​models associated with the first AI function. Performing a deactivation operation on the first AI function includes deactivating the first AI function and / or deactivating at least one AI model among the AI ​​models associated with the first AI function.

[0185] Exemplarily, if the terminal device activates the first AI function, the eighth information may include the function identifier and the third indication of the first AI function; if the terminal device activates at least one AI model among the AI ​​models associated with the first AI function, the eighth information may include the function identifier of the first AI function, the fourth indication, and the model identifier of the activated AI model associated with the first AI function in the terminal device.

[0186] If the terminal device deactivates the first AI function, the ninth information may include the function identifier of the first AI function and the fifth indication; if the terminal device deactivates at least one AI model among the AI ​​models associated with the first AI function, the ninth information may include the function identifier of the first AI function, the sixth indication, and the model identifier of the deactivated AI model associated with the first AI function in the terminal device.

[0187] In an embodiment of the present application, if an AI function is deactivated, all AI models associated with the AI ​​function are invalid models, or in other words, all AI models associated with the AI ​​function are deactivated; similarly, if an AI function is activated, all AI models associated with the AI ​​function are valid models, or in other words, all AI models associated with the AI ​​function are activated.

[0188] In addition, the deactivation operation and the activation operation can be independent of each other. For example, if the terminal device activates the first AI function, and the AI ​​model associated with the first AI function includes the AI ​​model currently running in the terminal device, then there is no need to activate the AI ​​model currently running in the terminal device. In this case, the normal operation of the currently running AI model can be maintained, and then the AI ​​models associated with the first AI function, except for the currently running AI model, can be activated. Alternatively, if the terminal device activates the first AI function, the AI ​​models associated with the first AI function only include the AI ​​model currently running in the terminal device, and then there is no need to perform the activation operation.

[0189] In an embodiment of the present application, the terminal device tells the network side device through the eighth information or the ninth information that the first AI function on the terminal side has undergone a state switch (including activation or deactivation), and the network side device should assume that the terminal device cannot use the first AI function within the state switching delay. Among them, the model activation delay and the model deactivation delay of the terminal device can be reported to the network side device by the terminal device through the terminal capability information. The network side device can suspend sending the reasoning task related to the first AI function to the terminal device within the state switching delay to avoid the terminal device being unable to execute the reasoning task related to the first AI function in time due to the state switch, resulting in a period of waiting for the reasoning result, affecting the overall processing efficiency.

[0190] In summary, the embodiments of the present application provide a performance supervision method for an AI function, wherein a terminal device performs performance supervision on a first AI function, determines first information, and sends the first information to a network side device, and reports the validity of the first AI function and / or the accuracy of the reasoning result corresponding to the first AI function to the network side device. While ensuring the reasoning performance of the first AI function in the terminal device, the model performance of the AI ​​model associated with the first AI function can be promptly notified to the network side device, so as to facilitate the network side device to perform corresponding configuration.

[0191] The present application embodiment provides another method for monitoring the performance of an AI function. Referring to FIG5 , a flow chart of a method for monitoring the performance of an AI function provided by an embodiment of the present application is shown. The method is applied to a network-side device, as shown in FIG5 , and the method may specifically include:

[0192] Step 201: A network-side device receives first information sent by a terminal device, where the first information is used to indicate the validity of a first AI function and / or the accuracy of an inference result of an AI model corresponding to the first AI function.

[0193] The first AI function is associated with at least one AI model, and the AI ​​models include activated AI models and inactivated AI models.

[0194] The network side device can be the access network device in Figure 1, such as a base station or a newly defined artificial intelligence processing node on the access network side, or it can be the core network device in Figure 1, such as a network data analysis function (Network Data Analytics Function, NWDAF), a positioning management function (Location Management Function, LMF), or a newly defined processing node on the core network side, or it can be a combination of the above multiple nodes.

[0195] In an embodiment of the present invention, an AI function may be associated with one or more AI models. The first information is generated by a terminal device based on the results of model performance supervision of the AI ​​model associated with the first AI function. The first information is used to indicate the effectiveness of the first AI function in the terminal device, the accuracy of the inference result, and other performance.

[0196] Optionally, the first information includes at least one of the following:

[0197] a function identifier of the first AI function;

[0198] a validity flag of the first AI function, where the validity flag is used to indicate whether the first AI function is valid;

[0199] Second information, where the second information is used to indicate resource information associated with the first AI function;

[0200] A first indication is used to indicate the accuracy of an inference result of the AI ​​model corresponding to the first AI function.

[0201] Among them, the function identifier of the first AI function can be an AI model identifier, an AI structure identifier, an AI algorithm identifier, or an identifier of a specific data set associated with the AI ​​unit / AI model, or an identifier of a specific scenario, environment, channel feature, or device related to the AI / ML, or an identifier of a function, feature, capability, or module related to the AI / ML. This embodiment of the present application does not specifically limit this.

[0202] The validity flag of the first AI function is used to indicate whether the first AI function is valid. It is understandable that the validity of the first AI function can be determined based on the validity of the AI ​​model associated with the first AI function. For example, if the number of valid AI models in the AI ​​model associated with the first AI function is greater than a preset threshold, the first AI function can be considered valid. The validity of the AI ​​model can be determined based on the model performance supervision method. If an AI model is an invalid AI model, the AI ​​model can be deactivated; if an AI model is a valid AI model, the AI ​​model can be activated.

[0203] In addition, as an example, the validity identifier of the first AI function can be a number between 0 and 1 with an interval of 0.1. The closer the value is to 1, the better the performance of the first AI function and the higher the reliability; the larger the value of the validity identifier of an AI function, the higher the proportion of accurate results in the multiple inference results obtained using the AI ​​function. The validity identifier of the first AI function can be determined based on the loss value, number of training times, etc. of the AI ​​model associated with the first AI function during the training process. Before the training reaches the plateau, the more training times and the smaller the loss value, the larger the value of the validity identifier of the first AI function. It is understandable that the loss value of the AI ​​model will not decrease indefinitely with the increase in the number of training times. When the number of training times reaches a certain threshold, the training process reaches a plateau, the loss value of the AI ​​model will no longer change significantly, and the model performance has become relatively stable and cannot be further improved.

[0204] Alternatively, the validity flag of the first AI function can be a fixed number. For example, the validity flag can be "1," indicating that the first AI function is valid, or "0," indicating that the first AI function is invalid. It is understood that the validity flag can be specified by the protocol, configured by the network device or the terminal device, or configured collaboratively by the network device and the terminal device.

[0205] The first indication may be a numerical value representing the accuracy of a single inference result corresponding to the first AI function, such as 80% or 75.6%, or an identifier associated with the accuracy of the inference result corresponding to the first AI function, such as A, B, or C, where each identifier corresponds to a range of accuracy values. The accuracy of the inference result corresponding to the first AI function may be determined based on the similarity between the inference result corresponding to the first AI function and a reference result, or based on the loss value of the AI ​​model associated with the first AI function during training, etc.

[0206] Optionally, the second information includes at least one of the following:

[0207] the cell identifier associated with the first AI function;

[0208] an identifier of the sending / receiving point associated with the first AI function;

[0209] a reference signal identifier associated with the first AI function;

[0210] an identifier of a reference signal resource set associated with the first AI function;

[0211] The port number associated with the first AI function.

[0212] It is understandable that the cell identifier associated with the first AI function may be one or more, and the identifier of the transmission and reception point (TRP) associated with the first AI function may also be one or more.

[0213] The reference signals associated with the first AI function may include, but are not limited to, positioning reference signals, CSI-RS, SRS, SSB, TRS, etc.

[0214] Optionally, the method further includes:

[0215] The network side device receives the third information sent by the terminal device.

[0216] The third information includes at least one of the following:

[0217] the number of valid AI models among the AI ​​models associated with the first AI function;

[0218] a model identifier of a valid AI model among the AI ​​models associated with the first AI function;

[0219] the number of invalid AI models among the AI ​​models associated with the first AI function;

[0220] a model identifier of an invalid AI model among the AI ​​models associated with the first AI function;

[0221] a first identifier associated with the number of valid AI models among the AI ​​models associated with the first AI function;

[0222] a second identifier associated with a number of invalid AI models among the AI ​​models associated with the first AI function;

[0223] Fourth information, the fourth information being used to indicate resource information associated with the valid AI model;

[0224] Fifth information, the fifth information is used to indicate resource information associated with the invalid AI model.

[0225] In an embodiment of the present application, the network side device can obtain the validity of the first AI function in the terminal device through the third information, such as the number of valid AI models in the AI ​​models associated with the first AI function, the model ID of the valid AI model, the number of invalid AI models, the model ID of the invalid AI model, the area information associated with the first AI function (such as the fourth information, the fifth information), etc. The area information can be the cell ID, tracking area (Tracking Area) ID, TRP ID, etc. associated with the first AI function. The network side device does not need to know the specific meaning of the model ID, but the network side device can know which AI models in the terminal device are valid in which areas through the fourth information, and know which AI models in the terminal device are invalid in which areas through the fifth information, which is beneficial to model-level LCM.

[0226] Among them, the first identifier is associated with the number of valid AI models in the AI ​​models associated with the first AI function. For example, the first identifier can be the ratio of the number of valid AI models associated with the first AI function to the total number of AI models, or the first identifier can be identifiers such as A, B, and C. Each identifier corresponds to a value range. The first identifier is the identifier corresponding to the value range in which the number of valid AI models associated with the first AI function falls.

[0227] Similarly, the second identifier can be the ratio of the number of invalid AI models associated with the first AI function to the total number of AI models, or identifiers such as A, B, and C. Each identifier corresponds to a value range. The second identifier is the identifier corresponding to the value range in which the number of invalid AI models associated with the first AI function falls.

[0228] It should be noted that the total number of models in the embodiment of the present application can be the number of models simultaneously supervised by the terminal device, or the number of AI models associated with the first AI function in the terminal device.

[0229] Optionally, the fourth information includes at least one of the following:

[0230] The cell identifier associated with the valid AI model;

[0231] The identifier of the sending and receiving point associated with the valid AI model;

[0232] A reference signal identifier associated with the valid AI model;

[0233] An identifier of a reference signal resource set associated with the valid AI model;

[0234] The port number associated with the valid AI model.

[0235] Optionally, the fifth information includes at least one of the following:

[0236] The cell identifier associated with the invalid AI model;

[0237] The identifier of the sending and receiving point associated with the invalid AI model;

[0238] A reference signal identifier associated with the invalid AI model;

[0239] an identifier of a reference signal resource set associated with the invalid AI model;

[0240] The port number associated with the invalid AI model.

[0241] Optionally, the method further includes:

[0242] The network side device sends a second indication to the terminal device, where the second indication is used to indicate the length of the performance monitoring time window and the second instance number.

[0243] Optionally, the method further includes:

[0244] The network side device receives sixth information sent by the terminal device, where the sixth information carries a time window length T and a number of instances Y used by the terminal device to perform performance supervision on the first AI function.

[0245] In an embodiment of the present application, the network side device can indicate the minimum length of the performance supervision time window and the minimum number of instances (i.e., the second number of instances) to the terminal device through a second indication. The length of the performance supervision time window actually used by the terminal device cannot be less than the length of the performance supervision time window indicated by the second indication, and the number of instances Y actually used by the terminal device cannot be less than the second number of instances indicated by the second indication.

[0246] Optionally, the method further includes:

[0247] The network side device receives the terminal capability information sent by the terminal device.

[0248] In an embodiment of the present application, the network side device may also receive terminal capability information related to model performance supervision reported by the terminal device. The terminal capability information includes at least one of the following:

[0249] The functional performance supervision capability of the AI ​​function level of the terminal device, where the functional performance supervision capability is used to indicate whether the terminal device supports performance supervision of the AI ​​function level;

[0250] The AI ​​model-level model performance supervision capability of the terminal device, where the model performance supervision capability is used to indicate whether the terminal device supports performing AI model-level performance supervision;

[0251] Model performance supervision method supported by the terminal device;

[0252] an indication of the reliability of the model performance supervision method supported by the terminal device;

[0253] The maximum time window length and maximum number of instances of model performance supervision supported by the terminal device;

[0254] The number of AI models supported by the terminal device to run simultaneously;

[0255] The number of AI models that the terminal device supports simultaneous supervision;

[0256] The delay of model activation and model deactivation of the terminal device;

[0257] The activation and deactivation delay of the first AI function of the terminal device;

[0258] The type of the validity indicator or accuracy indicator of the AI ​​function in the terminal device;

[0259] The granularity of the validity identifier or accuracy identifier of the AI ​​function in the terminal device, where the granularity is used to indicate the interval value between two consecutive numerical values ​​in the validity identifier or accuracy identifier.

[0260] Among them, the model performance supervision methods supported by the terminal device may include but are not limited to: methods based on model reasoning accuracy, methods based on system performance, methods based on data distribution, methods based on application conditions, etc.

[0261] Whether the AI ​​model is effective can be determined through the model performance supervision method, which depends on the specific algorithm adopted by the terminal device or is specified by the protocol.

[0262] The identification type of the validity identifier or accuracy identifier of the AI ​​function in the terminal device can include a variable value (soft value) and a fixed value (hard value). Among them, the variable value means that the value of the validity identifier or accuracy identifier can float within a value range. For example, the validity identifier can be a value between 0 and 1. The closer the value is to 1, the better the performance of the AI ​​function and the higher the reliability. The fixed value means that the value of the validity identifier or accuracy identifier is a fixed value. For example, the validity identifier can be "1", indicating that the AI ​​function is valid; the validity identifier can also be "0", indicating that the AI ​​function is invalid.

[0263] The granularity of the validity flag or accuracy flag of the AI ​​function in the terminal device is used to indicate the interval between two consecutive values ​​in the validity flag or accuracy flag. For example, if the validity flag type is 0 and the granularity is 0.1, then the validity flag value of the AI ​​function in the terminal device can be determined to be a number between 0 and 1 with an interval of 0.1.

[0264] In an embodiment of the present application, the terminal device can report information such as the identification type and granularity of the validity identifier or accuracy identifier of the AI ​​function in the terminal device to the network side device, so that the network side device can determine the validity of the first AI function based on the received validity identifier of the first AI function, and / or determine the accuracy of the inference result corresponding to the first AI function based on the accuracy identifier of the first AI function.

[0265] Optionally, the method further includes:

[0266] The network side device receives seventh information sent by the terminal device, where the seventh information is used to indicate whether the first model has been activated.

[0267] It should be noted that the first model in the embodiments of the present application is not limited to a specific AI model. In other words, the first model in the embodiments of the present application may include one or more AI models, or may be an AI function, and an AI function may be associated with one or more AI models. Activating the first model may mean simultaneously activating one or more AI models included in the first model, or simultaneously activating one or more AI functions referred to by the first model.

[0268] After activating the first model, the terminal device can report to the network side device through the seventh information that the first model has been activated; or, after deactivating the first model, the terminal device can report to the network side device through the seventh information that the first model has been deactivated.

[0269] Optionally, the method further includes:

[0270] The network side device receives the eighth information or the ninth information sent by the terminal device.

[0271] The eighth information includes at least one of the following:

[0272] a function identifier of the first AI function;

[0273] a third indication, where the third indication is used to indicate that the terminal device has activated the first AI function;

[0274] a fourth indication, where the fourth indication is used to indicate that the terminal device has activated the AI ​​model associated with the first function;

[0275] a model identifier of an activated AI model associated with the first AI function in the terminal device;

[0276] The ninth information includes at least one of the following:

[0277] a function identifier of the first AI function;

[0278] a fifth indication, where the fifth indication is used to instruct the terminal device to deactivate the first AI function;

[0279] a sixth indication, where the sixth indication is used to instruct the terminal device to deactivate the AI ​​model associated with the first AI function;

[0280] A model identifier of a deactivated AI model associated with the first AI function in the terminal device.

[0281] It should be noted that, in the embodiment of the present application, performing an activation operation on the first AI function includes activating the first AI function and / or activating at least one AI model among the AI ​​models associated with the first AI function. Performing a deactivation operation on the first AI function includes deactivating the first AI function and / or deactivating at least one AI model among the AI ​​models associated with the first AI function.

[0282] Exemplarily, if the terminal device activates the first AI function, the eighth information may include the function identifier and the third indication of the first AI function; if the terminal device activates at least one AI model among the AI ​​models associated with the first AI function, the eighth information may include the function identifier of the first AI function, the fourth indication, and the model identifier of the activated AI model associated with the first AI function in the terminal device.

[0283] If the terminal device deactivates the first AI function, the ninth information may include the function identifier of the first AI function and the fifth indication; if the terminal device deactivates at least one AI model among the AI ​​models associated with the first AI function, the ninth information may include the function identifier of the first AI function, the sixth indication, and the model identifier of the deactivated AI model associated with the first AI function in the terminal device.

[0284] In an embodiment of the present application, if an AI function is deactivated, all AI models associated with the AI ​​function are invalid models, or in other words, all AI models associated with the AI ​​function are deactivated; similarly, if an AI function is activated, all AI models associated with the AI ​​function are valid models, or in other words, all AI models associated with the AI ​​function are activated.

[0285] In an embodiment of the present application, the terminal device tells the network side device through the eighth information or the ninth information that the first AI function on the terminal side has undergone a state switch (including activation or deactivation), and the network side device should assume that the terminal device cannot use the first AI function within the state switching delay. Among them, the model activation delay and the model deactivation delay of the terminal device can be reported to the network side device by the terminal device through the terminal capability information. The network side device can suspend sending the reasoning task related to the first AI function to the terminal device within the state switching delay to avoid the terminal device being unable to execute the reasoning task related to the first AI function in time due to the state switch, resulting in a period of waiting for the reasoning result, affecting the overall processing efficiency.

[0286] In summary, the embodiments of the present application provide a performance supervision method for an AI function. The network side device can determine the validity of the first AI function in the terminal device and / or the accuracy of the inference result corresponding to the first AI function through the first information, so that the network side device can perform corresponding configuration according to the model performance of the AI ​​model associated with the first AI function.

[0287] The AI ​​function performance monitoring method provided in the embodiment of the present application can be executed by a performance monitoring device for the AI ​​function. In the embodiment of the present application, the AI ​​function performance monitoring device executing the model performance monitoring method is used as an example to illustrate the AI ​​function performance monitoring device provided in the embodiment of the present application.

[0288] The embodiment of the present application provides a performance monitoring device for AI functions. Referring to FIG6 , a block diagram of a performance monitoring device for AI functions provided by the embodiment of the present application is shown, which can be applied to a terminal device. As shown in FIG6 , the device may specifically include:

[0289] A performance monitoring module 301 is configured to monitor the performance of a first artificial intelligence (AI) function and determine first information; the first information is configured to indicate the effectiveness of the first AI function and / or the accuracy of a reasoning result corresponding to the first AI function.

[0290] A first sending module 302 is configured to send the first information to a network-side device;

[0291] The first AI function is associated with at least one AI model, and the AI ​​models include activated AI models and inactivated AI models.

[0292] Optionally, the first information includes at least one of the following:

[0293] a function identifier of the first AI function;

[0294] a validity flag of the first AI function, where the validity flag is used to indicate whether the first AI function is valid;

[0295] Second information, where the second information is used to indicate resource information associated with the first AI function;

[0296] A first indication is used to indicate the accuracy of the inference result corresponding to the first AI function.

[0297] Optionally, the second information includes at least one of the following:

[0298] the cell identifier associated with the first AI function;

[0299] an identifier of the sending / receiving point associated with the first AI function;

[0300] a reference signal identifier associated with the first AI function;

[0301] an identifier of a reference signal resource set associated with the first AI function;

[0302] The port number associated with the first AI function.

[0303] Optionally, the device further comprises:

[0304] a first determining module, configured to determine that the first AI function is valid if the AI ​​model associated with the first AI function satisfies a first condition;

[0305] The first condition includes at least one of the following:

[0306] The number of valid AI models among the M AI models associated with the first AI function is greater than or equal to a first threshold; M is a positive integer;

[0307] A ratio of the number of valid AI models in the M AI models associated with the first AI to M is greater than or equal to a second threshold.

[0308] Optionally, the device further comprises:

[0309] a second determining module, configured to determine that the first AI function is invalid if the AI ​​model associated with the first AI function satisfies a second condition;

[0310] The second condition includes at least one of the following:

[0311] The number of valid AI models among the M AI models associated with the first AI function is less than or equal to a third threshold; M is a positive integer;

[0312] A ratio of the number of valid AI models in the M AI models associated with the first AI to M is less than or equal to a fourth threshold.

[0313] Optionally, the device further comprises:

[0314] The second sending module is configured to send third information to the network-side device, where the third information includes at least one of the following:

[0315] the number of valid AI models among the AI ​​models associated with the first AI function;

[0316] a model identifier of a valid AI model among the AI ​​models associated with the first AI function;

[0317] the number of invalid AI models among the AI ​​models associated with the first AI function;

[0318] a model identifier of an invalid AI model among the AI ​​models associated with the first AI function;

[0319] a first identifier associated with the number of valid AI models among the AI ​​models associated with the first AI function;

[0320] a second identifier associated with a number of invalid AI models among the AI ​​models associated with the first AI function;

[0321] Fourth information, the fourth information being used to indicate resource information associated with the valid AI model;

[0322] Fifth information, the fifth information is used to indicate resource information associated with the invalid AI model.

[0323] Optionally, the fourth information includes at least one of the following:

[0324] The cell identifier associated with the valid AI model;

[0325] The identifier of the sending and receiving point associated with the valid AI model;

[0326] A reference signal identifier associated with the valid AI model;

[0327] An identifier of a reference signal resource set associated with the valid AI model;

[0328] The port number associated with the valid AI model.

[0329] Optionally, the fifth information includes at least one of the following:

[0330] The cell identifier associated with the invalid AI model;

[0331] The identifier of the sending and receiving point associated with the invalid AI model;

[0332] A reference signal identifier associated with the invalid AI model;

[0333] an identifier of a reference signal resource set associated with the invalid AI model;

[0334] The port number associated with the invalid AI model.

[0335] Optionally, the device further comprises:

[0336] a third determining module, configured to determine that the AI ​​model is a valid AI model if the AI ​​model satisfies a third condition;

[0337] The third condition includes at least one of the following:

[0338] The number of first instances for determining that the model is valid is greater than or equal to a fifth threshold within T consecutive time units; T is a positive integer;

[0339] Within T consecutive time units, the ratio of the first number of instances for determining that the model is valid to the total number of instances is greater than or equal to a sixth threshold;

[0340] In Y consecutive instances, the first number of instances in which the model is determined to be valid is greater than or equal to a seventh threshold; Y is a positive integer;

[0341] In Y consecutive instances, the ratio of the first number of instances in which the model is determined to be valid to the total number of instances is greater than or equal to an eighth threshold.

[0342] Optionally, the device further comprises:

[0343] a fourth determining module, configured to determine that the AI ​​model is an invalid AI model if the AI ​​model satisfies a fourth condition;

[0344] The fourth condition includes at least one of the following:

[0345] The number of first instances of determining that the model is valid is less than or equal to a ninth threshold value within T consecutive time units; T is a positive integer;

[0346] Within T consecutive time units, the ratio of the first number of instances that determine the model to be valid to the total number of instances is less than or equal to the tenth threshold;

[0347] In Y consecutive instances, the first number of instances in which the model is determined to be valid is less than or equal to an eleventh threshold; Y is a positive integer and greater than 1;

[0348] In Y consecutive instances, the ratio of the first number of instances in which the model is determined to be valid to the total number of instances is less than or equal to a twelfth threshold.

[0349] Optionally, the device further comprises:

[0350] an indication receiving module, configured to receive a second indication sent by the network side device, where the second indication is used to indicate the length of the performance monitoring time window and the number of second instances;

[0351] The terminal device determines a value of T according to the length of the performance monitoring time window, and determines a value of Y according to the second number of instances;

[0352] The value of T is greater than or equal to the length of the performance monitoring time window, and the value of Y is greater than or equal to the second instance number.

[0353] Optionally, the device further comprises:

[0354] A third sending module is configured to send sixth information to the network-side device, where the sixth information carries a time window length T and a number of instances Y used by the terminal device to perform performance supervision on the first AI function.

[0355] Optionally, the device further comprises:

[0356] A fourth sending module is configured to send terminal capability information to the network side device, where the terminal capability information includes at least one of the following:

[0357] The functional performance supervision capability of the AI ​​function level of the terminal device, where the functional performance supervision capability is used to indicate whether the terminal device supports performance supervision of the AI ​​function level;

[0358] The AI ​​model-level model performance supervision capability of the terminal device, where the model performance supervision capability is used to indicate whether the terminal device supports performing AI model-level performance supervision;

[0359] Model performance supervision method supported by the terminal device;

[0360] an indication of the reliability of the model performance supervision method supported by the terminal device;

[0361] The maximum time window length and maximum number of instances of model performance supervision supported by the terminal device;

[0362] The number of AI models supported by the terminal device to run simultaneously;

[0363] The number of AI models that the terminal device supports simultaneous supervision;

[0364] The delay of model activation and model deactivation of the terminal device;

[0365] The activation and deactivation delay of the first AI function of the terminal device;

[0366] The type of the validity indicator or accuracy indicator of the AI ​​function in the terminal device;

[0367] The granularity of the validity identifier or accuracy identifier of the AI ​​function in the terminal device, where the granularity is used to indicate the interval value between two consecutive numerical values ​​in the validity identifier or accuracy identifier.

[0368] Optionally, the device further comprises:

[0369] The fifth sending module is used to send seventh information to the network side device, where the seventh information is used to indicate whether the first model has been activated.

[0370] Optionally, the device further comprises:

[0371] a sixth sending module, configured to send eighth information to the network-side device when an activation operation is performed on the first AI function; or to send ninth information to the network-side device when a deactivation operation is performed on the first AI function;

[0372] The eighth information includes at least one of the following:

[0373] a function identifier of the first AI function;

[0374] a third indication, where the third indication is used to indicate that the terminal device has activated the first AI function;

[0375] a fourth indication, where the fourth indication is used to indicate that the terminal device has activated the AI ​​model associated with the first AI function;

[0376] a model identifier of an activated AI model associated with the first AI function in the terminal device;

[0377] The ninth information includes at least one of the following:

[0378] a function identifier of the first AI function;

[0379] a fifth indication, where the fifth indication is used to instruct the terminal device to deactivate the first AI function;

[0380] a sixth indication, where the sixth indication is used to instruct the terminal device to deactivate the AI ​​model associated with the first AI function;

[0381] A model identifier of a deactivated AI model associated with the first AI function in the terminal device.

[0382] The performance monitoring device for AI functions in the embodiments of the present application can be an electronic device, such as an electronic device with an operating system, or a component in an electronic device, such as an integrated circuit or chip. The electronic device can be a terminal device. For example, the terminal device can include, but is not limited to, the types of terminal devices 11 listed above.

[0383] The performance monitoring device for AI functions provided in the embodiment of the present application can implement the various processes implemented in the method embodiment of Figure 1 and achieve the same technical effects. To avoid repetition, they will not be described here.

[0384] The present application also provides another AI function performance monitoring device. Referring to Figure 7, a block diagram of a performance monitoring device for an AI function provided by the present application is shown. The device can be applied to a network-side device. As shown in Figure 7, the device may specifically include:

[0385] A first receiving module 401 is configured to receive first information sent by a terminal device, where the first information is used to indicate the validity of a first AI function and / or the accuracy of an inference result of an AI model corresponding to the first AI function;

[0386] The first AI function is associated with at least one AI model, and the AI ​​models include activated AI models and inactivated AI models.

[0387] Optionally, the first information includes at least one of the following:

[0388] a function identifier of the first AI function;

[0389] a validity flag of the first AI function, where the validity flag is used to indicate whether the first AI function is valid;

[0390] Second information, where the second information is used to indicate resource information associated with the first AI function;

[0391] A first indication is used to indicate the accuracy of an inference result of the AI ​​model corresponding to the first AI function.

[0392] Optionally, the second information includes at least one of the following:

[0393] the cell identifier associated with the first AI function;

[0394] an identifier of the sending / receiving point associated with the first AI function;

[0395] a reference signal identifier associated with the first AI function;

[0396] an identifier of a reference signal resource set associated with the first AI function;

[0397] The port number associated with the first AI function.

[0398] Optionally, the device further comprises:

[0399] The second receiving module is configured to receive third information sent by the terminal device, where the third information includes at least one of the following:

[0400] the number of valid AI models among the AI ​​models associated with the first AI function;

[0401] a model identifier of a valid AI model among the AI ​​models associated with the first AI function;

[0402] the number of invalid AI models among the AI ​​models associated with the first AI function;

[0403] a model identifier of an invalid AI model among the AI ​​models associated with the first AI function;

[0404] a first identifier associated with the number of valid AI models among the AI ​​models associated with the first AI function;

[0405] a second identifier associated with a number of invalid AI models among the AI ​​models associated with the first AI function;

[0406] Fourth information, the fourth information being used to indicate resource information associated with the valid AI model;

[0407] Fifth information, the fifth information is used to indicate resource information associated with the invalid AI model.

[0408] Optionally, the fourth information includes at least one of the following:

[0409] The cell identifier associated with the valid AI model;

[0410] The identifier of the sending and receiving point associated with the valid AI model;

[0411] A reference signal identifier associated with the valid AI model;

[0412] An identifier of a reference signal resource set associated with the valid AI model;

[0413] The port number associated with the valid AI model.

[0414] Optionally, the fifth information includes at least one of the following:

[0415] The cell identifier associated with the invalid AI model;

[0416] The identifier of the sending and receiving point associated with the invalid AI model;

[0417] A reference signal identifier associated with the invalid AI model;

[0418] an identifier of a reference signal resource set associated with the invalid AI model;

[0419] The port number associated with the invalid AI model.

[0420] Optionally, the device further comprises:

[0421] An indication sending module is used to send a second indication to the terminal device, where the second indication is used to indicate the length of the performance monitoring time window and the second number of instances.

[0422] Optionally, the device further comprises:

[0423] A third receiving module is configured to receive sixth information sent by the terminal device, where the sixth information carries a time window length T and a number of instances Y used by the terminal device to perform performance supervision on the first AI function.

[0424] Optionally, the device further comprises:

[0425] A fourth receiving module is configured to receive terminal capability information sent by the terminal device, where the terminal capability information includes at least one of the following:

[0426] The functional performance supervision capability of the AI ​​function level of the terminal device, where the functional performance supervision capability is used to indicate whether the terminal device supports performance supervision of the AI ​​function level;

[0427] The AI ​​model-level model performance supervision capability of the terminal device, where the model performance supervision capability is used to indicate whether the terminal device supports performing AI model-level performance supervision;

[0428] Model performance supervision method supported by the terminal device;

[0429] an indication of the reliability of the model performance supervision method supported by the terminal device;

[0430] The maximum time window length and maximum number of instances of model performance supervision supported by the terminal device;

[0431] The number of AI models supported by the terminal device to run simultaneously;

[0432] The number of AI models that the terminal device supports simultaneous supervision;

[0433] The delay of model activation and model deactivation of the terminal device;

[0434] The activation and deactivation delay of the first AI function of the terminal device;

[0435] The type of the validity indicator or accuracy indicator of the AI ​​function in the terminal device;

[0436] The granularity of the validity identifier or accuracy identifier of the AI ​​function in the terminal device, where the granularity is used to indicate the interval value between two consecutive numerical values ​​in the validity identifier or accuracy identifier.

[0437] Optionally, the device further comprises:

[0438] The fifth receiving module is used to receive seventh information sent by the terminal device, where the seventh information is used to indicate whether the first model has been activated.

[0439] Optionally, the device further comprises:

[0440] a sixth receiving module, configured to receive the eighth information or the ninth information sent by the terminal device;

[0441] The eighth information includes at least one of the following:

[0442] a function identifier of the first AI function;

[0443] a third indication, where the third indication is used to indicate that the terminal device has activated the first AI function;

[0444] a fourth indication, where the fourth indication is used to indicate that the terminal device has activated the AI ​​model associated with the first function;

[0445] a model identifier of an activated AI model associated with the first AI function in the terminal device;

[0446] The ninth information includes at least one of the following:

[0447] a function identifier of the first AI function;

[0448] a fifth indication, where the fifth indication is used to instruct the terminal device to deactivate the first AI function;

[0449] a sixth indication, where the sixth indication is used to instruct the terminal device to deactivate the AI ​​model associated with the first AI function;

[0450] A model identifier of a deactivated AI model associated with the first AI function in the terminal device.

[0451] The performance monitoring device for AI functions provided in the embodiment of the present application can implement the various processes implemented in the method embodiment of Figure 5 and achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0452] Optionally, as shown in FIG8 , an embodiment of the present application further provides a communication device 900, comprising a processor 901 and a memory 902, wherein the memory 902 stores programs or instructions that can be run on the processor 901. For example, when the communication device 900 is a terminal device, the program or instruction is executed by the processor 901 to implement the various steps of the embodiment of the performance supervision method of the AI ​​function described in the first aspect above, and can achieve the same technical effect. When the communication device 900 is a network-side device, the program or instruction is executed by the processor 901 to implement the various steps of the embodiment of the performance supervision method of the AI ​​function described in the second aspect above, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0453] As shown in FIG9 , it is a schematic diagram of the hardware structure of a terminal device implementing an embodiment of the present application.

[0454] The terminal device 1000 includes but is not limited to: a radio frequency unit 1001, a network module 1002, an audio output unit 1003, an input unit 1004, a sensor 1005, a display unit 1006, a user input unit 1007, an interface unit 1008, a memory 1009 and at least some of the components of the processor 1010.

[0455] Those skilled in the art will appreciate that the terminal device 1000 may further include a power source (such as a battery) to power various components. The power source may be logically connected to the processor 1010 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The terminal device structure shown in FIG9 does not limit the terminal device. The terminal device may include more or fewer components than shown, or may combine certain components, or have different component arrangements, which will not be described in detail here.

[0456] It should be understood that in an embodiment of the present application, the input unit 1004 may include a graphics processing unit (GPU) 10041 and a microphone 10042, and the graphics processor 10041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 1006 may include a display panel 10061, and the display panel 10061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1007 includes a touch panel 10071 and at least one of other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include two parts: a touch detection device and a touch controller. Other input devices 10072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and an operating stick, which will not be repeated here.

[0457] In the embodiment of the present application, after receiving downlink data from a network-side device, the RF unit 1001 may transmit the data to the processor 1010 for processing. Furthermore, the RF unit 1001 may send uplink data to the network-side device. Typically, the RF unit 1001 includes, but is not limited to, an antenna, an amplifier, a transceiver, a coupler, a low-noise amplifier, a duplexer, and the like.

[0458] The memory 1009 can be used to store software programs or instructions and various data. The memory 1009 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, applications or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 1009 may include a volatile memory or a non-volatile memory, or the memory 1009 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM). The memory 1009 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0459] Processor 1010 may include one or more processing units. Optionally, processor 1010 integrates an application processor and a modem processor. The application processor primarily handles operations related to the operating system, user interface, and application programs, while the modem processor primarily processes wireless communication signals, such as a baseband processor. It is understood that the modem processor may not be integrated into processor 1010.

[0460] The processor 1010 is configured to perform performance monitoring on a first artificial intelligence (AI) function and determine first information; the first information is configured to indicate the effectiveness of the first AI function and / or the accuracy of a reasoning result corresponding to the first AI function.

[0461] The radio frequency unit 1001 is configured to send the first information to the network side device;

[0462] The first AI function is associated with at least one AI model, and the AI ​​models include activated AI models and inactivated AI models.

[0463] Optionally, the first information includes at least one of the following:

[0464] a function identifier of the first AI function;

[0465] a validity flag of the first AI function, where the validity flag is used to indicate whether the first AI function is valid;

[0466] Second information, where the second information is used to indicate resource information associated with the first AI function;

[0467] A first indication is used to indicate the accuracy of the inference result corresponding to the first AI function.

[0468] Optionally, the second information includes at least one of the following:

[0469] the cell identifier associated with the first AI function;

[0470] an identifier of the sending / receiving point associated with the first AI function;

[0471] a reference signal identifier associated with the first AI function;

[0472] an identifier of a reference signal resource set associated with the first AI function;

[0473] The port number associated with the first AI function.

[0474] Optionally, the processor 1010 is further configured to determine that the first AI function is valid if the AI ​​model associated with the first AI function meets a first condition;

[0475] The first condition includes at least one of the following:

[0476] The number of valid AI models among the M AI models associated with the first AI function is greater than or equal to a first threshold; M is a positive integer;

[0477] A ratio of the number of valid AI models in the M AI models associated with the first AI to M is greater than or equal to a second threshold.

[0478] Optionally, the processor 1010 is further configured to determine that the first AI function is invalid if the AI ​​model associated with the first AI function meets a second condition;

[0479] The second condition includes at least one of the following:

[0480] The number of valid AI models among the M AI models associated with the first AI function is less than or equal to a third threshold; M is a positive integer;

[0481] A ratio of the number of valid AI models in the M AI models associated with the first AI to M is less than or equal to a fourth threshold.

[0482] Optionally, the radio frequency unit 1001 is further configured to send third information to the network side device, where the third information includes at least one of the following:

[0483] the number of valid AI models among the AI ​​models associated with the first AI function;

[0484] a model identifier of a valid AI model among the AI ​​models associated with the first AI function;

[0485] the number of invalid AI models among the AI ​​models associated with the first AI function;

[0486] a model identifier of an invalid AI model among the AI ​​models associated with the first AI function;

[0487] a first identifier associated with the number of valid AI models among the AI ​​models associated with the first AI function;

[0488] a second identifier associated with a number of invalid AI models among the AI ​​models associated with the first AI function;

[0489] Fourth information, the fourth information being used to indicate resource information associated with the valid AI model;

[0490] Fifth information, the fifth information is used to indicate resource information associated with the invalid AI model.

[0491] Optionally, the fourth information includes at least one of the following:

[0492] The cell identifier associated with the valid AI model;

[0493] The identifier of the sending and receiving point associated with the valid AI model;

[0494] A reference signal identifier associated with the valid AI model;

[0495] An identifier of a reference signal resource set associated with the valid AI model;

[0496] The port number associated with the valid AI model.

[0497] Optionally, the fifth information includes at least one of the following:

[0498] The cell identifier associated with the invalid AI model;

[0499] The identifier of the sending and receiving point associated with the invalid AI model;

[0500] A reference signal identifier associated with the invalid AI model;

[0501] an identifier of a reference signal resource set associated with the invalid AI model;

[0502] The port number associated with the invalid AI model.

[0503] Optionally, the processor 1010 is further configured to determine that the AI ​​model is a valid AI model when the AI ​​model satisfies a third condition;

[0504] The third condition includes at least one of the following:

[0505] The number of first instances for determining that the model is valid is greater than or equal to a fifth threshold within T consecutive time units; T is a positive integer;

[0506] Within T consecutive time units, the ratio of the first number of instances for determining that the model is valid to the total number of instances is greater than or equal to a sixth threshold;

[0507] In Y consecutive instances, the first number of instances in which the model is determined to be valid is greater than or equal to a seventh threshold; Y is a positive integer;

[0508] In Y consecutive instances, the ratio of the first number of instances in which the model is determined to be valid to the total number of instances is greater than or equal to an eighth threshold.

[0509] Optionally, the processor 1010 is further configured to, when the AI ​​model satisfies a fourth condition, determine that the AI ​​model is an invalid AI model;

[0510] The fourth condition includes at least one of the following:

[0511] The number of first instances of determining that the model is valid is less than or equal to a ninth threshold value within T consecutive time units; T is a positive integer;

[0512] Within T consecutive time units, the ratio of the first number of instances that determine the model to be valid to the total number of instances is less than or equal to the tenth threshold;

[0513] In Y consecutive instances, the first number of instances in which the model is determined to be valid is less than or equal to an eleventh threshold; Y is a positive integer and greater than 1;

[0514] In Y consecutive instances, the ratio of the first number of instances in which the model is determined to be valid to the total number of instances is less than or equal to a twelfth threshold.

[0515] Optionally, the radio frequency unit 1001 is further configured to receive a second indication sent by the network side device, where the second indication is used to indicate the length of the performance monitoring time window and the number of second instances;

[0516] The processor 1010 is further configured to determine a value of T according to the length of the performance monitoring time window, and determine a value of Y according to the second number of instances;

[0517] The value of T is greater than or equal to the length of the performance monitoring time window, and the value of Y is greater than or equal to the second instance number.

[0518] Optionally, the radio frequency unit 1001 is further used to send sixth information to the network side device, where the sixth information carries the time window length T and number of instances Y used by the terminal device to perform performance supervision on the first AI function.

[0519] Optionally, the radio frequency unit 1001 is further configured to send terminal capability information to the network side device, where the terminal capability information includes at least one of the following:

[0520] The functional performance supervision capability of the AI ​​function level of the terminal device, where the functional performance supervision capability is used to indicate whether the terminal device supports performance supervision of the AI ​​function level;

[0521] The AI ​​model-level model performance supervision capability of the terminal device, where the model performance supervision capability is used to indicate whether the terminal device supports performing AI model-level performance supervision;

[0522] Model performance supervision method supported by the terminal device;

[0523] an indication of the reliability of the model performance supervision method supported by the terminal device;

[0524] The maximum time window length and maximum number of instances of model performance supervision supported by the terminal device;

[0525] The number of AI models supported by the terminal device to run simultaneously;

[0526] The number of AI models that the terminal device supports simultaneous supervision;

[0527] The delay of model activation and model deactivation of the terminal device;

[0528] The activation and deactivation delay of the first AI function of the terminal device;

[0529] The type of the validity indicator or accuracy indicator of the AI ​​function in the terminal device;

[0530] The granularity of the validity identifier or accuracy identifier of the AI ​​function in the terminal device, where the granularity is used to indicate the interval value between two consecutive numerical values ​​in the validity identifier or accuracy identifier.

[0531] Optionally, the radio frequency unit 1001 is further configured to send seventh information to the network side device, where the seventh information is used to indicate whether the first model has been activated.

[0532] Optionally, the radio frequency unit 1001 is further configured to send eighth information to the network side device when an activation operation is performed on the first AI function; or send ninth information to the network side device when a deactivation operation is performed on the first AI function;

[0533] The eighth information includes at least one of the following:

[0534] a function identifier of the first AI function;

[0535] a third indication, where the third indication is used to indicate that the terminal device has activated the first AI function;

[0536] a fourth indication, where the fourth indication is used to indicate that the terminal device has activated the AI ​​model associated with the first AI function;

[0537] a model identifier of an activated AI model associated with the first AI function in the terminal device;

[0538] The ninth information includes at least one of the following:

[0539] a function identifier of the first AI function;

[0540] a fifth indication, where the fifth indication is used to instruct the terminal device to deactivate the first AI function;

[0541] a sixth indication, where the sixth indication is used to instruct the terminal device to deactivate the AI ​​model associated with the first AI function;

[0542] A model identifier of a deactivated AI model associated with the first AI function in the terminal device.

[0543] The present application also provides a network-side device, including a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is configured to execute a program or instruction to implement the steps of the method embodiment shown in FIG5 . This network-side device embodiment corresponds to the aforementioned network-side device method embodiment, and each implementation process and implementation method of the aforementioned method embodiment are applicable to this network-side device embodiment and can achieve the same technical effects.

[0544] Specifically, an embodiment of the present application further provides a network-side device. As shown in FIG10 , the network-side device 1100 includes an antenna 111, a radio frequency device 112, a baseband device 113, a processor 114, and a memory 115. Antenna 111 is connected to radio frequency device 112. In the uplink direction, radio frequency device 112 receives information via antenna 111 and sends the received information to baseband device 113 for processing. In the downlink direction, baseband device 113 processes the information to be transmitted and sends it to radio frequency device 112. Radio frequency device 112 processes the received information and then sends it through antenna 111.

[0545] The method executed by the network-side device in the above embodiment may be implemented in the baseband device 113 , which includes a baseband processor.

[0546] The baseband device 113 may include, for example, at least one baseband board, on which multiple chips are arranged, as shown in Figure 10, one of which is, for example, a baseband processor, which is connected to the memory 115 through a bus interface to call the program in the memory 115 and execute the network device operations shown in the above method embodiment.

[0547] The network side device may further include a network interface 116, which is, for example, a common public radio interface (CPRI).

[0548] Specifically, the network side device 1100 of an embodiment of the present invention also includes: instructions or programs stored in the memory 115 and executable on the processor 114. The processor 114 calls the instructions or programs in the memory 115 to execute the methods executed by the modules shown in FIG7 and achieve the same technical effect. To avoid repetition, they will not be elaborated here.

[0549] The embodiment of the present application further provides a network side device. As shown in FIG11 , the network side device 1200 includes: a processor 1201, a network interface 1202, and a memory 1203. The network interface 1202 is, for example, a common public radio interface (CPRI).

[0550] Specifically, the network side device 1200 of an embodiment of the present invention also includes: instructions or programs stored in the memory 1203 and executable on the processor 1201. The processor 1201 calls the instructions or programs in the memory 1203 to execute the methods executed by the modules shown in FIG7 and achieve the same technical effect. To avoid repetition, they will not be elaborated here.

[0551] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the performance supervision method embodiment of the above-mentioned AI function are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0552] The processor is the processor in the terminal device described in the above embodiment. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0553] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the embodiment of the performance supervision method of the above-mentioned AI function, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0554] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0555] An embodiment of the present application further provides a computer program / program product, which is stored in a storage medium. The computer program / program product is executed by at least one processor to implement the various processes of the performance supervision method embodiment of the above-mentioned AI function, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0556] An embodiment of the present application also provides a performance monitoring system for an AI function, including: a terminal device and a network side device, wherein the terminal device can be used to execute the steps of the performance monitoring method for the AI ​​function as described in the first aspect above, and the network side device can be used to execute the steps of the performance monitoring method for the AI ​​function as described in the second aspect above.

[0557] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0558] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0559] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0560] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0561] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0562] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0563] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0564] It is understood that the embodiments described in the embodiments of the present disclosure may be implemented using hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the modules, units, and sub-units may be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, or other electronic units or combinations thereof for performing the functions described in the present disclosure.

[0565] For software implementation, the techniques described in the embodiments of the present disclosure can be implemented by modules (e.g., procedures, functions, etc.) that perform the functions described in the embodiments of the present disclosure. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

[0566] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A method for supervising the performance of an AI function, wherein: The method comprises: The terminal device performs performance monitoring on the first artificial intelligence AI function to determine first information; the first information is used to indicate the effectiveness of the first AI function and / or the accuracy of the reasoning result corresponding to the first AI function; The terminal device sends the first information to the network side device; The first AI function is associated with at least one AI model, and the AI ​​model includes an activated AI model and an inactivated AI model.

2. The method according to claim 1, wherein: The first information includes at least one of the following: a function identifier of the first AI function; a validity flag of the first AI function, where the validity flag is used to indicate whether the first AI function is valid; second information, where the second information is used to indicate resource information associated with the first AI function; A first indication, where the first indication is used to indicate the accuracy of the reasoning result corresponding to the first AI function.

3. The method according to claim 2, wherein: The second information includes at least one of the following: a cell identifier associated with the first AI function; An identifier of a sending / receiving point associated with the first AI function; a reference signal identifier associated with the first AI function; an identifier of a reference signal resource set associated with the first AI function; The port number associated with the first AI function.

4. The method according to claim 1, wherein: The method further comprises: When the AI ​​model associated with the first AI function satisfies a first condition, the terminal device determines that the first AI function is valid; The first condition includes at least one of the following: The number of valid AI models among the M AI models associated with the first AI function is greater than or equal to a first threshold; M is a positive integer; A ratio of the number of valid AI models among the M AI models associated with the first AI function to M is greater than or equal to a second threshold.

5. The method according to claim 1, wherein: The method further comprises: When the AI ​​model associated with the first AI function satisfies a second condition, the terminal device determines that the first AI function is invalid; The second condition includes at least one of the following: The number of valid AI models among the M AI models associated with the first AI function is less than or equal to a third threshold; M is a positive integer; A ratio of the number of valid AI models among the M AI models associated with the first AI to M is less than or equal to a fourth threshold.

6. The method according to claim 1, wherein: The method further comprises: The terminal device sends third information to the network side device, where the third information includes at least one of the following: the number of valid AI models among the AI ​​models associated with the first AI function; a model identifier of a valid AI model among the AI ​​models associated with the first AI function; the number of invalid AI models among the AI ​​models associated with the first AI function; a model identifier of an invalid AI model among the AI ​​models associated with the first AI function; a first identifier, where the first identifier is associated with the number of valid AI models in the AI ​​model associated with the first AI function; a second identifier, where the second identifier is associated with the number of invalid AI models among the AI ​​models associated with the first AI function; Fourth information, the fourth information is used to indicate resource information associated with the valid AI model; The fifth information is used to indicate resource information associated with the invalid AI model.

7. The method according to claim 6, wherein: The fourth information includes at least one of the following: The cell identifier associated with the valid AI model; The identifier of the sending and receiving point associated with the valid AI model; A reference signal identifier associated with the valid AI model; An identifier of a reference signal resource set associated with the valid AI model; The port number associated with the valid AI model; The fifth information includes at least one of the following: The cell identifier associated with the invalid AI model; The identifier of the sending and receiving point associated with the invalid AI model; A reference signal identifier associated with the invalid AI model; An identifier of a reference signal resource set associated with the invalid AI model; The port number associated with the invalid AI model.

8. The method according to any one of claims 4 to 7, wherein: The method further comprises: The terminal device determines, when the AI ​​model satisfies a third condition, that the AI ​​model is a valid AI model; When the AI ​​model satisfies a fourth condition, the terminal device determines that the AI ​​model is an invalid AI model; The third condition includes at least one of the following: Within T consecutive time units, the number of first instances for determining that the model is valid is greater than or equal to a fifth threshold; T is a positive integer; Within T consecutive time units, the ratio of the first number of instances for determining that the model is valid to the total number of instances is greater than or equal to a sixth threshold; In Y consecutive instances, the first number of instances in which the model is determined to be valid is greater than or equal to a seventh threshold; Y is a positive integer; In Y consecutive instances, the ratio of the first number of instances for which the model is determined to be effective to the total number of instances is greater than or equal to an eighth threshold; The fourth condition includes at least one of the following: Within T consecutive time units, the number of first instances for determining that the model is valid is less than or equal to a ninth threshold; T is a positive integer; Within T consecutive time units, the ratio of the first number of instances for which the model is determined to be effective to the total number of instances is less than or equal to the tenth threshold; In Y consecutive instances, the number of first instances for which the model is determined to be valid is less than or equal to an eleventh threshold; Y is a positive integer, and Y is greater than 1; In Y consecutive instances, the ratio of the first number of instances in which the model is determined to be valid to the total number of instances is less than or equal to a twelfth threshold.

9. The method according to claim 8, wherein: The method further comprises: The terminal device receives a second indication sent by the network side device, where the second indication is used to indicate the length of the performance monitoring time window and the number of second instances; The terminal device determines a value of T according to the length of the performance monitoring time window, and determines a value of Y according to the second number of instances; Among them, the value of T is greater than or equal to the length of the performance monitoring time window, and the value of Y is greater than or equal to the second instance number.

10. The method according to claim 8 or 9, wherein: The method further comprises: The terminal device sends sixth information to the network side device, where the sixth information carries the time window length T and the number of instances Y used by the terminal device to perform performance supervision on the first AI function.

11. The method according to any one of claims 1 to 10, wherein: The method further comprises: The terminal capability information sent by the terminal device to the network side device includes at least one of the following: The functional performance supervision capability of the AI ​​function level of the terminal device, where the functional performance supervision capability is used to indicate whether the terminal device supports the performance supervision of the AI ​​function level; The AI ​​model-level model performance supervision capability of the terminal device, wherein the model performance supervision capability is used to indicate whether the terminal device supports the execution of AI model-level performance supervision; A model performance supervision method supported by the terminal device; an indication of the reliability of a model performance supervision method supported by the terminal device; The maximum time window length and maximum number of instances of model performance supervision supported by the terminal device; The number of AI models supported by the terminal device to run simultaneously; The number of AI models that the terminal device supports to be supervised simultaneously; The delay of model activation and model deactivation of the terminal device; The activation and deactivation delay of the first AI function of the terminal device; The type of the identification of the validity identification or accuracy identification of the AI ​​function in the terminal device; The granularity of the validity mark or accuracy mark of the AI ​​function in the terminal device, and the granularity is used to indicate the interval value between two consecutive numerical values ​​in the validity mark or accuracy mark.

12. The method according to claim 1, wherein: The method further comprises: The terminal device sends seventh information to the network side device, where the seventh information is used to indicate whether the first model has been activated.

13. The method according to any one of claims 1 to 12, wherein: The method further comprises: The terminal device sends eighth information to the network side device when performing an activation operation on the first AI function; or When the terminal device performs a deactivation operation on the first AI function, the terminal device sends ninth information to the network side device; The eighth information includes at least one of the following: a function identifier of the first AI function; a third indication, where the third indication is used to indicate that the terminal device has activated the first AI function; a fourth indication, where the fourth indication is used to indicate that the terminal device has activated an AI model associated with the first AI function; A model identifier of an activated AI model associated with the first AI function in the terminal device; The ninth information includes at least one of the following: a function identifier of the first AI function; a fifth indication, where the fifth indication is used to indicate that the terminal device has deactivated the first AI function; a sixth indication, where the sixth indication is used to indicate that the terminal device has deactivated the AI ​​model associated with the first AI function; A model identifier of a deactivated AI model associated with the first AI function in the terminal device.

14. A method for supervising the performance of an AI function, wherein: The method comprises: The network side device receives first information sent by the terminal device, where the first information is used to indicate the validity of the first AI function and / or the accuracy of the inference result of the AI ​​model corresponding to the first AI function; The first AI function is associated with at least one AI model, and the AI ​​model includes an activated AI model and an inactivated AI model.

15. The method according to claim 14, wherein: The first information includes at least one of the following: a function identifier of the first AI function; a validity flag of the first AI function, where the validity flag is used to indicate whether the first AI function is valid; second information, where the second information is used to indicate resource information associated with the first AI function; A first indication, where the first indication is used to indicate the accuracy of the inference result of the AI ​​model corresponding to the first AI function.

16. The method according to claim 15, wherein: The second information includes at least one of the following: a cell identifier associated with the first AI function; An identifier of a sending / receiving point associated with the first AI function; a reference signal identifier associated with the first AI function; an identifier of a reference signal resource set associated with the first AI function; The port number associated with the first AI function.

17. The method according to claim 14, wherein: The method further comprises: The network side device receives third information sent by the terminal device, where the third information includes at least one of the following: the number of valid AI models among the AI ​​models associated with the first AI function; a model identifier of a valid AI model among the AI ​​models associated with the first AI function; the number of invalid AI models among the AI ​​models associated with the first AI function; a model identifier of an invalid AI model among the AI ​​models associated with the first AI function; a first identifier, where the first identifier is associated with the number of valid AI models in the AI ​​model associated with the first AI function; a second identifier, where the second identifier is associated with the number of invalid AI models among the AI ​​models associated with the first AI function; Fourth information, the fourth information is used to indicate resource information associated with the valid AI model; The fifth information is used to indicate resource information associated with the invalid AI model.

18. The method according to claim 17, wherein: The fourth information includes at least one of the following: The cell identifier associated with the valid AI model; The identifier of the sending and receiving point associated with the valid AI model; A reference signal identifier associated with the valid AI model; An identifier of a reference signal resource set associated with the valid AI model; The port number associated with the valid AI model; The fifth information includes at least one of the following: The cell identifier associated with the invalid AI model; The identifier of the sending and receiving point associated with the invalid AI model; A reference signal identifier associated with the invalid AI model; An identifier of a reference signal resource set associated with the invalid AI model; The port number associated with the invalid AI model.

19. The method according to claim 14, wherein: The method further comprises: The network side device sends a second indication to the terminal device, where the second indication is used to indicate the length of the performance monitoring time window and the second instance number.

20. The method according to claim 14, wherein: The method further comprises: The network side device receives the sixth information sent by the terminal device, where the sixth information carries the time window length T and the number of instances Y used by the terminal device to perform performance supervision on the first AI function.

21. The method according to any one of claims 14 to 20, wherein: The method further comprises: The network side device receives the terminal capability information sent by the terminal device, where the terminal capability information includes at least one of the following: The functional performance supervision capability of the AI ​​function level of the terminal device, where the functional performance supervision capability is used to indicate whether the terminal device supports the performance supervision of the AI ​​function level; The AI ​​model-level model performance supervision capability of the terminal device, wherein the model performance supervision capability is used to indicate whether the terminal device supports the execution of AI model-level performance supervision; A model performance supervision method supported by the terminal device; an indication of the reliability of a model performance supervision method supported by the terminal device; The maximum time window length and maximum number of instances of model performance supervision supported by the terminal device; The number of AI models supported by the terminal device to run simultaneously; The number of AI models that the terminal device supports to be supervised simultaneously; The delay of model activation and model deactivation of the terminal device; The activation and deactivation delay of the first AI function of the terminal device; The type of the identification of the validity identification or accuracy identification of the AI ​​function in the terminal device; The granularity of the validity mark or accuracy mark of the AI ​​function in the terminal device, and the granularity is used to indicate the interval value between two consecutive numerical values ​​in the validity mark or accuracy mark.

22. The method according to claim 14, wherein: The method further comprises: The network side device receives seventh information sent by the terminal device, where the seventh information is used to indicate whether the first model has been activated.

23. The method according to any one of claims 14 to 22, wherein: The method further comprises: The network side device receives the eighth information or the ninth information sent by the terminal device; The eighth information includes at least one of the following: a function identifier of the first AI function; a third indication, where the third indication is used to indicate that the terminal device has activated the first AI function; a fourth indication, where the fourth indication is used to indicate that the terminal device has activated an AI model associated with the first function; A model identifier of an activated AI model associated with the first AI function in the terminal device; The ninth information includes at least one of the following: a function identifier of the first AI function; a fifth indication, where the fifth indication is used to indicate that the terminal device has deactivated the first AI function; a sixth indication, where the sixth indication is used to indicate that the terminal device has deactivated the AI ​​model associated with the first AI function; A model identifier of a deactivated AI model associated with the first AI function in the terminal device.

24. A performance monitoring device for an AI function, wherein: Applied to a terminal device, the device comprises: A performance monitoring module, configured to perform performance monitoring on a first artificial intelligence (AI) function and determine first information; the first information is used to indicate the effectiveness of the first AI function and / or the accuracy of a reasoning result corresponding to the first AI function; A first sending module, used to send the first information to a network side device; The first AI function is associated with at least one AI model, and the AI ​​model includes an activated AI model and an inactivated AI model.

25. The device according to claim 24, wherein: The first information includes at least one of the following: a function identifier of the first AI function; a validity flag of the first AI function, where the validity flag is used to indicate whether the first AI function is valid; second information, where the second information is used to indicate resource information associated with the first AI function; a first indication, where the first indication is used to indicate the accuracy of a reasoning result corresponding to the first AI function; The second information includes at least one of the following: a cell identifier associated with the first AI function; An identifier of a sending / receiving point associated with the first AI function; a reference signal identifier associated with the first AI function; an identifier of a reference signal resource set associated with the first AI function; The port number associated with the first AI function.

26. The device according to claim 24, wherein: The device also includes: A first determining module is configured to determine that the first AI function is valid when the AI ​​model associated with the first AI function satisfies a first condition; or a second determining module, configured to determine that the first AI function is invalid if the AI ​​model associated with the first AI function satisfies a second condition; The first condition includes at least one of the following: The number of valid AI models among the M AI models associated with the first AI function is greater than or equal to a first threshold; M is a positive integer; The ratio of the number of valid AI models among the M AI models associated with the first AI to M is greater than or equal to a second threshold; The second condition includes at least one of the following: The number of valid AI models among the M AI models associated with the first AI function is less than or equal to a third threshold; M is a positive integer; A ratio of the number of valid AI models among the M AI models associated with the first AI to M is less than or equal to a fourth threshold.

27. The device according to claim 24, wherein: The device also includes: The second sending module is configured to send third information to the network side device, where the third information includes at least one of the following: the number of valid AI models among the AI ​​models associated with the first AI function; a model identifier of a valid AI model among the AI ​​models associated with the first AI function; the number of invalid AI models among the AI ​​models associated with the first AI function; a model identifier of an invalid AI model among the AI ​​models associated with the first AI function; a first identifier, where the first identifier is associated with the number of valid AI models in the AI ​​model associated with the first AI function; a second identifier, where the second identifier is associated with the number of invalid AI models among the AI ​​models associated with the first AI function; Fourth information, the fourth information is used to indicate resource information associated with the valid AI model; The fifth information is used to indicate resource information associated with the invalid AI model.

28. The device according to claim 27, wherein The device also includes: A third determination module is configured to determine that the AI ​​model is a valid AI model if the AI ​​model satisfies a third condition; or a fourth determining module, configured to determine that the AI ​​model is an invalid AI model if the AI ​​model satisfies a fourth condition; The third condition includes at least one of the following: Within T consecutive time units, the number of first instances for determining that the model is valid is greater than or equal to a fifth threshold; T is a positive integer; Within T consecutive time units, the ratio of the first number of instances for determining that the model is valid to the total number of instances is greater than or equal to a sixth threshold; In Y consecutive instances, the first number of instances in which the model is determined to be valid is greater than or equal to a seventh threshold; Y is a positive integer; In Y consecutive instances, the ratio of the first number of instances for which the model is determined to be effective to the total number of instances is greater than or equal to an eighth threshold; The fourth condition includes at least one of the following: Within T consecutive time units, the number of first instances for determining that the model is valid is less than or equal to a ninth threshold; T is a positive integer; Within T consecutive time units, the ratio of the first number of instances for which the model is determined to be effective to the total number of instances is less than or equal to the tenth threshold; In Y consecutive instances, the number of first instances for which the model is determined to be valid is less than or equal to an eleventh threshold; Y is a positive integer, and Y is greater than 1; In Y consecutive instances, the ratio of the first number of instances in which the model is determined to be valid to the total number of instances is less than or equal to a twelfth threshold.

29. A performance monitoring device for an AI function, wherein: Applied to a network side device, the device comprises: A first receiving module, configured to receive first information sent by a terminal device, where the first information is used to indicate the validity of a first AI function and / or the accuracy of an inference result of an AI model corresponding to the first AI function; The first AI function is associated with at least one AI model, and the AI ​​model includes an activated AI model and an inactivated AI model.

30. The device according to claim 29, wherein: The first information includes at least one of the following: a function identifier of the first AI function; a validity flag of the first AI function, where the validity flag is used to indicate whether the first AI function is valid; second information, where the second information is used to indicate resource information associated with the first AI function; a first indication, where the first indication is used to indicate the accuracy of a reasoning result of the AI ​​model corresponding to the first AI function; The second information includes at least one of the following: a cell identifier associated with the first AI function; An identifier of a sending / receiving point associated with the first AI function; a reference signal identifier associated with the first AI function; an identifier of a reference signal resource set associated with the first AI function; The port number associated with the first AI function.

31. The device according to claim 29, wherein: The device also includes: An indication sending module is used to send a second indication to the terminal device, where the second indication is used to indicate the length of the performance monitoring time window and the second instance number.

32. The device according to claim 29, wherein: The device also includes: The third receiving module is used to receive sixth information sent by the terminal device, where the sixth information carries the time window length T and the number of instances Y used by the terminal device to perform performance supervision on the first AI function.

33. A terminal device, wherein: It includes a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the performance supervision method of the AI ​​function as described in any one of claims 1 to 13 are implemented.

34. A network side device, wherein: It includes a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the performance supervision method of the AI ​​function as described in any one of claims 14 to 23 are implemented.

35. A readable storage medium, wherein: The readable storage medium stores a program or instruction, which, when executed by a processor, implements the model performance supervision method as described in any one of claims 1 to 13, or implements the steps of the performance supervision method of the AI ​​function as described in any one of claims 14 to 23.

Images