Ai function or ai model performance evaluation method, and device

Abstract

Provided in the embodiments of the present disclosure are an AI function or AI model performance evaluation method, and a device. The method comprises: on the basis of AI function-related first information from a network device, determining a performance evaluation result of an AI function or of an AI model in the AI function; and sending second information to the network device, the second information comprising at least one of the following: the performance evaluation result of the AI function, the performance evaluation result of the AI model in the AI function and first indication information, the first indication information being used for indicating whether the AI function or the AI model in the AI function is available or whether the performance thereof satisfies performance requirements, and the first indication information being determined on the basis of the performance evaluation result of the AI function or of the AI model in the AI function.

Description

Performance evaluation method and device for AI function or AI model

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese patent application No. 202410369122.1, filed on March 28, 2024, entitled “Performance evaluation method and device for AI function or AI model”, which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates to the field of communication technologies, and in particular to a method and device for evaluating the performance of an AI function or AI model. Background Art

[0004] With the advancement of science and technology, artificial intelligence (AI) and machine learning (ML) are increasingly being used. As data-driven algorithms, AI / ML suffer from generalization issues. This means that an AI / ML model trained in scenario 1 may be difficult to use in scenario 2. For example, in AI / ML-based beam management, if the transmit beam codebook corresponding to the reference signal used for model inference is inconsistent with the transmit beam codebook used during model training, model inference performance may be poor. The transmit beam codebook can be considered a network-side condition or additional condition.

[0005] If the AI / ML model is deployed on the network side, the network side can ensure the consistency of network-side conditions / additional conditions during the model training phase and the model inference phase. However, if the AI / ML model is deployed on the terminal side, when the network-side conditions / additional conditions during the training phase and the model inference phase are inconsistent, the AI / ML model or AI / ML function on the terminal side may be unavailable. Therefore, for those skilled in the art, how to determine whether the AI / ML model or AI / ML function on the terminal side is available is a technical problem that needs to be solved. Summary of the Invention

[0006] In response to the problems of related technologies, the embodiments of the present disclosure provide a method and device for performance evaluation (or performance monitoring) of AI functions or AI models.

[0007] In a first aspect, an embodiment of the present disclosure provides a performance evaluation method for an AI function or AI model, applied to a terminal, the method comprising:

[0008] Determining, based on first information related to the AI ​​function from the network device, a performance evaluation result of the AI ​​function or an AI model in the AI ​​function;

[0009] Sending second information to the network device, the second information including at least one of the following: a performance evaluation result of the AI ​​function; a performance evaluation result of the AI ​​model in the AI ​​function; first indication information; the first indication information is used to indicate whether the AI ​​function or the AI ​​model in the AI ​​function is available or whether the performance meets the performance requirements; the first indication information is determined based on the performance evaluation result of the AI ​​function or the AI ​​model in the AI ​​function.

[0010] In some embodiments, according to the performance evaluation method of an AI function or AI model according to one embodiment of the present disclosure, the first information includes at least one of the following:

[0011] Configuration information for performance evaluation;

[0012] performance requirements;

[0013] Dataset information used for performance evaluation.

[0014] In some embodiments, according to the performance evaluation method of an AI function or AI model according to one embodiment of the present disclosure, the configuration information for performance evaluation includes at least one of the following:

[0015] The identification of the AI ​​function;

[0016] An identifier of the radio resource control (RRC) configuration information corresponding to the AI ​​function;

[0017] The identifier of the AI ​​model in the AI ​​function;

[0018] An identifier of the RRC configuration information corresponding to the AI ​​model in the AI ​​function;

[0019] Reference signals for performance evaluation;

[0020] Configuration information for reporting information;

[0021] performance metrics for performance evaluation;

[0022] Reporting metrics for performance evaluation;

[0023] performance requirements;

[0024] Performance threshold.

[0025] In some embodiments, according to the performance evaluation method of an AI function or AI model according to one embodiment of the present disclosure, the dataset information includes at least one of the following:

[0026] Second indication information, used to indicate that the data set is used for performance evaluation;

[0027] The identification of the AI ​​function;

[0028] An identifier of the RRC configuration information corresponding to the AI ​​function;

[0029] Input data for the AI ​​function;

[0030] The identifier of the AI ​​model in the AI ​​function;

[0031] An identifier of the RRC configuration information corresponding to the AI ​​model in the AI ​​function;

[0032] Input data of the AI ​​model in the AI ​​function;

[0033] Reference information or true value information used to calculate performance indicators;

[0034] The number of samples used for performance evaluation;

[0035] performance metrics for performance evaluation;

[0036] Reporting metrics for performance evaluation;

[0037] Configuration information used for reporting information.

[0038] In some embodiments, according to the performance evaluation method of an AI function or AI model according to one embodiment of the present disclosure, before determining the performance evaluation result of the AI ​​function or the AI ​​model in the AI ​​function based on the first information related to the AI ​​function from the network device, the method further includes:

[0039] Sending third information to the network device; the third information includes at least one of the following:

[0040] The identification of the AI ​​function;

[0041] An identifier of the RRC configuration information corresponding to the AI ​​function;

[0042] The identifier of the AI ​​model in the AI ​​function;

[0043] An identifier of the RRC configuration information corresponding to the AI ​​model in the AI ​​function;

[0044] Request information for configuration information used for performance evaluation;

[0045] performance metrics for performance evaluation;

[0046] Reporting metrics for performance evaluation;

[0047] The number of samples used for performance evaluation;

[0048] The request information is used to request the network device to indicate a performance requirement or a performance threshold.

[0049] In some embodiments, according to the performance evaluation method of an AI function or AI model according to one embodiment of the present disclosure, the performance evaluation result of the AI ​​function or the AI ​​model in the AI ​​function includes at least one of the following:

[0050] Performance evaluation results of all AI models in the AI ​​function;

[0051] Performance evaluation results of N AI models in the AI ​​function; N is an integer greater than 0;

[0052] Performance evaluation results of AI models that meet the performance requirements of the AI ​​function;

[0053] The number of AI models in the AI ​​function that meet the performance requirements.

[0054] In some embodiments, according to the performance evaluation method of an AI function or AI model according to one embodiment of the present disclosure, before sending the second information to the network device, the method further includes:

[0055] determining whether the AI ​​function is available or meets the performance requirement based on the performance evaluation result of the AI ​​function and the performance requirement; or

[0056] determining, based on a performance evaluation result of the AI ​​model in the AI ​​function and the performance requirement, whether the AI ​​model in the AI ​​function is usable or meets the performance requirement;

[0057] The first indication information is used to indicate at least one of the following: the AI ​​function is unavailable or does not meet the performance requirements; the AI ​​function is available or meets the performance requirements; the AI ​​model in the AI ​​function is unavailable or does not meet the performance requirements; the AI ​​model in the AI ​​function is available or meets the performance requirements.

[0058] In some embodiments, according to the performance evaluation method of an AI function or AI model according to one embodiment of the present disclosure, the second information further includes at least one of the following:

[0059] Performance evaluation reference results obtained from non-AI functions;

[0060] The identifier of the AI ​​model in the AI ​​function.

[0061] In some embodiments, according to the performance evaluation method of an AI function or AI model of an embodiment of the present disclosure, the identifier of the AI ​​model in the AI ​​function includes an index, and the length of the index of the AI ​​model in the AI ​​function is obtained according to a preset rule, and the preset rule is to determine the length of the index of the AI ​​model according to the number of AI models in the AI ​​function or the number of reported AI models.

[0062] In some embodiments, according to the performance evaluation method of an AI function or AI model according to one embodiment of the present disclosure, after sending the second information to the network device, the method further includes:

[0063] Receive fourth information sent by the network device, where the fourth information includes at least one of the following:

[0064] activation instruction information of the AI ​​function;

[0065] Activation indication information of the AI ​​model in the AI ​​function;

[0066] third indication information, used to indicate that the AI ​​function or the AI ​​model in the AI ​​function is available, or that the AI ​​function or the AI ​​model in the AI ​​function meets performance requirements;

[0067] Fourth indication information, used to indicate that the AI ​​function or the AI ​​model in the AI ​​function is unavailable, or that the AI ​​function or the AI ​​model in the AI ​​function does not meet performance requirements;

[0068] The identifier of the target model in the AI ​​function;

[0069] The number of target models in the AI ​​function;

[0070] The target model is an available model, an activatable model, or a model that meets performance requirements.

[0071] In some embodiments, according to the performance evaluation method of an AI function or AI model according to one embodiment of the present disclosure, before sending the second information to the network device, the method further includes:

[0072] The number of AI models included in the AI ​​function is sent to the network device.

[0073] In a second aspect, the embodiments of the present disclosure further provide a method for evaluating the performance of an AI function or AI model, including:

[0074] Sending first information related to the AI ​​function to the terminal, where the first information is used to determine a performance evaluation result of the AI ​​function or an AI model in the AI ​​function;

[0075] Receive second information sent by the terminal, where the second information includes at least one of the following: a performance evaluation result of the AI ​​function; a performance evaluation result of the AI ​​model in the AI ​​function; first indication information; the first indication information is used to indicate whether the AI ​​function or the AI ​​model in the AI ​​function is available or whether the performance meets performance requirements; the first indication information is determined based on the performance evaluation result of the AI ​​function or the AI ​​model in the AI ​​function.

[0076] In some embodiments, according to the performance evaluation method of an AI function or AI model according to one embodiment of the present disclosure, the first information includes at least one of the following:

[0077] Configuration information for performance evaluation;

[0078] performance requirements;

[0079] Dataset information used for performance evaluation.

[0080] In some embodiments, according to the performance evaluation method of an AI function or AI model according to one embodiment of the present disclosure, the configuration information for performance evaluation includes at least one of the following:

[0081] The identification of the AI ​​function;

[0082] An identifier of the RRC configuration information corresponding to the AI ​​function;

[0083] The identifier of the AI ​​model in the AI ​​function;

[0084] An identifier of the RRC configuration information corresponding to the AI ​​model in the AI ​​function;

[0085] Reference signals for performance evaluation;

[0086] Configuration information for reporting information;

[0087] performance metrics for performance evaluation;

[0088] Reporting metrics for performance evaluation;

[0089] performance requirements;

[0090] Performance threshold.

[0091] In some embodiments, according to the performance evaluation method of an AI function or AI model according to one embodiment of the present disclosure, the dataset information includes at least one of the following:

[0092] Second indication information, used to indicate that the data set is used for performance evaluation;

[0093] The identification of the AI ​​function;

[0094] An identifier of the RRC configuration information corresponding to the AI ​​function;

[0095] Input data for the AI ​​function;

[0096] The identifier of the AI ​​model in the AI ​​function;

[0097] An identifier of the RRC configuration information corresponding to the AI ​​model in the AI ​​function;

[0098] Input data of the AI ​​model in the AI ​​function;

[0099] Reference information or true value information used to calculate performance indicators;

[0100] The number of samples used for performance evaluation;

[0101] performance metrics for performance evaluation;

[0102] Reporting metrics for performance evaluation;

[0103] Configuration information used for reporting information.

[0104] In some embodiments, the operations further include:

[0105] Receive third information sent by the terminal, where the third information includes at least one of the following:

[0106] The identification of the AI ​​function;

[0107] An identifier of the RRC configuration information corresponding to the AI ​​function;

[0108] The identifier of the AI ​​model in the AI ​​function;

[0109] An identifier of the RRC configuration information corresponding to the AI ​​model in the AI ​​function;

[0110] Request information for configuration information used for performance evaluation;

[0111] performance metrics for performance evaluation;

[0112] Reporting metrics for performance evaluation;

[0113] The number of samples used for performance evaluation;

[0114] The request information is used to request the network device to indicate a performance requirement or a performance threshold.

[0115] In some embodiments, according to the performance evaluation method of an AI function or AI model according to one embodiment of the present disclosure, the performance evaluation result of the AI ​​function or the AI ​​model in the AI ​​function includes at least one of the following:

[0116] Performance evaluation results of all AI models in the AI ​​function;

[0117] Performance evaluation results of N AI models in the AI ​​function; N is an integer greater than 0;

[0118] Performance evaluation results of AI models that meet the performance requirements of the AI ​​function;

[0119] The number of AI models in the AI ​​function that meet the performance requirements.

[0120] In some embodiments, the second information further includes at least one of the following:

[0121] Performance evaluation reference results obtained from non-AI functions;

[0122] The identifier of the AI ​​model in the AI ​​function.

[0123] In some embodiments, according to the performance evaluation method of an AI function or AI model of an embodiment of the present disclosure, the identifier of the AI ​​model in the AI ​​function includes an index, and the length of the index of the AI ​​model in the AI ​​function is obtained according to a preset rule, and the preset rule is to determine the length of the index of the AI ​​model according to the number of AI models in the AI ​​function or the number of reported AI models.

[0124] In some embodiments, according to the performance evaluation method of an AI function or AI model according to one embodiment of the present disclosure, after receiving the second information sent by the terminal, the operation further includes:

[0125] Sending fourth information to the terminal, where the fourth information includes at least one of the following:

[0126] activation instruction information of the AI ​​function;

[0127] Activation indication information of the AI ​​model in the AI ​​function;

[0128] third indication information, used to indicate that the AI ​​function or the AI ​​model in the AI ​​function is available, or that the AI ​​function or the AI ​​model in the AI ​​function meets performance requirements;

[0129] Fourth indication information, used to indicate that the AI ​​function or the AI ​​model in the AI ​​function is unavailable, or that the AI ​​function or the AI ​​model in the AI ​​function does not meet performance requirements;

[0130] The identifier of the target model in the AI ​​function;

[0131] The number of target models in the AI ​​function;

[0132] The target model is an available model, an activatable model, or a model that meets performance requirements.

[0133] In some embodiments, according to the performance evaluation method of an AI function or AI model according to one embodiment of the present disclosure, before receiving the second information sent by the terminal, the operation further includes:

[0134] Receive the number of AI models included in the AI ​​function sent by the terminal.

[0135] In a third aspect, an embodiment of the present disclosure further provides a terminal, including a memory, a transceiver, and a processor, wherein:

[0136] A memory for storing a computer program; a transceiver for sending and receiving data under the control of the processor; and a processor for reading the computer program in the memory and implementing the steps of the performance evaluation method for the AI ​​function or AI model as described in the first aspect above.

[0137] In a fourth aspect, an embodiment of the present disclosure further provides a network device, including a memory, a transceiver, and a processor, wherein:

[0138] A memory for storing computer programs; a transceiver for sending and receiving data under the control of the processor; and a processor for reading the computer program in the memory and implementing the steps of the performance evaluation method for the AI ​​function or AI model as described in the second aspect above.

[0139] In a fifth aspect, an embodiment of the present disclosure further provides a performance evaluation device for an AI function or AI model, the device comprising:

[0140] a processing unit, configured to determine a performance evaluation result of the AI ​​function or an AI model in the AI ​​function based on first information related to the AI ​​function from a network device;

[0141] A first sending unit is used to send second information to the network device, where the second information includes at least one of the following: a performance evaluation result of the AI ​​function; a performance evaluation result of the AI ​​model in the AI ​​function; first indication information; the first indication information is used to indicate whether the AI ​​function or the AI ​​model in the AI ​​function is available or whether the performance meets performance requirements; the first indication information is determined based on the performance evaluation result of the AI ​​function or the AI ​​model in the AI ​​function.

[0142] In a sixth aspect, an embodiment of the present disclosure further provides a performance evaluation device for an AI function or AI model, the device comprising:

[0143] a second sending unit, configured to send first information related to the AI ​​function to the terminal, where the first information is used to determine a performance evaluation result of the AI ​​function or an AI model in the AI ​​function;

[0144] A receiving unit is configured to receive second information sent by the terminal, where the second information includes at least one of the following: a performance evaluation result of the AI ​​function; a performance evaluation result of the AI ​​model in the AI ​​function; and first indication information; the first indication information is used to indicate whether the AI ​​function or the AI ​​model in the AI ​​function is available or whether the performance meets performance requirements; the first indication information is determined based on the performance evaluation result of the AI ​​function or the AI ​​model in the AI ​​function.

[0145] In the seventh aspect, an embodiment of the present disclosure also provides a processor-readable storage medium, which stores a computer program, and the computer program is used to enable the processor to execute the steps of the performance evaluation method of the AI ​​function or AI model described in the first aspect or the second aspect above.

[0146] In an eighth aspect, an embodiment of the present disclosure further provides a non-transitory readable storage medium, wherein the non-transitory readable storage medium stores a computer program, and the computer program is used to enable a processor to execute the steps of the performance evaluation method of the AI ​​function or AI model as described in any one of the first aspect or the second aspect above.

[0147] In the ninth aspect, an embodiment of the present disclosure further provides a chip product, in which a computer program is stored, and the computer program is used to enable the chip product to execute the steps of the performance evaluation method of the AI ​​function or AI model as described in any one of the first aspect or the second aspect above.

[0148] The performance evaluation method and device of an AI function or AI model provided by the embodiments of the present disclosure determine a performance evaluation result of the AI ​​function or the AI ​​model in the AI ​​function based on first information related to the AI ​​function from a network device; send second information to the network device, where the second information includes at least one of the following: the performance evaluation result of the AI ​​function; the performance evaluation result of the AI ​​model in the AI ​​function; first indication information; the first indication information is used to indicate whether the AI ​​function or the AI ​​model in the AI ​​function is available or whether the performance meets the performance requirements; the first indication information is determined based on the performance evaluation result of the AI ​​function or the AI ​​model in the AI ​​function. In the above scheme, the terminal determines the AI ​​function or the AI ​​model in the AI ​​function. The terminal can determine whether the AI ​​function or the AI ​​model in the AI ​​function is available or whether the performance meets the performance requirements, and then the terminal reports the result of whether the AI ​​function or the AI ​​model in the AI ​​function is available or whether the performance meets the performance requirements to the network device, or the terminal can report the performance evaluation result of the AI ​​function or the AI ​​model in the AI ​​function to the network device, and the network device can further determine whether the AI ​​function or the AI ​​model in the AI ​​function is available or whether the performance meets the performance requirements, so that under the lifecycle management method based on the AI ​​function, the terminal can autonomously switch to the AI ​​model that meets the performance requirements, avoid switching to the AI ​​model that does not meet the performance requirements, thereby ensuring the performance of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0149] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or related technologies, the following is a brief introduction to the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0150] FIG1 is a flow chart of a method for evaluating the performance of an AI function or AI model according to an embodiment of the present disclosure;

[0151] FIG2 is a second flow chart of a method for evaluating the performance of an AI function or AI model provided in an embodiment of the present disclosure;

[0152] FIG3 is a schematic structural diagram of a terminal provided in an embodiment of the present disclosure;

[0153] FIG4 is a schematic diagram of the structure of a network device provided in an embodiment of the present disclosure;

[0154] FIG5 is a schematic diagram of a structure of a device for evaluating the performance of an AI function or AI model provided by an embodiment of the present disclosure;

[0155] FIG6 is a second structural diagram of the device for evaluating the performance of an AI function or AI model provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0156] In the embodiments of the present disclosure, the term "and / or" describes the association relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0157] In the embodiments of the present disclosure, the term "plurality" refers to two or more than two, and other quantifiers are similar thereto.

[0158] The following will be combined with the accompanying drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the embodiments described are only part of the embodiments of the present disclosure and not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0159] The technical solutions provided by the embodiments of the present disclosure can be applicable to a variety of systems, such as 5G systems or 6G systems. For example, applicable systems may be global system of mobile communication (GSM) systems, code division multiple access (CDMA) systems, wideband code division multiple access (WCDMA) general packet radio service (GPRS) systems, long term evolution (LTE) systems, LTE frequency division duplex (FDD) systems, LTE time division duplex (TDD) systems, long term evolution advanced (LTE-A) systems, universal mobile telecommunication systems (UMTS), worldwide interoperability for microwave access (WiMAX) systems, 5G new radio (NR) systems, etc. These various systems include terminal devices and network devices. The system may also include a core network part, such as an evolved packet system (EPS), a 5G system (5GS), etc.

[0160] The terminal device involved in the embodiments of the present disclosure may be a device that provides voice and / or data connectivity to a user, a handheld device with wireless connection function, or other processing device connected to a wireless modem. In different systems, the name of the terminal device may also be different. For example, in a 5G system, the terminal device may be called User Equipment (UE). A wireless terminal device can communicate with one or more core networks (CN) via a radio access network (RAN). The wireless terminal device may be a mobile terminal device, such as a mobile phone (or "cellular" phone) and a computer with a mobile terminal device. For example, it may be a portable, pocket-sized, handheld, computer-built-in or vehicle-mounted mobile device that exchanges voice and / or data with a radio access network. For example, Personal Communication Service (PCS) phones, cordless phones, Session Initiated Protocol (SIP) phones, Wireless Local Loop (WLL) stations, Personal Digital Assistants (PDAs), and other devices. The wireless terminal device may also be referred to as a system, a subscriber unit, a subscriber station, a mobile station, a mobile station, a remote station, an access point, a remote terminal device, an access terminal device, a user terminal device, a user agent, or a user device, but is not limited in the embodiments of the present disclosure.

[0161] The network device involved in the embodiments of the present disclosure may be a base station or a core network device, and the base station may include multiple cells providing services to the terminal. Depending on the specific application scenario, the base station may also be called an access point, or may be a device in the access network that communicates with the wireless terminal device through one or more sectors on the air interface, or other names. The network device may be used to interchange received air frames with Internet Protocol (IP) packets, acting as a router between the wireless terminal device and the rest of the access network, wherein the rest of the access network may include an Internet Protocol (IP) communication network. The network device may also coordinate the attribute management of the air interface. For example, the network device involved in the embodiments of the present disclosure may be a base transceiver station (BTS) in the Global System for Mobile communications (GSM) or code division multiple access (CDMA), a network device (NodeB) in wide-band code division multiple access (WCDMA), an evolutionary Node B (eNB or e-NodeB) in the long term evolution (LTE) system, a 5G base station (gNB) in the 5G network architecture (next generation system), a home evolved Node B (HeNB), a relay node, a femto, a pico, etc., and is not limited in the embodiments of the present disclosure. In some network structures, the network device may include a centralized unit (CU) node and a distributed unit (DU) node, and the centralized unit and the distributed unit may also be geographically separated.In some embodiments, the core network device may include but is not limited to at least one of the following: core network node, core network function, Mobility Management Entity (MME), Access and Mobility Management Function (AMF), Session Management Function (SMF), User Plane Function (UPF), Policy Control Function (PCF), Policy and Charging Rules Function (PCRF), Edge Application Server Discovery Function (EASDF), Unified Data Management (UDM), Unified Data Repository (UDR), Network Repository Function (NRF), Network Exposure Function (NEF), Application Function (AF), Sensing Requirement Function (SRF), Location Management Function (LMF), etc.

[0162] In order to facilitate a clearer understanding of the technical solutions provided by the various embodiments of the present disclosure, some relevant knowledge is first introduced as follows.

[0163] The lifecycle of an AI / ML model refers to the complete process from start to finish, including data collection, model training, model updates, model identification, model transmission, model monitoring, model activation / deactivation / selection / switching / rollback, etc. Related technologies have studied the following two AI / ML lifecycle management (LCM) approaches:

[0164] 1. Lifecycle management based on AI / ML model ID

[0165] Lifecycle management based on AI / ML model ID refers to the recognition of AI / ML models by terminals and network devices through the model identification process. The model ID can be used to indicate or manage operations such as model updates, model monitoring, model activation / deactivation / selection / switching / rollback.

[0166] 2. Lifecycle management based on AI / ML functionality

[0167] AI / ML function-based lifecycle management refers to the network device side controlling the activation / deactivation / switching / rollback of specific AI / ML functions on the terminal side (such as AI beam management or AI positioning), but does not care about the lifecycle management of the AI / ML model level within each AI / ML function. Through the functionality identification process, the AI / ML function on the terminal side is identified by the network device side, but the AI / ML model within it is invisible to the network device side, and the AI / ML model is managed by the terminal side itself. For example, an AI / ML function includes multiple AI / ML models. The network device side instructs the activation of the AI / ML function. When the terminal uses the AI / ML function for inference, the terminal side decides which AI / ML model to use for inference.

[0168] In the lifecycle management of AI / ML functions, network devices determine the AI / ML function-level management, while the terminal determines the model-level management within the AI / ML function. The terminal can autonomously activate / deactivate / switch AI / ML models within the AI / ML function. If the terminal switches to a model that does not match the network device-side conditions / additional conditions during model switching within the AI / ML function, AI / ML inference performance will degrade. To avoid this, when using / activating the AI ​​function, it is necessary to determine which models within the AI ​​function match the current network-side conditions / additional conditions and which do not. The terminal then switches models only to those that match. In other words, the network device cannot allow the terminal to switch models arbitrarily.

[0169] In the lifecycle management of AI / ML functions, when performance monitoring or performance evaluation is used to determine whether network conditions / additional conditions are consistent during the training and inference phases, the relevant technology has not yet provided a solution for assisting the terminal side in managing the AI / ML model level. Therefore, the embodiments of the present disclosure address the above issues and propose a performance evaluation (or performance monitoring) method for AI functions or AI models. In this disclosure, "performance evaluation" can be used interchangeably with similar terms such as "performance monitoring."

[0170] In the embodiments of the present disclosure, AI functions may also be referred to as ML functions, AI / ML modules, or AI / ML units. AI models may also be referred to as AI structures, AI features, machine learning models, or operational rules with AI / ML functions. Alternatively, the AI ​​model may refer to a model capable of implementing specific AI-related algorithms, formulas, processing flows, capabilities, etc., which are not specifically limited in the embodiments of the present disclosure.

[0171] FIG1 is a flow chart of a method for evaluating the performance of an AI function or AI model provided by an embodiment of the present disclosure. As shown in FIG1 , an embodiment of the present disclosure provides a method for evaluating the performance of an AI function or AI model, the execution subject of which may be a terminal, such as a mobile phone. The method includes:

[0172] Step 101: Determine a performance evaluation result of the AI ​​function or an AI model in the AI ​​function based on first information related to the AI ​​function from a network device.

[0173] Specifically, the first information related to the AI ​​function may be information configured by a network device, or information sent based on a request from a terminal, and is used for performance monitoring / performance evaluation.

[0174] The terminal determines the performance evaluation result of the AI ​​function or the AI ​​model in the AI ​​function based on the first information related to the AI ​​function, that is, the terminal can determine the performance evaluation result with the AI ​​function as the granularity or determine the performance evaluation result with the AI ​​model as the granularity.

[0175] The performance evaluation results can be expressed through performance indicators, such as beam prediction accuracy, positioning accuracy, reference signal received power (RSRP), reference signal received quality (RSRQ), etc.

[0176] Step 102: Send second information to the network device, where the second information includes at least one of the following: a performance evaluation result of the AI ​​function; a performance evaluation result of the AI ​​model in the AI ​​function; first indication information; the first indication information is used to indicate whether the AI ​​function or the AI ​​model in the AI ​​function is available or whether the performance meets the performance requirements; the first indication information is determined based on the performance evaluation result of the AI ​​function or the AI ​​model in the AI ​​function.

[0177] Specifically, after determining the performance evaluation result of the AI ​​function or the AI ​​model in the AI ​​function, the terminal can report relevant second information to the network device, such as directly reporting the performance evaluation result of the AI ​​function or the AI ​​model in the AI ​​function. The network device can determine whether the AI ​​function or the AI ​​model in the AI ​​function is available / whether it meets the performance requirements based on the reported performance evaluation result.

[0178] In some embodiments, the terminal may also report to the network device first indication information determined based on the performance evaluation result of the AI ​​function or the AI ​​model in the AI ​​function, which is used to indicate whether the AI ​​function or the AI ​​model in the AI ​​function is available, or whether the performance of the AI ​​function or the AI ​​model in the AI ​​function meets the performance requirements. This can enable the network device side to identify whether the AI ​​function or the AI ​​model in the AI ​​function is available / whether it meets the performance requirements. In some embodiments, the network device side can assist the terminal in identifying which AI models are available AI models.

[0179] Among them, the AI ​​function is available means that there are available AI models in the AI ​​function. If all AI models in the AI ​​function are unavailable, the AI ​​function is unavailable.

[0180] In some embodiments, the uplink resources used by the terminal to report the second information are configured / instructed by the network device or requested by the terminal.

[0181] In the method of this embodiment, the terminal determines the performance evaluation result of the AI ​​function or the AI ​​model in the AI ​​function, and the terminal can determine whether the AI ​​function or the AI ​​model in the AI ​​function is available or whether the performance meets the performance requirements. Then, the terminal reports the result of whether the AI ​​function or the AI ​​model in the AI ​​function is available or whether the performance meets the performance requirements to the network device, or the terminal can report the performance evaluation result of the AI ​​function or the AI ​​model in the AI ​​function to the network device, and the network device can further determine whether the AI ​​function or the AI ​​model in the AI ​​function is available or whether the performance meets the performance requirements. In this way, under the lifecycle management method based on the AI ​​function, the terminal can autonomously switch to an AI model that meets the performance requirements, avoiding switching to an AI model that does not meet the performance requirements, thereby ensuring the performance of the network.

[0182] In some embodiments, the first information includes at least one of the following:

[0183] (1) Configuration information for performance evaluation;

[0184] (2) Performance requirements;

[0185] (3) Dataset information used for performance evaluation.

[0186] Specifically, the terminal performs performance monitoring / evaluation on the AI ​​function or the AI ​​model in the AI ​​function according to the configuration information for performance evaluation configured by the network device, and reports the second information.

[0187] The performance requirements may be, for example, performance indicator requirements or performance thresholds. The terminal may compare the performance evaluation results of the determined AI function or the AI ​​model in the AI ​​function with the performance requirements to determine whether the AI ​​function or the AI ​​model in the AI ​​function is available, or whether the performance of the AI ​​function or the AI ​​model in the AI ​​function meets the performance requirements.

[0188] Among them, (3) takes into account that after the terminals with AI models deployed in a cell access the network, these terminals need to verify whether the network side conditions / additional conditions match, or whether the AI ​​function or the AI ​​model in the AI ​​function is available or meets the performance requirements through performance monitoring / evaluation. In order to avoid the network equipment repeatedly and multiple times sending the same configuration reference signal to these terminals, the network equipment can send the data set used for performance monitoring / evaluation to the terminals in a unified manner.

[0189] If an AI model is available, the terminal reports that the AI ​​function is available. Otherwise, the terminal reports that the AI ​​function is unavailable. In this case, the information about the available AI models in the AI ​​function is only known to the terminal side, and the terminal will subsequently switch among these available AI models.

[0190] In the above implementation, the terminal can determine the performance evaluation result of the AI ​​function or the AI ​​model in the AI ​​function based on the first information sent by the network device, and then independently determine whether the AI ​​function or the AI ​​model in the AI ​​function is available, or whether it meets the performance requirements; or, the terminal determines whether the AI ​​function or the AI ​​model in the AI ​​function is available, or whether it meets the performance requirements with the assistance of the network device side.

[0191] In some embodiments, before step 102, the method further includes:

[0192] determining whether the AI ​​function is available or meets the performance requirement based on the performance evaluation result of the AI ​​function and the performance requirement; or

[0193] determining, based on a performance evaluation result of the AI ​​model in the AI ​​function and the performance requirement, whether the AI ​​model in the AI ​​function is usable or meets the performance requirement;

[0194] The first indication information is used to indicate at least one of the following: the AI ​​function is unavailable or does not meet the performance requirements; the AI ​​function is available or meets the performance requirements; the AI ​​model in the AI ​​function is unavailable or does not meet the performance requirements; the AI ​​model in the AI ​​function is available or meets the performance requirements.

[0195] Specifically, with respect to the AI ​​function granularity, the terminal may determine whether the AI ​​function is available or whether the AI ​​function meets the performance requirement based on the performance evaluation result of the AI ​​function and the performance requirement corresponding to the AI ​​function.

[0196] Regarding the AI ​​model granularity, the terminal can determine whether the AI ​​model in the AI ​​function is available or whether the AI ​​model in the AI ​​function meets the performance requirements based on the performance evaluation results of the AI ​​model in the AI ​​function and the performance requirements corresponding to the AI ​​model in the AI ​​function; for example, for each AI model in a certain AI function, the terminal can determine whether the AI ​​model is available or whether the AI ​​model meets the performance requirements based on the performance evaluation results of the AI ​​model and the performance requirements corresponding to the AI ​​model, and the terminal can subsequently switch models among these available models.

[0197] In some embodiments, the performance requirement may be configured or predefined by the network device side, such as a performance evaluation result being better than a reference / benchmark or a performance evaluation result being better than a performance threshold.

[0198] In some embodiments, the configuration information for performance evaluation includes at least one of the following:

[0199] The identification of the AI ​​function;

[0200] An identifier of the radio resource control (RRC) configuration information corresponding to the AI ​​function;

[0201] The identifier of the AI ​​model in the AI ​​function;

[0202] An identifier of the RRC configuration information corresponding to the AI ​​model in the AI ​​function;

[0203] Reference signals for performance evaluation;

[0204] Configuration information for reporting information;

[0205] performance metrics for performance evaluation;

[0206] Reporting metrics for performance evaluation;

[0207] performance requirements;

[0208] Performance threshold.

[0209] Among them, the identification of the AI ​​function / AI model in the AI ​​function can be represented by a name, an identifier (Identity, ID), index information, etc.

[0210] Among them, the RRC configuration information may correspond to the AI ​​function or the AI ​​model in the AI ​​function. For example, different AI functions or AI models in the AI ​​function may be determined through different RRC configuration information.

[0211] The configuration information used for reporting information, that is, the configuration information used for reporting the second information, may be, for example, resource information used for reporting the information.

[0212] For example, the performance indicator of the AI ​​function for beam management is the beam prediction accuracy or the difference in the predicted layer L1-RSRP, the performance indicator of the AI ​​function for positioning is the positioning accuracy, and the performance indicator of the AI ​​function for channel state information (CSI) prediction / feedback is the minimization of squared cosine similarity (SGCS).

[0213] For example, a performance requirement or performance threshold may be a beam prediction accuracy greater than or equal to 85% in beam management or a positioning accuracy error less than or equal to 1 meter in a positioning use case.

[0214] In some embodiments, the dataset information includes at least one of the following:

[0215] Second indication information, used to indicate that the data set is used for performance evaluation;

[0216] The identification of the AI ​​function;

[0217] An identifier of the RRC configuration information corresponding to the AI ​​function;

[0218] Input data for the AI ​​function;

[0219] The identifier of the AI ​​model in the AI ​​function;

[0220] An identifier of the RRC configuration information corresponding to the AI ​​model in the AI ​​function;

[0221] Input data of the AI ​​model in the AI ​​function;

[0222] Reference information or true value information used to calculate performance indicators;

[0223] The number of samples used for performance evaluation;

[0224] performance metrics for performance evaluation;

[0225] Reporting metrics for performance evaluation;

[0226] Configuration information used for reporting information.

[0227] The second indication information is data set function indication information, which is used to indicate that the data set is used for performance monitoring / evaluation.

[0228] For example, the AI ​​function is a function for beam management, and the input data of the AI ​​function or the AI ​​model in the AI ​​function can be M L1-RSRP values; M is an integer greater than 0.

[0229] For example, the AI ​​function is a function for positioning, and the input data of the AI ​​function or the AI ​​model in the AI ​​function can be M channel impulse responses (CIRs) or M power delay profiles (PDPs) or M delay profiles (DPs).

[0230] For example, the AI ​​function is a function for CSI feedback or prediction, and the input data of the AI ​​function or the AI ​​model in the AI ​​function can be M channel information matrices or M channel feature vectors.

[0231] The reference information may be referred to as reference / benchmark information, and the true value information may be, for example, a ground truth label.

[0232] For example, if the AI ​​function is used for beam management, the ground truth information may be the indexes of the top-K beams and / or L1-RSRP values. For example, the top-K beams are the top K beams with the best performance, such as the best signal strength. For example, when the performance metric is beam prediction accuracy, the ground truth information is the indexes of the top-K beams in SetA. When the performance metric is the difference in L1-RSRP between the predicted top-K beams, the ground truth information may include the corresponding L1-RSRP measurement value.

[0233] For example, the AI ​​function is a function for positioning, and the true value information may be N pieces of position information.

[0234] For example, the AI ​​function is a function for CSI feedback or prediction, and the true value information may be X channel information matrices or channel eigenvectors.

[0235] In some embodiments, the performance indicators or reporting indicators used for performance monitoring / evaluation may be the same indicators, or indicators of different granularities and dimensions, which is not limited in the embodiments of the present disclosure.

[0236] In the above implementation, the data set information includes at least one of multiple items, which is more flexible.

[0237] In some embodiments, before step 101, the method further includes:

[0238] Sending third information to the network device; the third information includes at least one of the following:

[0239] The identification of the AI ​​function;

[0240] An identifier of the RRC configuration information corresponding to the AI ​​function;

[0241] The identifier of the AI ​​model in the AI ​​function;

[0242] An identifier of the RRC configuration information corresponding to the AI ​​model in the AI ​​function;

[0243] Request information for configuration information used for performance evaluation;

[0244] performance metrics for performance evaluation;

[0245] Reporting metrics for performance evaluation;

[0246] The number of samples used for performance evaluation;

[0247] The request information is used to request the network device to indicate a performance requirement or a performance threshold.

[0248] Specifically, the terminal may request performance monitoring / performance evaluation by sending third information. The terminal sends request information for configuration information for performance evaluation, and the network device may send corresponding configuration information to the terminal based on the request information for the configuration information.

[0249] For example, in the beam management use case, a request is made to calculate the beam prediction accuracy based on 50 inference results. That is, the number of samples used for performance evaluation in this beam management use case is 50.

[0250] The terminal can request the network device side to indicate performance requirements or performance thresholds. For example, when the terminal side requests to send a reference signal for performance monitoring, it can also request to send the corresponding performance requirements or performance thresholds, so that the terminal side can independently perform performance monitoring / performance evaluation according to the performance requirements.

[0251] In the above implementation, performance monitoring / evaluation can be triggered by a request from a terminal, which reduces implementation complexity and increases flexibility.

[0252] In some embodiments, the performance evaluation result of the AI ​​function or the AI ​​model in the AI ​​function includes at least one of the following:

[0253] Performance evaluation results of all AI models in the AI ​​function;

[0254] Performance evaluation results of N AI models in the AI ​​function; N is an integer greater than 0;

[0255] Performance evaluation results of AI models that meet the performance requirements of the AI ​​function;

[0256] The number of AI models in the AI ​​function that meet the performance requirements.

[0257] Specifically, the performance evaluation results of the AI ​​model reported by the terminal may be reported in a prescribed order or an indicated order.

[0258] When a terminal reports the performance evaluation results of an AI function or AI model, in some embodiments, it may also report index information. When instructed by a network device, the network device indicates the index information of available AI models, and the terminal can determine which AI models are available based on the index information. The length of the index information can be determined based on a rule, such as the number of AI models in the AI ​​function or the number of AI models reported by the terminal. For example, in this case, the index information is functionality-specific, indicating only the AI ​​models included in the AI ​​function.

[0259] For example, the N AI models are the top N AI models with better performance evaluation results, or the N AI models with performance evaluation results greater than or equal to the performance threshold, or the terminal determines according to other rules, which is not limited in the embodiments of the present disclosure.

[0260] In some embodiments, the second information further includes at least one of the following:

[0261] Performance evaluation reference results obtained from non-AI functions;

[0262] The identifier of the AI ​​model in the AI ​​function.

[0263] Specifically, the terminal can also report performance evaluation reference results (also called performance evaluation benchmark results). For example, the performance evaluation reference results are obtained according to traditional algorithms, that is, algorithms without AI functions. The performance evaluation reference results obtained with non-AI functions can assist network devices in making decisions.

[0264] For example, the base station reconfirms which AI functions are available or which AI models are available models in the reported performance evaluation results based on the performance evaluation results of the AI ​​function, the performance evaluation results of the AI ​​model in the AI ​​function, the performance evaluation reference results and the current network conditions, and then indicates the information of the available models to the terminal.

[0265] For example, if the performance of a non-AI function is comparable to that of an AI function, the network device may choose to use the non-AI function.

[0266] In some embodiments, the identifier of the AI ​​model in the AI ​​function includes an index, and the length of the index of the AI ​​model in the AI ​​function is obtained according to a preset rule, and the preset rule is to determine the length of the index of the AI ​​model according to the number of AI models in the AI ​​function or the number of reported AI models.

[0267] Specifically, when reporting the performance evaluation results of N AI models, the length of the AI ​​model index can be determined based on the number N. N can be the number of AI models in the AI ​​function, or it can be the number of reported AI models.

[0268] For example, if the AI ​​function includes four AI models, the index length is 2, and 00, 01, 10, and 11 are used to represent the indexes of these four AI models.

[0269] In some embodiments, before step 102, the method further includes:

[0270] The number of AI models included in the AI ​​function is sent to the network device.

[0271] Specifically, the terminal can report the number of AI models included in the AI ​​function to the network device, and the network device can configure reporting resources, etc. according to the number of AI models included in the AI ​​function.

[0272] In the above implementation, the terminal can be assisted in achieving performance evaluation / monitoring by sending the performance evaluation reference result obtained by the non-AI function, the identifier of the AI ​​model in the AI ​​function, the number of AI models included in the AI ​​function, etc.

[0273] In some embodiments, after sending the second information to the network device, the method further includes:

[0274] Receive fourth information sent by the network device, where the fourth information includes at least one of the following:

[0275] activation instruction information of the AI ​​function;

[0276] Activation indication information of the AI ​​model in the AI ​​function;

[0277] third indication information, used to indicate that the AI ​​function or the AI ​​model in the AI ​​function is available, or that the AI ​​function or the AI ​​model in the AI ​​function meets performance requirements;

[0278] Fourth indication information, used to indicate that the AI ​​function or the AI ​​model in the AI ​​function is unavailable, or that the AI ​​function or the AI ​​model in the AI ​​function does not meet performance requirements;

[0279] The identifier of the target model in the AI ​​function;

[0280] The number of target models in the AI ​​function;

[0281] The target model is an available model, an activatable model, or a model that meets performance requirements.

[0282] Specifically, after receiving the second information sent by the terminal, the network device can send activation indication information to activate the AI ​​function or the AI ​​model in the AI ​​function.

[0283] Alternatively, when the second information includes the performance evaluation result of the AI ​​function or the performance evaluation result of the AI ​​model in the AI ​​function, that is, the terminal does not determine whether the AI ​​function or the AI ​​model in the AI ​​function is available or meets the performance requirements, the network device may further send third indication information to indicate that the AI ​​function or the AI ​​model in the AI ​​function is available or meets the performance requirements; the network device may further send fourth indication information to indicate that the AI ​​function or the AI ​​model in the AI ​​function is not available or does not meet the performance requirements.

[0284] Alternatively, the network device may also send the identifier and / or number of target models in the AI ​​function, where the target model refers to an available model, an activatable model, or a model that meets performance requirements. For example, the identifier is index information. In some embodiments, the length of the index information may be determined based on the number of AI models in the AI ​​function or the number of AI models in the AI ​​function reported by the terminal.

[0285] In the above implementation, after the terminal sends the second information to the network device, it can also receive fourth information, such as activation indication information, the results of whether the AI ​​function or the AI ​​model in the AI ​​function determined by the network device side is available or meets the performance requirements, etc., thereby realizing performance evaluation / monitoring of the AI ​​function or the AI ​​model in the AI ​​function.

[0286] Example 1: After the terminal performs AI function recognition, the network device sends a data set for performance monitoring / performance evaluation:

[0287] 1. The terminal identifies the AI ​​function and informs the network device that the AI ​​function (denoted as AI function #k) is deployed on the terminal side. This AI function is used for spatial beam management. Specifically, the optimal beam of the n beam set can be predicted based on the L1-RSRP values ​​of M beams / reference signals.

[0288] 2. The network device sends a data set, which includes one or more of the following information:

[0289] The second indication information indicates that the data set is used for performance monitoring / performance evaluation;

[0290] The identifier of the AI ​​function or the AI ​​model in the AI ​​function, such as the AI ​​function index or the configuration index #k associated with the AI ​​function, indicating that the dataset is used to monitor / evaluate the performance of the AI ​​function corresponding to AI function #k or configuration #k;

[0291] Number of samples: S, that is, the dataset includes the input data of S AI models and the corresponding reference information / true value information;

[0292] Input data of the AI ​​function or the AI ​​model in the AI ​​function, for example, the input data of each AI model is M L1-RSRP values;

[0293] Reference information / true value information: the top-1 beam index value corresponding to each input data, for example, the index value of the beam with the largest signal strength;

[0294] Performance indicators or reporting indicators used for performance monitoring / performance evaluation, such as beam prediction accuracy.

[0295] 3. After receiving the data set, the terminal uses the S input data as model input in sequence for inference, obtains the inference results of each model, and compares them with the S true value information in the data set to calculate the beam prediction accuracy.

[0296] In example 2, after the terminal identifies the AI ​​function, the network device sends a data set or configuration information for performance evaluation for performance monitoring / performance evaluation. The terminal reports the performance evaluation results of all AI models in the AI ​​function, and the network device indicates which AI models are available or whose performance meets the requirements.

[0297] 1. The terminal performs AI function identification and informs the network device that the AI ​​function (denoted as AI function #k) is deployed on the terminal side. In some embodiments, the terminal reports AI function #k including d AI models.

[0298] 2. The network device configures the corresponding configuration information for performance evaluation or sends a data set.

[0299] 3. The terminal measures the reference signal or performs performance monitoring / performance evaluation on the AI ​​model in AI function #k based on the received data set to obtain the beam prediction accuracy of d AI models.

[0300] 4. Assuming d = 4, the terminal reports the performance evaluation results shown in Table 1 below:

[0301] Table 1

[0302] Among them, 00, 01, 10, and 11 are the indexes of the AI ​​models in AI function #k, and X1%, X2%, X3%, and X4% are the beam prediction accuracies of the corresponding models.

[0303] In some embodiments, the terminal may not report the model index, but only report four beam prediction accuracy rates X1%, X2%, X3%, and X4%.

[0304] 5. The network device indicates the following information based on the performance evaluation results reported by the terminal:

[0305] If it is determined that the performance of X1% and X4% meets the performance requirement, indication information 00, 11 is sent, indicating that the prediction accuracy of the first and fourth beams meets the performance requirement.

[0306] If the terminal is required to report the prediction accuracy in a certain order in step 4, such as from large to small, the network device can indicate the number of prediction accuracy rates that meet the performance requirements, such as 2, which means that X1% and X2% meet the performance requirements.

[0307] 6. The terminal determines which AI models meet the performance requirements based on the results indicated by the network device. For example, in step 5, the network device sends the indication information 00 and 11, indicating that the AI ​​models corresponding to 00 and 01 meet the performance requirements. When using AI function #1, the terminal can independently select the AI ​​model corresponding to 00 or 01 for inference.

[0308] Example three: After the terminal identifies the AI ​​function, the network device configures and sends a reference signal for performance monitoring / performance evaluation. The terminal reports whether the AI ​​function meets the performance requirements or reports the performance evaluation results of the AI ​​function or AI model that meets the performance requirements and the reference / benchmark results. The network device indicates whether to activate the AI ​​function or which AI models can be activated / used.

[0309] 1. The terminal performs AI function identification and informs the network device that the AI ​​function is deployed on the terminal side (indicated as AI function #k).

[0310] 2. The network equipment is configured with reference signals for performance monitoring / performance evaluation, and the network equipment is configured or specifies a performance threshold of beam prediction accuracy greater than or equal to 85%. The reference signals configured on the network equipment include:

[0311] Option 1: Two reference signal sets are used. The reference signal measurements of set 1 are used as model input, and the reference signal measurements of set 2 are used to obtain the optimal beam. That is, the optimal beam in set 2 is predicted by measuring the reference signals of set 1.

[0312] Option 2: 1 reference signal set, the measurement of some reference signals in the set is used as the input of the model, and the optimal beam is the beam corresponding to the reference signal with the maximum L1-RSRP in the reference signal set, that is, the beam corresponding to the reference signal with the maximum L1-RSRP in the reference signal set is predicted by using some reference signals.

[0313] 3. The terminal measures the reference signal or monitors / performs the AI ​​model in AI Function #1 based on the received data set to obtain the beam prediction accuracy and the beam prediction accuracy obtained by traditional beam management (non-AI) methods.

[0314] 4. The beam prediction accuracy of AI function #1 reported by the terminal is 90%, while the beam prediction accuracy of the traditional beam management algorithm is 85%.

[0315] 5. The network device activates or deactivates AI Function #1 based on the results reported by the terminal. If the network device determines that AI Function #1 outperforms the traditional algorithm, it can indicate activation of AI Function #1. If the base station determines that the performance of AI Function #1 is similar to that of the traditional algorithm, it can deactivate the AI / ML function.

[0316] In Example 4, after the terminal identifies the AI ​​function, the network device sends a data set or configures a reference signal for performance monitoring / performance evaluation, and the terminal reports whether the AI ​​function meets the performance requirements.

[0317] 1. The terminal performs AI function identification and informs the network device that the AI ​​function is deployed on the terminal side (indicated as AI function #k).

[0318] 2. The network device is configured with a corresponding reference signal or transmits a data set, and the network device is configured or specifies a performance threshold of beam prediction accuracy greater than or equal to 85%.

[0319] 3. The terminal measures the reference signal or monitors / performs the AI ​​model in function A#k based on the received data set to obtain the beam prediction accuracy.

[0320] 4. If the result obtained by the terminal in step 3 is that the beam prediction accuracy of the AI ​​model in function #k is greater than or equal to 85%, the terminal reports the second information indicating that the AI ​​function #k is available or meets the performance requirements: If the result obtained by the terminal in step 3 is that the beam prediction accuracy of the AI ​​model in function #k is greater than or equal to 85%, the terminal reports the second information indicating that the AI ​​function #k is unavailable or does not meet the performance requirements.

[0321] 5. If the terminal reports an indication in step 4 that AI function #k meets the performance requirements, the network device configures / activates the corresponding configuration of AI function #k. When using AI function #k, the terminal may autonomously select a model for inference from among the AI ​​models with a beam prediction accuracy greater than or equal to 85% in step 4.

[0322] In summary, the embodiments of the present disclosure propose a method for determining whether the network-side conditions / additional conditions in the training and reasoning stages of an AI function or AI model are consistent through performance monitoring or performance evaluation, that is, whether the performance of the AI ​​function or AI model meets the requirements or is available through performance monitoring / evaluation. The method provided by the embodiments of the present disclosure can assist the terminal side in managing the AI ​​function or AI model level, so that under the lifecycle management method based on the AI ​​function or AI model, the terminal can autonomously switch to an AI function or AI model that meets the performance requirements, avoiding switching to an AI model in the AI ​​function that does not meet the performance requirements, thereby ensuring the performance of the network.

[0323] FIG2 is a second flow chart of a method for evaluating the performance of an AI function or AI model provided by an embodiment of the present disclosure. As shown in FIG2 , an embodiment of the present disclosure provides a method for evaluating the performance of an AI function or AI model, the execution subject of which may be a network device, such as a base station. The method includes:

[0324] Step 201: Send first information related to an AI function to a terminal, where the first information is used to determine a performance evaluation result of the AI ​​function or an AI model in the AI ​​function.

[0325] Step 202: Receive second information sent by the terminal, where the second information includes at least one of the following: a performance evaluation result of the AI ​​function; a performance evaluation result of the AI ​​model in the AI ​​function; first indication information; the first indication information is used to indicate whether the AI ​​function or the AI ​​model in the AI ​​function is available or whether the performance meets performance requirements; the first indication information is determined based on the performance evaluation result of the AI ​​function or the AI ​​model in the AI ​​function.

[0326] In some embodiments, when the second information includes a performance evaluation result of the AI ​​function, whether the AI ​​function is available or whether the performance meets the performance requirements is determined based on the performance evaluation result of the AI ​​function.

[0327] In some embodiments, when the second information includes a performance evaluation result of the AI ​​model in the AI ​​function, whether the AI ​​model in the AI ​​function is available or whether the performance meets the performance requirements is determined based on the performance evaluation result of the AI ​​model in the AI ​​function.

[0328] In some embodiments, the first information includes at least one of the following:

[0329] Configuration information for performance evaluation;

[0330] performance requirements;

[0331] Dataset information used for performance evaluation.

[0332] In some embodiments, the configuration information for performance evaluation includes at least one of the following:

[0333] The identification of the AI ​​function;

[0334] An identifier of the RRC configuration information corresponding to the AI ​​function;

[0335] The identifier of the AI ​​model in the AI ​​function;

[0336] An identifier of the RRC configuration information corresponding to the AI ​​model in the AI ​​function;

[0337] Reference signals for performance evaluation;

[0338] Configuration information for reporting information;

[0339] performance metrics for performance evaluation;

[0340] Reporting metrics for performance evaluation;

[0341] performance requirements;

[0342] Performance threshold.

[0343] In some embodiments, the dataset information includes at least one of the following:

[0344] Second indication information, used to indicate that the data set is used for performance evaluation;

[0345] The identification of the AI ​​function;

[0346] An identifier of the RRC configuration information corresponding to the AI ​​function;

[0347] Input data for the AI ​​function;

[0348] The identifier of the AI ​​model in the AI ​​function;

[0349] An identifier of the RRC configuration information corresponding to the AI ​​model in the AI ​​function;

[0350] Input data of the AI ​​model in the AI ​​function;

[0351] Reference information or true value information used to calculate performance indicators;

[0352] The number of samples used for performance evaluation;

[0353] performance metrics for performance evaluation;

[0354] Reporting metrics for performance evaluation;

[0355] Configuration information used for reporting information.

[0356] In some embodiments, the method further comprises:

[0357] Receive third information sent by the terminal, where the third information includes at least one of the following:

[0358] The identification of the AI ​​function;

[0359] An identifier of the RRC configuration information corresponding to the AI ​​function;

[0360] The identifier of the AI ​​model in the AI ​​function;

[0361] An identifier of the RRC configuration information corresponding to the AI ​​model in the AI ​​function;

[0362] Request information for configuration information used for performance evaluation;

[0363] performance metrics for performance evaluation;

[0364] Reporting metrics for performance evaluation;

[0365] The number of samples used for performance evaluation;

[0366] The request information is used to request the network device to indicate a performance requirement or a performance threshold.

[0367] In some embodiments, the performance evaluation result of the AI ​​function or the AI ​​model in the AI ​​function includes at least one of the following:

[0368] Performance evaluation results of all AI models in the AI ​​function;

[0369] Performance evaluation results of N AI models in the AI ​​function; N is an integer greater than 0;

[0370] Performance evaluation results of AI models that meet the performance requirements of the AI ​​function;

[0371] The number of AI models in the AI ​​function that meet the performance requirements.

[0372] In some embodiments, the second information further includes at least one of the following:

[0373] Performance evaluation reference results obtained from non-AI functions;

[0374] The identifier of the AI ​​model in the AI ​​function.

[0375] In some embodiments, the identifier of the AI ​​model in the AI ​​function includes an index, and the length of the index of the AI ​​model in the AI ​​function is obtained according to a preset rule, and the preset rule is to determine the length of the index of the AI ​​model according to the number of AI models in the AI ​​function or the number of reported AI models.

[0376] In some embodiments, after receiving the second information sent by the terminal, the method further includes:

[0377] Sending fourth information to the terminal, where the fourth information includes at least one of the following:

[0378] activation instruction information of the AI ​​function;

[0379] Activation indication information of the AI ​​model in the AI ​​function;

[0380] third indication information, used to indicate that the AI ​​function or the AI ​​model in the AI ​​function is available, or that the AI ​​function or the AI ​​model in the AI ​​function meets performance requirements;

[0381] Fourth indication information, used to indicate that the AI ​​function or the AI ​​model in the AI ​​function is unavailable, or that the AI ​​function or the AI ​​model in the AI ​​function does not meet performance requirements;

[0382] The identifier of the target model in the AI ​​function;

[0383] The number of target models in the AI ​​function;

[0384] The target model is an available model, an activatable model, or a model that meets performance requirements.

[0385] In some embodiments, before receiving the second information sent by the terminal, the method further includes:

[0386] Receive the number of AI models included in the AI ​​function sent by the terminal.

[0387] It should be noted here that the above method provided by the embodiment of the present disclosure has the same principle as the method implemented by the aforementioned method embodiment in which the execution subject is a terminal, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as the aforementioned embodiments will not be described in detail here.

[0388] FIG3 is a schematic diagram of the structure of a terminal provided by an embodiment of the present disclosure. As shown in FIG3 , the terminal includes a memory 320, a transceiver 300, and a processor 310, wherein:

[0389] The memory 320 is used to store computer programs; the transceiver 300 is used to send and receive data under the control of the processor 310; the processor 310 is used to read the computer program in the memory 320 and perform the following operations:

[0390] Determining, based on first information related to the AI ​​function from the network device, a performance evaluation result of the AI ​​function or an AI model in the AI ​​function;

[0391] Sending second information to the network device, the second information including at least one of the following: a performance evaluation result of the AI ​​function; a performance evaluation result of the AI ​​model in the AI ​​function; first indication information; the first indication information is used to indicate whether the AI ​​function or the AI ​​model in the AI ​​function is available or whether the performance meets the performance requirements; the first indication information is determined based on the performance evaluation result of the AI ​​function or the AI ​​model in the AI ​​function.

[0392] Specifically, the transceiver 300 is configured to receive and send data under the control of the processor 310 .

[0393] In FIG3 , the bus architecture may include any number of interconnected buses and bridges, specifically various circuits connected together by one or more processors represented by processor 310 and memory represented by memory 320. The bus architecture may also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and are therefore not described further herein. The bus interface provides an interface. The transceiver 300 may be a plurality of components, including a transmitter and a receiver, providing a unit for communicating with various other devices on a transmission medium, such as a wireless channel, a wired channel, an optical cable, and the like. For different user devices, the user interface 330 may also be an interface capable of connecting external or internal devices as required, and the connected devices include but are not limited to a keypad, a display, a speaker, a microphone, a joystick, and the like.

[0394] The processor 310 is responsible for managing the bus architecture and general processing, and the memory 320 can store data used by the processor 310 when performing operations.

[0395] In some embodiments, the processor 310 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or a complex programmable logic device (CPLD). The processor may also adopt a multi-core architecture.

[0396] The processor calls the computer program stored in the memory to execute any of the methods provided by the embodiments of the present disclosure according to the obtained executable instructions. The processor and the memory can also be arranged physically separately.

[0397] In some embodiments, the first information includes at least one of the following:

[0398] Configuration information for performance evaluation;

[0399] performance requirements;

[0400] Dataset information used for performance evaluation.

[0401] In some embodiments, the configuration information for performance evaluation includes at least one of the following:

[0402] The identification of the AI ​​function;

[0403] An identifier of the radio resource control (RRC) configuration information corresponding to the AI ​​function;

[0404] The identifier of the AI ​​model in the AI ​​function;

[0405] An identifier of the RRC configuration information corresponding to the AI ​​model in the AI ​​function;

[0406] Reference signals for performance evaluation;

[0407] Configuration information for reporting information;

[0408] performance metrics for performance evaluation;

[0409] Reporting metrics for performance evaluation;

[0410] performance requirements;

[0411] Performance threshold.

[0412] In some embodiments, the dataset information includes at least one of the following:

[0413] Second indication information, used to indicate that the data set is used for performance evaluation;

[0414] The identification of the AI ​​function;

[0415] An identifier of the RRC configuration information corresponding to the AI ​​function;

[0416] Input data for the AI ​​function;

[0417] The identifier of the AI ​​model in the AI ​​function;

[0418] An identifier of the RRC configuration information corresponding to the AI ​​model in the AI ​​function;

[0419] Input data of the AI ​​model in the AI ​​function;

[0420] Reference information or true value information used to calculate performance indicators;

[0421] The number of samples used for performance evaluation;

[0422] performance metrics for performance evaluation;

[0423] Reporting metrics for performance evaluation;

[0424] Configuration information used for reporting information.

[0425] In some embodiments, before determining, based on the first information related to the AI ​​function from the network device, a performance evaluation result of the AI ​​function or an AI model in the AI ​​function, the operation further includes:

[0426] Sending third information to the network device; the third information includes at least one of the following:

[0427] The identification of the AI ​​function;

[0428] An identifier of the RRC configuration information corresponding to the AI ​​function;

[0429] The identifier of the AI ​​model in the AI ​​function;

[0430] An identifier of the RRC configuration information corresponding to the AI ​​model in the AI ​​function;

[0431] Request information for configuration information used for performance evaluation;

[0432] performance metrics for performance evaluation;

[0433] Reporting metrics for performance evaluation;

[0434] The number of samples used for performance evaluation;

[0435] The request information is used to request the network device to indicate a performance requirement or a performance threshold.

[0436] In some embodiments, the performance evaluation result of the AI ​​function or the AI ​​model in the AI ​​function includes at least one of the following:

[0437] Performance evaluation results of all AI models in the AI ​​function;

[0438] Performance evaluation results of N AI models in the AI ​​function; N is an integer greater than 0;

[0439] Performance evaluation results of AI models that meet the performance requirements of the AI ​​function;

[0440] The number of AI models in the AI ​​function that meet the performance requirements.

[0441] In some embodiments, before sending the second information to the network device, the operation further includes:

[0442] determining whether the AI ​​function is available or meets the performance requirement based on the performance evaluation result of the AI ​​function and the performance requirement; or

[0443] determining, based on a performance evaluation result of the AI ​​model in the AI ​​function and the performance requirement, whether the AI ​​model in the AI ​​function is usable or meets the performance requirement;

[0444] The first indication information is used to indicate at least one of the following: the AI ​​function is unavailable or does not meet the performance requirements; the AI ​​function is available or meets the performance requirements; the AI ​​model in the AI ​​function is unavailable or does not meet the performance requirements; the AI ​​model in the AI ​​function is available or meets the performance requirements.

[0445] In some embodiments, the second information further includes at least one of the following:

[0446] Performance evaluation reference results obtained from non-AI functions;

[0447] The identifier of the AI ​​model in the AI ​​function.

[0448] In some embodiments, the identifier of the AI ​​model in the AI ​​function includes an index, and the length of the index of the AI ​​model in the AI ​​function is obtained according to a preset rule, and the preset rule is to determine the length of the index of the AI ​​model according to the number of AI models in the AI ​​function or the number of reported AI models.

[0449] In some embodiments, after sending the second information to the network device, the operation further includes:

[0450] Receive fourth information sent by the network device, where the fourth information includes at least one of the following:

[0451] activation instruction information of the AI ​​function;

[0452] Activation indication information of the AI ​​model in the AI ​​function;

[0453] third indication information, used to indicate that the AI ​​function or the AI ​​model in the AI ​​function is available, or that the AI ​​function or the AI ​​model in the AI ​​function meets performance requirements;

[0454] Fourth indication information, used to indicate that the AI ​​function or the AI ​​model in the AI ​​function is unavailable, or that the AI ​​function or the AI ​​model in the AI ​​function does not meet performance requirements;

[0455] The identifier of the target model in the AI ​​function;

[0456] The number of target models in the AI ​​function;

[0457] The target model is an available model, an activatable model, or a model that meets performance requirements.

[0458] In some embodiments, before sending the second information to the network device, the operation further includes:

[0459] The number of AI models included in the AI ​​function is sent to the network device.

[0460] It should be noted here that the above-mentioned terminal provided in the embodiment of the present disclosure can implement all the method steps implemented by the method embodiment in which the execution subject is the terminal, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as those of the method embodiment will not be described in detail here.

[0461] FIG4 is a schematic diagram of the structure of a network device provided by an embodiment of the present disclosure. As shown in FIG4 , the network device includes a memory 420, a transceiver 400, and a processor 410, wherein:

[0462] The memory 420 is used to store computer programs; the transceiver 400 is used to send and receive data under the control of the processor 410; the processor 410 is used to read the computer program in the memory 420 and perform the following operations:

[0463] Sending first information related to the AI ​​function to the terminal, where the first information is used to determine a performance evaluation result of the AI ​​function or an AI model in the AI ​​function;

[0464] Receive second information sent by the terminal, where the second information includes at least one of the following: a performance evaluation result of the AI ​​function; a performance evaluation result of the AI ​​model in the AI ​​function; first indication information; the first indication information is used to indicate whether the AI ​​function or the AI ​​model in the AI ​​function is available or whether the performance meets performance requirements; the first indication information is determined based on the performance evaluation result of the AI ​​function or the AI ​​model in the AI ​​function.

[0465] Specifically, the transceiver 400 is configured to receive and send data under the control of the processor 410 .

[0466] In FIG4 , the bus architecture may include any number of interconnected buses and bridges, specifically various circuits linked together by one or more processors represented by processor 410 and memory represented by memory 420. The bus architecture may also link together various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface. The transceiver 400 may be a plurality of components, namely, a transmitter and a receiver, providing a unit for communicating with various other devices on a transmission medium, such as a wireless channel, a wired channel, an optical cable, and the like. The processor 410 is responsible for managing the bus architecture and general processing, and the memory 420 may store data used by the processor 410 when performing operations.

[0467] The processor 410 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or a complex programmable logic device (CPLD). The processor may also adopt a multi-core architecture.

[0468] In some embodiments, the first information includes at least one of the following:

[0469] Configuration information for performance evaluation;

[0470] performance requirements;

[0471] Dataset information used for performance evaluation.

[0472] In some embodiments, the configuration information for performance evaluation includes at least one of the following:

[0473] The identification of the AI ​​function;

[0474] An identifier of the RRC configuration information corresponding to the AI ​​function;

[0475] The identifier of the AI ​​model in the AI ​​function;

[0476] An identifier of the RRC configuration information corresponding to the AI ​​model in the AI ​​function;

[0477] Reference signals for performance evaluation;

[0478] Configuration information for reporting information;

[0479] performance metrics for performance evaluation;

[0480] Reporting metrics for performance evaluation;

[0481] performance requirements;

[0482] Performance threshold.

[0483] In some embodiments, the dataset information includes at least one of the following:

[0484] Second indication information, used to indicate that the data set is used for performance evaluation;

[0485] The identification of the AI ​​function;

[0486] An identifier of the RRC configuration information corresponding to the AI ​​function;

[0487] Input data for the AI ​​function;

[0488] The identifier of the AI ​​model in the AI ​​function;

[0489] An identifier of the RRC configuration information corresponding to the AI ​​model in the AI ​​function;

[0490] Input data of the AI ​​model in the AI ​​function;

[0491] Reference information or true value information used to calculate performance indicators;

[0492] The number of samples used for performance evaluation;

[0493] performance metrics for performance evaluation;

[0494] Reporting metrics for performance evaluation;

[0495] Configuration information used for reporting information.

[0496] In some embodiments, the operations further include:

[0497] Receive third information sent by the terminal, where the third information includes at least one of the following:

[0498] The identification of the AI ​​function;

[0499] An identifier of the RRC configuration information corresponding to the AI ​​function;

[0500] The identifier of the AI ​​model in the AI ​​function;

[0501] An identifier of the RRC configuration information corresponding to the AI ​​model in the AI ​​function;

[0502] Request information for configuration information used for performance evaluation;

[0503] performance metrics for performance evaluation;

[0504] Reporting metrics for performance evaluation;

[0505] The number of samples used for performance evaluation;

[0506] The request information is used to request the network device to indicate a performance requirement or a performance threshold.

[0507] In some embodiments, the performance evaluation result of the AI ​​function or the AI ​​model in the AI ​​function includes at least one of the following:

[0508] Performance evaluation results of all AI models in the AI ​​function;

[0509] Performance evaluation results of N AI models in the AI ​​function; N is an integer greater than 0;

[0510] Performance evaluation results of AI models that meet the performance requirements of the AI ​​function;

[0511] The number of AI models in the AI ​​function that meet the performance requirements.

[0512] In some embodiments, the second information further includes at least one of the following:

[0513] Performance evaluation reference results obtained from non-AI functions;

[0514] The identifier of the AI ​​model in the AI ​​function.

[0515] In some embodiments, the identifier of the AI ​​model in the AI ​​function includes an index, and the length of the index of the AI ​​model in the AI ​​function is obtained according to a preset rule, and the preset rule is to determine the length of the index of the AI ​​model according to the number of AI models in the AI ​​function or the number of reported AI models.

[0516] In some embodiments, after receiving the second information sent by the terminal, the operation further includes:

[0517] Sending fourth information to the terminal, where the fourth information includes at least one of the following:

[0518] activation instruction information of the AI ​​function;

[0519] Activation indication information of the AI ​​model in the AI ​​function;

[0520] third indication information, used to indicate that the AI ​​function or the AI ​​model in the AI ​​function is available, or that the AI ​​function or the AI ​​model in the AI ​​function meets performance requirements;

[0521] Fourth indication information, used to indicate that the AI ​​function or the AI ​​model in the AI ​​function is unavailable, or that the AI ​​function or the AI ​​model in the AI ​​function does not meet performance requirements;

[0522] The identifier of the target model in the AI ​​function;

[0523] The number of target models in the AI ​​function;

[0524] The target model is an available model, an activatable model, or a model that meets performance requirements.

[0525] In some embodiments, before receiving the second information sent by the terminal, the operation further includes:

[0526] Receive the number of AI models included in the AI ​​function sent by the terminal.

[0527] It should be noted here that the above-mentioned network device provided in the embodiment of the present disclosure can implement all the method steps implemented by the method embodiment in which the execution subject is the network device, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as those of the method embodiment will not be described in detail here.

[0528] FIG5 is a schematic diagram of a structure of a device for evaluating the performance of an AI function or AI model provided by an embodiment of the present disclosure. As shown in FIG5 , the device for evaluating the performance of an AI function or AI model includes a processing unit 510 and a first sending unit 520, wherein:

[0529] The processing unit 510 is configured to determine a performance evaluation result of the AI ​​function or an AI model in the AI ​​function based on first information related to the AI ​​function from the network device;

[0530] The first sending unit 520 is used to send second information to the network device, where the second information includes at least one of the following: a performance evaluation result of the AI ​​function; a performance evaluation result of the AI ​​model in the AI ​​function; first indication information; the first indication information is used to indicate whether the AI ​​function or the AI ​​model in the AI ​​function is available or whether the performance meets the performance requirements; the first indication information is determined based on the performance evaluation result of the AI ​​function or the AI ​​model in the AI ​​function.

[0531] In some embodiments, the first information includes at least one of the following:

[0532] Configuration information for performance evaluation;

[0533] performance requirements;

[0534] Dataset information used for performance evaluation.

[0535] In some embodiments, the configuration information for performance evaluation includes at least one of the following:

[0536] The identification of the AI ​​function;

[0537] An identifier of the radio resource control (RRC) configuration information corresponding to the AI ​​function;

[0538] The identifier of the AI ​​model in the AI ​​function;

[0539] An identifier of the RRC configuration information corresponding to the AI ​​model in the AI ​​function;

[0540] Reference signals for performance evaluation;

[0541] Configuration information for reporting information;

[0542] performance metrics for performance evaluation;

[0543] Reporting metrics for performance evaluation;

[0544] performance requirements;

[0545] Performance threshold.

[0546] In some embodiments, the dataset information includes at least one of the following:

[0547] Second indication information, used to indicate that the data set is used for performance evaluation;

[0548] The identification of the AI ​​function;

[0549] An identifier of the RRC configuration information corresponding to the AI ​​function;

[0550] Input data for the AI ​​function;

[0551] The identifier of the AI ​​model in the AI ​​function;

[0552] An identifier of the RRC configuration information corresponding to the AI ​​model in the AI ​​function;

[0553] Input data of the AI ​​model in the AI ​​function;

[0554] Reference information or true value information used to calculate performance indicators;

[0555] The number of samples used for performance evaluation;

[0556] performance metrics for performance evaluation;

[0557] Reporting metrics for performance evaluation;

[0558] Configuration information used for reporting information.

[0559] In some embodiments, the first sending unit 520 is further configured to:

[0560] Before determining a performance evaluation result of the AI ​​function or an AI model in the AI ​​function based on the first information related to the AI ​​function from the network device, third information is sent to the network device; the third information includes at least one of the following:

[0561] The identification of the AI ​​function;

[0562] An identifier of the RRC configuration information corresponding to the AI ​​function;

[0563] The identifier of the AI ​​model in the AI ​​function;

[0564] An identifier of the RRC configuration information corresponding to the AI ​​model in the AI ​​function;

[0565] Request information for configuration information used for performance evaluation;

[0566] performance metrics for performance evaluation;

[0567] Reporting metrics for performance evaluation;

[0568] The number of samples used for performance evaluation;

[0569] The request information is used to request the network device to indicate a performance requirement or a performance threshold.

[0570] In some embodiments, the performance evaluation result of the AI ​​function or the AI ​​model in the AI ​​function includes at least one of the following:

[0571] Performance evaluation results of all AI models in the AI ​​function;

[0572] Performance evaluation results of N AI models in the AI ​​function; N is an integer greater than 0;

[0573] Performance evaluation results of AI models that meet the performance requirements of the AI ​​function;

[0574] The number of AI models in the AI ​​function that meet the performance requirements.

[0575] In some embodiments, the processing unit 510 is further configured to:

[0576] Before sending the second information to the network device, determining whether the AI ​​function is available or meets the performance requirement based on the performance evaluation result of the AI ​​function and the performance requirement; or,

[0577] determining, based on a performance evaluation result of the AI ​​model in the AI ​​function and the performance requirement, whether the AI ​​model in the AI ​​function is usable or meets the performance requirement;

[0578] The first indication information is used to indicate at least one of the following: the AI ​​function is unavailable or does not meet the performance requirements; the AI ​​function is available or meets the performance requirements; the AI ​​model in the AI ​​function is unavailable or does not meet the performance requirements; the AI ​​model in the AI ​​function is available or meets the performance requirements.

[0579] In some embodiments, the second information further includes at least one of the following:

[0580] Performance evaluation reference results obtained from non-AI functions;

[0581] The identifier of the AI ​​model in the AI ​​function.

[0582] In some embodiments, the identifier of the AI ​​model in the AI ​​function includes an index, and the length of the index of the AI ​​model in the AI ​​function is obtained according to a preset rule, and the preset rule is to determine the length of the index of the AI ​​model according to the number of AI models in the AI ​​function or the number of reported AI models.

[0583] In some embodiments, the apparatus further comprises:

[0584] a receiving unit, configured to receive fourth information sent by the network device after sending the second information to the network device, where the fourth information includes at least one of the following:

[0585] activation instruction information of the AI ​​function;

[0586] Activation indication information of the AI ​​model in the AI ​​function;

[0587] third indication information, used to indicate that the AI ​​function or the AI ​​model in the AI ​​function is available, or that the AI ​​function or the AI ​​model in the AI ​​function meets performance requirements;

[0588] Fourth indication information, used to indicate that the AI ​​function or the AI ​​model in the AI ​​function is unavailable, or that the AI ​​function or the AI ​​model in the AI ​​function does not meet performance requirements;

[0589] The identifier of the target model in the AI ​​function;

[0590] The number of target models in the AI ​​function;

[0591] The target model is an available model, an activatable model, or a model that meets performance requirements.

[0592] In some embodiments, the first sending unit 520 is further configured to:

[0593] Before sending the second information to the network device, the number of AI models included in the AI ​​function is sent to the network device.

[0594] It should be noted that the performance evaluation device for the above-mentioned AI function or AI model provided in the embodiment of the present disclosure can implement all the method steps implemented in the method embodiment in which the execution subject is the terminal, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as those of the method embodiment will not be described in detail here.

[0595] FIG6 is a second structural diagram of the performance evaluation device for an AI function or AI model provided by an embodiment of the present disclosure. As shown in FIG6 , the performance evaluation device for an AI function or AI model includes a second sending unit 610 and a receiving unit 620, wherein:

[0596] The second sending unit 610 is configured to send first information related to the AI ​​function to the terminal, where the first information is used to determine a performance evaluation result of the AI ​​function or an AI model in the AI ​​function;

[0597] The receiving unit 620 is used to receive second information sent by the terminal, where the second information includes at least one of the following: a performance evaluation result of the AI ​​function; a performance evaluation result of the AI ​​model in the AI ​​function; and first indication information. The first indication information is used to indicate whether the AI ​​function or the AI ​​model in the AI ​​function is available or whether the performance meets performance requirements. The first indication information is determined based on the performance evaluation result of the AI ​​function or the AI ​​model in the AI ​​function.

[0598] In some embodiments, the first information includes at least one of the following:

[0599] Configuration information for performance evaluation;

[0600] performance requirements;

[0601] Dataset information used for performance evaluation.

[0602] In some embodiments, the configuration information for performance evaluation includes at least one of the following:

[0603] The identification of the AI ​​function;

[0604] An identifier of the RRC configuration information corresponding to the AI ​​function;

[0605] The identifier of the AI ​​model in the AI ​​function;

[0606] An identifier of the RRC configuration information corresponding to the AI ​​model in the AI ​​function;

[0607] Reference signals for performance evaluation;

[0608] Configuration information for reporting information;

[0609] performance metrics for performance evaluation;

[0610] Reporting metrics for performance evaluation;

[0611] performance requirements;

[0612] Performance threshold.

[0613] In some embodiments, the dataset information includes at least one of the following:

[0614] Second indication information, used to indicate that the data set is used for performance evaluation;

[0615] The identification of the AI ​​function;

[0616] An identifier of the RRC configuration information corresponding to the AI ​​function;

[0617] Input data for the AI ​​function;

[0618] The identifier of the AI ​​model in the AI ​​function;

[0619] An identifier of the RRC configuration information corresponding to the AI ​​model in the AI ​​function;

[0620] Input data of the AI ​​model in the AI ​​function;

[0621] Reference information or true value information used to calculate performance indicators;

[0622] The number of samples used for performance evaluation;

[0623] performance metrics for performance evaluation;

[0624] Reporting metrics for performance evaluation;

[0625] Configuration information used for reporting information.

[0626] In some embodiments, the receiving unit 620 is further configured to:

[0627] Receive third information sent by the terminal, where the third information includes at least one of the following:

[0628] The identification of the AI ​​function;

[0629] An identifier of the RRC configuration information corresponding to the AI ​​function;

[0630] The identifier of the AI ​​model in the AI ​​function;

[0631] An identifier of the RRC configuration information corresponding to the AI ​​model in the AI ​​function;

[0632] Request information for configuration information used for performance evaluation;

[0633] performance metrics for performance evaluation;

[0634] Reporting metrics for performance evaluation;

[0635] The number of samples used for performance evaluation;

[0636] The request information is used to request the network device to indicate a performance requirement or a performance threshold.

[0637] In some embodiments, the performance evaluation result of the AI ​​function or the AI ​​model in the AI ​​function includes at least one of the following:

[0638] Performance evaluation results of all AI models in the AI ​​function;

[0639] Performance evaluation results of N AI models in the AI ​​function; N is an integer greater than 0;

[0640] Performance evaluation results of AI models that meet the performance requirements of the AI ​​function;

[0641] The number of AI models in the AI ​​function that meet the performance requirements.

[0642] In some embodiments, the second information further includes at least one of the following:

[0643] Performance evaluation reference results obtained from non-AI functions;

[0644] The identifier of the AI ​​model in the AI ​​function.

[0645] In some embodiments, the identifier of the AI ​​model in the AI ​​function includes an index, and the length of the index of the AI ​​model in the AI ​​function is obtained according to a preset rule, and the preset rule is to determine the length of the index of the AI ​​model according to the number of AI models in the AI ​​function or the number of reported AI models.

[0646] In some embodiments, the second sending unit 610 is further configured to:

[0647] After receiving the second information sent by the terminal, fourth information is sent to the terminal, where the fourth information includes at least one of the following:

[0648] activation instruction information of the AI ​​function;

[0649] Activation indication information of the AI ​​model in the AI ​​function;

[0650] third indication information, used to indicate that the AI ​​function or the AI ​​model in the AI ​​function is available, or that the AI ​​function or the AI ​​model in the AI ​​function meets performance requirements;

[0651] Fourth indication information, used to indicate that the AI ​​function or the AI ​​model in the AI ​​function is unavailable, or that the AI ​​function or the AI ​​model in the AI ​​function does not meet performance requirements;

[0652] The identifier of the target model in the AI ​​function;

[0653] The number of target models in the AI ​​function;

[0654] The target model is an available model, an activatable model, or a model that meets performance requirements.

[0655] In some embodiments, the receiving unit 620 is further configured to:

[0656] Before receiving the second information sent by the terminal, the number of AI models included in the AI ​​function sent by the terminal is received.

[0657] It should be noted that the performance evaluation device for the above-mentioned AI function or AI model provided in the embodiment of the present disclosure can implement all the method steps implemented in the method embodiment in which the execution subject is a network device, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as those of the method embodiment will not be described in detail here.

[0658] It should be noted that the division of units / modules in the above-mentioned embodiments of the present disclosure is schematic and is merely a logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional units in the various embodiments of the present disclosure may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The above-mentioned integrated units may be implemented in the form of hardware or software functional units.

[0659] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a processor-readable storage medium. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0660] In some embodiments, a non-transitory readable storage medium is also provided, which stores a computer program, and the computer program is used to enable the processor to execute the performance evaluation method of the AI ​​function or AI model provided by the above-mentioned method embodiments.

[0661] Specifically, the above-mentioned non-transitory readable storage medium provided by the embodiment of the present disclosure can implement all the method steps implemented by the above-mentioned method embodiments, and can achieve the same technical effects. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.

[0662] It should be noted that the non-transitory readable storage medium can be any available medium or data storage device that can be accessed by the processor, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (such as CDs, DVDs, BDs, HVDs, etc.), and semiconductor storage (such as ROMs, EPROMs, EEPROMs, non-volatile memories (NAND FLASH), solid-state drives (SSDs)), etc.

[0663] In some embodiments, a processor-readable storage medium is also provided, which stores a computer program, and the computer program is used to enable the processor to execute the performance evaluation method of the AI ​​function or AI model provided by the above-mentioned method embodiments.

[0664] Specifically, the processor-readable storage medium provided in the embodiment of the present disclosure can implement all the method steps implemented in the above-mentioned method embodiments and can achieve the same technical effects. The parts and beneficial effects of this embodiment that are the same as those of the method embodiment will not be described in detail here.

[0665] In some embodiments, a computer-readable storage medium is also provided, wherein the computer-readable storage medium stores a computer program, and the computer program is used to enable a computer to execute the above-mentioned method embodiments to provide a performance evaluation method for AI functions or AI models.

[0666] Specifically, the above-mentioned computer-readable storage medium provided by the embodiment of the present disclosure can implement all the method steps implemented by the above-mentioned method embodiments, and can achieve the same technical effects. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.

[0667] In some embodiments, a chip product is also provided, in which a computer program is stored, and the computer program is used to enable the chip product to execute the performance evaluation method of the AI ​​function or AI model provided by the above-mentioned method embodiments.

[0668] Specifically, the above-mentioned chip product provided by the embodiment of the present disclosure can implement all the method steps implemented by the above-mentioned method embodiments, and can achieve the same technical effects. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.

[0669] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage, etc.) containing computer-usable program code.

[0670] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer-executable instructions. These computer-executable instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0671] These processor-executable instructions may also be stored in a processor-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the processor-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0672] These processor-executable instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0673] Obviously, those skilled in the art may make various changes and modifications to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is intended to include these modifications and variations.

Claims

1. A performance evaluation method for an AI function or AI model, applied to a terminal, comprising: Determining, based on first information related to the AI ​​function from the network device, a performance evaluation result of the AI ​​function or an AI model in the AI ​​function; Sending second information to the network device, the second information including at least one of the following: a performance evaluation result of the AI ​​function; a performance evaluation result of the AI ​​model in the AI ​​function; first indication information; the first indication information is used to indicate whether the AI ​​function or the AI ​​model in the AI ​​function is available or whether the performance meets the performance requirements; the first indication information is determined based on the performance evaluation result of the AI ​​function or the AI ​​model in the AI ​​function.

2. The method according to claim 1, wherein The first information includes at least one of the following: Configuration information for performance evaluation; performance requirements; Dataset information used for performance evaluation.

3. The method according to claim 2, wherein: The configuration information for performance evaluation includes at least one of the following: The identification of the AI ​​function; An identifier of the radio resource control (RRC) configuration information corresponding to the AI ​​function; The identifier of the AI ​​model in the AI ​​function; An identifier of the RRC configuration information corresponding to the AI ​​model in the AI ​​function; Reference signals for performance evaluation; Configuration information for reporting information; performance metrics for performance evaluation; Reporting metrics for performance evaluation; performance requirements; Performance threshold.

4. The method according to claim 2, wherein: The data set information includes at least one of the following: Second indication information, used to indicate that the data set is used for performance evaluation; The identification of the AI ​​function; An identifier of the RRC configuration information corresponding to the AI ​​function; Input data for the AI ​​function; The identifier of the AI ​​model in the AI ​​function; An identifier of the RRC configuration information corresponding to the AI ​​model in the AI ​​function; Input data of the AI ​​model in the AI ​​function; Reference information or true value information used to calculate performance indicators; The number of samples used for performance evaluation; performance metrics for performance evaluation; Reporting metrics for performance evaluation; Configuration information used for reporting information.

5. The method according to any one of claims 1 to 4, wherein: Before determining a performance evaluation result of the AI ​​function or an AI model in the AI ​​function based on the first information related to the AI ​​function from the network device, the method further includes: Sending third information to the network device; the third information includes at least one of the following: The identification of the AI ​​function; An identifier of the RRC configuration information corresponding to the AI ​​function; The identifier of the AI ​​model in the AI ​​function; An identifier of the RRC configuration information corresponding to the AI ​​model in the AI ​​function; Request information for configuration information used for performance evaluation; performance metrics for performance evaluation; Reporting metrics for performance evaluation; The number of samples used for performance evaluation; The request information is used to request the network device to indicate a performance requirement or a performance threshold.

6. The method according to any one of claims 1 to 5, wherein: The performance evaluation result of the AI ​​function or the AI ​​model in the AI ​​function includes at least one of the following: Performance evaluation results of all AI models in the AI ​​function; Performance evaluation results of N AI models in the AI ​​function; N is an integer greater than 0; Performance evaluation results of AI models that meet the performance requirements of the AI ​​function; The number of AI models in the AI ​​function that meet the performance requirements.

7. The method according to any one of claims 1 to 6, wherein: Before sending the second information to the network device, the method further includes: determining whether the AI ​​function is available or meets the performance requirement based on the performance evaluation result of the AI ​​function and the performance requirement; or determining, based on a performance evaluation result of the AI ​​model in the AI ​​function and the performance requirement, whether the AI ​​model in the AI ​​function is usable or meets the performance requirement; The first indication information is used to indicate at least one of the following: the AI ​​function is unavailable or does not meet the performance requirements; the AI ​​function is available or meets the performance requirements; the AI ​​model in the AI ​​function is unavailable or does not meet the performance requirements; the AI ​​model in the AI ​​function is available or meets the performance requirements.

8. The method according to any one of claims 1 to 7, wherein: The second information further includes at least one of the following: Performance evaluation reference results obtained from non-AI functions; The identifier of the AI ​​model in the AI ​​function.

9. The method according to claim 8, wherein The identifier of the AI ​​model in the AI ​​function includes an index, and the length of the index of the AI ​​model in the AI ​​function is obtained according to a preset rule, and the preset rule is to determine the length of the index of the AI ​​model according to the number of AI models in the AI ​​function or the number of reported AI models.

10. The method according to any one of claims 1 to 9, wherein: After sending the second information to the network device, the method further includes: Receive fourth information sent by the network device, where the fourth information includes at least one of the following: activation instruction information of the AI ​​function; Activation indication information of the AI ​​model in the AI ​​function; third indication information, used to indicate that the AI ​​function or the AI ​​model in the AI ​​function is available, or that the AI ​​function or the AI ​​model in the AI ​​function meets performance requirements; Fourth indication information, used to indicate that the AI ​​function or the AI ​​model in the AI ​​function is unavailable, or that the AI ​​function or the AI ​​model in the AI ​​function does not meet performance requirements; The identifier of the target model in the AI ​​function; The number of target models in the AI ​​function; The target model is an available model, an activatable model, or a model that meets performance requirements.

11. The method according to any one of claims 1 to 10, wherein: Before sending the second information to the network device, the method further includes: The number of AI models included in the AI ​​function is sent to the network device.

12. A performance evaluation method for an AI function or AI model, applied to a network device, comprising: Sending first information related to the AI ​​function to the terminal, where the first information is used to determine a performance evaluation result of the AI ​​function or an AI model in the AI ​​function; Receive second information sent by the terminal, where the second information includes at least one of the following: a performance evaluation result of the AI ​​function; a performance evaluation result of the AI ​​model in the AI ​​function; first indication information; the first indication information is used to indicate whether the AI ​​function or the AI ​​model in the AI ​​function is available or whether the performance meets performance requirements; the first indication information is determined based on the performance evaluation result of the AI ​​function or the AI ​​model in the AI ​​function.

13. The method according to claim 12, wherein: The first information includes at least one of the following: Configuration information for performance evaluation; performance requirements; Dataset information used for performance evaluation.

14. The method according to claim 13, wherein: The configuration information for performance evaluation includes at least one of the following: The identification of the AI ​​function; An identifier of the RRC configuration information corresponding to the AI ​​function; The identifier of the AI ​​model in the AI ​​function; An identifier of the RRC configuration information corresponding to the AI ​​model in the AI ​​function; Reference signals for performance evaluation; Configuration information for reporting information; performance metrics for performance evaluation; Reporting metrics for performance evaluation; performance requirements; Performance threshold.

15. The method according to claim 13, wherein The data set information includes at least one of the following: Second indication information, used to indicate that the data set is used for performance evaluation; The identification of the AI ​​function; An identifier of the RRC configuration information corresponding to the AI ​​function; Input data for the AI ​​function; The identifier of the AI ​​model in the AI ​​function; An identifier of the RRC configuration information corresponding to the AI ​​model in the AI ​​function; Input data of the AI ​​model in the AI ​​function; Reference information or true value information used to calculate performance indicators; The number of samples used for performance evaluation; performance metrics for performance evaluation; Reporting metrics for performance evaluation; Configuration information used for reporting information.

16. The method according to any one of claims 12 to 15, wherein: The method further comprises: Receive third information sent by the terminal, where the third information includes at least one of the following: The identification of the AI ​​function; An identifier of the RRC configuration information corresponding to the AI ​​function; The identifier of the AI ​​model in the AI ​​function; An identifier of the RRC configuration information corresponding to the AI ​​model in the AI ​​function; Request information for configuration information used for performance evaluation; performance metrics for performance evaluation; Reporting metrics for performance evaluation; The number of samples used for performance evaluation; The request information is used to request the network device to indicate a performance requirement or a performance threshold.

17. The method according to any one of claims 12 to 16, wherein: The performance evaluation result of the AI ​​function or the AI ​​model in the AI ​​function includes at least one of the following: Performance evaluation results of all AI models in the AI ​​function; Performance evaluation results of N AI models in the AI ​​function; N is an integer greater than 0; Performance evaluation results of AI models that meet the performance requirements of the AI ​​function; The number of AI models in the AI ​​function that meet the performance requirements.

18. The method according to any one of claims 12 to 17, wherein: The second information further includes at least one of the following: Performance evaluation reference results obtained from non-AI functions; The identifier of the AI ​​model in the AI ​​function.

19. The method according to claim 18, wherein The identifier of the AI ​​model in the AI ​​function includes an index, and the length of the index of the AI ​​model in the AI ​​function is obtained according to a preset rule, and the preset rule is to determine the length of the index of the AI ​​model according to the number of AI models in the AI ​​function or the number of reported AI models.

20. The method according to any one of claims 12 to 19, wherein: After receiving the second information sent by the terminal, the method further includes: Sending fourth information to the terminal, where the fourth information includes at least one of the following: activation instruction information of the AI ​​function; Activation indication information of the AI ​​model in the AI ​​function; third indication information, used to indicate that the AI ​​function or the AI ​​model in the AI ​​function is available, or that the AI ​​function or the AI ​​model in the AI ​​function meets performance requirements; Fourth indication information, used to indicate that the AI ​​function or the AI ​​model in the AI ​​function is unavailable, or that the AI ​​function or the AI ​​model in the AI ​​function does not meet performance requirements; The identifier of the target model in the AI ​​function; The number of target models in the AI ​​function; The target model is an available model, an activatable model, or a model that meets performance requirements.

21. The method according to any one of claims 12 to 20, wherein: Before receiving the second information sent by the terminal, the method further includes: Receive the number of AI models included in the AI ​​function sent by the terminal.

22. A terminal comprising a memory, a transceiver, and a processor: A memory for storing a computer program; a transceiver for transmitting and receiving data under the control of the processor; and a processor for reading the computer program in the memory and performing the following operations: Determining, based on first information related to the AI ​​function from the network device, a performance evaluation result of the AI ​​function or an AI model in the AI ​​function; Sending second information to the network device, the second information including at least one of the following: a performance evaluation result of the AI ​​function; a performance evaluation result of the AI ​​model in the AI ​​function; first indication information; the first indication information is used to indicate whether the AI ​​function or the AI ​​model in the AI ​​function is available or whether the performance meets the performance requirements; the first indication information is determined based on the performance evaluation result of the AI ​​function or the AI ​​model in the AI ​​function.

23. The terminal according to claim 22, wherein: The first information includes at least one of the following: Configuration information for performance evaluation; performance requirements; Dataset information used for performance evaluation. The terminal according to claim 23 , wherein: The configuration information for performance evaluation includes at least one of the following: The identification of the AI ​​function; An identifier of the radio resource control (RRC) configuration information corresponding to the AI ​​function; The identifier of the AI ​​model in the AI ​​function; An identifier of the RRC configuration information corresponding to the AI ​​model in the AI ​​function; Reference signals for performance evaluation; Configuration information for reporting information; performance metrics for performance evaluation; Reporting metrics for performance evaluation; performance requirements; Performance threshold.

25. The terminal according to claim 23, wherein: The data set information includes at least one of the following: Second indication information, used to indicate that the data set is used for performance evaluation; The identification of the AI ​​function; An identifier of the RRC configuration information corresponding to the AI ​​function; Input data for the AI ​​function; The identifier of the AI ​​model in the AI ​​function; An identifier of the RRC configuration information corresponding to the AI ​​model in the AI ​​function; Input data of the AI ​​model in the AI ​​function; Reference information or true value information used to calculate performance indicators; The number of samples used for performance evaluation; performance metrics for performance evaluation; Reporting metrics for performance evaluation; Configuration information used for reporting information.

26. The terminal according to any one of claims 22 to 25, wherein: Before determining, based on the first information related to the AI ​​function from the network device, a performance evaluation result of the AI ​​function or an AI model in the AI ​​function, the operation further includes: Sending third information to the network device; the third information includes at least one of the following: The identification of the AI ​​function; An identifier of the RRC configuration information corresponding to the AI ​​function; The identifier of the AI ​​model in the AI ​​function; An identifier of the RRC configuration information corresponding to the AI ​​model in the AI ​​function; Request information for configuration information used for performance evaluation; performance metrics for performance evaluation; Reporting metrics for performance evaluation; The number of samples used for performance evaluation; The request information is used to request the network device to indicate a performance requirement or a performance threshold.

27. The terminal according to any one of claims 22 to 26, wherein: The performance evaluation result of the AI ​​function or the AI ​​model in the AI ​​function includes at least one of the following: Performance evaluation results of all AI models in the AI ​​function; Performance evaluation results of N AI models in the AI ​​function; N is an integer greater than 0; Performance evaluation results of AI models that meet the performance requirements of the AI ​​function; The number of AI models in the AI ​​function that meet the performance requirements.

28. The terminal according to any one of claims 22 to 27, wherein: After sending the second information to the network device, the operation further includes: Receive fourth information sent by the network device, where the fourth information includes at least one of the following: activation instruction information of the AI ​​function; Activation indication information of the AI ​​model in the AI ​​function; third indication information, used to indicate that the AI ​​function or the AI ​​model in the AI ​​function is available, or that the AI ​​function or the AI ​​model in the AI ​​function meets performance requirements; Fourth indication information, used to indicate that the AI ​​function or the AI ​​model in the AI ​​function is unavailable, or that the AI ​​function or the AI ​​model in the AI ​​function does not meet performance requirements; The identifier of the target model in the AI ​​function; The number of target models in the AI ​​function; The target model is an available model, an activatable model, or a model that meets performance requirements.

29. A network device comprising a memory, a transceiver, and a processor: A memory for storing a computer program; a transceiver for transmitting and receiving data under the control of the processor; and a processor for reading the computer program in the memory and performing the following operations: Sending first information related to the AI ​​function to the terminal, where the first information is used to determine a performance evaluation result of the AI ​​function or an AI model in the AI ​​function; Receive second information sent by the terminal, where the second information includes at least one of the following: a performance evaluation result of the AI ​​function; a performance evaluation result of the AI ​​model in the AI ​​function; first indication information; the first indication information is used to indicate whether the AI ​​function or the AI ​​model in the AI ​​function is available or whether the performance meets performance requirements; the first indication information is determined based on the performance evaluation result of the AI ​​function or the AI ​​model in the AI ​​function.

30. A device for evaluating the performance of an AI function or AI model, the device comprising: a processing unit, configured to determine a performance evaluation result of the AI ​​function or an AI model in the AI ​​function based on first information related to the AI ​​function from a network device; A first sending unit is used to send second information to the network device, where the second information includes at least one of the following: a performance evaluation result of the AI ​​function; a performance evaluation result of the AI ​​model in the AI ​​function; first indication information; the first indication information is used to indicate whether the AI ​​function or the AI ​​model in the AI ​​function is available or whether the performance meets performance requirements; the first indication information is determined based on the performance evaluation result of the AI ​​function or the AI ​​model in the AI ​​function.

31. A device for evaluating the performance of an AI function or AI model, the device comprising: a second sending unit, configured to send first information related to the AI ​​function to the terminal, where the first information is used to determine a performance evaluation result of the AI ​​function or an AI model in the AI ​​function; A receiving unit is configured to receive second information sent by the terminal, where the second information includes at least one of the following: a performance evaluation result of the AI ​​function; a performance evaluation result of the AI ​​model in the AI ​​function; and first indication information; the first indication information is used to indicate whether the AI ​​function or the AI ​​model in the AI ​​function is available or whether the performance meets performance requirements; the first indication information is determined based on the performance evaluation result of the AI ​​function or the AI ​​model in the AI ​​function.

32. A processor-readable storage medium storing a computer program, wherein the computer program is configured to cause the processor to execute the method according to any one of claims 1 to 11 or any one of claims 12 to 21.

Images