Model monitoring method and communication apparatus

Abstract

Provided in the present application are a model monitoring method and a communication apparatus. On the basis of the present application, a terminal device and / or a network device can acquire performance information of a first model, wherein the performance information of the first model is determined on the basis of at least one validation data set, so as to facilitate the terminal device and / or the network device to determine, on the basis of the performance information of the first model, a first monitoring parameter that better matches the performance of the first model, thereby improving the efficiency or reliability of model monitoring, or reducing energy consumption or resource waste, wherein the first monitoring parameter is used for monitoring the performance of the first model.

Description

Model monitoring method and communication apparatus

[0001] The present application claims priority to the Chinese patent application No. 202410634263.1, filed on May 20, 2024, and entitled "Model monitoring method and communication apparatus", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the field of communication, and more particularly, to a model monitoring method and a communication apparatus. BACKGROUND

[0003] When artificial intelligence (AI) is applied to a communication link, the performance of the communication link is closely related to the performance of the AI model. Before the model monitoring is run, the performance of the AI model is judged by observing its performance on a pre-provided (i.e., collected in a non-real environment) data set. After the training based on the static data set (i.e., training data) is completed, the AI model is put into a continuously changing real scene to perform an inference task. The difference between the static data set in the training process and the dynamically changing data in the actual use causes the performance of the AI model to possibly decrease over time, which requires the model monitoring to be carried out on the running AI model. SUMMARY

[0004] The present application provides a model monitoring method and a communication apparatus, which are beneficial to determine a model monitoring parameter that is more matched to the performance of the model.

[0005] In a first aspect, a model monitoring method is provided. The method can be executed by a terminal device or a network device, or can also be executed by a component (such as a chip, a circuit or a module, etc.) of the terminal device or the network device.

[0006] The method includes: obtaining performance information of a first model, the performance information of the first model being determined according to at least one verification data set, the performance information of the first model being used for determination of a first monitoring parameter; and the first monitoring parameter being used for performance monitoring of the first model.

[0007] Based on the above technical solution, the terminal device and / or the network device can obtain the performance information of the first model, thereby being beneficial to obtain the first monitoring parameter that is more matched to the performance of the first model according to the performance information of the first model.

[0008] Exemplarily, the performance information of the first model can include one or more of the following: a performance of the first model on the at least one validation dataset, a generalization capability of the first model on the at least one validation dataset, a performance level corresponding to the performance of the first model on the at least one validation dataset, or a generalization capability level corresponding to the generalization capability of the first model on the at least one validation dataset.

[0009] Exemplarily, the first monitoring parameter can include one or more of the following: a first performance threshold, a first monitoring frequency or period, a first monitoring duration, or a first monitoring error tolerance.

[0010] In a possible implementation, the first monitoring parameter is further related to a type of the first model, in other words, the first monitoring parameter is determined according to the performance information of the first model and the type of the first model. The type of the first model is a basic general model or a cell-specific model.

[0011] Based on the above technical solutions, it is beneficial to determine the first monitoring parameter that is more matched with the type of the first model according to the type of the first model.

[0012] With reference to the first aspect, in some implementations of the first aspect, the method further includes: sending or receiving first indication information, the first indication information being used to indicate the type of the first model.

[0013] With reference to the first aspect, in some implementations of the first aspect, the performance information of the first model is obtained by: determining the performance information of the first model according to the at least one validation dataset.

[0014] Based on the above technical solutions, it is beneficial to enable different devices to determine the performance information of the model using the same at least one validation dataset, thereby ensuring the consistency and rationality of the performance evaluation of the model.

[0015] Optionally, the performance information of the first model includes a performance level (denoted as performance level #1) corresponding to the performance of the first model on the at least one validation dataset, and the performance of the first model on the at least one validation dataset has a first correspondence relationship with the performance level #1. Alternatively, the performance information of the first model includes a generalization capability level (denoted as generalization capability #1) corresponding to the generalization capability of the first model on the at least one validation dataset, and the generalization capability of the first model on the at least one validation dataset has a first correspondence relationship with the generalization capability #1.

[0016] In some implementations of the first aspect, the method further includes receiving second indication information, the second indication information including a correspondence between performance of the model on the at least one validation dataset and a performance level, and / or a correspondence between generalization capability of the model on the at least one validation dataset and a generalization level. The correspondence between performance of the model on the at least one validation dataset and the performance level includes a correspondence between performance of the first model on the at least one validation dataset and a performance level #1, and / or the correspondence between generalization capability of the model on the at least one validation dataset and the generalization level includes a correspondence between generalization capability of the first model on the at least one validation dataset and a generalization level #1.

[0017] In some implementations of the first aspect, the method further includes receiving at least one of: third indication information, the third indication information being used to indicate the at least one validation dataset; or the at least one validation dataset.

[0018] In some implementations of the first aspect, the first model is of a first type, and the method further includes receiving fourth indication information, the fourth indication information being used to indicate that the at least one validation dataset is used for determination of performance information of the model of the first type.

[0019] In some implementations of the first aspect, the performance information of the first model is obtained by receiving fifth indication information, the fifth indication information being used to indicate the performance information of the first model.

[0020] In some implementations of the first aspect, the performance information of the first model has a second correspondence with the first monitoring parameter, and the method further includes determining the first monitoring parameter based on the performance information of the first model and a correspondence between performance information of a model and a monitoring parameter, the correspondence between performance information of the model and the monitoring parameter including the second correspondence.

[0021] Based on the above technical solutions, the terminal device and / or the network device can determine a first monitoring parameter that is more matched with the performance of the first model. For example, if the performance of the first model is better, the terminal device and / or the network device can determine a longer first monitoring period, thereby reducing energy consumption or resource waste. For another example, if the generalization capability of the first model is poor, the terminal device and / or the network device can determine a shorter first monitoring period, thereby avoiding a decline in communication quality caused by failure to timely discover a decline in performance of the first model.

[0022] In some implementations of the first aspect, the method further includes receiving sixth indication information, the sixth indication information including a correspondence between performance information of a model and a monitoring parameter.

[0023] With reference to the first aspect, in some implementations of the first aspect, the method further includes sending the first monitoring parameter.

[0024] Based on the above technical solution, the terminal device sends the first monitoring parameter to the network device, so that the network device can perform performance monitoring on the first model according to the first monitoring parameter. Alternatively, the network device sends the first monitoring parameter to the terminal device, so that the terminal device can perform performance monitoring on the first model according to the first monitoring parameter.

[0025] With reference to the first aspect, in some implementations of the first aspect, the method further includes sending fifth indication information, the fifth indication information being used to indicate performance information of the first model; and receiving the first monitoring parameter.

[0026] Based on the above technical solution, after the terminal device sends the fifth indication information to the network device, the terminal device can receive the first monitoring parameter from the network device, thereby facilitating performance monitoring on the first model according to the first monitoring parameter. Alternatively, after the network device sends the fifth indication information to the terminal device, the network device can receive the first monitoring parameter from the terminal device, thereby facilitating performance monitoring on the first model according to the first monitoring parameter.

[0027] With reference to the first aspect, in some implementations of the first aspect, the method further includes performing performance monitoring on the first model according to the first monitoring parameter.

[0028] Based on the above technical solution, the terminal device and / or the network device perform performance monitoring on the first model according to the first monitoring parameter, which is beneficial to timely learn the performance change of the first model, and is beneficial to triggering the first model switching or updating in the case that the performance of the first model decreases, thereby avoiding the inference performance decrease caused by the performance decrease of the first model.

[0029] The second aspect provides a communication apparatus, which can be a terminal device or a network device, or can be a component (such as a chip, a circuit or a module, etc.) for a terminal device or a network device.

[0030] The apparatus includes an obtaining unit configured to obtain performance information of a first model, the performance information of the first model being determined according to at least one verification data set, and the performance information of the first model being used for determination of a first monitoring parameter; and the first monitoring parameter being used for performance monitoring on the first model.

[0031] With reference to the second aspect, in some implementations of the second aspect, the first monitoring parameter is determined according to the performance information of the first model and a type of the first model.

[0032] With reference to the second aspect, in some implementations of the second aspect, the type of the first model is a basic general model or a cell-specific model.

[0033] With reference to the second aspect, in some implementations of the second aspect, the apparatus further includes a transceiver configured to transmit or receive the first indication information, the first indication information being configured to indicate the type of the first model.

[0034] With reference to the second aspect, in some implementations of the second aspect, the performance information of the first model includes one or more of a performance of the first model on the at least one validation dataset, a generalization ability of the first model on the at least one validation dataset, a performance level corresponding to the performance of the first model on the at least one validation dataset, or a generalization ability level corresponding to the generalization ability of the first model on the at least one validation dataset.

[0035] With reference to the second aspect, in some implementations of the second aspect, the obtaining unit includes a processing unit configured to determine the performance information of the first model according to the at least one validation dataset.

[0036] With reference to the second aspect, in some implementations of the second aspect, the performance information of the first model has a first correspondence relationship with the performance and / or the generalization ability of the first model on the at least one validation dataset.

[0037] With reference to the second aspect, in some implementations of the second aspect, the transceiver is further configured to receive second indication information, the second indication information including a correspondence relationship between a performance and / or a generalization ability of a model on the at least one validation dataset and performance information of the model, the correspondence relationship between the performance and / or the generalization ability of the model on the at least one validation dataset and the performance information of the model including the first correspondence relationship.

[0038] With reference to the second aspect, in some implementations of the second aspect, the transceiver is further configured to receive at least one of third indication information or the at least one validation dataset, the third indication information being configured to indicate the at least one validation dataset.

[0039] With reference to the second aspect, in some implementations of the second aspect, the type of the first model is a first type, and the transceiver is further configured to receive fourth indication information, the fourth indication information being configured to indicate that the at least one validation dataset is used for determining the performance information of the model of the first type.

[0040] With reference to the second aspect, in some implementations of the second aspect, the obtaining unit includes a transceiver configured to receive fifth indication information, the fifth indication information being configured to indicate the performance information of the first model.

[0041] In some implementations of the second aspect, in combination with the second aspect, the performance information of the first model has a second correspondence with the first monitoring parameter, and the processing unit is further configured to determine the first monitoring parameter according to the performance information of the first model and the correspondence between the performance information of the model and the monitoring parameter, the correspondence between the performance information of the model and the monitoring parameter including the second correspondence.

[0042] In some implementations of the second aspect, in combination with the second aspect, the transceiving unit is further configured to receive sixth indication information, the sixth indication information including the correspondence between the performance information of the model and the monitoring parameter.

[0043] In some implementations of the second aspect, in combination with the second aspect, the transceiving unit is further configured to send the first monitoring parameter.

[0044] In some implementations of the second aspect, in combination with the second aspect, the transceiving unit is further configured to send fifth indication information, the fifth indication information being used to indicate the performance information of the first model; and receive the first monitoring parameter.

[0045] In some implementations of the second aspect, in combination with the second aspect, the processing unit is further configured to perform performance monitoring on the first model according to the first monitoring parameter.

[0046] In some implementations of the second aspect, in combination with the second aspect, the first monitoring parameter includes one or more of the following: a first performance threshold, a first monitoring frequency or period, a first monitoring duration, or a first monitoring error tolerance.

[0047] In a third aspect, a communication apparatus is provided, which includes at least one processor configured to execute computer programs or instructions to perform the method in the first aspect and any possible implementation of the first aspect. Optionally, the apparatus further includes a memory configured to store the computer programs or instructions. Optionally, the apparatus further includes a communication interface through which the processor reads the computer programs or instructions.

[0048] In one implementation, the apparatus is a communication device (e.g., a terminal device, or a network device).

[0049] In another implementation, the apparatus is a chip, a chip system, or a circuit for a communication device (e.g., a terminal device, or a network device).

[0050] In a fourth aspect, a processor is provided, which is configured to perform the method provided in the first aspect.

[0051] For the sending and obtaining / receiving operations involved by the processor, if no special description is made, or if it does not conflict with the actual role or internal logic in the related description, it can be understood as the processor output and receiving, input operations, and can also be understood as the sending and receiving operations performed by the radio frequency circuit and the antenna, and the present application does not limit this.

[0052] Optionally, the apparatus further comprises a memory for storing a program; correspondingly, the at least one processor is configured to execute the computer program or instructions in the memory.

[0053] Optionally, the apparatus further comprises a communication interface. The communication interface is coupled with the processor, and can be used for inputting information to the processor, or outputting information in the processor.

[0054] In a fifth aspect, a computer readable storage medium is provided, which stores program codes for an apparatus to execute, and the program codes are used to execute the method in the first aspect and any possible implementation manner of the first aspect.

[0055] In a sixth aspect, a computer program product containing instructions is provided, which, when executed on a computer, causes the computer to execute the method in the first aspect and any possible implementation manner of the first aspect.

[0056] In a seventh aspect, a chip is provided, which comprises a processor and a communication interface. The processor reads instructions on a memory through the communication interface, and executes the method provided by the first aspect and any implementation manner of the first aspect.

[0057] Optionally, as an implementation manner, the chip further comprises a memory, and the memory stores computer programs or instructions. The processor is configured to execute the computer programs or instructions on the memory. When the computer programs or instructions are executed, the processor is configured to execute the method provided by the first aspect and any implementation manner of the first aspect.

[0058] In an eighth aspect, a communication system is provided, which comprises a terminal device and / or a network device. The terminal device and / or the network device are configured to implement the method provided by the first aspect and any possible implementation manner of the first aspect.

[0059] It should be understood that the beneficial effects of the second aspect to the eighth aspect and any implementation manner thereof can refer to the first aspect and any implementation manner thereof. BRIEF DESCRIPTION OF DRAWINGS

[0060] FIG. 1 is a schematic diagram of a possible application framework in a communication system.

[0061] FIG. 2 is a schematic diagram of a possible application framework in a communication system.

[0062] Figure 3 is a schematic diagram of a communication system suitable for use in the model monitoring method of embodiments of the application.

[0063] Figure 4 is a schematic diagram of a communication system suitable for use in the model monitoring method of embodiments of the application.

[0064] Figure 5 is a schematic diagram of a neuron structure.

[0065] Figure 6 is a schematic diagram of a model monitoring method 600 provided by embodiments of the application.

[0066] Figure 7 is a schematic diagram of a model monitoring method 700 provided by embodiments of the application.

[0067] Figure 8 is a schematic diagram of a model monitoring method 800 provided by embodiments of the application.

[0068] Figure 9 is a schematic diagram of a model monitoring method 900 provided by embodiments of the application.

[0069] Figure 10 is a schematic diagram of a communication apparatus 2000 provided by embodiments of the application.

[0070] Figure 11 is a schematic diagram of another communication apparatus 3000 provided by embodiments of the application. DETAILED DESCRIPTION

[0071] The technical solutions in the application will be described below with reference to the accompanying drawings.

[0072] The technical solutions provided in the present application can be applied to various communication systems, for example, a 5th generation (5G) or new radio (NR) system, a long term evolution (LTE) system, an LTE frequency division duplex (FDD) system, an LTE time division duplex (TDD) system, a wireless local area network (WLAN) system, a satellite communication system, a future communication system, or a fusion system of multiple systems, and the like. The technical solutions provided in the present application can also be applied to device to device (D2D) communication, vehicle-to-everything (V2X) communication, machine to machine (M2M) communication, machine type communication (MTC), and an internet of things (IoT) communication system or other communication systems.

[0073] A device in a communication system can send a signal to another device or receive a signal from another device. The signal can include information, signaling, or data, and the like. The device can also be replaced by an entity, a network entity, a communication device, a mobile device, a network element, a communication module, a node, a communication node, a communication apparatus, and the like. The device is taken as an example for description in the present disclosure. For example, the communication system can include at least one terminal device and at least one network device. The network device can send a downlink signal to the terminal device, and / or the terminal device can send an uplink signal to the network device.

[0074] In the embodiments of the present application, the terminal device can also be referred to as a user equipment (UE), an access terminal, a user unit, a user station, a mobile station, a mobile station, a remote station, a remote terminal, a mobile device, a user terminal, a terminal, a wireless communication device, a user agent, or a user apparatus.

[0075] The terminal device can be a device providing voice / data, for example, a handheld device with wireless connection function, a vehicle-mounted device, etc. At present, some examples of terminals are: mobile phone, tablet computer, notebook computer, palm computer, mobile internet device (MID), wearable device, virtual reality (VR) device, augmented reality (AR) device, wireless terminal in industrial control, wireless terminal in self driving, wireless terminal in remote medical surgery, wireless terminal in smart grid, wireless terminal in transportation safety, wireless terminal in smart city, wireless terminal in smart home, cellular phone, cordless phone, session initiation protocol (SIP) phone, wireless local loop (WLL) station, personal digital assistant (PDA), handheld device with wireless communication function, computing device or other processing device connected to a wireless modem, wearable device, terminal device in a 5G network, or terminal device in a future evolved public land mobile network (PLMN), etc. The embodiments of the present application are not limited thereto.

[0076] By way of example and not limitation, in the embodiments of the present application, the terminal device can also be a wearable device. The wearable device can also be referred to as a wearable smart device, which is a general term for devices that are designed and developed by applying wearable technology to daily wear, such as glasses, gloves, watches, clothing, and shoes. The wearable device is a portable device that is directly worn on the body or integrated into the user's clothes or accessories. The wearable device is not only a hardware device, but also a device that realizes powerful functions through software support and data interaction and cloud interaction. The general wearable smart device includes devices with full functions, large size, and the ability to realize complete or partial functions without relying on a smart phone, such as smart watches or smart glasses, and devices that focus on a certain application function and need to be used in cooperation with other devices, such as smart phones, such as various smart wristbands and smart jewelry for monitoring vital signs.

[0077] In the embodiments of the present application, the apparatus for implementing the function of the terminal device can be a terminal device, or can be an apparatus capable of supporting the terminal device to implement the function, for example, a chip system, which can be installed in the terminal device or used in matching with the terminal device. In the embodiments of the present application, the chip system can be composed of a chip, or can include the chip and other discrete devices. In the embodiments of the present application, only the apparatus for implementing the function of the terminal device is taken as an example for description, and the scheme of the embodiments of the present application is not limited in this way.

[0078] The network device in the embodiments of the present application can include a device for communicating with a terminal device, for example, the network device can include an access network device or a radio access network device, for example, the network device can be a base station. The radio access network device in the embodiments of the present application can refer to a radio access network (RAN) node (or device) that accesses a terminal device to a wireless network. The base station can broadly cover various names in the following or be replaced by the following names, such as: Node B (NodeB), evolved Node B (eNB), next generation Node B (gNB), relay station, access point, transmitting and receiving point (TRP), transmitting point (TP), primary station, secondary station, motor slide retainer (MSR) node, home base station, network controller, access node, wireless node, access point (AP), transmission node, transceiver node, baseband unit (BBU), remote radio unit (RRU), active antenna unit (AAU), remote radio head (RRH), central unit (CU), distributed unit (DU), radio unit (RU), positioning node, etc. The base station can be a macro base station, a micro base station, a relay node, a donor node or the like, or a combination thereof. The base station can also refer to a communication module, modem or chip used in the above-mentioned devices or apparatuses. The base station can also be a mobile switching center and a device that performs the function of a base station in D2D, V2X, M2M communication, a device that performs the function of a base station in future communication systems, etc. The base station can support networks of the same or different access technologies. Optionally, the RAN node can also be a server, a wearable device, a vehicle or a vehicle-mounted device, etc. For example, the access network device in the vehicle to everything (V2X) technology can be a road side unit (RSU). The embodiments of the present application do not limit the specific technology and specific device form of the network device.

[0079] The base station can be fixed or mobile. For example, a helicopter or a drone can be configured to act as a mobile base station, and one or more cells can move according to the location of the mobile base station. In other examples, the helicopter or the drone can be configured to serve as a device that communicates with another base station.

[0080] In some deployments, the network device mentioned by embodiments of the present application can be a device including a CU, or a DU, or a device including a CU and a DU, or a control plane CU node (central unit-control plane (CU-CP)) and a user plane CU node (central unit-user plane (CU-UP)), and a DU node. For example, the network device can include a gNB-CU-CP, a gNB-CU-UP, and a gNB-DU.

[0081] In some deployments, wireless access is assisted by multiple RAN nodes cooperating to assist a terminal, and different RAN nodes respectively implement part of the functions of a base station. For example, the RAN node can be a CU, a DU, a CU-CP, a CU-UP, or an RU, etc. The CU and the DU can be separately arranged, or can also be included in the same network element, such as a BBU. The RU can be included in a radio frequency device or a radio frequency unit, such as an RRU, an AAU, or an RRH.

[0082] The RAN node can support one or more types of fronthaul interfaces, different fronthaul interfaces respectively corresponding to DUs and RUs having different functions. If the fronthaul interface between the DU and the RU is a common public radio interface (CPRI), the DU is configured to implement one or more of baseband functions, and the RU is configured to implement one or more of radio frequency functions. If the fronthaul interface between the DU and the RU is another interface, which, relative to the CPRI, moves one or more of partial baseband functions of the downlink and / or uplink, such as, for the downlink, one or more of precoding, digital beamforming (BF), or inverse fast Fourier transform (IFFT) / add cyclic prefix (CP), from the DU to the RU for implementation, and for the uplink, one or more of digital beamforming (BF), or fast Fourier transform (FFT) / remove cyclic prefix (CP), from the DU to the RU for implementation. In a possible implementation, the interface can be an enhanced common public radio interface (eCPRI). Under the eCPRI architecture, the splitting manner between the DU and the RU is different, corresponding to different categories (Cat) of eCPRI, such as eCPRI Cat A, B, C, D, E, and F.

[0083] Taking eCPRI Cat A as an example, for downlink transmission, with layer mapping as the cut, the DU is configured to implement layer mapping and one or more functions (i.e., one or more of encoding, rate matching, scrambling, modulation, layer mapping) before layer mapping, while other functions (e.g., one or more of RE mapping, digital beamforming (BF), or inverse fast Fourier transform (IFFT) / addition of cyclic prefix (CP)) after layer mapping are implemented in the RU. For uplink transmission, with de-RE mapping as the cut, the DU is configured to implement de-mapping and one or more functions (i.e., one or more of decoding, de-rate matching, de-scrambling, de-modulation, inverse discrete Fourier transform (IDFT), channel equalization, de-RE mapping) before de-mapping, while other functions (e.g., one or more of digital BF or fast Fourier transform (FFT) / CP removal) after de-mapping are implemented in the RU. It can be understood that the function description of the DU and the RU corresponding to various types of eCPRI can refer to the eCPRI protocol, which is not described here.

[0084] In a possible design, the processing unit in the BBU for implementing baseband functions is referred to as a base band high (BBH) unit, and the processing unit in the RRU / AAU / RRH for implementing baseband functions is referred to as a base band low (BBL) unit.

[0085] In different systems, the CU (or CU-CP and CU-UP), DU or RU can also have different names, but those skilled in the art can understand their meanings. For example, in an open RAN (ORAN) system, the CU can also be referred to as an O-CU (open CU), the DU can also be referred to as an O-DU, the CU-CP can also be referred to as an O-CU-CP, the CU-UP can also be referred to as an O-CU-UP, and the RU can also be referred to as an O-RU. Any of the CU (or CU-CP, CU-UP), DU and RU in this application can be implemented by a software module, a hardware module, or a combination of a software module and a hardware module.

[0086] In the embodiments of the present application, the apparatus for implementing the function of the network device can be a network device, or an apparatus capable of supporting the network device to implement the function, such as a chip system, a hardware circuit, a software module, or a hardware circuit plus a software module. The apparatus can be installed in the network device or used in combination with the network device. In the embodiments of the present application, only the apparatus for implementing the function of the network device is taken as an example for illustration, and the scheme of the embodiments of the present application is not limited in this way.

[0087] The network device and / or the terminal device can be deployed on land, including indoors or outdoors, handheld or vehicle-mounted; can also be deployed on water; and can also be deployed on airplanes, balloons and satellites in the air. The scenarios in which the network device and the terminal device are located are not limited in the embodiments of the present application. In addition, the terminal device and the network device can be hardware devices, or software functions running on special hardware, software functions running on general hardware, such as virtualized functions instantiated on a platform (for example, a cloud platform), or entities including special or general hardware devices and software functions. The specific forms of the terminal device and the network device are not limited in the present application.

[0088] In a wireless communication network, for example, in a mobile communication network, the services supported by the network are increasingly diverse, and therefore the needs to be met are increasingly diverse. For example, the network needs to be able to support ultra-high rates, ultra-low latencies, and / or ultra-large connections. This feature makes network planning, network configuration, and / or resource scheduling increasingly complex. In addition, as the functions of the network become increasingly powerful, for example, supporting increasingly high frequency spectrums, supporting high-order multiple input multiple output (MIMO) technology, supporting beamforming, and / or supporting new technologies such as beam management, network energy saving has become a hot research topic. These new needs, new scenarios and new features bring unprecedented challenges to network planning, operation and efficient operation. In order to meet this challenge, artificial intelligence technology can be introduced into the wireless communication network, thereby realizing network intelligentization.

[0089] In order to support artificial intelligence (AI) technology in the wireless network, an AI node can also be introduced into the network.

[0090] Optionally, the AI node can be deployed in one or more of the following positions in the communication system: an access network device, a terminal device, or a core network device, etc., or the AI node can also be deployed separately, for example, deployed in a position other than any of the above devices, such as a host or a cloud server of an over the top (OTT) system. The AI node can communicate with other devices in the communication system, which can be one or more of the following: a wireless access network device, a terminal device, or a network element of a core network, etc.

[0091] It can be understood that the present application does not limit the number of AI nodes. For example, when there are multiple AI nodes, the multiple AI nodes can be divided based on functions, such as different AI nodes being responsible for different functions.

[0092] It can also be understood that the AI node can be a separate device, can be integrated into the same device to implement different functions, or can be a network element in a hardware device, or a software function running on a dedicated hardware, or a virtualized function instantiated on a platform (e.g., a cloud platform), and the present application does not limit the specific form of the AI node.

[0093] The AI node can be an AI network element or an AI module.

[0094] FIG. 1 is a schematic diagram of a possible application framework in a communication system. As shown in FIG. 1, the network elements in the communication system are connected through interfaces (such as next generation (NG) interfaces, Xn interfaces), or air interfaces. One or more AI modules (only one is shown in FIG. 1 for clarity) are provided in one or more of the following network element nodes: a core network device, an access network node or device (RAN node or device), a terminal, or one or more devices in operation administration and maintenance (OAM). The access network node can be a separate RAN node, or can include multiple RAN nodes, for example, including a CU and a DU. The CU and / or DU can also be provided with one or more AI modules. Optionally, the CU can also be split into a CU-CP and a CU-UP. One or more AI models are provided in the CU-CP and / or the CU-UP.

[0095] The AI module is used to implement a corresponding AI function. The AI modules deployed in different network elements can be the same or different. The AI module can implement different functions according to different parameter configurations of the model of the AI module. The model of the AI module can be configured based on one or more of the following parameters: a structure parameter (for example, at least one of a number of neural network layers, a width of a neural network, a connection relationship between layers, a weight of a neuron, an activation function of a neuron, or a bias in the activation function), an input parameter (for example, a type of input parameter and / or a dimension of the input parameter), or an output parameter (for example, a type of output parameter and / or a dimension of the output parameter). The bias in the activation function can also be referred to as a bias of the neural network.

[0096] One AI module can have one or more models. One model can infer an output including one parameter or multiple parameters. The learning process, the training process, or the inference process of different models can be deployed in different nodes or devices, or can be deployed in the same node or device.

[0097] FIG. 2 is a schematic diagram of a possible application framework in a communication system. As shown in FIG. 2, the communication system includes a RAN intelligent controller (RIC). For example, the RIC can be the AI module shown in FIG. 1, which is used to implement AI-related functions. The RIC includes a near-real time RIC (near-RT RIC) and a non-real time RIC (Non-RT RIC). The non-real time RIC mainly processes non-real-time information, such as data that is not sensitive to latency, which can be on the order of seconds. The real-time RIC mainly processes near-real-time information, such as data that is relatively sensitive to latency, which is on the order of tens of milliseconds.

[0098] The near-real-time RIC is used for model training and inference. For example, for training an AI model, inference is performed using the AI model. The near-real-time RIC can obtain network-side and / or terminal-side information from RAN nodes (such as CUs, CU-CPs, CU-UPs, DUs, and / or RUs) and / or terminals. This information can be used as training data or inference data. Optionally, the near-real-time RIC can submit inference results to RAN nodes and / or terminals. Optionally, the inference results can be exchanged between CUs and DUs, and / or between DUs and RUs. For example, the near-real-time RIC submits the inference results to the DU, and the DU sends the inference results to the RU.

[0099] The non-real-time RIC is also used for model training and inference. For example, the non-real-time RIC is used for training an AI model, and inference is performed using the model. The non-real-time RIC can obtain network-side and / or terminal-side information from a RAN node (e.g., a CU, a CU-CP, a CU-UP, a DU, and / or a RU) and / or a terminal. The information can be used as training data or inference data, and the inference result can be delivered to the RAN node and / or the terminal. Alternatively, the inference result can be exchanged between a CU and a DU, and / or between a DU and a RU, for example, the non-real-time RIC delivers the inference result to the DU, and the inference result is further delivered to the RU by the DU.

[0100] The near-real-time RIC and the non-real-time RIC can be respectively configured as a network element alone. Alternatively, the near-real-time RIC and the non-real-time RIC can be part of other devices, for example, the near-real-time RIC is configured in a RAN node (e.g., a CU, a DU), and the non-real-time RIC is configured in an OAM, a cloud server, a core network device, or other network devices.

[0101] FIG. 3 is a schematic diagram of a communication system suitable for the model monitoring method according to the embodiments of the present application. As shown in FIG. 3, the communication system 100 can include at least one network device, for example, the network device 110 shown in FIG. 3, and the communication system 100 can also include at least one terminal device, for example, the terminal device 120 and the terminal device 130 shown in FIG. 3. The network device 110 and the terminal devices (e.g., the terminal device 120 and the terminal device 130) can communicate with each other through wireless links. The communication devices in the communication system can communicate with each other through multi-antenna technology, for example, the network device 110 and the terminal device 120.

[0102] FIG. 4 is a schematic diagram of a communication system suitable for the model monitoring method according to the embodiments of the present application. Compared with the communication system 100 shown in FIG. 3, the communication system 200 shown in FIG. 4 further includes an AI network element 140. The AI network element 140 is used to perform AI-related operations, for example, constructing a training data set or training an AI model.

[0103] In a possible implementation, the network device 110 can send data related to the training of the AI model to the AI network element 140, the AI network element 140 constructs a training data set and trains the AI model. For example, the data related to the training of the AI model can include data reported by the terminal device. The AI network element 140 can send the result of the AI model related operation to the network device 110 and forward it to the terminal device through the network device 110. For example, the result of the AI model related operation can include at least one of the following: a trained AI model, an evaluation result or a test result of the model, and the like. For example, part of the trained AI model can be deployed on the network device 110, and the other part can be deployed on the terminal device. Alternatively, the trained AI model can be deployed on the network device 110. Alternatively, the trained AI model can be deployed on the terminal device.

[0104] It should be understood that FIG. 4 is only used as an example to illustrate that the AI network element 140 is directly connected to the network device 110, and in other scenarios, the AI network element 140 can also be connected to the terminal device. Alternatively, the AI network element 140 can be connected to the network device 110 and the terminal device at the same time. Alternatively, the AI network element 140 can also be connected to the network device 110 through a third-party network element. The connection relationship between the AI network element and other network elements is not limited in the embodiments of the present application.

[0105] The AI network element 140 can also be set as a module in the network device and / or the terminal device, for example, in the network device 110 or the terminal device shown in FIG. 3.

[0106] It should be noted that FIGS. 3 and 4 are only simplified schematic diagrams for understanding, for example, the communication system can also include other devices, such as wireless relay devices and / or wireless backhaul devices, which are not shown in FIGS. 3 and 4. In actual application, the communication system can include multiple network devices and multiple terminal devices. The number of network devices and terminal devices included in the communication system is not limited in the embodiments of the present application.

[0107] In order to facilitate understanding of the scheme of the embodiments of the present application, the terms that can be involved in the embodiments of the present application are explained as follows.

[0108] (1) Artificial intelligence: It is to make the machine have learning ability and can accumulate experience to solve the problems that can be solved by human experience, such as natural language understanding, image recognition and chess playing. Artificial intelligence can be understood as the intelligence shown by the machine made by human. Artificial intelligence usually refers to the technology of presenting human intelligence through computer program. The goal of artificial intelligence includes understanding intelligence by constructing symbolic reasoning or reasoning computer program.

[0109] (2) Machine Learning (ML): is a way of implementing artificial intelligence. Machine learning is a method that can give a machine the ability to learn and complete functions that cannot be completed by direct programming. In a practical sense, machine learning is a method of training a model by using data and then using the model for prediction. There are many methods of machine learning, such as neural networks (NN), decision trees, support vector machines, etc. Machine learning theory is mainly about designing and analyzing algorithms that allow computers to automatically learn. Machine learning algorithms are a class of algorithms that automatically analyze rules from data and use the rules to predict unknown data.

[0110] (3) Neural Network: Neural network is a specific embodiment of machine learning method. Neural network is a mathematical model that simulates the behavior characteristics of animal neural network for information processing. As shown in FIG. 5, neural network is a network that can be composed of three types of calculation layers, input layer, hidden layer and output layer. Each layer has one or more logical judgment units, which are called neurons. Common neural network structures include feedforward neural network (FNN), convolutional neural network (CNN) and recurrent neural network (RNN), etc., which are all based on neurons. Among them, each neuron can perform weighted summation operation on its input value, and the result of the weighted summation operation is output through a nonlinear function. The weights of the neurons in the neural network and the nonlinear function can be referred to as the parameters of the neural network, the connection relationship between the neurons in the neural network can be referred to as the structure of the neural network, and all the parameters of the neurons in the neural network constitute the parameters of the neural network.

[0111] (4) Deep Neural Network: Neural network with multiple hidden layers.

[0112] (5) Deep Learning: Machine learning using deep neural networks.

[0113] (6) AI Model: is an algorithm or computer program that can realize AI function. The AI model represents the mapping relationship between the input and output of the model, or in other words, the AI model is a function model that maps a certain dimension of input to a certain dimension of output. The parameters of the function model can be obtained by machine learning training. For example, f(x) = ax 2+b is a quadratic function model, which can be regarded as an AI model, and a and b are parameters of the AI model, and a and b can be obtained by machine learning training. Illustratively, the AI model mentioned in the embodiments below is not limited to a neural network, a linear regression model, a decision tree model, a support vector machine (SVM), a Bayesian network, a Q-learning model, or other machine learning (ML) models.

[0114] The implementation of the AI model can be hardware circuit, software, or a combination of software and hardware, without limitation. Non-limiting examples of software include program code, programs, subprograms, instructions, instruction sets, codes, code segments, software modules, applications, or software applications, etc.

[0115] (7) Training data set:

[0116] In the field of machine learning, ground truth generally refers to data that is considered accurate or true.

[0117] The training data set is used for training of the AI model, and the training data set can include the input of the AI model or the input and target output of the AI model. The training data set includes one or more training data, and the training data can include a training sample input to the AI model or a target output of the AI model. The target output can also be referred to as a label, a sample label, or a labeled sample. The label is the ground truth.

[0118] In the field of communication, the training data set can include simulation data collected through a simulation platform, experimental data collected in an experimental scenario, or measured data collected in an actual communication network. Due to differences in geographical environment and channel conditions where data is generated, such as differences in indoor, outdoor, mobile speed, frequency band, or antenna configuration, the collected data can be classified when obtaining the data. For example, data with the same channel propagation environment and antenna configuration are classified into one category.

[0119] Model training is essentially learning its features from training data. In the process of training an AI model (such as a neural network model), because the output of the AI model is expected to be as close as possible to the value that is actually intended to be predicted, the weight vector of each layer of the AI model can be updated according to the difference between the predicted value of the current network and the target value that is actually intended to be predicted (of course, before the first update, there is usually an initialization process, that is, the parameters of each layer of the AI model are pre-configured), for example, if the predicted value of the network is too high, the weight vector is adjusted to make it predict lower, and the adjustment is continuously made until the AI model can predict the target value that is actually intended to be predicted or a value very close to the target value that is actually intended to be predicted. Therefore, it is necessary to define "how to compare the difference between the predicted value and the target value", which is the loss function or the objective function, which is an important equation for measuring the difference between the predicted value and the target value. Taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, and the training of the AI model becomes a process of minimizing the loss so that the value of the loss function is less than a threshold or meets the target requirement. For example, the AI model is a neural network, and adjusting the model parameters of the neural network includes adjusting at least one of the following parameters: the number of layers of the neural network, the width, the weight of the neuron, or the parameters of the activation function of the neuron.

[0120] (8) Model training: training the model parameters by selecting a suitable loss function and using an optimization algorithm to make the value of the loss function less than a threshold or meet the target requirement.

[0121] (9) Model monitoring: used to observe the running AI model and ensure its performance and reliability. Before model monitoring is run, the performance of the AI model is judged by observing its performance on a pre-provided (i.e. non-real environment collected) data set. After the training based on the static data set (i.e. training data) is completed, the AI model is put into the changing real scene to perform inference tasks. The difference between the static data set in the training process and the dynamically changing data in actual use causes the performance of the AI model to possibly decrease over time, which requires model monitoring of the running AI model.

[0122] The present application provides a model monitoring method to improve the accuracy of model monitoring and thus improve the inference performance of the model.

[0123] Before introducing the scheme of the present application, the following points are explained.

[0124] (1) In the present application, "indication" can include direct indication, indirect indication, display indication, and implicit indication. When describing that certain indication information is used to indicate A, it can be understood that the indication information carries A, directly indicates A, or indirectly indicates A.

[0125] In the present application, the information indicated by the indication information is referred to as to-be-indicated information. In the specific implementation process, there are many ways to indicate the to-be-indicated information, for example, but not limited to, the to-be-indicated information can be directly indicated, such as the to-be-indicated information itself or the index of the to-be-indicated information. The to-be-indicated information can also be indirectly indicated by indicating other information, where the other information and the to-be-indicated information have an association relationship. The to-be-indicated information can also be indicated only by a part, and the other part of the to-be-indicated information is known or agreed in advance. For example, the indication of a specific information can also be achieved by means of the arrangement order of each information agreed in advance (for example, the protocol stipulates), thereby reducing the indication overhead to a certain extent. In addition, the to-be-indicated information can be sent as a whole, or can be sent separately into multiple sub-information, and the sending period and / or sending time of these sub-information can be the same or different.

[0126] (2) In the present application, "sending" and "receiving" represent the direction of signal transmission. For example, "sending information to XX" can be understood as that the destination of the information is XX, which can include direct sending through the air interface, and also includes indirect sending through the air interface by other units or modules. "Receiving information from YY" can be understood as that the source of the information is YY, which can include direct receiving from YY through the air interface, and also includes indirect receiving from YY through the air interface from other units or modules. "Sending" can also be understood as the "output" of the chip interface, and "receiving" can also be understood as the "input" of the chip interface. In other words, sending and receiving can be carried out between devices, for example, between network devices and terminal devices, or can be carried out within a device, for example, between components, modules, chips, software modules or hardware modules within a device through a bus, wire or interface.

[0127] (3) In various embodiments of the present application, the terms and / or descriptions of different embodiments have consistency and can be mutually referenced if there is no special description and logical conflict. The technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationship.

[0128] (4) In the present application, "first", "second", and "#1", "#2", etc. are only for convenience of description and are used to distinguish objects, and are not used to limit the scope of the embodiments of the present application. They are not used to describe the order or sequence of features. It should be understood that the objects thus described can be interchanged under appropriate circumstances, so as to be able to describe schemes other than the embodiments of the present application.

[0129] (5) In the present application, "predefined" can mean a standard protocol predefined, or can also mean pre-agreed or pre-negotiated between devices.

[0130] (6) In the present application, the words such as "exemplarily", "such as" and the like are used to represent examples, illustrations or descriptions. Any embodiment or design scheme described as "exemplary" in the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the word "exemplary" is intended to present the concept in a specific manner. In the embodiments of the present application, "of", "corresponding" and "corresponding" can be used interchangeably at times, and it should be pointed out that when the distinction is not emphasized, the meanings expressed are consistent. In addition, "corresponding to" in the present application can also be replaced by "for" or replaced by "determined according to xx" or replaced by "for determining", for example, "corresponding to" in the following embodiment "the first storage resource information corresponds to the number of storage resources available for executing tasks" can be replaced by "for determining", and for another example, "corresponding to" in the following embodiment "the storage resources available for executing tasks correspond to the storage resources available for running AI models / functional execution tasks" can be replaced by "for".

[0131] The model monitoring method provided by the embodiments of the present application will be described in detail below with reference to the accompanying drawings. The embodiments provided by the present application can be applied to the communication system shown in FIG. 3 or FIG. 4, without limitation.

[0132] It should be noted that the first device or the second device in the following embodiments can be a terminal device, or a component part of the terminal device, such as a chip or a circuit. The first device or the second device in the following embodiments can be a network device, or a component part of the network device, such as a chip or a circuit. Alternatively, the first device or the second device in the following embodiments can be an access network node, such as RIC, or CU, or DU, or RU, etc., or the second communication device can be a component part of the access network node, such as a chip or a circuit.

[0133] It should also be noted that the first device and the second device in the following embodiments are different, for example, the first device is a terminal device and the second device is a network device.

[0134] FIG. 6 shows a schematic flowchart of the model monitoring method provided by the embodiments of the present application. As shown in FIG. 6, the method 600 can include the following steps.

[0135] S610, the first device determines the performance information of the first model.

[0136] The first model is deployed at the first device side, such as a first device or an OTT device at the first device side. For example, the first model can be an AI model. The application does not limit the function of the first model. For example, the first model can be a model for channel state information (CSI) compression, or a model for beam management, or a model for channel estimation.

[0137] In a possible implementation, the first model is a model obtained by model training of the first device.

[0138] For example, the first device receives a training data set from the second device or the cloud or a data node, or the first device receives a training data set and at least one validation data set from the second device or the cloud or a data node, or the first device receives a training data set and indication information #1 from the second device or the cloud or a data node, or the first device receives a training data set, at least one validation data set and indication information #2 from the second device or the cloud or a data node; and the first device performs model training according to the training data set to obtain the first model.

[0139] The indication information #1 is used to indicate that the training data set is used for training of the first type of model, and the indication information #2 is used to indicate that the training data set is used for training of the first type of model and is used to indicate that the at least one validation data set is used for determination of performance information of the first type of model.

[0140] The first type of model is a basic general model or a cell-specific model. The basic general model is not associated with a specific scene or a specific cell, that is, the basic general model refers to a model that can be used in any scene or any cell. The cell-specific model is associated with a specific scene or a specific cell, that is, the cell-specific model refers to a model that can be used in a specific scene or a specific cell, or in other words, the cell-specific model can be configured only when the first device accesses a specific cell or enters a specific scene.

[0141] In a possible implementation, the first model is configured by the second device or the cloud, such as an OTT device or a server.

[0142] Optionally, in this implementation, the method 600 further includes that the second device or the cloud sends information of the first model to the first device, and correspondingly, the first device receives the information of the first model. The information of the first model can include parameters of the first model, such as weights, biases, etc.

[0143] The performance information of the first model can include one or more of the following: performance of the first model on the at least one validation dataset, a performance level corresponding to the performance of the first model on the at least one validation dataset, generalization ability of the first model on the at least one validation dataset, or a generalization level corresponding to the generalization ability of the first model on the at least one validation dataset.

[0144] For example, the performance of the first model on the at least one validation dataset can be represented by one or more of the following parameters: mean of accuracy, variance of accuracy, mean of performance, variance of performance, and the like on the at least one validation dataset. The accuracy of the first model on the at least one validation dataset can be represented by a similarity (for example, square of generalized cosine similarity, SGCS) between the true value and the output of the first model. Taking a validation dataset #1 in the at least one validation dataset as an example, the validation dataset #1 can include a sample #1 input to the first model, and can also include a true value #1 corresponding to the sample #1. In determining the performance of the first model on the validation dataset #1, the first device inputs the sample #1 to the first model and obtains a corresponding output #1, and then the first device calculates the similarity (for example, SGCS) between the output #1 and the true value #1. If the validation dataset #1 includes multiple samples and multiple true values, the SGCS of the first model on the validation dataset #1 can be the mean or variance of the SGCS between the outputs corresponding to the multiple samples and the true values. The mean of the SGCS represents the average performance of the first model on the validation dataset #1, and the variance of the SGCS represents the performance floating range of the first model.

[0145] The performance level (or performance grade) can correspond to a performance range. In other words, different performance values in different performance ranges correspond to different performance levels. For example, different SGCS values in the range [0.6, 0.7] correspond to the same performance level, and SGCS values in the range [0.6, 0.7] and SGCS values in the range [0.7, 0.8] correspond to different performance levels. It should be noted that the representation of the performance level is not limited in the embodiments of the present application. For example, the performance levels from high to low can be represented as “high”, “medium”, and “low” in turn. For another example, the performance levels from high to low can be represented as “1”, “2”, or “3” in turn. For another example, the performance levels from high to low can be represented as “A”, “B”, or “C” in turn.

[0146] The generalization capability of the model refers to the level of the processing capability of the model on different scenes or categories of data. For example, the generalization capability of the first model on the at least one validation data set can be the adaptability of the first model to the at least one validation data set. The generalization capability of the first model on the at least one validation data set can be represented by the number of target validation data sets in the at least one validation data set that match the first model, and the performance of the first model on the matching target validation data set is not lower than a performance threshold #1. When the performance of the first model on the target validation data set is not lower than the performance threshold #1, it can be understood that the first model matches the target validation data set. The performance threshold #1 can be preconfigured or pre-defined by a protocol, or indicated by the second device or the cloud to the first device, which is not limited in the present application.

[0147] The generalization capability level (or generalization capability gear) can correspond to the number range of the target validation data set, in other words, different numbers of the target validation data set in the number range can correspond to the same generalization capability level. For example, different numbers of the target validation data set in [number #1*2 / 3, number #1] can correspond to the same generalization capability level, and number #1 is the number of the at least one validation data set. The representation form of the generalization capability level can refer to the representation form of the performance level, which is not limited in the present application.

[0148] The following describes a manner in which the first device determines the performance information of the first model.

[0149] For example, the first device determines the performance information of the first model, including: the first device determines the performance information of the first model according to the at least one validation data set.

[0150] It can be understood that, in the case that the first device can determine the performance information of the first model according to the at least one validation data set, the performance information of the first model is related to the at least one validation data set, or the performance information of the first model has a corresponding relationship with the at least one validation data set.

[0151] Based on the above technical solutions, it is beneficial to realize that different devices use the same validation data set to determine the performance information of the model, so as to ensure the consistency and rationality of the performance evaluation of the model.

[0152] In a possible implementation manner, the performance information of the first model includes a performance level (denoted as performance level #1) corresponding to the performance of the first model on the at least one validation data set, and the performance of the first model on the at least one validation data set has a first corresponding relationship with the performance level #1.

[0153] In the implementation, the first device determines the performance information of the first model according to the at least one verification data set, including: the first device determines the performance of the first model on the at least one verification data set according to the at least one verification data set; and the first device determines the performance level #1 according to the performance of the first model on the at least one verification data set and a correspondence between the performance of a model on the at least one verification data set and a performance level. The correspondence between the performance of the model on the at least one verification data set and the performance level includes the first correspondence.

[0154] For example, if the performance of the first model on the at least one verification data set is represented by the SGCS, and the value of the SGCS of the first model on the at least one verification data set is 0.6, the first device determines the performance level #1 as “1” according to Table 1.

[0155] Table 1

[0156] It should be understood that the performance of the first model on the at least one verification data set is represented by the SGCS in Table 1 as an example, and the present application is not limited thereto. The performance of the first model on the at least one verification data set can include one or more performance values, for example, the performance of the first model on the at least one verification data set includes a performance value corresponding to the at least one verification data set, and each performance value represents the performance of the first model on the verification data set corresponding to the performance value. In the present application, [a, b) represents a set, which can include all values greater than or equal to a and less than b.

[0157] Optionally, the correspondence between the performance of the model on the at least one verification data set and the performance level corresponds to the first type of model, in other words, the correspondence between the performance of the model on the at least one verification data set and the performance level is the correspondence between the performance of the first type of model on the at least one verification data set and the performance level.

[0158] For example, if the type of the first model is a basic general model, the performance of the first model on the at least one verification data set is represented by the SGCS, and the value of the SGCS of the first model on the at least one verification data set is 0.6, the first device determines the performance level #1 as “1” according to Table 2.

[0159] Table 2

[0160] The correspondence between the performance of the model on the at least one validation data set and the performance level is predefined or preconfigured by the second device or the cloud or the data node, and the present application does not limit this.

[0161] Optionally, if the first device does not pre-acquire the correspondence between the performance of the model on the at least one validation data set and the performance level, the method 600 further includes S611: the first device receives second indication information from the second device or the cloud or the data node, and the second indication information includes the correspondence between the performance of the model on the at least one validation data set and the performance level. It should be understood that FIG. 6 only illustrates the first device receiving the second indication information from the second device as an example.

[0162] In a possible implementation, the performance information of the first model includes a generalization capability level (denoted as performance level #1) corresponding to the generalization capability of the first model on the at least one validation data set, and the generalization capability of the first model on the at least one validation data set has a first correspondence with the generalization capability level #1.

[0163] In this implementation, the first device determines the performance information of the first model according to the at least one validation data set, including: the first device determines the generalization capability of the first model on the at least one validation data set according to the at least one validation data set; and the first device determines the generalization capability level #1 according to the generalization capability of the first model on the at least one validation data set and the correspondence between the generalization capability of the model on the at least one validation data set and the generalization capability level. The correspondence between the generalization capability of the model on the at least one validation data set and the generalization capability level includes the first correspondence.

[0164] For example, the correspondence between the generalization capability of the model on the at least one validation data set and the generalization capability level is shown in Table 3 below. For example, if the number of target validation data sets in the at least one validation data set that match the first model is 2 / 3 of the number of the at least one validation data set, the first device determines the generalization capability level #1 as “2” according to Table 3. The number #1 is the number of the at least one validation data set.

[0165] Table 3

[0166] Optionally, the correspondence between the generalization capability of the model on the at least one validation data set and the generalization capability level corresponds to the first type of model, in other words, the correspondence between the generalization capability of the model on the at least one validation data set and the generalization capability level is the correspondence between the generalization capability of the first type of model on the at least one validation data set and the generalization capability level.

[0167] Exemplarily, the correspondence between the generalization capability of the different types of models on the at least one validation dataset and the generalization capability level is shown in Table 4 below. For example, if the type of the first model is the basic general model, the number of target validation datasets in the at least one validation dataset that match the first model is 2 / 3 of the number of the at least one validation dataset, and the generalization capability level #1 determined by the first device according to Table 4 is “2”.

[0168] Table 4

[0169] The correspondence between the generalization capability of the model on the at least one validation dataset and the generalization capability level is predefined or preconfigured by the second device or the cloud or the data node, which is not limited in the present application.

[0170] Optionally, if the first device does not pre-acquire the correspondence between the generalization capability of the model on the at least one validation dataset and the generalization capability level, the method 600 further includes S611: the first device receives second indication information from the second device or the cloud or the data node, and the second indication information includes the correspondence between the generalization capability of the model on the at least one validation dataset and the generalization capability level. It should be understood that FIG. 6 only illustrates the first device receiving the second indication information from the second device as an example.

[0171] Optionally, if the first device does not pre-acquire the at least one validation dataset, the method 600 further includes S612: the first device receives the at least one validation dataset or third indication information from the second device or the cloud or the data node, and the third indication information is used to indicate the at least one validation dataset. For example, the third indication information can include the identification or index of the at least one validation dataset. It should be understood that FIG. 6 only illustrates the first device receiving the third indication information or the at least one validation dataset from the second device as an example.

[0172] For example, the first model is trained by the first device, and the first device does not acquire the at least one validation dataset when acquiring the training dataset used to train the first model, and the method 600 further includes S612. For another example, the first model is configured by the second device or the cloud or the data node, and the method 600 further includes S612.

[0173] Optionally, if the at least one validation dataset is used for the determination of the performance information of the first type of model, the method 600 further includes S613: receiving fourth indication information from the second device or the cloud or the data node, and the fourth indication information is used to indicate that the at least one validation dataset is used for the determination of the performance information of the first type of model. For example, the fourth indication information includes the identification or index of the first type. It should be understood that FIG. 6 only illustrates the first device receiving the fourth indication information from the second device as an example.

[0174] In the embodiments of the present application, the first device can determine the performance information of the first model according to the at least one verification data set, so as to facilitate obtaining the first monitoring parameter that is more matched with the performance of the first model according to the performance information of the first model.

[0175] Optionally, the method 600 further includes S620.

[0176] S620, the first device determines the first monitoring parameter.

[0177] After the first device determines the performance information of the first model, the first monitoring parameter can be determined according to the performance information of the first model.

[0178] For example, the first monitoring parameter includes one or more of the following: the first performance threshold, the first monitoring frequency or period, the first monitoring duration, the first monitoring duration number, or the first monitoring error tolerance.

[0179] The performance threshold can be used to trigger model switching or model updating. For example, the performance threshold refers to the minimum performance for triggering model switching or model updating, in other words, if the performance of the model is less than or equal to the performance threshold, model switching or model updating is triggered. The first performance threshold can be the performance threshold for triggering the first model switching or the first model updating.

[0180] The monitoring frequency refers to the frequency of monitoring the model. The monitoring period refers to the period of monitoring the model. For example, the monitoring period is T, which means that the model is monitored at a time interval of T, that is, every time interval T, the performance of the model is monitored, for example, whether the model is invalid. T is a positive number. The first monitoring frequency or period refers to the frequency or period of monitoring the first model.

[0181] The monitoring duration refers to the duration of monitoring the model. The monitoring duration can be the total duration of monitoring the model, or the duration of monitoring the model in a model monitoring period. The first monitoring duration refers to the duration of monitoring the first model.

[0182] The monitoring duration refers to the number of times of model monitoring on the model. The monitoring duration can be the total number of times of model monitoring on the model, or the number of times of model monitoring on the model in a model monitoring period. One model monitoring can include the following steps: model inference using the model; and performance calculation of the model according to the inference result. Taking the model used for channel compression feedback as an example, one model monitoring can include the following steps: the base station issues a reference signal for monitoring, the UE determines channel information according to the reference signal, and inputs the channel information into the encoder, and then determines the performance of the model according to the output of the encoder and the channel information. The first monitoring duration refers to the number of times of model monitoring on the first model.

[0183] The monitoring error tolerance refers to the tolerance of the performance of the model being lower than the performance threshold. For example, the monitoring error tolerance refers to the tolerance of the difference between the performance of the model and the performance threshold, or the tolerance of the number of times of the performance of the model being lower than the performance threshold. For example, the monitoring error tolerance refers to the tolerance of the difference between the performance of the model and the performance threshold, and the value of the monitoring error tolerance is 10%, which means that the model switching or model updating can be triggered in the case that the difference between the performance of the model and the performance threshold exceeds 10% of the performance threshold. For another example, the monitoring error tolerance refers to the tolerance of the number of times of the performance of the model being lower than the performance threshold, and the value of the monitoring error tolerance is 3, which means that the model switching or model updating can be triggered in the case that the number of times of the performance of the model being lower than the performance threshold exceeds 3. The first monitoring error tolerance refers to the tolerance of the performance of the first model being lower than the performance threshold.

[0184] In a possible implementation, the first device determines the first monitoring parameter according to the performance information of the first model.

[0185] For example, the performance information of the first model comprises the performance of the first model on the at least one validation dataset, and the first monitoring parameter comprises the first performance threshold, the first device determines the first performance threshold as a threshold equal to or smaller than the performance of the first model on the at least one validation dataset, or the first performance threshold is equal to the sum of the performance of the first model on the at least one validation dataset and a bias, or the performance information of the first model comprises the mean and variance of the accuracy of the first model on the at least one validation dataset, and the first device determines the first performance threshold as the weighted sum or difference of the mean and variance of the accuracy of the first model on the at least one validation dataset. For another example, the performance information of the first model comprises the performance of the first model on the at least one validation dataset, and the first monitoring parameter comprises the first monitoring period, and the first device determines the first monitoring period to be greater than the monitoring period threshold #1 when the performance of the first model on the at least one validation dataset is better, for example, the performance of the first model on the at least one validation dataset is not smaller than the performance threshold #2. For another example, the performance information of the first model comprises the generalization ability of the first model on the at least one validation dataset, and the first monitoring parameter comprises the first monitoring period, and the first device determines the first monitoring period to be smaller than the monitoring period threshold #2 when the number of target validation datasets matched by the first model is smaller than the validation dataset number threshold #1, for example, the number of target validation datasets is not more than half of the number of the at least one validation dataset. Or the first monitoring period is the weighted sum of the number of validation datasets and a bias.

[0186] The performance threshold #2 can be preconfigured or pre-defined by a protocol, or indicated by the second device or the cloud to the first device, which is not limited in the present application.

[0187] The monitoring period threshold #1 and / or the monitoring period threshold #2 can be preconfigured or pre-defined by a protocol, or indicated by the second device or the cloud to the first device, which is not limited in the present application.

[0188] The validation dataset number threshold #1 can be preconfigured or pre-defined by a protocol, or indicated by the second device or the cloud to the first device, which is not limited in the present application.

[0189] It can be understood that the performance information of the first model is related to the first monitoring parameter, or the performance information of the first model has a corresponding relationship with the first monitoring parameter, for example, the performance information of the first model has a second corresponding relationship with the first monitoring parameter, when the first device determines the first monitoring parameter according to the performance information of the first model.

[0190] In a possible implementation, the first device determines the first monitoring parameter according to the performance information of the first model and the type of the first model.

[0191] For example, if the type of the first model is a basic general model, the first device can determine that the first monitoring period is greater than a monitoring period threshold #3. For another example, if the type of the first model is a cell-specific model, the first device can determine that the first monitoring period is less than a monitoring period threshold #4. The monitoring period threshold #3 and / or the monitoring period threshold #4 can be pre-configured or pre-defined by a protocol, or indicated by the second device or the cloud to the first device, which is not limited in the present application.

[0192] It can be understood that, in the case that the first device determines the first monitoring parameter according to the performance information of the first model and the type of the first model, the performance information of the first model and the type of the first model are related to the first monitoring parameter, or the performance information of the first model and the type of the first model have a corresponding relationship with the first monitoring parameter.

[0193] In a possible implementation, the first device determines the first monitoring parameter according to the performance information of the first model and the corresponding relationship between the performance information of the model and the monitoring parameter. The corresponding relationship between the performance information of the model and the monitoring parameter includes a second corresponding relationship, and the second corresponding relationship is a corresponding relationship between the performance information of the first model and the first monitoring parameter.

[0194] For example, if the performance information of the first model is a performance level #1, and the performance level #1 is “1”, the first monitoring parameter determined by the first device according to Table 5 includes a first performance threshold, and the first performance threshold is 0.6-Δ.

[0195] Table 5

[0196] Table 6

[0197] It should be understood that, in Table 5, the monitoring parameter is taken as an example of a performance threshold, and in Table 6, the monitoring parameter is taken as an example of a monitoring frequency, a monitoring duration number and a monitoring error tolerance.

[0198] Optionally, the corresponding relationship between the performance information of the model and the monitoring parameter corresponds to a first type of model, in other words, the corresponding relationship between the performance information of the model and the monitoring parameter is a corresponding relationship between the performance information of the first type of model and the monitoring parameter.

[0199] For example, if the type of the first model is a basic general model, the performance information of the first model is a performance level #1, and the performance level #1 is “1”, the first monitoring parameter determined by the first device according to Table 7 includes a first performance threshold, and the first performance threshold is 0.6-Δ.

[0200] Table 7

[0201] Table 8

[0202] The correspondence between the performance information of the model and the monitoring parameter is predefined, or is preconfigured by the second device or the cloud or the data node, and the present application does not limit this.

[0203] Optionally, if the first device does not pre-acquire the correspondence between the performance information of the model and the monitoring parameter, the method 600 further includes S621: the first device receives sixth indication information from the second device or the cloud or the data node, and the sixth indication information includes the correspondence between the performance information of the model and the monitoring parameter. It should be understood that FIG. 6 only illustrates the first device receiving the sixth indication information from the second device as an example.

[0204] Optionally, the method 600 further includes S630.

[0205] S630, the first device sends the first monitoring parameter.

[0206] Correspondingly, the second device receives the first monitoring parameter.

[0207] Illustratively, if the first device jointly performs performance monitoring on the first model with the second device, the first device can send the first monitoring parameter to the second device.

[0208] It should be understood that FIG. 6 illustrates the first device sending the first monitoring parameter to the second device as an example, and in actual implementation, the first device can send all or part of the parameters included in the first monitoring parameter to the second device.

[0209] Optionally, the method 600 further includes S640.

[0210] S630, the first device and / or the second device performs performance monitoring on the first model.

[0211] In one possible implementation, the first device performs performance monitoring on the first model according to the first monitoring parameter.

[0212] For example, the first device is a UE, the second device is a network device, the first model is used for CSI compression, and the first monitoring parameter determined by the first device includes a first performance threshold. The process of performance monitoring of the first device on the first model can include the following steps: the first device receives a reference signal from the second device; the first device determines CSI according to the reference signal; the first device takes the CSI or the reference signal as an input of the first model and obtains corresponding compressed CSI; the first device determines the performance of the first model by comparing the CSI and the compressed CSI; and the first device determines whether to trigger switching or updating of the first model by comparing the performance of the first model with the first performance threshold.

[0213] In a possible implementation, the first device and the second device perform performance monitoring on the first model according to the first monitoring parameter.

[0214] For example, the first device is a UE, the second device is a network device, the first model is used for CSI compression, and the first monitoring parameter determined by the first device includes a first performance threshold and a first monitoring period. The process of performance monitoring of the first device and the second device on the first model can include the following steps: the first device determines the transmission time of a reference signal according to the monitoring frequency / monitoring duration indication, to receive the reference signal from the second device; the first device determines CSI according to the reference signal; the first device takes the CSI or the reference signal as an input of the first model and obtains corresponding compressed CSI#1; the first device feeds back the CSI and the compressed CSI#1 to the second device; the second device inputs the compressed CSI#1 into a decoding model to obtain recovered channel information, and determines the performance of the first model by comparing the CSI#1 and the recovered channel information; and the second device determines whether to trigger switching or updating of the first model by comparing the performance of the first model with the first performance threshold. It should be noted that the first device determines to feed back the CSI and the compressed CSI to the second device in the process of performance monitoring of the first model according to the first monitoring period, and can not feed back the CSI to the second device when the first model is not monitored.

[0215] In the embodiments of the present application, the first device can determine the first monitoring parameter that is more matched to the performance of the first model according to the performance information of the first model, thereby facilitating to improve the efficiency and reliability of model monitoring, or reducing energy consumption or resource waste. For example, if the performance of the first model is better, the first device can determine to perform performance monitoring on the first model at a longer period, thereby reducing energy consumption or resource waste. For another example, if the generalization ability of the first model is poor, the first device can determine to perform performance monitoring on the first model at a shorter period, thereby avoiding the decline of communication quality caused by the decline of the performance of the first model.

[0216] The model monitoring method provided by the embodiment of the present application is described below in combination with FIG. 7. The method 700 shown in FIG. 7 is different from the method 600 in that the method 700 is that the device using the first model (i.e., the first device) sends the performance information of the first model to the other device (e.g., the second device) different from the device using the first model, and the other device determines the first monitoring parameter according to the performance information of the first model, while the method 600 is that the device using the first model (i.e., the first device) determines the first monitoring parameter according to the performance information of the first model.

[0217] FIG. 7 shows a schematic flowchart of the model monitoring method provided by the embodiment of the present application. As shown in FIG. 7, the method 700 can include the following steps.

[0218] Optionally, the method 700 includes one or more steps in S711-S713.

[0219] S711, the second device sends the second indication information.

[0220] Correspondingly, the first device receives the second indication information.

[0221] The second indication information includes the correspondence between the performance information of the model and the performance level, and the performance information of the model includes the performance or generalization ability of the model on at least one validation data set.

[0222] More description of S711 and the correspondence between the performance information of the model and the performance level can be referred to the method 600 above.

[0223] It should be noted that FIG. 7 takes the second device sending the second indication information to the first device as an example for description, and in actual implementation, the cloud or the data node can send the second indication information to the first device.

[0224] S712, the second device sends at least one validation data set or third indication information.

[0225] Correspondingly, the first device receives the at least one validation data set or the third indication information.

[0226] More description of S712 can be referred to S612 in the method 600 above.

[0227] It should be noted that FIG. 7 takes the second device sending the third indication information to the first device as an example for description, and in actual implementation, the cloud or the data node can send the third indication information to the first device.

[0228] S713, the second device sends fourth indication information.

[0229] Correspondingly, the first device receives the fourth indication information.

[0230] More description of S713 can be referred to S613 in method 600 above.

[0231] It should be noted that FIG. 7 takes the second device sending the fourth indication information to the first device as an example for description, and in actual implementation, the cloud or the data node can send the fourth indication information to the first device.

[0232] S710, the first device determines the performance information of the first model.

[0233] More description of S710 can be referred to S610 in method 600 above.

[0234] Optionally, the method 700 further includes S720.

[0235] S720, the first device sends fifth indication information.

[0236] Correspondingly, the second device receives the fifth indication information.

[0237] The fifth indication information is used to indicate the performance information of the first model. For example, the fifth indication information can include the performance information of the first model, or the fifth indication information can indicate the index of the performance information of the first model. For example, the performance information of the first model includes the performance or generalization ability of the first model on at least one validation dataset, and the fifth indication information can include the performance information of the first model. For another example, the performance information of the first model includes the performance level corresponding to the performance of the first model on at least one validation dataset, and the fifth indication information can indicate the index of the performance information of the first model. For example, the index indicated by the fifth indication information is “1”, which means that the fifth indication information is used to indicate that the performance level corresponding to the performance of the first model on at least one validation dataset is “1”.

[0238] Optionally, the method 700 further includes S721.

[0239] S721, the first device sends first indication information.

[0240] Correspondingly, the second device receives the first indication information.

[0241] The first indication information is used to indicate the type of the first model.

[0242] For example, if the first model deployed on the first device is obtained by model training of the first device or is configured by the cloud, the second device can not know the type of the first model, and therefore the method 700 can include S721.

[0243] Optionally, the method 700 further includes S730.

[0244] S730, the second device determines the first monitoring parameter.

[0245] The manner in which the second device determines the first monitoring parameter can refer to the manner in which the first device determines the first monitoring parameter described above in S620.

[0246] Optionally, the method 700 further includes S740.

[0247] S740, the second device sends the first monitoring parameter.

[0248] Correspondingly, the first device receives the first monitoring parameter.

[0249] Illustratively, if the second device instructs the first device to perform performance monitoring on the first model, the second device can send the first monitoring parameter to the first device.

[0250] It should be understood that FIG. 7 takes the second device sending the first monitoring parameter to the first device as an example, and in actual implementation, the second device can send all or part of the parameters included in the first monitoring parameter to the first device.

[0251] Optionally, the method 700 further includes S750.

[0252] S750, the first device and / or the second device performs performance monitoring on the first model.

[0253] S750 can refer to S640 in the method 600 above.

[0254] In the embodiments of the present application, the first device sends the performance information of the first model to the second device, so that the second device can determine the first monitoring parameter that is more matched with the performance of the first model according to the performance information of the first model, thereby facilitating to improve the efficiency and reliability of model monitoring, or reducing energy consumption or resource waste.

[0255] The model monitoring method provided in the embodiments of the present application is described below in combination with FIG. 8. The method 800 shown in FIG. 8 is different from the method 600 in that the method 800 is to determine the performance information of the first model by another device (for example, a second device) different from the device using the first model, and send the performance information of the first model to the device (that is, the first device) using the first model, while the method 600 is to determine the performance information of the first model by the device (that is, the first device) using the first model. The method 800 shown in FIG. 8 is different from the method 700 in that the method 800 is to determine the performance information of the first model by another device (for example, a second device) different from the device using the first model, and send the performance information of the first model to the device (that is, the first device) using the first model, so that the device using the first model can determine the first monitoring parameter according to the performance information of the first model, while the method 700 is to determine the performance information of the first model by the device (that is, the first device) using the first model, and send the performance information of the first model to another device (for example, a second device) different from the device using the first model, so that the other device (for example, the second device) different from the device using the first model can determine the first monitoring parameter according to the performance information of the first model.

[0256] FIG. 8 shows a schematic flowchart of the model monitoring method provided in the embodiments of the present application. As shown in FIG. 8, the method 800 can include the following steps.

[0257] Optionally, the method 800 includes one or more steps in S811 to S813.

[0258] S811, the cloud or the data node sends the second indication information.

[0259] Correspondingly, the second device receives the second indication information.

[0260] The second indication information includes the correspondence between the performance information of the model and the performance level, and the performance information of the model includes the performance or generalization ability of the model on at least one validation data set. More description of the correspondence between the performance information of the model and the performance level can be referred to the method 600 above.

[0261] It should be understood that if the second device does not pre-acquire the correspondence between the performance information of the model and the performance level, the method 800 can include S811. Or the second device pre-acquires the correspondence between the performance information of the model and the performance level, or the correspondence between the performance information of the model and the performance level is predefined or preconfigured, then the method 800 can not include S811.

[0262] S812, the cloud or the data node sends at least one validation data set or third indication information.

[0263] Correspondingly, the second device receives the at least one validation dataset or third indication information. The third indication information is used to indicate the at least one validation dataset.

[0264] It should be understood that if the second device does not obtain the at least one validation dataset in advance, the method 800 can include S812. Or the second device obtains the at least one validation dataset in advance, or the at least one validation dataset is predefined or preconfigured, the method 800 can not include S812.

[0265] S813, the cloud or the data node sends fourth indication information.

[0266] Correspondingly, the second device receives the fourth indication information.

[0267] The fourth indication information is used to indicate that the at least one validation dataset is used for determining the performance information of the first type of model. For example, the fourth indication information includes an identification or an index of the first type.

[0268] S810, the second device determines the performance information of the first model.

[0269] The way in which the second device determines the performance information of the first model can refer to the way in which the first device determines the performance information of the first model described in S610 above.

[0270] It can be understood that if the first model deployed on the side of the first device is configured by the second device, the second device can obtain the first model. In the case that the second device obtains the first model, the second device can determine the performance information of the first model according to the at least one validation dataset.

[0271] Optionally, the method 800 further includes S820.

[0272] S820, the second device sends fifth indication information.

[0273] Correspondingly, the first device receives the fifth indication information.

[0274] The fifth indication information is used to indicate the performance information of the first model. More description of the fifth indication information can refer to S720 in the method 700 above.

[0275] Optionally, the method 800 further includes S830.

[0276] S830, the first device determines the first monitoring parameter.

[0277] S830 can refer to S620 in the method 600 above.

[0278] Optionally, the method 800 further includes S840.

[0279] S840, the first device sends the first monitoring parameter.

[0280] Correspondingly, the second device receives the first monitoring parameter.

[0281] S840 can refer to S630 in method 600 above.

[0282] Optionally, the method 800 further includes S850.

[0283] S850, the first device and / or the second device performs performance monitoring on the first model.

[0284] S850 can refer to S640 in method 600 above.

[0285] In the embodiments of the present application, the second device sends the performance information of the first model to the first device, so that the first device can determine the first monitoring parameter that is more matched with the performance of the first model according to the performance information of the first model, thereby facilitating to improve the efficiency and reliability of model monitoring, or reducing energy consumption or resource waste.

[0286] The model monitoring method provided by the embodiments of the present application is described below in combination with FIG. 9. The method 900 shown in FIG. 9 is different from the method 600 in that the method 900 is to determine the performance information of the first model and the first monitoring parameter by the other device (for example, the second device) different from the device using the first model, while the method 600 is to determine the performance information of the first model and the first monitoring parameter by the device (i.e., the first device) using the first model. The method 900 shown in FIG. 9 is different from the method 700 in that the method 900 is to determine the performance information of the first model by the other device (for example, the second device) different from the device using the first model, while the method 700 is to determine the performance information of the first model by the device (i.e., the first device) using the first model and send the performance information of the first model to the other device (for example, the second device) different from the device using the first model. The method 900 shown in FIG. 9 is different from the method 800 in that the method 900 is to determine the first monitoring parameter by the other device (for example, the second device) different from the device using the first model, while the method 800 is to send the performance information of the first model by the other device (for example, the second device) different from the device using the first model to the device (i.e., the first device) using the first model, and then determine the first monitoring parameter by the device according to the performance information of the first model.

[0287] FIG. 9 shows a schematic flowchart of the model monitoring method provided by the embodiments of the present application. As shown in FIG. 9, the method 900 can include the following steps.

[0288] Optionally, the method 900 includes one or more steps of S911 to S913.

[0289] S911, the cloud or the data node sends second indication information.

[0290] Correspondingly, the second device receives the second indication information.

[0291] The second indication information includes a correspondence between performance information of the model and a performance level, and the performance information of the model includes performance or generalization capability of the model on at least one validation dataset. More description of the correspondence between the performance information of the model and the performance level can refer to the method 600 described above.

[0292] It should be understood that if the second device does not pre-acquire the correspondence between the performance information of the model and the performance level, the method 900 can include S911. Or the second device pre-acquires the correspondence between the performance information of the model and the performance level, or the correspondence between the performance information of the model and the performance level is predefined or preconfigured, the method 900 can not include S911.

[0293] S912, the cloud or the data node sends at least one validation dataset or third indication information.

[0294] Correspondingly, the second device receives the at least one validation dataset or the third indication information. The third indication information is used to indicate the at least one validation dataset.

[0295] It should be understood that if the second device does not pre-acquire the at least one validation dataset, the method 900 can include S912. Or the second device pre-acquires the at least one validation dataset, or the at least one validation dataset is predefined or preconfigured, the method 900 can not include S912.

[0296] S913, the cloud or the data node sends fourth indication information.

[0297] Correspondingly, the second device receives the fourth indication information.

[0298] The fourth indication information is used to indicate that the at least one validation dataset is used for determination of the performance information of the model of the first type. For example, the fourth indication information includes an identification or an index of the first type.

[0299] S910, the second device determines the performance information of the first model.

[0300] The way in which the second device determines the performance information of the first model can refer to the way in which the first device determines the performance information of the first model described in S610 above.

[0301] It can be understood that if the first model deployed at the first device side is configured by the second device, the second device can obtain the first model. In the case that the second device obtains the first model, the second device can determine the performance information of the first model according to the at least one verification data set.

[0302] Optionally, the method 900 further includes S920.

[0303] S920, the second device determines the first monitoring parameter.

[0304] S920 can refer to S620 in the method 600 above.

[0305] Optionally, the method 900 further includes S930.

[0306] S930, the second device sends the first monitoring parameter.

[0307] Correspondingly, the first device receives the first monitoring parameter.

[0308] S930 can refer to S740 in the method 700 above.

[0309] Optionally, the method 900 further includes S940.

[0310] S940, the first device and / or the second device performs performance monitoring on the first model.

[0311] S940 can refer to S640 in the method 600 above.

[0312] In the embodiments of the present application, the second device can determine the performance information of the first model, and determine the first monitoring parameter that is more matched with the performance of the first model according to the performance information of the first model, thereby facilitating to improve the efficiency and reliability of model monitoring, or reducing energy consumption or resource waste.

[0313] It should be understood that the size of the serial number of the above processes does not mean the order of execution, and the execution order of the processes should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0314] It should also be understood that in various embodiments of the present application, the terms and / or descriptions of different embodiments are consistent and can be mutually referred to if there is no special description and no logical conflict, and the technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationship.

[0315] It can be understood that the methods and operations implemented by the devices (such as the first device and the second device) in the above various method embodiments can also be implemented by components (such as chips or circuits) of the devices.

[0316] The model monitoring method provided by the embodiments of the present application is described in detail above in combination with FIG. 6 to FIG. 9. The model monitoring method is mainly introduced from the perspective of the interaction between the first device and the second device. It can be understood that the first device and the second device include the hardware structure and / or software module corresponding to the execution of each function in order to achieve the above functions.

[0317] It can be understood that, in order to implement the functions in the above embodiments, the first device and the second device include the hardware structure and / or software module corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and method steps of each example described in the embodiments disclosed in the present application, the present application can be realized in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the form of hardware or computer software driving hardware depends on the specific application scenario and design constraints of the technical solution.

[0318] FIG. 10 and FIG. 11 are schematic block diagrams of communication devices provided by the embodiments of the present application. These communication devices can be used to implement the functions of the first device or the second device in the above method embodiments, and thus can also achieve the beneficial effects possessed by the above method embodiments.

[0319] FIG. 10 is a schematic block diagram of a communication device 2000 provided by the embodiments of the present application. As shown in FIG. 10, the communication device 2000 includes a transceiver unit (or communication unit) 2020, and optionally, the communication device 2000 further includes a processing unit 2010. The communication device 2000 is used to implement the functions of the first device or the second device in the method embodiments shown in FIG. 6, FIG. 7, FIG. 8 or FIG. 9.

[0320] When the communication device 2000 is used to implement the functions of the first device or the second device in the method embodiments shown in FIG. 6, FIG. 7, FIG. 8 or FIG. 9, the transceiver unit 2020 and / or the processing unit 2010 are used to obtain performance information of a first model. The performance information of the first model is determined according to at least one verification data set, and the performance information of the first model is used for the determination of a first monitoring parameter; the first monitoring parameter is used for performance monitoring of the first model.

[0321] For more detailed descriptions of the processing unit 2010 and the transceiver unit 2020, reference can be made to the related descriptions in the method embodiments shown in FIG. 6, FIG. 7, FIG. 8 or FIG. 9.

[0322] The apparatus 2000 of each of the above-mentioned solutions has a function of implementing the corresponding steps performed by the first device in the above-mentioned methods, or the apparatus 2000 of each of the above-mentioned solutions has a function of implementing the corresponding steps performed by the second device in the above-mentioned methods. The function can be implemented by hardware, or the corresponding software is executed by hardware. The hardware or software includes one or more modules corresponding to the above-mentioned functions; for example, the transceiver unit can be replaced by a transceiver (for example, the transmitting unit in the transceiver unit can be replaced by a transmitter, and the receiving unit in the transceiver unit can be replaced by a receiver), and other units such as the processing unit can be replaced by a processor, which respectively performs the transceiving operation and the related processing operation in each method embodiment.

[0323] In addition, the transceiver unit can also be a transceiver circuit (for example, which can include a receiving circuit and a transmitting circuit), and the processing unit can be a processing circuit. The processing circuit can be one or more processors, or all or part of the circuit for control or processing function in the one or more processors. In the embodiment of the present application, the apparatus in FIG. 10 can be the first device or the second device in the foregoing embodiments, or can be a chip or a chip system, for example, a system on chip (SoC). Wherein, the transceiver unit can be an input / output circuit or a communication interface; and the processing unit is a processor or a microprocessor integrated on the chip or an integrated circuit. In this regard, no limitation is made.

[0324] FIG. 11 is a schematic block diagram of a communication apparatus 3000 provided by the embodiment of the present application. The apparatus 3000 includes a processing circuit. The apparatus can also include a communication circuit. Wherein, the processing circuit and the communication circuit communicate with each other through an internal connection path, and the processing circuit is used to execute instructions to control the communication circuit to transmit and / or receive signals.

[0325] Taking the processing circuit including one or more processors and the communication circuit being a transceiver as an example, as shown in FIG. 11, the communication apparatus 3000 includes a processor 3010 and a transceiver 3020. The processor 3010 and the transceiver 3020 are coupled with each other. It can be understood that the transceiver 3020 can be a transceiver or an input / output interface. Optionally, the communication apparatus 3000 can also include a memory 3030 for storing instructions executed by the processor 3010 or storing input data required by the processor 3010 to run instructions or storing data generated after the processor 3010 runs instructions. Sometimes, the transceiver 3020 can also be understood as a part of the processor 3010, and at this time, the communication apparatus 3000 includes the processor 3010.

[0326] In a possible implementation, the apparatus 3000 is configured to implement the procedures and steps corresponding to the first device in the above method embodiments. In another possible implementation, the apparatus 3000 is configured to implement the procedures and steps corresponding to the second device in the above method embodiments.

[0327] It can be understood that the apparatus 3000 can be specifically the first device or the second device in the above embodiments, and can also be a chip or a chip system. Correspondingly, the communication circuit can be an interface circuit of the chip, or an input / output circuit, which is not limited herein. Specifically, the apparatus 3000 can be configured to perform the procedures and steps corresponding to the first device or the second device in the above method embodiments.

[0328] When the apparatus 3000 is configured to implement the method shown in FIG. 6, FIG. 7, FIG. 8 or FIG. 9, the processor 3010 is configured to implement the functions of the processing unit 2010, and the transceiver 3020 is configured to implement the functions of the transceiving unit 2020.

[0329] When the apparatus is a chip or an OTT device applied to the first device, the chip or the OTT device of the first device implements the functions of the first device in the above method embodiments, for example, implements the processing functions of the first device. The chip or the OTT device of the first device receives information from the second device, which can be understood as that the information is first received by other modules (such as a radio frequency module or an antenna) in the first device, and then transmitted to the chip or the OTT device of the first device by the modules. The chip or the OTT device of the first device transmits information to the second device, which can be understood as that the information is first transmitted by the chip or the OTT device of the first device to other modules (such as a radio frequency module or an antenna) in the first device, and then transmitted to the second device by the modules.

[0330] When the apparatus is a chip or an OTT device applied to the second device, the chip or the OTT device of the second device implements the functions of the second device in the above method embodiments, for example, implements the processing functions of the second device. The chip or the OTT device of the second device receives information from the first device, which can be understood as that the information is first received by other modules (such as a radio frequency module or an antenna) in the second device, and then transmitted to the chip or the OTT device of the second device by the modules. The chip or the OTT device of the second device transmits information to the first device, which can be understood as that the information is first transmitted by the chip or the OTT device of the second device to other modules (such as a radio frequency module or an antenna) in the second device, and then transmitted to the first device by the modules.

[0331] It can be understood that, in order to implement the functions in the above embodiments, the first device and the second device comprise hardware structures and / or software modules corresponding to the respective functions. Those skilled in the art should easily understand that, in combination with the units and method steps of the examples described in the embodiments disclosed in the present application, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is implemented in hardware or computer software driving hardware depends on the specific application scenarios and design constraints of the technical solutions.

[0332] It can be understood that the processor in the embodiments of the present application can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA), image processors, artificial intelligence processors or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor can be a microprocessor or any conventional processor.

[0333] The method steps in the embodiments of the present application can be implemented in hardware or in software instructions executable by a processor. The software instructions can be composed of corresponding software modules, which can be stored in a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, a register, a hard disk, a mobile hard disk, a CD-ROM or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor, so that the processor can read information from the storage medium and write information to the storage medium. The storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in the first device or the second device. The processor and the storage medium can also exist as discrete components in the first device or the second device.

[0334] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer programs or instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments are performed. The computer can be a general purpose computer, a special purpose computer, a computer network, a network device, a user equipment or other programmable apparatus. The computer programs or instructions can be stored in a computer readable storage medium or transferred from one computer readable storage medium to another computer readable storage medium, for example, the computer programs or instructions can be transferred from one website site, computer, server or data center to another website site, computer, server or data center through wired or wireless manner. The computer readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center and the like integrated with one or more available media. The available media can be a magnetic medium, such as a floppy disk, a hard disk, a magnetic tape; or an optical medium, such as a digital video disc; or a semiconductor medium, such as a solid state disk. The computer readable storage medium can be a volatile or non-volatile storage medium, or can include both volatile and non-volatile storage media.

[0335] In the above various embodiments, the terms and / or descriptions of different embodiments are consistent and can be referred to each other if there is no special description and logical conflict. The technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationship.

[0336] “at least one” herein means one or more. “more” means two or more. “and / or” describes the association relationship of the associated objects, which means that there can be three kinds of relationships, for example, A and / or B, which can represent the following cases: A exists alone, A and B exist together, B exists alone, where A and B can be singular or plural. In the literal description of the present application, the character “ / ” generally represents that the front and rear associated objects are in an “or” relationship; in the formula of the present application, the character “ / ” represents that the front and rear associated objects are in a “division” relationship. “including at least one of A, B and C” can mean: including A; including B; including C; including A and B; including A and C; including B and C; including A, B and C.

[0337] Those skilled in the art can clearly understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0338] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0339] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the above-described device embodiments are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0340] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0341] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit.

[0342] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0343] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A model monitoring method characterized by, The method comprises: obtaining performance information of a first model, the performance information of the first model being determined according to at least one validation data set, the performance information of the first model being used for determination of a first monitoring parameter; the first monitoring parameter is used for performance monitoring of the first model.

2. The method of claim 1, wherein, The first monitoring parameter is determined according to the performance information of the first model and the type of the first model.

3. The method of claim 2, wherein, The type of the first model is a basic general model or a cell-specific model.

4. The method according to claim 2 or 3, characterized in that, The method further comprises: sending or receiving first indication information, the first indication information being used for indicating the type of the first model.

5. The method according to any one of claims 1 to 4, characterized in that, The performance information of the first model comprises one or more of the following: performance of the first model on the at least one validation data set, generalization ability of the first model on the at least one validation data set, performance level corresponding to the performance of the first model on the at least one validation data set, or generalization ability level corresponding to the generalization ability of the first model on the at least one validation data set.

6. The method according to any one of claims 1 to 5, characterized in that, The method further comprises: determining the performance information of the first model according to the at least one validation data set.

7. The method of claim 6, wherein, The performance information of the first model has a first correspondence relationship with the performance and / or generalization ability of the first model on the at least one validation data set.

8. The method of claim 7, wherein, The method further comprises: receiving second indication information, the second indication information comprising a correspondence relationship between performance and / or generalization ability of a model on the at least one validation data set and performance information of the model, the correspondence relationship between performance and / or generalization ability of the model on the at least one validation data set and the performance information of the model comprising the first correspondence relationship.

9. The method according to any one of claims 1 to 8, characterized in that, The method further comprises: receiving at least one of the following: third indication information, the third indication information being used for indicating the at least one validation data set; or the at least one validation data set.

10. The method of claim 9, wherein, The type of the first model is a first type, and the method further comprises: receiving fourth indication information, the fourth indication information being used for indicating that the at least one validation data set is used for determination of performance information of a model of the first type.

11. The method according to any one of claims 1 to 5, characterized in that, The method further comprises: receiving fifth indication information, the fifth indication information being used for indicating the performance information of the first model.

12. The method according to any one of claims 1 to 11, characterized in that, The performance information of the first model has a second correspondence relationship with the first monitoring parameter, and the method further comprises: determining the first monitoring parameter according to the performance information of the first model and a correspondence relationship between performance information of a model and a monitoring parameter, the correspondence relationship between the performance information of the model and the monitoring parameter comprising the second correspondence relationship.

13. The method of claim 12, wherein, The method further comprises: receiving sixth indication information, the sixth indication information comprising a correspondence relationship between performance information of a model and a monitoring parameter.

14. The method according to claim 12 or 13, characterized in that, The method further comprises: sending the first monitoring parameter.

15. The method according to any one of claims 1 to 10, characterized in that, The method further comprises: sending fifth indication information, the fifth indication information being used for indicating the performance information of the first model; receiving the first monitoring parameter.

16. The method according to any one of claims 12 to 15, characterized in that, The method further comprises: performing performance monitoring on the first model according to the first monitoring parameter.

17. The method of any one of claims 1 to 16, wherein, The first monitoring parameter comprises one or more of: a first performance threshold, a first monitoring frequency or period, a first monitoring duration, or a first monitoring error tolerance.

18. A communications device, characterized by comprises means or units for performing the method of any of claims 1 to 17.

19. A communications device, characterized by comprises a processor for executing computer programs or instructions to cause the method of any of claims 1 to 17 to be performed.

20. The communication apparatus according to claim 19, wherein, The communication device further comprises a memory.

21. A computer-readable storage medium, characterized in that, The computer readable storage medium stores program code for execution by an apparatus, the program code for performing the method of any of claims 1 to 17.

22. A computer program product, characterised in that, comprises instructions which, when the computer program product is run on a computer, cause the computer to carry out the method of any of claims 1 to 17.

23. A chip, characterized by comprises a processor and a communication interface, the processor reading instructions on a memory through the communication interface to perform the method of any of claims 1 to 17.

Images