Model monitoring method and device and communication equipment

Abstract

The invention discloses a model monitoring method and device and communication equipment, and belongs to the technical field of communication, and the monitoring method comprises the steps that first equipment monitors a first AI model according to a preset monitoring type; wherein the preset monitoring type comprises at least one of the following items: a first monitoring type, the first monitoring type is to monitor a plurality of AI models by using the same monitoring index, and the first AI model belongs to the plurality of AI models; and a second monitoring type, wherein the second monitoring type is to monitor the first AI model by adopting a special monitoring index.

Description

Technical Field

[0001] The present application belongs to the field of communication technology, and specifically relates to a model monitoring method, device and communication equipment. Background Art

[0002] With the continuous development of communication technology, artificial intelligence (AI) has also been widely used in various parts of communication systems to improve the performance of communication systems.

[0003] However, in communication-related technologies, usually only a single model monitoring solution can be used to monitor AI models, resulting in poor flexibility in model monitoring and limiting the performance of the communication system. Summary of the invention

[0004] The embodiments of the present application provide a model monitoring method, apparatus, and communication device, which can achieve effective monitoring of the AI ​​model while ensuring the flexibility of model monitoring and avoiding limitations on the performance of the communication system.

[0005] In a first aspect, a model monitoring method is provided, including: a first device monitors a first artificial intelligence (AI) model according to a predetermined monitoring type; wherein the predetermined monitoring type includes at least one of the following: a first monitoring type, wherein the first monitoring type is to monitor multiple AI models using the same monitoring indicator, wherein the first AI model belongs to the multiple AI models; a second monitoring type, wherein the second monitoring type is to monitor the first AI model using a dedicated monitoring indicator.

[0006] In a second aspect, a model monitoring method is provided, including: a second device sends first information to a first device; wherein the first information is used to configure relevant information of a first monitoring type, and the first monitoring type is to monitor multiple AI models using the same monitoring indicators.

[0007] In a third aspect, a model monitoring device is provided, comprising: a monitoring module, used to monitor a first artificial intelligence (AI) model according to a predetermined monitoring type; wherein the predetermined monitoring type includes at least one of the following: a first monitoring type, wherein the first monitoring type is to monitor multiple AI models using the same monitoring indicator, wherein the first AI model belongs to the multiple AI models; a second monitoring type, wherein the second monitoring type is to monitor the first AI model using a dedicated monitoring indicator.

[0008] In a fourth aspect, a model monitoring device is provided, including: a transmission module for sending first information to a first device; wherein the first information is used to configure relevant information of a first monitoring type, and the first monitoring type is to monitor multiple AI models using the same monitoring indicators.

[0009] In a fifth aspect, a communication device is provided, which includes a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the method described in the first aspect or the second aspect are implemented.

[0010] In a sixth aspect, a communication device is provided, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the steps of the method described in the first aspect, or to implement the steps of the method described in the second aspect.

[0011] In the seventh aspect, a readable storage medium is provided, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented, or the steps of the method described in the second aspect are implemented.

[0012] In an eighth aspect, a wireless communication system is provided, comprising: a first device and a second device, wherein the first device can be used to execute the steps of the method described in the first aspect, and the second device can be used to execute the steps of the method described in the second aspect.

[0013] In the ninth aspect, a chip is provided, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instructions to implement the steps of the method described in the first aspect, or to implement the steps of the method described in the second aspect.

[0014] In a tenth aspect, a computer program / program product is provided, wherein the computer program / program product is stored in a storage medium, and the program / program product is executed by at least one processor to implement the steps of the method described in the first aspect or the second aspect.

[0015] In an embodiment of the present application, the first device monitors the first AI model according to a predetermined monitoring type, and the predetermined monitoring type includes at least one of a first monitoring type and a second monitoring type. Thus, the first device can determine different monitoring types for model monitoring according to different needs, thereby effectively improving the flexibility of model monitoring to adapt to different model monitoring scenarios and avoid limiting the performance of the communication system. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a structural diagram of a wireless communication system provided by an exemplary embodiment of the present application.

[0017] Figure 2 It is one of the flow charts of the model monitoring method provided by an exemplary embodiment of the present application.

[0018] Figure 3 This is the second flowchart of the model monitoring method provided by an exemplary embodiment of the present application.

[0019] Figure 4a It is one of the interactive flow diagrams of the model monitoring method provided by an exemplary embodiment of the present application.

[0020] Figure 4b This is the second interactive flow diagram of the model monitoring method provided by an exemplary embodiment of the present application.

[0021] Figure 4c This is the third interactive flow diagram of the model monitoring method provided by an exemplary embodiment of the present application.

[0022] Figure 4d This is the fourth interactive flow diagram of the model monitoring method provided by an exemplary embodiment of the present application.

[0023] Figure 5 This is the third flow chart of the model monitoring method provided by an exemplary embodiment of the present application.

[0024] Figure 6 It is one of the structural schematic diagrams of a model device provided by an exemplary embodiment of the present application.

[0025] Figure 7 This is the second structural schematic diagram of the model device provided by an exemplary embodiment of the present application.

[0026] Figure 8 It is a structural diagram of a communication device provided by an exemplary embodiment of the present application.

[0027] Fig. 9 It is a schematic diagram of the structure of a terminal provided by an exemplary embodiment of the present application.

[0028] Fig.10 It is a schematic diagram of the structure of a network side device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of this application.

[0030] The terms "first", "second", etc. of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of one type, and the number of objects is not limited, for example, the first object can be one or more. In addition, "or" in the present application represents at least one of the connected objects. For example, "A or B" covers three schemes, namely, Scheme 1: including A but not including B; Scheme 2: including B but not including A; Scheme 3: including both A and B. The character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0031] The term "indication" in this application can be a direct indication (or explicit indication) or an indirect indication (or implicit indication). A direct indication can be understood as the sender explicitly informing the receiver of specific information, operations to be performed, or request results in the sent indication; an indirect indication can be understood as the receiver determining the corresponding information according to the indication sent by the sender, or making a judgment and determining the operation to be performed or the request result according to the judgment result.

[0032] It is worth noting that the technology described in the embodiments of the present application is not limited to the Long Term Evolution (LTE) / LTE-Advanced (LTE-A) system, but can also be used in other wireless communication systems, such as Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal Frequency Division Multiple Access (OFDMA), Single-carrier Frequency-Division Multiple Access (SC-FDMA) or other systems. The terms "system" and "network" in the embodiments of the present application are often used interchangeably, and the described technology can be used for the systems and radio technologies mentioned above as well as for other systems and radio technologies. The following description describes a New Radio (NR) system for illustrative purposes, and NR terminology is used in most of the following descriptions, but these technologies can also be applied to systems other than NR systems, such as the 6th generation (6 thGeneration, 6G) communication system.

[0033] Figure 1A block diagram of a wireless communication system applicable to an embodiment of the present application is shown. The wireless communication system includes a terminal 11 and a network side device 12. Among them, the terminal 11 can be a mobile phone, a tablet computer (Tablet Personal Computer), a laptop computer (Laptop Computer), a notebook computer, a personal digital assistant (Personal Digital Assistant, PDA), a handheld computer, a netbook, an ultra-mobile personal computer (Ultra-mobile Personal Computer, UMPC), a mobile Internet device (Mobile Internet Device, MID), an augmented reality (Augmented Reality, AR), a virtual reality (Virtual Reality, VR) device, a robot, a wearable device (Wearable Device), an aircraft (flight vehicle), a vehicle-mounted device (Vehicle User Equipment, VUE), a ship-mounted device, a pedestrian terminal (Pedestrian User Equipment, PUE), a smart home (home appliances with wireless communication functions, such as refrigerators, televisions, washing machines or furniture, etc.), a game console, a personal computer (Personal Computer, PC), a teller machine or a self-service machine and other terminal side devices. Wearable devices include: smart watches, smart bracelets, smart headphones, smart glasses, smart jewelry (smart bracelets, smart bracelets, smart rings, smart necklaces, smart anklets, smart anklets, etc.), smart wristbands, smart clothing, etc. Among them, the vehicle-mounted device can also be called a vehicle-mounted terminal, a vehicle-mounted controller, a vehicle-mounted module, a vehicle-mounted component, a vehicle-mounted chip or a vehicle-mounted unit, etc. It should be noted that the specific type of the terminal 11 is not limited in the embodiment of the present application. The network side device 12 may include an access network device or a core network device, wherein the access network device may also be referred to as a radio access network (Radio Access Network, RAN) device, a radio access network function or a radio access network unit. The access network device may include a base station, a wireless local area network (Wireless Local Area Network, WLAN) access point (Access Point, AS) or a wireless fidelity (Wireless Fidelity, WiFi) node, etc.Among them, the base station may be referred to as a Node B (NB), an evolved Node B (eNB), a next generation Node B (gNB), a New Radio Node B (NR Node B), an access point, a Relay Base Station (RBS), a Serving Base Station (SBS), a Base Transceiver Station (BTS), a radio base station, a radio transceiver, a Basic Service Set (BSS), an Extended Service Set (ESS), a Home Node B (HNB), a Home Evolved Node B, a Transmission Reception Point (TRP) or other appropriate terms in the field. As long as the same technical effect is achieved, the base station is not limited to specific technical terms. It should be noted that in the embodiments of the present application, only the base station in the NR system is used as an example for introduction, and the specific type of the base station is not limited.

[0034] It is worth noting that Figure 1 The communication scenario shown in is one of the application scenarios of the technical solution provided by this application. For example, in this application, the combination of the first device and the second device mentioned later or the application scenario of the technical solution provided by this application can be but not limited to the following.

[0035] a) The first device is Figure 1 The terminal 11 and the second device are Figure 1 The network side device 12 is shown.

[0036] b) The first device is Figure 1 The network side device 12 and the second device are Figure 1 Terminal 11 is shown.

[0037] c) Both the first device and the second device are terminals.

[0038] d) Both the first device and the second device are network side devices.

[0039] In addition, the AI ​​model mentioned in the context of this application may also be referred to as an AI unit, an AI structure, etc., or the AI ​​model may also refer to a processing unit that can implement specific algorithms, formulas, processing procedures, capabilities, etc. related to AI, or the AI ​​model may also be a processing method, algorithm, function, module or unit for a specific data set, or the AI ​​model may be a processing method, algorithm, function, module or unit running on AI-related hardware such as a graphics processing unit (GPU), a neural network processing unit (NPU), a tensor processing unit (TPU), an application-specific integrated circuit (ASIC), etc., and this application does not make specific restrictions on this. Optionally, the specific data set may include but is not limited to the input or output of the AI ​​model.

[0040] Correspondingly, the subsequent mentioned first AI model, etc. can be described by an AI model identifier. Among them, the AI ​​model identifier can be an AI unit identifier, an AI structure identifier, an AI algorithm identifier, a functional identifier (functionality ID), a physical identifier, a logical identifier, a global identifier, a local identifier, or an identifier of a specific data set associated with the AI ​​model, or an identifier of a specific scene related to the AI, an environment identifier related to the AI ​​model, a channel feature identifier related to the AI ​​model, an identifier of a device related to the AI ​​model, or an identifier of a function, feature, capability or module related to the AI, which is not specifically limited in this application.

[0041] Based on this, the technical solution provided by the embodiments of the present application is described in detail below through some embodiments and their application scenarios in combination with the accompanying drawings.

[0042] like Figure 2 As shown, it is a flow chart of a model monitoring method 200 provided by an exemplary embodiment of the present application, and the method 200 can be, but is not limited to, executed by a first device, and specifically can be executed by hardware or software installed in the first device. In this embodiment, the method 200 can at least include the following steps.

[0043] S210: The first device monitors the first AI model according to a predetermined monitoring type.

[0044] Before performing model monitoring, the first device may determine an AI model to be monitored from multiple configured AI models, such as the first AI model, according to a model monitoring period, etc. In this embodiment, the first AI model may be used to calculate, predict, and evaluate communication data, thereby improving the performance of the communication system.

[0045] Optionally, the first AI model can be but is limited to a neural network, a decision tree, a support vector machine, a Bayesian classifier, etc., and the neural network can be but is not limited to a deep neural network, a convolutional neural network, a recurrent neural network, etc.

[0046] Based on this, when the present application performs model monitoring, the predetermined monitoring type adopted may include but is not limited to at least one of the first monitoring type and the second monitoring type.

[0047] Among them, the first monitoring type is to monitor multiple AI models using the same monitoring indicators, and the first AI model belongs to the multiple AI models. In other words, the first monitoring type is to use unified or the same monitoring indicators for different AI models to perform model monitoring. Therefore, on the one hand, it can avoid the problem of using different AI model monitoring solutions for different wireless AI use cases to monitor AI models. On the other hand, it can effectively reduce the complexity and redundancy of model monitoring, reduce signaling interaction overhead, and ensure the performance of the communication system.

[0048] The second monitoring type is to monitor the first AI model using dedicated monitoring indicators. In other words, the second monitoring model can use different or dedicated monitoring indicators for different AI models to perform model monitoring, thereby ensuring the reliability and accuracy of the model monitoring results corresponding to each AI model.

[0049] Of course, when the terminal monitors the first AI model, the first device may monitor the first AI model only based on the first monitoring type or the second monitoring type, or may monitor the first AI model by combining the first monitoring type and the second monitoring type. Thus, the flexibility of model monitoring can be effectively improved to adapt to different model monitoring scenarios and avoid limiting the performance of the communication system.

[0050] It can be understood that as to which monitoring type the first device adopts to monitor the first AI model, it can be determined by the first device according to a preconfigured trigger condition, or the first device can apply to the second device to adopt the first monitoring type or the second monitoring type for model monitoring, or the second device can instruct the first device to adopt the first monitoring type or the second monitoring type for model monitoring, or the first device can autonomously determine whether to adopt the first monitoring type or the second monitoring type for model monitoring, etc., and there is no limitation here.

[0051] In this embodiment, the first device monitors the first AI model according to a predetermined monitoring type, and the predetermined monitoring type includes at least one of the first monitoring type and the second monitoring type. Thus, the first device can determine different monitoring types for model monitoring according to different needs, thereby effectively improving the flexibility of model monitoring to adapt to different model monitoring scenarios and avoid limiting the performance of the communication system.

[0052] In addition, in the case where the first device monitors the first AI model according to the first monitoring type, it can also effectively reduce the complexity and redundancy of model monitoring, reduce signaling interaction overhead, and ensure communication system performance.

[0053] like Figure 3 As shown, it is a flow chart of a model monitoring method 300 provided by an exemplary embodiment of the present application, and the method 300 can be, but is not limited to, executed by a first device, and specifically can be executed by hardware or software installed in the first device. In this embodiment, the method 300 can at least include the following steps.

[0054] S310: The first device monitors the first AI model according to a predetermined monitoring type.

[0055] Among them, the predetermined monitoring type includes at least one of the following: a first monitoring type, wherein the first monitoring type is to monitor multiple AI models using the same monitoring indicators, wherein the first AI model belongs to the multiple AI models; a second monitoring type, wherein the second monitoring type is to monitor the first AI model using dedicated monitoring indicators.

[0056] It can be understood that in addition to referring to the relevant description in method embodiment 200, in some embodiments, for the scenario where the first device monitors the first AI model based on the first monitoring type or the second monitoring type, considering that the monitoring granularity of the first monitoring type is relatively coarse, it can only reflect the approximate performance of the AI ​​model, while the monitoring granularity of the second monitoring type is relatively fine, it can reflect the specific performance of the AI ​​model. Therefore, for the first device that supports both the first monitoring type and the second monitoring type, if the accuracy requirement of the first AI model is high, then the first monitoring type can be called first to monitor the first AI model in general scenarios, and the second monitoring type can be called in scenarios where performance deteriorates severely, to achieve combined monitoring of the AI ​​models, and vice versa.

[0057] Based on this, the aforementioned combined monitoring can be implemented in this embodiment through signaling or pre-configured trigger conditions, etc. For example, the first device can switch the monitoring type according to at least one of the following methods 1 to 3.

[0058] Method 1: The first device sends fourth information to the second device, and the fourth information is used to apply to the second device to switch from the third monitoring type to the fourth monitoring type; wherein the third monitoring type is any one of the first monitoring type and the second monitoring type, and the fourth monitoring type is one of the first monitoring type and the second monitoring type except the third monitoring type.

[0059] Correspondingly, after receiving the fourth information, the second device may send a confirmation message to the first device to confirm whether it agrees that the first device adopts the new monitoring type to monitor the first AI model.

[0060] That is, when the monitoring type needs to be switched, the first device cannot switch the monitoring type autonomously, but needs to apply to the second device for a new monitoring type to switch from the current third monitoring type (such as the first monitoring type or the second monitoring type) to the fourth monitoring type. Wherein, when the third monitoring type is the first monitoring type, the fourth monitoring type can be the second monitoring type, and vice versa.

[0061] Mode 2: The second device sends the fifth information to the first device, and correspondingly, the first device receives the fifth information sent by the second device, and the fifth information is used to instruct the first device to switch from the third monitoring type to the fourth monitoring type. The third monitoring type is any one of the first monitoring type and the second monitoring type, and the fourth monitoring type is one of the first monitoring type and the second monitoring type except the third monitoring type.

[0062] That is, when the monitoring type needs to be switched, the second device can indicate a new monitoring type to the first device, so that the first device switches from the current third monitoring type (such as the first monitoring type or the second monitoring type) to the fourth monitoring type. Wherein, when the third monitoring type is the first monitoring type, the fourth monitoring type can be the second monitoring type, and vice versa.

[0063] It is worth noting that the fifth information can be determined autonomously by the second device word, or can be determined based on the received fourth information, which is not limited here.

[0064] Mode 3: Switching from the third monitoring type to the fourth monitoring type according to a preconfigured monitoring type switching condition, wherein the third monitoring type is any one of the first monitoring type and the second monitoring type, and the fourth monitoring type is one of the first monitoring type and the second monitoring type except the third monitoring type.

[0065] That is, the first device may switch from the current third monitoring type to the fourth monitoring type when determining that the monitoring type switching condition is met. Wherein, when the third monitoring type is the first monitoring type, the fourth monitoring type may be the second monitoring type, and vice versa.

[0066] The monitoring type switching condition may be, but is not limited to, related to the current monitoring result of the first AI model. For example, if the first AI model is currently monitored based on the first monitoring type, and the monitoring result indicates that the performance of the first AI model has deteriorated, then the monitoring type may be switched to the second monitoring type to monitor the first AI model; conversely, if the first AI model is currently monitored based on the second monitoring type, and the monitoring result indicates that the performance of the first AI model is better, then the monitoring type may be switched to the first monitoring type to monitor the first AI model.

[0067] Optionally, the monitoring type switching condition may be configured by a second device (such as subsequent first information), a protocol agreement, etc., and is not limited here.

[0068] It is worth noting that which of the aforementioned methods 1 to 3 is used to switch the monitoring type can be determined by protocol agreement, high-level configuration, etc., and there is no restriction on this.

[0069] In some embodiments, there may be multiple forms of configuration methods for the aforementioned first monitoring type and second monitoring type. Taking the first monitoring type as an example, the first monitoring type may be, but is not limited to, configured by the second device. For example, the second device may send first information to the first device, and the first information is used to configure the relevant information of the first monitoring type. Correspondingly, the first device receives the first information sent by the second device, and determines the relevant information of the first monitoring type according to the first information.

[0070] Optionally, in this embodiment, the first information may include but is not limited to at least one of the following 11)-17).

[0071] 11) A first monitoring indicator, which is used to indicate a monitoring indicator that the first device needs to send and can reflect the reasoning performance of the first AI model. In this embodiment, the first monitoring indicator can also be understood as a unified monitoring indicator, corresponding to the first monitoring type.

[0072] In this embodiment, the first monitoring indicator may include at least one of a first value, a second value, a third value, a fourth value, and a fifth value.

[0073] The first value includes a graded score, which is a scoring result of the performance of the first AI model. For example, the performance monitoring indicator of the first AI model can be scored according to the output of the first AI model, such as a score of 0 to 5, where a lower score indicates a worse performance of the first AI model, and vice versa.

[0074] The second value is used to indicate the number of abnormal samples that appear in the monitored samples. For example, it is possible to determine whether one or more monitored samples are abnormal based on the abnormal condition information, and then count the number of abnormal samples. In this embodiment, the number of abnormal samples can be a numerical value or an interval value, and the larger the value, the worse the current performance of the first AI model. The interval value refers to dividing into multiple intervals first, each interval corresponds to a result value, and the specific monitoring result value is determined by looking at which interval the number of abnormalities or the proportion falls into.

[0075] The third value is used to indicate the proportion or percentage of abnormal samples in the monitored samples. For example, it is possible to determine whether one or more monitored samples are abnormal based on the abnormal condition information, and then count the proportion or percentage of abnormal samples. In this embodiment, the proportion or percentage of abnormal samples can be a numerical value or an interval numerical value, and the larger the value, the worse the current performance of the first AI model. The interval numerical value refers to dividing into multiple intervals first, each interval corresponds to a result value, and the specific monitoring result value is determined by looking at which interval the number of abnormal times or the proportion falls into.

[0076] The fourth numerical value is used to indicate the number of normal samples that appear in the monitored samples. For example, it is possible to determine whether one or more monitored samples are normal based on the normal condition information, and then count the number of normal samples. In this embodiment, the number of normal samples can be a numerical value or an interval numerical value, and the larger the value, the better the current performance of the first AI model. The interval numerical value refers to dividing into multiple intervals first, each interval corresponds to a result value, and the specific monitoring result value is determined by looking at which interval the normal number or proportion falls into.

[0077] The fifth value is used to indicate the proportion or percentage of normal samples in the monitored samples. For example, it is possible to determine whether one or more monitored samples are normal based on the normal condition information, and then count the proportion or percentage of normal samples. In this embodiment, the proportion or percentage of normal samples can be a numerical value or an interval numerical value, and the larger the value, the better the current performance of the first AI model. The interval numerical value refers to dividing into multiple intervals first, each interval corresponds to a result value, and the specific monitoring result value is determined by looking at which interval the normal number or percentage falls into.

[0078] It can be understood that the aforementioned monitoring samples are used to monitor the first AI model. In addition, the aforementioned abnormal condition information and normal condition information can be determined by protocol agreement, high-level configuration or network side configuration. Taking the abnormal condition information as an example, the abnormal condition information can be whether the correlation coefficient between the output of the first AI model and its true value is lower than a preset threshold, such as the number of abnormal samples is increased by 1 when it is lower than the predetermined threshold. The correlation coefficient between the output of the first AI model and its true value can be obtained based on statistics of one or more monitoring samples.

[0079] 12) Monitoring indicator conversion information, used to convert the second monitoring indicator dedicated to the first AI model monitoring into the first monitoring indicator.

[0080] Optionally, the second monitoring indicator may include but is not limited to at least one of the following 121)-123).

[0081] 121) Error information between the output of the first AI model and its true value, such as error, normalized error, absolute value of error, cross-entropy loss (CEL), root mean square error (RSME), and average results of the foregoing information. Of course, the smaller the value of the error information between the output of the first AI model and its true value, the better the performance of the first AI model.

[0082] 122) Similarity information between the output of the first AI model and its true value, such as precision, accuracy, Top-K accuracy, similarity, cosine similarity, correlation, correlation coefficient, area under the curve score (Area Under the Curvescore, AUC score) and the average result of the above information. Of course, the larger the value of the similarity information between the output of the first AI model and its true value, the better the performance of the first AI model.

[0083] 123) Communication system performance determined according to the first AI model, such as throughput, spectrum efficiency, signal to interference plus noise ratio (SINR), signal-to-noise ratio (SNR), bit error rate, block error rate, packet loss rate, transmission rate (uplink / downlink), peak rate (uplink / downlink), etc. The aforementioned communication system performance can be calculated based on a single monitoring sample or based on statistics of multiple monitoring samples, which is not limited here.

[0084] In this embodiment, the monitoring indicator conversion information may include, but is not limited to, at least one of a monitoring indicator conversion method and auxiliary information required to implement the monitoring indicator conversion. The auxiliary information may be understood as a threshold, a weighting parameter, a bias parameter, a polynomial parameter, various condition information, etc., which are necessary to implement the monitoring indicator conversion method.

[0085] In some embodiments, the monitoring indicator conversion method may include but is not limited to at least one of the following methods 1 to 3.

[0086] Mode 1: When the first monitoring indicator includes the first numerical value, the second monitoring indicator is mapped to a grading score according to a preset grading score to obtain the first numerical value.

[0087] For example, assuming the preset grading scores are 0, 1, and 2, if the mean square error between the output of the first AI model and its true value is less than or equal to a first threshold, then the first value is 0; if the mean square error between the output of the first AI model and its true value is greater than the first threshold and less than or equal to the second threshold, then the first value is 1; if the mean square error between the output of the first AI model and its true value is greater than the second threshold, then the first value is 2.

[0088] It is worth noting that the grading scores may be different for different monitoring scenarios. That is to say, although the same piecewise function is used to characterize the grading scores, different AI models can be assigned different auxiliary information to achieve similar performance of different AI models using the same grading scores. For example, the first AI model has high requirements for AI reasoning accuracy, that is, a lower mean square error is required to ensure performance, but the second AI model has relatively broad requirements for AI reasoning accuracy, that is, the mean square error does not necessarily require a particularly low value. Then, the first threshold corresponding to the first AI model can be 0.01 and the second threshold can be 0.1, and the first threshold corresponding to the second AI model can be 0.1 and the second threshold can be 0.3.

[0089] Method 2: When the first monitoring indicator includes the second value or the third value, the second monitoring indicator is compared with the abnormal condition information, and the second value or the third value is determined according to the comparison result.

[0090] Method 3: When the first monitoring indicator includes the fourth value or the fifth value, the second monitoring indicator is compared with the normal condition information, and the fourth value or the fifth value is determined according to the comparison result.

[0091] It is worth noting that the aforementioned method 1, method 2, and method 3 can be described by a formula. In this embodiment, the formula can be predefined by the protocol or configured through signaling. For example, a predefined identifier can be carried in the first information, such as "00" for method 1, "01" for method 2, "10" for method 3, etc., or the formula can be directly described in the signaling.

[0092] 13) Monitoring start conditions of the first AI model.

[0093] 14) Monitoring stop conditions of the first AI model.

[0094] Among them, the monitoring start condition and the monitoring stop condition may be, but are not limited to, periodic conditions or event conditions, etc. For example, the first AI model may be monitored periodically, and for another example, the monitoring of the first AI model may be performed or stopped when a predetermined event is satisfied, etc. Among them, for the monitoring start condition, the predetermined event may be, but is not limited to: the first device accesses a new cell, the change in the moving speed of the first device exceeds a preset threshold, the environment in which the first device is located changes, the first device updates the AI ​​model, the size between the communication performance indicator and the preset threshold does not meet the preset conditions, the size relationship between the bit error rate (or block error rate) and the preset threshold does not meet the preset conditions, etc.

[0095] 15) Monitoring window of the first AI model.

[0096] That is, the first device can only monitor the first AI model within the monitoring window.

[0097] 16) Monitoring resources of the first AI model.

[0098] The monitoring resource may be a time domain resource, a frequency resource, or a spatial domain resource on which the first device monitors the first AI model. The time domain resource may be, but is not limited to, a symbol, a time slot, etc., and the frequency domain resource may be, but is not limited to, a carrier, a resource block, etc. The spatial domain resource may be, but is not limited to, a port, etc.

[0099] 17) A monitoring type switching condition, used for the first device to switch between the first monitoring type and the second monitoring type. For example, when the monitoring type switching condition is met, the first device can switch from the current first monitoring type to the second monitoring type, or from the current second monitoring type to the first monitoring type.

[0100] In some embodiments, in addition to configuring the first device with the relevant information of the first monitoring type through the first information, the first device can also receive the seventh information sent by the second device, and send the monitoring result according to the seventh information; that is, the seventh information can be used to instruct the first device to send the monitoring result. Thus, the first device can accurately report the monitoring result. The monitoring result is determined by monitoring the first AI model based on the aforementioned monitoring indicators.

[0101] Optionally, the seventh information may include, but is not limited to, at least one of the type of the monitoring result, the sending condition of the monitoring result, and the transmission resource of the monitoring result.

[0102] The type of the monitoring result includes a monitoring result corresponding to the first monitoring type or a monitoring result corresponding to the second monitoring type.

[0103] The sending condition of the monitoring result may be a sending cycle, a trigger condition, a trigger instruction, etc. of the monitoring result.

[0104] The transmission resources of the monitoring results may be, but are not limited to, the resources on which the monitoring results are transmitted (such as, which time domain resources, which frequency domain resources, which spatial domain resources, etc.).

[0105] In some embodiments, when the first device monitors the first AI model, it may also send second information or third information to the second device to indicate to the second device the model monitoring capabilities it supports, so that the second device is aware of the model monitoring capabilities of the first device and achieves the accuracy of configurations such as the first information.

[0106] For example, the second information may be used to indicate the AI ​​model monitoring capability supported by the first device. In this embodiment, the AI ​​model monitoring capability includes at least one of the first monitoring capability and the second monitoring capability.

[0107] The first monitoring capability is to support monitoring the multiple AI models using the same monitoring indicators, that is, the first monitoring capability is that all AI models share the same monitoring indicators, which corresponds to the first monitoring type.

[0108] The second monitoring capability is to support monitoring different AI models using different monitoring indicators. In other words, the second monitoring capability is to monitor different AI models based on dedicated monitoring indicators, which corresponds to the second monitoring type.

[0109] In some embodiments, in order to save overhead, the second information may be a predefined or protocol-agreed identifier, etc. For example, "00" indicates that the first device supports the second monitoring capability, "01" indicates that the first device supports the first monitoring capability, "10" indicates that the first device supports both the first monitoring capability and the second monitoring capability, and "11" indicates that the first device does not support the first monitoring capability and the second monitoring capability.

[0110] Of course, in addition to sending the capability of the first device to the second device through the aforementioned second information, in another embodiment, it is also possible to not consider whether the first device supports the second monitoring capability, but only consider whether it supports the first monitoring capability. That is, the third information can be used to indicate whether the first device supports the first monitoring capability. For example, when the third information is "0", it indicates that the first device does not support the first monitoring capability, and when the third information is "1", it indicates that the first device supports the first monitoring capability.

[0111] In this embodiment, not only a flexible and effective model monitoring solution is provided, but also for the first monitoring type, model monitoring based on the monitoring indicators corresponding to the first monitoring type is implemented through the collaboration of the transceiver ends, which can greatly reduce the types and overheads of signaling interactions, and at the same time reduce the complexity of AI model monitoring.

[0112] Based on the model monitoring solutions provided in the aforementioned method embodiments 200-300, the aforementioned model monitoring method is exemplarily introduced below in combination with Examples 1-4, as follows: It is assumed that the first device is a terminal and the second device is a network side device.

[0113] Example 1

[0114] S411, such as Figure 4a As shown, the terminal sends the second information or the third information to the network side device. The second information is used to indicate the AI ​​model monitoring capability supported by the terminal, and the third information is used to indicate whether the terminal supports the first monitoring capability.

[0115] S412, the network side device sends the first information and the seventh information to the terminal, wherein the first information is determined according to the second information and the third information and is used to configure the relevant information of the first monitoring type, and the seventh information is used to indicate the relevant information when the terminal sends the monitoring result.

[0116] S413: The terminal monitors the first AI model according to the first information.

[0117] S414: The terminal reports the monitoring result to the network side device according to the seventh information.

[0118] Example 2

[0119] S421, such as Figure 4b As shown, the terminal sends the second information or the third information to the network side device. The second information is used to indicate the AI ​​model monitoring capability supported by the terminal, and the third information is used to indicate whether the terminal supports the first monitoring capability.

[0120] S422, the network side device sends first information to the terminal, wherein the first information is determined according to the second information and third information and is used to configure relevant information of the first monitoring type.

[0121] S423: The terminal monitors the first AI model according to the third monitoring type. The third monitoring type is the first monitoring type or the second monitoring type.

[0122] S424: The terminal reports the monitoring result to the network side device.

[0123] S425, the network side device sends fifth information to the terminal according to the monitoring result, and the fifth information is used to instruct the terminal to switch from the third monitoring type to the fourth monitoring type.

[0124] S426: The terminal switches the monitoring type according to the new fourth monitoring type indicated by the fifth information.

[0125] Example 3

[0126] S431, such as Figure 4c As shown, the terminal sends the second information or the third information to the network side device. The second information is used to indicate the AI ​​model monitoring capability supported by the terminal, and the third information is used to indicate whether the terminal supports the first monitoring capability.

[0127] S432, the network side device sends first information to the terminal, wherein the first information is determined according to the second information and third information and is used to configure relevant information of the first monitoring type.

[0128] S433: The terminal monitors the first AI model according to the third monitoring type; the third monitoring type is the first monitoring type or the second monitoring type.

[0129] S434, the terminal sends fourth information to the network side device to apply for a new monitoring type.

[0130] S435, the network side device sends fifth information to the terminal, where the fifth information corresponds to the fourth information and is used to instruct the first device to switch from the third monitoring type to the fourth monitoring type.

[0131] S436: The terminal switches the monitoring type according to the applied new fourth monitoring type or the fifth information.

[0132] Example 4

[0133] S441, such as Figure 4d As shown, the terminal sends the second information or the third information to the network side device. The second information is used to indicate the AI ​​model monitoring capability supported by the terminal, and the third information is used to indicate whether the terminal supports the first monitoring capability.

[0134] S442, the network side device sends first information to the terminal, wherein the first information is determined according to the second information and third information and is used to configure relevant information of the first monitoring type.

[0135] S443, the terminal monitors the first AI model according to the third monitoring type; the third monitoring type is the first monitoring type or the second monitoring type.

[0136] S444: The terminal switches the monitoring type according to a preconfigured monitoring type switching condition.

[0137] It is worth noting that several different monitoring type switching schemes are provided in the aforementioned Examples 2 to 4, respectively. Among them, the terminal in Example 2 can switch the monitoring type according to the monitoring type indicated by the network side device, that is, the terminal dynamically selects which monitoring mode to use based on the network side decision indication; Example 3 is a process for dynamically selecting which monitoring mode to use based on the terminal application; Example 4 is based on the preconfigured monitoring type switching conditions to switch the monitoring type.

[0138] In addition, the aforementioned examples 1 to 4 may include but are not limited to the aforementioned steps, such as including more or fewer steps than the aforementioned steps. The implementation process of the aforementioned examples 1 to 4 may refer to the relevant descriptions in the aforementioned method embodiments 200 to 300, and achieve the same or corresponding technical effects. To avoid repetition, they will not be described here.

[0139] like Figure 5 FIG. 5 is a flow chart of a model monitoring method 500 provided by an exemplary embodiment of the present application. The method 500 may be, but is not limited to, executed by a second device, and may be specifically executed by hardware or software installed in the second device. In this embodiment, the method 500 may at least include the following steps.

[0140] S510: The second device sends first information to the first device.

[0141] Among them, the first information is used to configure relevant information of the first monitoring type, and the first monitoring type is to monitor multiple AI models using the same monitoring indicators.

[0142] Optionally, the first information includes at least one of the following: a first monitoring indicator, used to indicate a monitoring indicator that the first device needs to send and can reflect the reasoning performance of the first AI model, and the first monitoring indicator corresponds to the first monitoring type; monitoring indicator conversion information, used to convert a second monitoring indicator dedicated to monitoring the first AI model into the first monitoring indicator; monitoring start conditions of the first AI model; monitoring stop conditions of the first AI model; monitoring window of the first AI model; monitoring resources of the first AI model; monitoring type switching conditions, used for the first device to switch between the first monitoring type and the second monitoring type.

[0143] Optionally, the first monitoring indicator includes at least one of the following: a first numerical value, the first numerical value includes a graded score, and the graded score is a scoring result of the performance of the first AI model; a second numerical value, used to indicate the number of abnormal samples appearing in the monitoring samples; a third numerical value, used to indicate the proportion of abnormal samples in the monitoring samples; a fourth numerical value, used to indicate the number of normal samples appearing in the monitoring samples; a fifth numerical value, used to indicate the proportion of normal samples in the monitoring samples; wherein the monitoring samples are used to monitor the first AI model.

[0144] Optionally, the monitoring indicator conversion information includes at least one of a monitoring indicator conversion method and auxiliary information required to implement the monitoring indicator conversion.

[0145] Optionally, the monitoring indicator conversion method includes at least one of the following: when the first monitoring indicator includes the first numerical value, mapping the second monitoring indicator to a grading score according to a preset grading score to obtain the first numerical value; when the first monitoring indicator includes the second numerical value or the third numerical value, comparing the second monitoring indicator with the abnormal condition information, and determining the second numerical value or the third numerical value based on the comparison result; when the first monitoring indicator includes the fourth numerical value or the fifth numerical value, comparing the second monitoring indicator with the normal condition information, and determining the fourth numerical value or the fifth numerical value based on the comparison result.

[0146] Optionally, the method also includes: receiving second information or third information sent by the first device; wherein the second information is used to indicate the AI ​​model monitoring capability supported by the first device, and the third information is used to indicate whether the first device has a first monitoring capability, and the first monitoring capability supports monitoring multiple AI models using the same monitoring indicators.

[0147] Optionally, the AI ​​model monitoring capability includes at least one of the following: a first monitoring capability, which supports monitoring the multiple AI models using the same monitoring indicators; and a second monitoring capability, which supports monitoring different AI models using different monitoring indicators.

[0148] Optionally, the method also includes: receiving fourth information sent by the first device, the fourth information being used to apply to the second device to switch from the third monitoring type to the fourth monitoring type; sending fifth information to the first device, the fifth information being used to instruct the first device to switch from the third monitoring type to the fourth monitoring type; wherein the third monitoring type is any one of the first monitoring type and the second monitoring type, the fourth monitoring type is one of the first monitoring type and the second monitoring type except the third monitoring type, and the second monitoring type is to monitor the first AI model using dedicated monitoring indicators.

[0149] Optionally, the method further includes: sending seventh information to the first device; wherein the seventh information is used by the first device to send monitoring results.

[0150] Optionally, the seventh information includes at least one of the following: the type of the monitoring result, the type of the monitoring result including the monitoring result corresponding to the first monitoring type or the monitoring result corresponding to the second monitoring type, the second monitoring type is to monitor the first AI model using dedicated monitoring indicators; the sending conditions of the monitoring result; the transmission resources of the monitoring result.

[0151] It can be understood that each implementation method in method embodiment 500 has the same or corresponding technical features as the aforementioned method embodiments 200-300. Therefore, the various implementation methods in method embodiment 500 can refer to the relevant descriptions in the aforementioned method embodiments 200-300, and achieve the same or corresponding technical effects. To avoid repetition, they will not be repeated here.

[0152] The model monitoring methods 200-500 provided in the embodiments of the present application may be executed by a model monitoring device. In the embodiments of the present application, the model monitoring device executing the model monitoring methods 200-500 is taken as an example to illustrate the model monitoring device provided in the embodiments of the present application.

[0153] like Figure 6 As shown, it is a structural schematic diagram of a model monitoring device 600 provided in an embodiment of the present application, and the device 600 includes: a monitoring module 610, used to monitor the first AI model according to a predetermined monitoring type; wherein the predetermined monitoring type includes at least one of the following: a first monitoring type, wherein the first monitoring type is to monitor multiple AI models using the same monitoring indicator, wherein the first AI model belongs to the multiple AI models; a second monitoring type, wherein the second monitoring type is to monitor the first AI model using a dedicated monitoring indicator.

[0154] Optionally, the apparatus 600 further includes: a transmission module, configured to receive first information sent by a second device; wherein the first information is used to configure relevant information of the first monitoring type.

[0155] Optionally, the first information includes at least one of the following: a first monitoring indicator, used to indicate a monitoring indicator that the first device needs to send and can reflect the reasoning performance of the first AI model, and the first monitoring indicator corresponds to the first monitoring type; monitoring indicator conversion information, used to convert a second monitoring indicator dedicated to monitoring the first AI model into the first monitoring indicator; monitoring start conditions of the first AI model; monitoring stop conditions of the first AI model; monitoring window of the first AI model; monitoring resources of the first AI model, and monitoring type switching conditions, used for the first device to switch between the first monitoring type and the second monitoring type.

[0156] Optionally, the first monitoring indicator includes at least one of the following: a first numerical value, the first numerical value includes a graded score, and the graded score is a scoring result of the performance of the first AI model; a second numerical value, used to indicate the number of abnormal samples appearing in the monitoring samples; a third numerical value, used to indicate the proportion of abnormal samples in the monitoring samples; a fourth numerical value, used to indicate the number of normal samples appearing in the monitoring samples; a fifth numerical value, used to indicate the proportion of normal samples in the monitoring samples; wherein the monitoring samples are used to monitor the first AI model.

[0157] Optionally, the monitoring indicator conversion information includes at least one of a monitoring indicator conversion method and auxiliary information required to implement the monitoring indicator conversion.

[0158] Optionally, the monitoring indicator conversion method includes at least one of the following: when the first monitoring indicator includes the first numerical value, mapping the second monitoring indicator to a grading score according to a preset grading score to obtain the first numerical value; when the first monitoring indicator includes the second numerical value or the third numerical value, comparing the second monitoring indicator with the abnormal condition information, and determining the second numerical value or the third numerical value based on the comparison result; when the first monitoring indicator includes the fourth numerical value or the fifth numerical value, comparing the second monitoring indicator with the normal condition information, and determining the fourth numerical value or the fifth numerical value based on the comparison result.

[0159] Optionally, the transmission module is also used to send second information or third information to the second device; wherein, the second information is used to indicate the AI ​​model monitoring capability supported by the first device, and the third information is used to indicate whether the first device has a first monitoring capability, and the first monitoring capability supports monitoring multiple AI models using the same monitoring indicators.

[0160] Optionally, the AI ​​model monitoring capability includes at least one of the following: a first monitoring capability, which supports monitoring the multiple AI models using the same monitoring indicators; and a second monitoring capability, which supports monitoring different AI models using different monitoring indicators.

[0161] Optionally, the monitoring module 610 is also used to: switch the monitoring type according to at least one of the following: send fourth information to the second device, the fourth information being used to apply to the second device for switching from the third monitoring type to the fourth monitoring type; receive fifth information sent by the second device, the fifth information being used to instruct the first device to switch from the third monitoring type to the fourth monitoring type; switch from the third monitoring type to the fourth monitoring type according to preconfigured monitoring type switching conditions; wherein the third monitoring type is any one of the first monitoring type and the second monitoring type, and the fourth monitoring type is one of the first monitoring type and the second monitoring type except the third monitoring type.

[0162] Optionally, the transmission module is further used to: receive seventh information sent by the second device; and send monitoring results according to the seventh information.

[0163] Optionally, the seventh information includes at least one of the following: the type of the monitoring result, the type of the monitoring result including the monitoring result corresponding to the first monitoring type or the monitoring result corresponding to the second monitoring type, the second monitoring type is to monitor the first AI model using dedicated monitoring indicators; the sending conditions of the monitoring result; the transmission resources of the monitoring result.

[0164] The model monitoring device 600 in the embodiment of the present application can be a communication device, such as a communication device with an operating system, or a component in a communication device, such as an integrated circuit or a chip. The communication device can be a terminal or a network side device. Exemplarily, the terminal can include but is not limited to the types of the terminal 11 listed above, and the network side device can include but is not limited to the types of the network side device 12 listed above, which are not specifically limited in the embodiment of the present application.

[0165] The model monitoring device 600 provided in the embodiment of the present application can achieve Figures 2 to 3 The various processes implemented by the method embodiment and achieving the same technical effect are not described here to avoid repetition.

[0166] like Figure 7As shown, it is a structural diagram of a model monitoring device 700 provided in an embodiment of the present application, and the device 700 includes: a transmission module 710, used to send first information to a first device; wherein, the first information is used to configure relevant information of a first monitoring type, and the first monitoring type is to monitor multiple AI models using the same monitoring indicators.

[0167] Optionally, the first information includes at least one of the following: a first monitoring indicator, used to indicate a monitoring indicator that the first device needs to send and can reflect the reasoning performance of the first AI model, and the first monitoring indicator corresponds to the first monitoring type; monitoring indicator conversion information, used to convert a second monitoring indicator dedicated to monitoring the first AI model into the first monitoring indicator; monitoring start conditions of the first AI model; monitoring stop conditions of the first AI model; monitoring window of the first AI model; monitoring resources of the first AI model; monitoring type switching conditions, used for the first device to switch between the first monitoring type and the second monitoring type.

[0168] Optionally, the first monitoring indicator includes at least one of the following: a first numerical value, the first numerical value includes a graded score, and the graded score is a scoring result of the performance of the first AI model; a second numerical value, used to indicate the number of abnormal samples appearing in the monitoring samples; a third numerical value, used to indicate the proportion of abnormal samples in the monitoring samples; a fourth numerical value, used to indicate the number of normal samples appearing in the monitoring samples; a fifth numerical value, used to indicate the proportion of normal samples in the monitoring samples; wherein the monitoring samples are used to monitor the first AI model.

[0169] Optionally, the monitoring indicator conversion information includes at least one of a monitoring indicator conversion method and auxiliary information required to implement the monitoring indicator conversion.

[0170] Optionally, the monitoring indicator conversion method includes at least one of the following: when the first monitoring indicator includes the first numerical value, mapping the second monitoring indicator to a grading score according to a preset grading score to obtain the first numerical value; when the first monitoring indicator includes the second numerical value or the third numerical value, comparing the second monitoring indicator with the abnormal condition information, and determining the second numerical value or the third numerical value based on the comparison result; when the first monitoring indicator includes the fourth numerical value or the fifth numerical value, comparing the second monitoring indicator with the normal condition information, and determining the fourth numerical value or the fifth numerical value based on the comparison result.

[0171] Optionally, the transmission module 710 is also used to receive second information or third information sent by the first device; wherein the second information is used to indicate the AI ​​model monitoring capability supported by the first device, and the third information is used to indicate whether the first device has a first monitoring capability, and the first monitoring capability supports monitoring multiple AI models using the same monitoring indicators.

[0172] Optionally, the AI ​​model monitoring capability includes at least one of the following: a first monitoring capability, which supports monitoring the multiple AI models using the same monitoring indicators; and a second monitoring capability, which supports monitoring different AI models using different monitoring indicators.

[0173] Optionally, the transmission module 710 is also used to receive fourth information sent by the first device, and the fourth information is used to apply to the second device to switch from the third monitoring type to the fourth monitoring type; send fifth information to the first device, and the fifth information is used to instruct the first device to switch from the third monitoring type to the fourth monitoring type; wherein the third monitoring type is any one of the first monitoring type and the second monitoring type, the fourth monitoring type is one of the first monitoring type and the second monitoring type except the third monitoring type, and the second monitoring type is to monitor the first AI model using dedicated monitoring indicators.

[0174] Optionally, the transmission module 710 is further used to send seventh information to the first device; wherein the seventh information is used for the first device to send monitoring results.

[0175] Optionally, the seventh information includes at least one of the following: the type of the monitoring result, the type of the monitoring result including the monitoring result corresponding to the first monitoring type or the monitoring result corresponding to the second monitoring type, the second monitoring type is to monitor the first AI model using dedicated monitoring indicators; the sending conditions of the monitoring result; the transmission resources of the monitoring result.

[0176] The model monitoring device 700 in the embodiment of the present application can be a communication device, such as a communication device with an operating system, or a component in a communication device, such as an integrated circuit or a chip. The communication device can be a terminal or a network side device. Exemplarily, the terminal can include but is not limited to the types of the terminal 11 listed above, and the network side device can include but is not limited to the types of the network side device 12 listed above, which are not specifically limited in the embodiment of the present application.

[0177] The model monitoring device 700 provided in the embodiment of the present application can achieve Figure 5The various processes implemented by the method embodiment and achieving the same technical effect are not described here to avoid repetition.

[0178] like Figure 8 As shown, the embodiment of the present application further provides a communication device 800, including a processor 801 and a memory 802, and the memory 802 stores a program or instruction that can be run on the processor 801. For example, when the communication device 800 is a terminal, the program or instruction is executed by the processor 801 to implement the various steps of the above-mentioned model monitoring method embodiment, and can achieve the same technical effect. When the communication device 800 is a network side device, the program or instruction is executed by the processor 801 to implement the various steps of the above-mentioned model monitoring method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0179] The embodiment of the present application also provides a terminal, including a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the following Figure 2-Figure 5 The steps in the method embodiment shown. This terminal embodiment corresponds to the above-mentioned terminal side method embodiment, and each implementation process and implementation method of the above-mentioned method embodiment can be applied to this terminal embodiment and can achieve the same technical effect. Specifically, Fig. 9 A schematic diagram of the hardware structure of a terminal for implementing an embodiment of the present application.

[0180] The terminal 900 includes but is not limited to: a radio frequency unit 901, a network module 902, an audio output unit 903, an input unit 904, a sensor 905, a display unit 906, a user input unit 907, an interface unit 908, a memory 909 and at least some of the components of a processor 910.

[0181] Those skilled in the art will appreciate that the terminal 900 may also include a power source (such as a battery) for supplying power to various components, and the power source may be logically connected to the processor 910 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig. 9 The terminal structure shown in the figure does not constitute a limitation on the terminal. The terminal may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0182] It should be understood that in the embodiments of the present application, the input unit 904 may include a GPU 9041 and a microphone 9042. The graphics processor 9041 processes the image data of static pictures or videos obtained by an image capturing device (such as a camera) in a video capturing mode or an image capturing mode. The display unit 906 may include a display panel 9061, and the display panel 9061 may be configured in the form of, for example, a liquid crystal display, an organic light emitting diode, etc. The user input unit 907 includes at least one of a touch panel 9071 and other input devices 9072. The touch panel 9071 is also referred to as a touch screen. The touch panel 9071 may include two parts: a touch detection device and a touch controller. The other input devices 9072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, which will not be elaborated here.

[0183] In the embodiments of the present application, after receiving downlink data from a network side device, the radio frequency unit 901 may transmit it to the processor 910 for processing; in addition, the radio frequency unit 901 may send uplink data to the network side device. Generally, the radio frequency unit 901 includes, but is not limited to, an antenna, an amplifier, a transceiver, a coupler, a low noise amplifier, a duplexer, etc.

[0184] The memory 909 can be used to store software programs or instructions and various data. The memory 909 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instruction required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 909 may include a volatile memory or a non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 909 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0185] The processor 910 may include one or more processing units; optionally, the processor 910 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 910.

[0186] In one implementation, the processor 910 is used to monitor the first AI model according to a predetermined monitoring type; wherein the predetermined monitoring type includes at least one of the following: a first monitoring type, wherein the first monitoring type is to monitor multiple AI models using the same monitoring indicator, wherein the first AI model belongs to the multiple AI models; a second monitoring type, wherein the second monitoring type is to monitor the first AI model using a dedicated monitoring indicator.

[0187] Optionally, the apparatus 600 further includes: a radio frequency unit 901, configured to receive first information sent by a second device; wherein the first information is used to configure relevant information of the first monitoring type.

[0188] Optionally, the first information includes at least one of the following: a first monitoring indicator, used to indicate a monitoring indicator that the first device needs to send and can reflect the reasoning performance of the first AI model, and the first monitoring indicator corresponds to the first monitoring type; monitoring indicator conversion information, used to convert a second monitoring indicator dedicated to monitoring the first AI model into the first monitoring indicator; monitoring start conditions of the first AI model; monitoring stop conditions of the first AI model; monitoring window of the first AI model; monitoring resources of the first AI model; monitoring type switching conditions, used for the first device to switch between the first monitoring type and the second monitoring type.

[0189] Optionally, the first monitoring indicator includes at least one of the following: a first numerical value, the first numerical value includes a graded score, and the graded score is a scoring result of the performance of the first AI model; a second numerical value, used to indicate the number of abnormal samples appearing in the monitoring samples; a third numerical value, used to indicate the proportion of abnormal samples in the monitoring samples; a fourth numerical value, used to indicate the number of normal samples appearing in the monitoring samples; a fifth numerical value, used to indicate the proportion of normal samples in the monitoring samples; wherein the monitoring samples are used to monitor the first AI model.

[0190] Optionally, the monitoring indicator conversion information includes at least one of a monitoring indicator conversion method and auxiliary information required to implement the monitoring indicator conversion.

[0191] Optionally, the monitoring indicator conversion method includes at least one of the following: when the first monitoring indicator includes the first numerical value, mapping the second monitoring indicator to a grading score according to a preset grading score to obtain the first numerical value; when the first monitoring indicator includes the second numerical value or the third numerical value, comparing the second monitoring indicator with the abnormal condition information, and determining the second numerical value or the third numerical value based on the comparison result; when the first monitoring indicator includes the fourth numerical value or the fifth numerical value, comparing the second monitoring indicator with the normal condition information, and determining the fourth numerical value or the fifth numerical value based on the comparison result.

[0192] Optionally, the radio frequency unit 901 is also used to send second information or third information to the second device; wherein the second information is used to indicate the AI ​​model monitoring capability supported by the first device, and the third information is used to indicate whether the first device has a first monitoring capability, and the first monitoring capability supports monitoring multiple AI models using the same monitoring indicators.

[0193] Optionally, the AI ​​model monitoring capability includes at least one of the following: a first monitoring capability, which supports monitoring the multiple AI models using the same monitoring indicators; and a second monitoring capability, which supports monitoring different AI models using different monitoring indicators.

[0194] Optionally, the processor 910 is also used to: switch the monitoring type according to at least one of the following: send fourth information to the second device, the fourth information being used to apply to the second device for switching from the third monitoring type to the fourth monitoring type; receive fifth information sent by the second device, the fifth information being used to instruct the first device to switch from the third monitoring type to the fourth monitoring type; switch from the third monitoring type to the fourth monitoring type according to a preconfigured monitoring type switching condition; wherein the third monitoring type is any one of the first monitoring type and the second monitoring type, and the fourth monitoring type is one of the first monitoring type and the second monitoring type except the third monitoring type.

[0195] Optionally, the radio frequency unit 901 is further used to: receive seventh information sent by the second device; and send monitoring results according to the seventh information.

[0196] Optionally, the seventh information includes at least one of the following: the type of the monitoring result, the type of the monitoring result including the monitoring result corresponding to the first monitoring type or the monitoring result corresponding to the second monitoring type, the second monitoring type is to monitor the first AI model using dedicated monitoring indicators; the sending conditions of the monitoring result; the transmission resources of the monitoring result.

[0197] In another implementation, the radio frequency unit 901 is used to send first information to a first device; wherein the first information is used to configure relevant information of a first monitoring type, and the first monitoring type is to monitor multiple AI models using the same monitoring indicators.

[0198] Optionally, the first information includes at least one of the following: a first monitoring indicator, used to indicate a monitoring indicator that the first device needs to send and can reflect the reasoning performance of the first AI model, and the first monitoring indicator corresponds to the first monitoring type; monitoring indicator conversion information, used to convert a second monitoring indicator dedicated to monitoring the first AI model into the first monitoring indicator; monitoring start conditions of the first AI model; monitoring stop conditions of the first AI model; monitoring window of the first AI model; monitoring resources of the first AI model; monitoring type switching conditions, used for the first device to switch between the first monitoring type and the second monitoring type.

[0199] Optionally, the first monitoring indicator includes at least one of the following: a first numerical value, the first numerical value includes a graded score, and the graded score is a scoring result of the performance of the first AI model; a second numerical value, used to indicate the number of abnormal samples appearing in the monitoring samples; a third numerical value, used to indicate the proportion of abnormal samples in the monitoring samples; a fourth numerical value, used to indicate the number of normal samples appearing in the monitoring samples; a fifth numerical value, used to indicate the proportion of normal samples in the monitoring samples; wherein the monitoring samples are used to monitor the first AI model.

[0200] Optionally, the monitoring indicator conversion information includes at least one of a monitoring indicator conversion method and auxiliary information required to implement the monitoring indicator conversion.

[0201] Optionally, the monitoring indicator conversion method includes at least one of the following: when the first monitoring indicator includes the first numerical value, mapping the second monitoring indicator to a grading score according to a preset grading score to obtain the first numerical value; when the first monitoring indicator includes the second numerical value or the third numerical value, comparing the second monitoring indicator with the abnormal condition information, and determining the second numerical value or the third numerical value based on the comparison result; when the first monitoring indicator includes the fourth numerical value or the fifth numerical value, comparing the second monitoring indicator with the normal condition information, and determining the fourth numerical value or the fifth numerical value based on the comparison result.

[0202] Optionally, the radio frequency unit 901 is also used to receive second information or third information sent by the first device; wherein the second information is used to indicate the AI ​​model monitoring capability supported by the first device, and the third information is used to indicate whether the first device has a first monitoring capability, and the first monitoring capability supports monitoring multiple AI models using the same monitoring indicators.

[0203] Optionally, the AI ​​model monitoring capability includes at least one of the following: a first monitoring capability, which supports monitoring the multiple AI models using the same monitoring indicators; and a second monitoring capability, which supports monitoring different AI models using different monitoring indicators.

[0204] Optionally, the radio frequency unit 901 is also used to receive fourth information sent by the first device, and the fourth information is used to apply to the second device to switch from the third monitoring type to the fourth monitoring type; send fifth information to the first device, and the fifth information is used to instruct the first device to switch from the third monitoring type to the fourth monitoring type; wherein the third monitoring type is any one of the first monitoring type and the second monitoring type, the fourth monitoring type is one of the first monitoring type and the second monitoring type except the third monitoring type, and the second monitoring type is to monitor the first AI model using dedicated monitoring indicators.

[0205] Optionally, the radio frequency unit 901 is further used to send seventh information to the first device; wherein the seventh information is used for the first device to send monitoring results.

[0206] Optionally, the seventh information includes at least one of the following: the type of the monitoring result, the type of the monitoring result including the monitoring result corresponding to the first monitoring type or the monitoring result corresponding to the second monitoring type, the second monitoring type is to monitor the first AI model using dedicated monitoring indicators; the sending conditions of the monitoring result; the transmission resources of the monitoring result.

[0207] It can be understood that the implementation process of each implementation method mentioned in this embodiment can refer to the relevant description of method embodiments 200-500, and achieve the same or corresponding technical effects. To avoid repetition, it will not be repeated here.

[0208] The embodiment of the present application also provides a network side device, including a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the following Figure 2-Figure 5 The steps of the method embodiment shown. This network side device embodiment corresponds to the above network side device method embodiment, and each implementation process and implementation mode of the above method embodiment can be applied to this network side device embodiment and can achieve the same technical effect.

[0209] Specifically, the embodiment of the present application also provides a network side device. Fig.10As shown, the network side device 1000 includes: an antenna 1001, a radio frequency device 1002, a baseband device 1003, a processor 1004 and a memory 1005. The antenna 1001 is connected to the radio frequency device 1002. In the uplink direction, the radio frequency device 1002 receives information through the antenna 1001 and sends the received information to the baseband device 1003 for processing. In the downlink direction, the baseband device 1003 processes the information to be sent and sends it to the radio frequency device 1002. The radio frequency device 1002 processes the received information and sends it out through the antenna 1001.

[0210] The method executed by the network-side device in the above embodiment may be implemented in the baseband device 1003 , which includes a baseband processor.

[0211] The baseband device 1003 may include, for example, at least one baseband board on which a plurality of chips are arranged. Fig.10 As shown, one of the chips is, for example, a baseband processor, which is connected to the memory 1005 through a bus interface to call the program in the memory 1005 to execute the network device operations shown in the above method embodiment.

[0212] The network side device may further include a network interface 1006, which is, for example, a Common Public Radio Interface (CPRI).

[0213] Specifically, the network side device 1000 of the embodiment of the present application further includes: instructions or programs stored in the memory 1005 and executable on the processor 1004, and the processor 1004 calls the instructions or programs in the memory 1005 to execute Figure 6 or Figure 7 The methods executed by the modules shown achieve the same technical effects, and therefore will not be described here in detail to avoid repetition.

[0214] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned model monitoring method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0215] The processor is the processor in the terminal described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk. In some examples, the readable storage medium may be a non-transient readable storage medium.

[0216] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned model monitoring method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0217] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0218] The embodiment of the present application further provides a computer program / program product, which is stored in a storage medium. The computer program / program product is executed by at least one processor to implement the various processes of the above-mentioned model monitoring method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0219] An embodiment of the present application also provides a wireless communication system, including: a first device and a second device, wherein the first device can be used to execute to implement the various processes of the above-mentioned model monitoring method embodiments 200-300, and the second device can be used to execute to implement the various processes of the above-mentioned model monitoring method embodiment 500, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0220] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises one..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiment of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0221] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of a computer software product plus a necessary general hardware platform, and of course, can also be implemented by hardware. The computer software product is stored in a storage medium (such as ROM, RAM, disk, CD, etc.), including several instructions to enable a terminal or a network-side device to execute the methods described in each embodiment of the present application.

[0222] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms of implementation methods without departing from the purpose of the present application and the scope of protection of the claims, and these implementation methods are all within the protection of the present application.

Claims

1. A model monitoring method, It is characterized in that include: The first device monitors the first artificial intelligence AI model according to a predetermined monitoring type; The predetermined monitoring type includes at least one of the following: A first monitoring type, wherein the first monitoring type is to monitor multiple AI models using the same monitoring indicator, wherein the first AI model belongs to the multiple AI models; The second monitoring type is to monitor the first AI model using dedicated monitoring indicators.

2. The method according to claim 1, It is characterized in that The method further comprises: receiving first information sent by a second device; The first information is used to configure relevant information of the first monitoring type.

3. The method according to claim 2, It is characterized in that The first information includes at least one of the following: A first monitoring indicator, used to indicate a monitoring indicator that the first device needs to send and can reflect the reasoning performance of the first AI model, wherein the first monitoring indicator corresponds to the first monitoring type; Monitoring indicator conversion information, used to convert a second monitoring indicator dedicated to monitoring the first AI model into the first monitoring indicator; Monitoring start conditions of the first AI model; Monitoring stop condition of the first AI model; a monitoring window of the first AI model; Monitoring resources of the first AI model; A monitoring type switching condition is used for the first device to switch between the first monitoring type and the second monitoring type.

4. The method according to claim 3, It is characterized in that The first monitoring indicator includes at least one of the following: A first value, the first value comprising a grading score, the grading score being a scoring result of the performance of the first AI model; The second value is used to indicate the number of times abnormal samples appear in the monitoring samples; The third value is used to indicate the proportion of abnormal samples in the monitoring samples; A fourth value is used to indicate the number of times a normal sample appears in the monitoring sample; The fifth value is used to indicate the proportion of normal samples in the monitored samples; Among them, the monitoring samples are used to monitor the first AI model.

5. The method according to claim 4, It is characterized in that The monitoring indicator conversion information includes at least one of a monitoring indicator conversion method and auxiliary information required to implement the monitoring indicator conversion.

6. The method according to claim 5, It is characterized in that The monitoring indicator conversion method includes at least one of the following: In the case where the first monitoring indicator includes the first value, mapping the second monitoring indicator to a grading score according to a preset grading score to obtain the first value; When the first monitoring indicator includes the second value or the third value, comparing the second monitoring indicator with the abnormal condition information, and determining the second value or the third value according to the comparison result; In the case where the first monitoring indicator includes the fourth value or the fifth value, the second monitoring indicator is compared with the normal condition information, and the fourth value or the fifth value is determined according to the comparison result.

7. The method according to any one of claims 1 to 6, It is characterized in that The method further comprises: sending the second information or the third information to the second device; Among them, the second information is used to indicate the AI ​​model monitoring capability supported by the first device, and the third information is used to indicate whether the first device has the first monitoring capability, and the first monitoring capability supports monitoring multiple AI models using the same monitoring indicators.

8. The method according to claim 7, It is characterized in that The AI ​​model monitoring capability includes at least one of the following: A first monitoring capability, wherein the first monitoring capability supports monitoring the multiple AI models using the same monitoring indicators; The second monitoring capability supports monitoring different AI models using different monitoring indicators.

9. The method according to any one of claims 1 to 8, It is characterized in that The method further comprises: Switch the monitoring type based on at least one of the following: Sending fourth information to the second device, where the fourth information is used to apply to the second device for switching from the third monitoring type to the fourth monitoring type; receiving fifth information sent by the second device, where the fifth information is used to instruct the first device to switch from the third monitoring type to the fourth monitoring type; According to a preconfigured monitoring type switching condition, switching from the third monitoring type to the fourth monitoring type; The third monitoring type is any one of the first monitoring type and the second monitoring type, and the fourth monitoring type is one of the first monitoring type and the second monitoring type except the third monitoring type.

10. The method according to any one of claims 1 to 9, It is characterized in that The method further comprises: receiving seventh information sent by the second device; The monitoring result is sent according to the seventh information.

11. The method according to claim 10, It is characterized in that The seventh information includes at least one of the following: The type of the monitoring result, the type of the monitoring result comprising a monitoring result corresponding to the first monitoring type or a monitoring result corresponding to the second monitoring type; The conditions for sending the monitoring results; The transmission resources of the monitoring results.

12. A model monitoring method, It is characterized in that include: The second device sends the first information to the first device; Among them, the first information is used to configure relevant information of the first monitoring type, and the first monitoring type is to monitor multiple AI models using the same monitoring indicators.

13. The method according to claim 12, It is characterized in that The first information includes at least one of the following: A first monitoring indicator, used to indicate a monitoring indicator that the first device needs to send and can reflect the reasoning performance of the first AI model, wherein the first monitoring indicator corresponds to the first monitoring type; Monitoring indicator conversion information, used to convert a second monitoring indicator dedicated to monitoring the first AI model into the first monitoring indicator; Monitoring start conditions of the first AI model; Monitoring stop condition of the first AI model; a monitoring window of the first AI model; Monitoring resources of the first AI model; A monitoring type switching condition is used for the first device to switch between the first monitoring type and the second monitoring type.

14. The method according to claim 13, It is characterized in that The first monitoring indicator includes at least one of the following: A first value, the first value comprising a grading score, the grading score being a scoring result of the performance of the first AI model; The second value is used to indicate the number of times abnormal samples appear in the monitoring samples; The third value is used to indicate the proportion of abnormal samples in the monitoring samples; A fourth value is used to indicate the number of times a normal sample appears in the monitoring sample; The fifth value is used to indicate the proportion of normal samples in the monitored samples; Among them, the monitoring samples are used to monitor the first AI model.

15. The method of claim 14, It is characterized in that The monitoring indicator conversion information includes at least one of a monitoring indicator conversion method and auxiliary information required to implement the monitoring indicator conversion.

16. The method of claim 15, It is characterized in that The monitoring indicator conversion method includes at least one of the following: In the case where the first monitoring indicator includes the first value, mapping the second monitoring indicator to a grading score according to a preset grading score to obtain the first value; When the first monitoring indicator includes the second value or the third value, comparing the second monitoring indicator with the abnormal condition information, and determining the second value or the third value according to the comparison result; In the case where the first monitoring indicator includes the fourth value or the fifth value, the second monitoring indicator is compared with the normal condition information, and the fourth value or the fifth value is determined according to the comparison result.

17. The method according to any one of claims 12 to 16, It is characterized in that The method further comprises: receiving second information or third information sent by the first device; Among them, the second information is used to indicate the AI ​​model monitoring capability supported by the first device, and the third information is used to indicate whether the first device has the first monitoring capability, and the first monitoring capability supports monitoring multiple AI models using the same monitoring indicators.

18. The method of claim 17, It is characterized in that The AI ​​model monitoring capability includes at least one of the following: A first monitoring capability, wherein the first monitoring capability supports monitoring the multiple AI models using the same monitoring indicators; The second monitoring capability supports the use of different monitoring indicators to monitor different AI models.

19. The method according to any one of claims 12 to 18, It is characterized in that The method further comprises: receiving fourth information sent by the first device, where the fourth information is used to apply to the second device for switching from the third monitoring type to the fourth monitoring type; Sending fifth information to the first device, where the fifth information is used to instruct the first device to switch from the third monitoring type to the fourth monitoring type; Among them, the third monitoring type is any one of the first monitoring type and the second monitoring type, the fourth monitoring type is one of the first monitoring type and the second monitoring type except the third monitoring type, and the second monitoring type is to monitor the first AI model using dedicated monitoring indicators.

20. The method according to any one of claims 12 to 19, It is characterized in that The method further comprises: Sending seventh information to the first device; The seventh information is used by the first device to send monitoring results.

21. The method of claim 20, It is characterized in that The seventh information includes at least one of the following: The type of the monitoring result, the type of the monitoring result including a monitoring result corresponding to the first monitoring type or a monitoring result corresponding to a second monitoring type, wherein the second monitoring type is to monitor the first AI model using a dedicated monitoring indicator; The conditions for sending the monitoring results; The transmission resources of the monitoring results.

22. A model monitoring device, It is characterized in that include: A monitoring module, used to monitor the first artificial intelligence AI model according to a predetermined monitoring type; The predetermined monitoring type includes at least one of the following: A first monitoring type, wherein the first monitoring type is to monitor multiple AI models using the same monitoring indicator, wherein the first AI model belongs to the multiple AI models; The second monitoring type is to monitor the first AI model using dedicated monitoring indicators.

23. A model monitoring device, It is characterized in that include: A transmission module, configured to send first information to a first device; Among them, the first information is used to configure relevant information of the first monitoring type, and the first monitoring type is to monitor multiple AI models using the same monitoring indicators.

24. A communication device, It is characterized in that The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the method according to any one of claims 1 to 21 are implemented.

25. A readable storage medium, It is characterized in that The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps as claimed in any one of claims 1 to 21 are implemented.

Images