Model monitoring method and apparatus, and communication device

Abstract

The present application relates to the technical field of communications, and discloses a model monitoring method and apparatus, and a communication device. The monitoring method in embodiments of the present application comprises: a first device monitors a first AI model on the basis of a predetermined monitoring type, wherein the predetermined monitoring type comprises at least one of the following: a first monitoring type, the first monitoring type relating to using a same monitoring index to monitor a plurality of AI models, wherein the first AI model belongs to the plurality of AI models; and a second monitoring type, the second monitoring type relating to using a special monitoring index to monitor the first AI model.

Description

Model monitoring method, device and communication equipment

[0001] Cross-references

[0002] The present invention claims priority to the Chinese patent application filed with the China Patent Office on November 24, 2023, with application number 202311585328X and invention name “Model Monitoring Method, Device and Communication Equipment”. The entire contents of the application are incorporated by reference into the present invention. Technical Field

[0003] The present application belongs to the field of communication technology, and specifically relates to a model monitoring method, device and communication equipment. Background Art

[0004] With the continuous development of communication technology, artificial intelligence (AI) has also been widely used in various parts of communication systems to improve the performance of communication systems.

[0005] However, in communication-related technologies, only a single model monitoring solution can usually be used to monitor AI models, resulting in poor flexibility in model monitoring and limiting the performance of the communication system. Summary of the Invention

[0006] The embodiments of the present application provide a model monitoring method, apparatus, and communication equipment, which can achieve effective monitoring of AI models while ensuring the flexibility of model monitoring and avoiding restrictions on the performance of the communication system.

[0007] In a first aspect, a model monitoring method is provided, including: a first device monitors a first AI model according to a predetermined monitoring type; wherein the predetermined monitoring type includes at least one of the following: a first monitoring type, wherein the first monitoring type is to monitor multiple AI models using the same monitoring indicator, wherein the first AI model belongs to the multiple AI models; a second monitoring type, wherein the second monitoring type is to monitor the first AI model using a dedicated monitoring indicator.

[0008] In the second aspect, a model monitoring method is provided, including: a second device sends first information to a first device; wherein, the first information is used to configure relevant information of a first monitoring type, and the first monitoring type is to monitor multiple AI models using the same monitoring indicators.

[0009] In a third aspect, a model monitoring device is provided, including: a monitoring module for monitoring a first AI model according to a predetermined monitoring type; wherein the predetermined monitoring type includes at least one of the following: a first monitoring type, wherein the first monitoring type is to monitor multiple AI models using the same monitoring indicator, wherein the first AI model belongs to the multiple AI models; a second monitoring type, wherein the second monitoring type is to monitor the first AI model using a dedicated monitoring indicator.

[0010] In a fourth aspect, a model monitoring device is provided, including: a transmission module for sending first information to a first device; wherein, the first information is used to configure relevant information of a first monitoring type, and the first monitoring type is to monitor multiple AI models using the same monitoring indicators.

[0011] In a fifth aspect, a communication device is provided, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect or the second aspect are implemented.

[0012] In a sixth aspect, a communication device is provided, comprising a processor and a communication interface, wherein the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the steps of the method described in the first aspect, or to implement the steps of the method described in the second aspect.

[0013] In the seventh aspect, a readable storage medium is provided, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented, or the steps of the method described in the second aspect are implemented.

[0014] In an eighth aspect, a wireless communication system is provided, comprising: a first device and a second device, wherein the first device can be used to execute the steps of the method described in the first aspect, and the second device can be used to execute the steps of the method described in the second aspect.

[0015] In the ninth aspect, a chip is provided, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the steps of the method described in the first aspect, or to implement the steps of the method described in the second aspect.

[0016] In a tenth aspect, a computer program / program product is provided, wherein the computer program / program product is stored in a storage medium, and the program / program product is executed by at least one processor to implement the steps of the method described in the first aspect or the second aspect.

[0017] In an embodiment of the present application, the first device monitors the first AI model according to a predetermined monitoring type, and the predetermined monitoring type includes at least one of the first monitoring type and the second monitoring type. Thus, the first device can determine different monitoring types for model monitoring according to different needs, thereby effectively improving the flexibility of model monitoring to adapt to different model monitoring scenarios and avoid limiting the performance of the communication system. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] FIG1 is a schematic structural diagram of a wireless communication system provided by an exemplary embodiment of the present application.

[0019] FIG2 is a flow chart of a model monitoring method according to an exemplary embodiment of the present application.

[0020] FIG3 is a second flow chart of a model monitoring method provided by an exemplary embodiment of the present application.

[0021] FIG4 a is one of the interactive flow diagrams of the model monitoring method provided by an exemplary embodiment of the present application.

[0022] FIG4 b is a second interactive flow diagram of a model monitoring method provided by an exemplary embodiment of the present application.

[0023] FIG4c is a third interactive flow diagram of a model monitoring method provided by an exemplary embodiment of the present application.

[0024] FIG4 d is a fourth interactive flow diagram of a model monitoring method provided by an exemplary embodiment of the present application.

[0025] FIG5 is a third flow chart of a model monitoring method provided by an exemplary embodiment of the present application.

[0026] FIG6 is one of the structural schematic diagrams of a model device provided by an exemplary embodiment of the present application.

[0027] FIG. 7 is a second schematic structural diagram of a model device provided by an exemplary embodiment of the present application.

[0028] FIG8 is a schematic structural diagram of a communication device provided by an exemplary embodiment of the present application.

[0029] FIG9 is a schematic structural diagram of a terminal provided by an exemplary embodiment of the present application.

[0030] FIG10 is a schematic structural diagram of a network-side device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0031] The following will be combined with the accompanying drawings in the embodiments of this application to clearly describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0032] The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same type, and do not limit the number of objects, for example, the first object can be one or more. In addition, "or" in this application represents at least one of the connected objects. For example, "A or B" covers three options, namely, Option 1: including A but not including B; Option 2: including B but not including A; Option 3: including both A and B. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0033] The term "indication" in this application can be either a direct indication (or explicit indication) or an indirect indication (or implicit indication). A direct indication can be understood as the sender explicitly informing the receiver of specific information, the operation to be performed, or the requested result, etc. in the instruction sent; an indirect indication can be understood as the receiver determining the corresponding information based on the instruction sent by the sender, or making a judgment and determining the operation to be performed or the requested result, etc. based on the judgment result.

[0034] It is worth noting that the technology described in the embodiments of the present application is not limited to the Long Term Evolution (LTE) / LTE-Advanced (LTE-A) system, but can also be used in other wireless communication systems, such as Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal Frequency Division Multiple Access (OFDMA), Single-carrier Frequency-Division Multiple Access (SC-FDMA) or other systems. The terms "system" and "network" in the embodiments of the present application are often used interchangeably, and the technology described can be used for the systems and radio technologies mentioned above, as well as for other systems and radio technologies. The following description describes a New Radio (NR) system for illustrative purposes, and NR terminology is used in most of the following description, but these technologies can also be applied to systems other than NR systems, such as 6th generation (6G) systems. th Generation, 6G) communication system.

[0035] FIG1 is a block diagram of a wireless communication system applicable to an embodiment of the present application. The wireless communication system includes a terminal 11 and a network-side device 12. The terminal 11 may be a mobile phone, a tablet computer (Tablet Personal Computer), a laptop computer (Laptop Computer), a notebook computer, a personal digital assistant (PDA), a handheld computer, a netbook, an ultra-mobile personal computer (UMPC), a mobile internet device (MID), an augmented reality (AR), a virtual reality (VR) device, a robot, a wearable device (Wearable Device), an aircraft (Flight Vehicle), a vehicle-mounted device (VUE), a ship-mounted device, a pedestrian user equipment (PUE), a smart home (home appliances with wireless communication capabilities, such as refrigerators, televisions, washing machines, or furniture), a game console, a personal computer (PC), an ATM, or a self-service machine, or other terminal-side devices. Wearable devices include: smart watches, smart bracelets, smart headphones, smart glasses, smart jewelry (smart bracelets, smart bracelets, smart rings, smart necklaces, smart anklets, smart anklets, etc.), smart wristbands, smart clothing, etc. Among them, the vehicle-mounted device can also be called a vehicle-mounted terminal, a vehicle-mounted controller, a vehicle-mounted module, a vehicle-mounted component, a vehicle-mounted chip or a vehicle-mounted unit, etc. It should be noted that the specific type of the terminal 11 is not limited in the embodiment of the present application. The network side device 12 may include an access network device or a core network device, wherein the access network device may also be called a radio access network (Radio Access Network, RAN) device, a radio access network function or a radio access network unit. The access network device may include a base station, a wireless local area network (Wireless Local Area Network, WLAN) access point (Access Point, AS) or a wireless fidelity (Wireless Fidelity, WiFi) node, etc.Among them, the base station can be referred to as Node B (NB), Evolved Node B (eNB), the next generation Node B (gNB), New Radio Node B (NR Node B), access point, Relay Base Station (RBS), Serving Base Station (SBS), Base Transceiver Station (BTS), radio base station, radio transceiver, Basic Service Set (BSS), Extended Service Set (ESS), Home Node B (HNB), Home Evolved Node B (home evolved Node B), Transmission Reception Point (TRP) or other appropriate terms in the field. As long as the same technical effect is achieved, the base station is not limited to specific technical vocabulary. It should be noted that in the embodiment of the present application, only the base station in the NR system is used as an example for introduction, and the specific type of the base station is not limited.

[0036] It is worth noting that the communication scenario shown in Figure 1 is one of the application scenarios of the technical solution provided in this application. For example, in this application, the combination of the first device and the second device mentioned later or the application scenario of the technical solution provided in this application can be, but is not limited to, the following.

[0037] a) The first device is the terminal 11 shown in FIG. 1 , and the second device is the network side device 12 shown in FIG. 1 .

[0038] b) The first device is the network side device 12 shown in FIG. 1 , and the second device is the terminal 11 shown in FIG. 1 .

[0039] c) Both the first device and the second device are terminals.

[0040] d) Both the first device and the second device are network-side devices.

[0041] In addition, the AI ​​model mentioned in the context of this application may also be referred to as an AI unit, AI structure, etc., or the AI ​​model may also refer to a processing unit that can implement specific algorithms, formulas, processing procedures, capabilities, etc. related to AI, or the AI ​​model may also be a processing method, algorithm, function, module or unit for a specific data set, or the AI ​​model may be a processing method, algorithm, function, module or unit running on AI-related hardware such as a graphics processing unit (GPU), a neural network processing unit (NPU), a tensor processing unit (TPU), an application-specific integrated circuit (ASIC), etc., and this application does not make specific restrictions on this. Optionally, the specific data set may include but is not limited to the input or output of the AI ​​model.

[0042] Correspondingly, the subsequent mentioned first AI model, etc. can be described by an AI model identifier. Among them, the AI ​​model identifier can be an AI unit identifier, an AI structure identifier, an AI algorithm identifier, a functional identifier (functionality ID), a physical identifier, a logical identifier, a global identifier, a local identifier, or an identifier of a specific data set associated with the AI ​​model, or an identifier of a specific scenario related to the AI, an environment identifier related to the AI ​​model, a channel feature identifier related to the AI ​​model, an identifier of a device related to the AI ​​model, or an identifier of a function, feature, capability or module related to the AI, which is not specifically limited in this application.

[0043] Based on this, the technical solutions provided by the embodiments of the present application are described in detail below through some embodiments and their application scenarios in combination with the accompanying drawings.

[0044] FIG2 is a flow chart of a model monitoring method 200 according to an exemplary embodiment of the present application. This method 200 may be, but is not limited to, executed by a first device, specifically hardware or software installed in the first device. In this embodiment, the method 200 may include at least the following steps.

[0045] S210: The first device monitors the first AI model according to a predetermined monitoring type.

[0046] Before performing model monitoring, the first device may determine an AI model to be monitored, such as the first AI model, from among multiple configured AI models based on the model monitoring period, etc. In this embodiment, the first AI model may be used to calculate, predict, and evaluate communication data, thereby improving communication system performance.

[0047] Optionally, the first AI model can be but is limited to a neural network, a decision tree, a support vector machine, a Bayesian classifier, etc., and the neural network can be but is not limited to a deep neural network, a convolutional neural network, a recurrent neural network, etc.

[0048] Based on this, when performing model monitoring in this application, the predetermined monitoring type adopted may include, but is not limited to, at least one of the first monitoring type and the second monitoring type.

[0049] The first monitoring type is to monitor multiple AI models using the same monitoring indicator, and the first AI model belongs to the multiple AI models. In other words, the first monitoring type is to monitor different AI models using the same or the same monitoring indicator. This, on the one hand, can avoid the problem of needing to use different AI model monitoring solutions for different wireless AI use cases, and on the other hand, can effectively reduce the complexity and redundancy of model monitoring, reduce signaling interaction overhead, and ensure communication system performance.

[0050] The second monitoring type is to monitor the first AI model using dedicated monitoring indicators. In other words, the second monitoring model can use different or dedicated monitoring indicators for different AI models to monitor the model, thereby ensuring the reliability and accuracy of the model monitoring results corresponding to each AI model.

[0051] In which, when the terminal monitors the first AI model, the first device can monitor the first AI model only based on the first monitoring type or the second monitoring type, or it can monitor the first AI model by combining the first monitoring type and the second monitoring type. Thus, the flexibility of model monitoring can be effectively improved to adapt to different model monitoring scenarios and avoid restrictions on the performance of the communication system.

[0052] In this embodiment, the first device may determine which monitoring type to use to monitor the first AI model based on a preconfigured trigger condition, or the first device may apply to the second device for whether to use the first monitoring type or the second monitoring type for model monitoring, or the second device may instruct the first device to use the first monitoring type or the second monitoring type for model monitoring, or the first device may autonomously determine whether to use the first monitoring type or the second monitoring type for model monitoring, etc., and there is no limitation here.

[0053] In this embodiment, the first device monitors the first AI model according to a predetermined monitoring type, and the predetermined monitoring type includes at least one of the first monitoring type and the second monitoring type. Thus, the first device can determine different monitoring types for model monitoring according to different needs, thereby effectively improving the flexibility of model monitoring to adapt to different model monitoring scenarios and avoid limiting the performance of the communication system.

[0054] In addition, in the case where the first device monitors the first AI model according to the first monitoring type, it can also effectively reduce the complexity and redundancy of model monitoring, reduce signaling interaction overhead, and ensure communication system performance.

[0055] FIG3 is a flow chart of a model monitoring method 300 according to an exemplary embodiment of the present application. This method 300 may be, but is not limited to, executed by a first device, specifically hardware or software installed in the first device. In this embodiment, the method 300 may include at least the following steps.

[0056] S310: The first device monitors the first AI model according to a predetermined monitoring type.

[0057] Among them, the predetermined monitoring type includes at least one of the following: a first monitoring type, wherein the first monitoring type is to monitor multiple AI models using the same monitoring indicators, wherein the first AI model belongs to the multiple AI models; a second monitoring type, wherein the second monitoring type is to monitor the first AI model using dedicated monitoring indicators.

[0058] It can be understood that in addition to referring to the relevant description in method embodiment 200 for the implementation process of step S310, in some embodiments, for the scenario where the first device monitors the first AI model based on the first monitoring type or the second monitoring type, considering that the monitoring granularity of the first monitoring type is relatively coarse and can only reflect the approximate performance of the AI ​​model, while the monitoring granularity of the second monitoring type is relatively fine and can reflect the specific performance of the AI ​​model. Therefore, for the first device that supports both the first monitoring type and the second monitoring type, if the accuracy requirement of the first AI model is high, then the first monitoring type can be called first to monitor the first AI model in general scenarios, and the second monitoring type can be called in scenarios where performance deteriorates severely, to achieve combined monitoring of the AI ​​models, and vice versa.

[0059] Based on this, the aforementioned combined monitoring can be implemented in this embodiment through signaling or pre-configured trigger conditions, etc. For example, the first device can switch the monitoring type according to at least one of the following methods 1 to 3.

[0060] Method 1: The first device sends a fourth message to the second device, and the fourth message is used to apply to the second device to switch from the third monitoring type to the fourth monitoring type; wherein, the third monitoring type is any one of the first monitoring type and the second monitoring type, and the fourth monitoring type is one of the first monitoring type and the second monitoring type except the third monitoring type.

[0061] Correspondingly, after receiving the fourth information, the second device may send a confirmation message to the first device to confirm whether it agrees that the first device adopts the new monitoring type to monitor the first AI model.

[0062] That is, when the monitoring type needs to be switched, the first device cannot switch the monitoring type independently, but needs to apply to the second device for a new monitoring type to switch from the current third monitoring type (such as the first monitoring type or the second monitoring type) to the fourth monitoring type. In particular, if the third monitoring type is the first monitoring type, the fourth monitoring type can be the second monitoring type, and vice versa.

[0063] Method 2: The second device sends fifth information to the first device, and the first device receives the fifth information sent by the second device. The fifth information is used to instruct the first device to switch from the third monitoring type to the fourth monitoring type. The third monitoring type is either the first monitoring type or the second monitoring type, and the fourth monitoring type is either the first monitoring type or the second monitoring type except the third monitoring type.

[0064] That is, when the monitoring type needs to be switched, the second device can indicate a new monitoring type to the first device, so that the first device switches from the current third monitoring type (such as the first monitoring type or the second monitoring type) to the fourth monitoring type. Wherein, if the third monitoring type is the first monitoring type, the fourth monitoring type can be the second monitoring type, and vice versa.

[0065] In this embodiment, the fifth information may be determined autonomously by the second device word, or may be determined based on the received fourth information, which is not limited here.

[0066] Mode 3: Switching from the third monitoring type to the fourth monitoring type according to a preconfigured monitoring type switching condition. The third monitoring type is any one of the first monitoring type and the second monitoring type, and the fourth monitoring type is one of the first monitoring type and the second monitoring type other than the third monitoring type.

[0067] That is, the first device may switch from the current third monitoring type to the fourth monitoring type when determining that the monitoring type switching condition is met. Wherein, if the third monitoring type is the first monitoring type, the fourth monitoring type may be the second monitoring type, and vice versa.

[0068] The monitoring type switching condition may be, but is not limited to, related to the current monitoring result of the first AI model. For example, if the first AI model is currently being monitored based on the first monitoring type and the monitoring result indicates that the performance of the first AI model has deteriorated, the monitoring type may be switched to the second monitoring type. Conversely, if the first AI model is currently being monitored based on the second monitoring type and the monitoring result indicates that the performance of the first AI model is better, the monitoring type may be switched to the first monitoring type.

[0069] Optionally, the monitoring type switching condition may be configured by the second device (such as the subsequent first information), by protocol agreement, etc., and is not limited here.

[0070] In this embodiment, which of the aforementioned methods 1 to 3 is used to switch the monitoring type can be determined by protocol agreement, high-level configuration, etc., and there is no limitation on this.

[0071] In some embodiments, the configuration methods for the aforementioned first and second monitoring types can be various. Taking the first monitoring type as an example, the first monitoring type can be, but is not limited to, configured by the second device. For example, the second device can send first information to the first device, where the first information is used to configure the relevant information for the first monitoring type. In response, the first device receives the first information sent by the second device and determines the relevant information for the first monitoring type based on the first information.

[0072] Optionally, in this embodiment, the first information may include, but is not limited to, at least one of the following 11)-17).

[0073] 11) A first monitoring indicator, which indicates a monitoring indicator that the first device needs to send and that can reflect the reasoning performance of the first AI model. In this embodiment, the first monitoring indicator can also be understood as a unified monitoring indicator corresponding to the first monitoring type.

[0074] In this embodiment, the first monitoring indicator may include at least one of a first value, a second value, a third value, a fourth value, and a fifth value.

[0075] The first numerical value includes a graded score, which is a score of the performance of the first AI model. For example, the performance monitoring indicator of the first AI model can be scored based on the output of the first AI model, such as a score from 0 to 5, where a lower score indicates worse performance of the first AI model, and vice versa.

[0076] The second value is used to indicate the number of abnormal samples that appear in the monitored samples. For example, one or more monitored samples can be judged to be abnormal based on the abnormal condition information, and then the number of abnormal samples can be counted. In this embodiment, the number of abnormal samples can be a numerical value or an interval value, and the larger the value, the worse the current performance of the first AI model. The interval value refers to first dividing into multiple intervals, each interval corresponding to a result value, and the specific monitoring result value is determined by looking at which interval the number of abnormalities or the proportion falls into.

[0077] The third value is used to indicate the proportion or percentage of abnormal samples in the monitored samples. For example, it is possible to determine whether one or more monitored samples are abnormal based on the abnormal condition information, and then calculate the proportion or percentage of abnormal samples. In this embodiment, the proportion or percentage of abnormal samples can be a numerical value or an interval value, and the larger the value, the worse the current performance of the first AI model. The interval value refers to dividing into multiple intervals, each interval corresponds to a result value, and the specific monitoring result value is determined by looking at which interval the number of abnormalities or the proportion falls into.

[0078] The fourth value is used to indicate the number of normal samples that appear in the monitored samples. For example, the normality of one or more monitored samples can be determined based on the normal condition information, and the number of normal samples can be counted. In this embodiment, the number of normal samples can be a numeric value or an interval value, and the larger the value, the better the current performance of the first AI model. The interval value refers to first dividing into multiple intervals, each interval corresponding to a result value, and the specific monitoring result value is determined by looking at which interval the normal number or proportion falls into.

[0079] The fifth value is used to indicate the ratio or proportion of normal samples in the monitored samples. For example, whether one or more monitored samples are normal can be determined based on normal condition information, and then the ratio or proportion of normal samples can be calculated. In this embodiment, the ratio or proportion of normal samples can be a numerical value or an interval value, and the larger the value, the better the current performance of the first AI model. The interval value refers to dividing into multiple intervals, each interval corresponding to a result value, and the specific monitoring result value is determined by looking at which interval the normal number or proportion falls into.

[0080] The aforementioned monitoring samples are used to monitor the first AI model. In addition, the aforementioned abnormal condition information and normal condition information can be determined by protocol agreement, high-level configuration, or network-side configuration. Taking the abnormal condition information as an example, the abnormal condition information can be whether the correlation coefficient between the output of the first AI model and its true value is lower than a preset threshold, such as when the correlation coefficient is lower than the predetermined threshold, the number of abnormal samples is increased by 1. The correlation coefficient between the output of the first AI model and its true value can be obtained based on statistics of one or more monitoring samples.

[0081] 12) Monitoring indicator conversion information, used to convert the second monitoring indicator dedicated to the first AI model monitoring into the first monitoring indicator.

[0082] Optionally, the second monitoring indicator may include but is not limited to at least one of the following 121)-123).

[0083] 121) Error information between the output of the first AI model and its true value, such as error, normalized error, absolute value of error, cross-entropy loss (CEL), root mean square error (RSME), and average of the foregoing information. Of course, smaller values ​​of the error information between the output of the first AI model and its true value indicate better performance of the first AI model.

[0084] 122) Similarity information between the output of the first AI model and its true value, such as precision, accuracy, Top-K accuracy, similarity, cosine similarity, correlation, correlation coefficient, Area Under the Curve score (AUC score), and average of the foregoing information. Of course, a larger value of the similarity information between the output of the first AI model and its true value indicates better performance of the first AI model.

[0085] 123) Communication system performance determined according to the first AI model, such as throughput, spectral efficiency, signal to interference plus noise ratio (SINR), signal-to-noise ratio (SNR), bit error rate, block error rate, packet loss rate, transmission rate (uplink / downlink), peak rate (uplink / downlink), etc. The aforementioned communication system performance can be calculated based on a single monitoring sample or based on statistics of multiple monitoring samples, and is not limited here.

[0086] In this embodiment, the monitoring indicator conversion information may include, but is not limited to, at least one of a monitoring indicator conversion method and auxiliary information required to implement the monitoring indicator conversion. The auxiliary information may be understood as a threshold, a weighting parameter, a bias parameter, a polynomial parameter, various conditional information, etc. required to implement the monitoring indicator conversion method.

[0087] In some embodiments, the monitoring indicator conversion method may include, but is not limited to, at least one of the following methods 1 to 3.

[0088] Mode 1: When the first monitoring indicator includes the first numerical value, the second monitoring indicator is mapped to a graded score according to a preset graded score to obtain the first numerical value.

[0089] For example, assuming the preset grading scores are 0, 1, and 2, if the mean square error between the output of the first AI model and its true value is less than or equal to a first threshold, then the first value is 0; if the mean square error between the output of the first AI model and its true value is greater than the first threshold and less than or equal to the second threshold, then the first value is 1; if the mean square error between the output of the first AI model and its true value is greater than the second threshold, then the first value is 2.

[0090] It is worth noting that the grading scores may be different for different monitoring scenarios. That is, although the same piecewise function is used to characterize the grading scores, different AI models can be assigned different auxiliary information to achieve similar performance of different AI models using the same grading scores. For example, the first AI model has high requirements for AI reasoning accuracy, that is, a lower mean square error is required to ensure performance, but the second AI model has relatively broad requirements for AI reasoning accuracy, that is, the mean square error does not necessarily need to be particularly low. Then, the first threshold corresponding to the first AI model can be 0.01 and the second threshold is 0.1, and the first threshold corresponding to the second AI model can be 0.1 and the second threshold is 0.3.

[0091] Method 2: When the first monitoring indicator includes the second value or the third value, the second monitoring indicator is compared with the abnormal condition information, and the second value or the third value is determined according to the comparison result.

[0092] Method 3: When the first monitoring indicator includes the fourth value or the fifth value, the second monitoring indicator is compared with the normal condition information, and the fourth value or the fifth value is determined according to the comparison result.

[0093] It is worth noting that the aforementioned methods 1, 2, and 3 can be described using formulas. In this embodiment, the formulas can be predefined in the protocol or configured through signaling. For example, a predefined identifier can be included in the first information, such as "00" for method 1, "01" for method 2, and "10" for method 3, etc., or the formula can be directly described in the signaling.

[0094] 13) Monitoring start conditions of the first AI model.

[0095] 14) Monitoring stop conditions of the first AI model.

[0096] Among them, the monitoring start condition and the monitoring stop condition may be, but are not limited to, periodic conditions or event conditions, etc. For example, the first AI model may be monitored periodically, and for another example, the monitoring of the first AI model may be performed or stopped when a predetermined event is satisfied. Among them, for the monitoring start condition, the predetermined event may be, but is not limited to: the first device accesses a new cell, the change in the moving speed of the first device exceeds a preset threshold, the environment in which the first device is located changes, the first device updates the AI ​​model, the size between the communication performance indicator and the preset threshold does not meet the preset conditions, the size relationship between the bit error rate (or block error rate) and the preset threshold does not meet the preset conditions, etc.

[0097] 15) Monitoring window of the first AI model.

[0098] That is, the first device can only monitor the first AI model within the monitoring window.

[0099] 16) Monitoring resources of the first AI model.

[0100] The monitoring resource may be a time domain resource, a frequency resource, or a spatial domain resource on which the first device monitors the first AI model. The time domain resource may be, but is not limited to, a symbol, a time slot, etc., the frequency domain resource may be, but is not limited to, a carrier, a resource block, etc., and the spatial domain resource may be, but is not limited to, a port, etc.

[0101] 17) A monitoring type switching condition, used for the first device to switch between the first monitoring type and the second monitoring type. For example, when the monitoring type switching condition is met, the first device may switch from the current first monitoring type to the second monitoring type, or from the current second monitoring type to the first monitoring type.

[0102] In some embodiments, in addition to configuring the first device with the relevant information of the first monitoring type through the first information, the first device can also receive the seventh information sent by the second device and send the monitoring results according to the seventh information; that is, the seventh information can be used to instruct the first device to send the monitoring results. As a result, the first device can accurately report the monitoring results. The monitoring results are determined by monitoring the first AI model based on the aforementioned monitoring indicators.

[0103] Optionally, the seventh information may include, but is not limited to, at least one of the type of the monitoring result, the sending condition of the monitoring result, and the transmission resource of the monitoring result.

[0104] The type of the monitoring result includes a monitoring result corresponding to the first monitoring type or a monitoring result corresponding to the second monitoring type.

[0105] The sending condition of the monitoring result may be the sending cycle, trigger condition, trigger instruction, etc. of the monitoring result.

[0106] The transmission resources of the monitoring results may be, but are not limited to, the resources (such as time domain resources, frequency domain resources, spatial domain resources, etc.) on which the monitoring results are transmitted.

[0107] In some embodiments, when the first device monitors the first AI model, it may also send second information or third information to the second device to indicate to the second device the model monitoring capabilities it supports, so that the second device is aware of the model monitoring capabilities of the first device and achieves the accuracy of configurations such as the first information.

[0108] For example, the second information may be used to indicate the AI ​​model monitoring capability supported by the first device. In this embodiment, the AI ​​model monitoring capability includes at least one of the first monitoring capability and the second monitoring capability.

[0109] The first monitoring capability is to support monitoring the multiple AI models using the same monitoring indicators. That is, the first monitoring capability is that all AI models share the same monitoring indicators, which corresponds to the first monitoring type.

[0110] The second monitoring capability supports monitoring different AI models using different monitoring indicators. In other words, the second monitoring capability monitors different AI models based on dedicated monitoring indicators, which corresponds to the second monitoring type.

[0111] In some embodiments, to save overhead, the second information may be a predefined or protocol-agreed identifier, etc. For example, "00" indicates that the first device supports the second monitoring capability, "01" indicates that the first device supports the first monitoring capability, "10" indicates that the first device supports both the first monitoring capability and the second monitoring capability, and "11" indicates that the first device does not support the first monitoring capability and the second monitoring capability.

[0112] Of course, in addition to transmitting the capabilities of the first device to the second device via the aforementioned second information, in another embodiment, it is also possible to ignore whether the first device supports the second monitoring capability and only consider whether it supports the first monitoring capability. In other words, the third information can be used to indicate whether the first device supports the first monitoring capability. For example, if the third information is "0", it indicates that the first device does not support the first monitoring capability, and if the third information is "1", it indicates that the first device supports the first monitoring capability.

[0113] In this embodiment, not only a flexible and effective model monitoring solution is provided, but also for the first monitoring type, model monitoring based on the monitoring indicators corresponding to the first monitoring type is realized through the collaboration of the transmitting and receiving ends, which can greatly reduce the type and overhead of signaling interaction, and at the same time reduce the complexity of AI model monitoring.

[0114] Based on the model monitoring solutions provided in the aforementioned method embodiments 200-300, the aforementioned model monitoring method is exemplarily introduced below in combination with Examples 1-4. It is assumed that the first device is a terminal and the second device is a network-side device.

[0115] Example 1

[0116] S411, as shown in Figure 4a, the terminal sends second information or third information to the network side device. The second information is used to indicate the AI ​​model monitoring capability supported by the terminal, and the third information is used to indicate whether the terminal supports the first monitoring capability.

[0117] S412, the network side device sends the first information and the seventh information to the terminal, wherein the first information is determined based on the second information and the third information and is used to configure the relevant information of the first monitoring type, and the seventh information is used to indicate the relevant information when the terminal sends the monitoring result.

[0118] S413: The terminal monitors the first AI model according to the first information.

[0119] S414: The terminal reports the monitoring result to the network side device according to the seventh information.

[0120] Example 2

[0121] S421, as shown in Figure 4b, the terminal sends second information or third information to the network side device. The second information is used to indicate the AI ​​model monitoring capability supported by the terminal, and the third information is used to indicate whether the terminal supports the first monitoring capability.

[0122] S422: The network-side device sends first information to the terminal, wherein the first information is determined based on the second information and third information and is used to configure relevant information of the first monitoring type.

[0123] S423: The terminal monitors the first AI model according to the third monitoring type, where the third monitoring type is the first monitoring type or the second monitoring type.

[0124] S424: The terminal reports the monitoring result to the network side device.

[0125] S425: The network-side device sends fifth information to the terminal according to the monitoring result, where the fifth information is used to instruct the terminal to switch from the third monitoring type to the fourth monitoring type.

[0126] S426: The terminal switches the monitoring type according to the new fourth monitoring type indicated by the fifth information.

[0127] Example 3

[0128] S431, as shown in Figure 4c, the terminal sends second information or third information to the network side device. The second information is used to indicate the AI ​​model monitoring capability supported by the terminal, and the third information is used to indicate whether the terminal supports the first monitoring capability.

[0129] S432: The network-side device sends first information to the terminal, wherein the first information is determined based on the second information and third information and is used to configure relevant information of the first monitoring type.

[0130] S433: The terminal monitors the first AI model according to the third monitoring type; the third monitoring type is the first monitoring type or the second monitoring type.

[0131] S434: The terminal sends fourth information to the network side device to apply for a new monitoring type.

[0132] S435: The network-side device sends fifth information to the terminal, where the fifth information corresponds to the fourth information and is used to instruct the first device to switch from the third monitoring type to the fourth monitoring type.

[0133] S436: The terminal switches the monitoring type according to the applied new fourth monitoring type or the fifth information.

[0134] Example 4

[0135] S441, as shown in Figure 4d, the terminal sends second information or third information to the network side device. The second information is used to indicate the AI ​​model monitoring capability supported by the terminal, and the third information is used to indicate whether the terminal supports the first monitoring capability.

[0136] S442: The network-side device sends first information to the terminal, wherein the first information is determined based on the second information and third information and is used to configure relevant information of the first monitoring type.

[0137] S443: The terminal monitors the first AI model according to the third monitoring type; the third monitoring type is the first monitoring type or the second monitoring type.

[0138] S444: The terminal switches the monitoring type according to a preconfigured monitoring type switching condition.

[0139] It is worth noting that several different monitoring type switching schemes are provided in the aforementioned Examples 2 to 4 respectively. Among them, the terminal in Example 2 can switch the monitoring type according to the monitoring type indicated by the network side device, that is, the terminal dynamically selects which monitoring mode to use based on the network side decision indication; Example 3 is a process of dynamically selecting which monitoring mode to use based on the terminal application; Example 4 is based on the preconfigured monitoring type switching conditions for switching the monitoring type.

[0140] Furthermore, Examples 1-4 may include, but are not limited to, the aforementioned steps, and may include more or fewer steps than those described above. The implementation processes of Examples 1-4 may refer to the relevant descriptions in Method Embodiments 200-300, and achieve the same or corresponding technical effects. To avoid repetition, these are not further described here.

[0141] FIG5 is a flow chart of a model monitoring method 500 according to an exemplary embodiment of the present application. This method 500 may be, but is not limited to, executed by a second device, specifically hardware or software installed in the second device. In this embodiment, the method 500 may include at least the following steps.

[0142] S510: The second device sends first information to the first device.

[0143] Among them, the first information is used to configure relevant information of the first monitoring type, and the first monitoring type is to monitor multiple AI models using the same monitoring indicators.

[0144] Optionally, the first information includes at least one of the following: a first monitoring indicator, used to indicate a monitoring indicator that the first device needs to send and can reflect the inference performance of the first AI model, and the first monitoring indicator corresponds to the first monitoring type; monitoring indicator conversion information, used to convert a second monitoring indicator dedicated to monitoring the first AI model into the first monitoring indicator; the monitoring start condition of the first AI model; the monitoring stop condition of the first AI model; the monitoring window of the first AI model; the monitoring resources of the first AI model; and a monitoring type switching condition, used for the first device to switch between the first monitoring type and the second monitoring type.

[0145] Optionally, the first monitoring indicator includes at least one of the following: a first numerical value, which includes a graded score, and the graded score is a scoring result of the performance of the first AI model; a second numerical value, used to indicate the number of abnormal samples appearing in the monitoring samples; a third numerical value, used to indicate the proportion of abnormal samples appearing in the monitoring samples; a fourth numerical value, used to indicate the number of normal samples appearing in the monitoring samples; a fifth numerical value, used to indicate the proportion of normal samples appearing in the monitoring samples; wherein, the monitoring samples are used to monitor the first AI model.

[0146] Optionally, the monitoring indicator conversion information includes at least one of a monitoring indicator conversion method and auxiliary information required to implement the monitoring indicator conversion.

[0147] Optionally, the monitoring indicator conversion method includes at least one of the following: when the first monitoring indicator includes the first numerical value, mapping the second monitoring indicator to a hierarchical score according to a preset hierarchical score to obtain the first numerical value; when the first monitoring indicator includes the second numerical value or the third numerical value, comparing the second monitoring indicator with the abnormal condition information, and determining the second numerical value or the third numerical value based on the comparison result; when the first monitoring indicator includes the fourth numerical value or the fifth numerical value, comparing the second monitoring indicator with the normal condition information, and determining the fourth numerical value or the fifth numerical value based on the comparison result.

[0148] Optionally, the method also includes: receiving second information or third information sent by the first device; wherein, the second information is used to indicate the AI ​​model monitoring capability supported by the first device, and the third information is used to indicate whether the first device has a first monitoring capability, and the first monitoring capability is to support monitoring of multiple AI models using the same monitoring indicators.

[0149] Optionally, the AI ​​model monitoring capability includes at least one of the following: a first monitoring capability, which supports monitoring the multiple AI models using the same monitoring indicators; and a second monitoring capability, which supports monitoring different AI models using different monitoring indicators.

[0150] Optionally, the method also includes: receiving fourth information sent by the first device, the fourth information being used to apply to the second device to switch from the third monitoring type to the fourth monitoring type; sending fifth information to the first device, the fifth information being used to instruct the first device to switch from the third monitoring type to the fourth monitoring type; wherein the third monitoring type is any one of the first monitoring type and the second monitoring type, the fourth monitoring type is one of the first monitoring type and the second monitoring type except the third monitoring type, and the second monitoring type is to monitor the first AI model using dedicated monitoring indicators.

[0151] Optionally, the method further includes: sending seventh information to the first device; wherein the seventh information is used by the first device to send monitoring results.

[0152] Optionally, the seventh information includes at least one of the following: the type of the monitoring result, the type of the monitoring result including the monitoring result corresponding to the first monitoring type or the monitoring result corresponding to the second monitoring type, the second monitoring type is to monitor the first AI model using dedicated monitoring indicators; the sending conditions of the monitoring result; the transmission resources of the monitoring result.

[0153] It can be understood that each implementation method in method embodiment 500 has the same or corresponding technical features as the aforementioned method embodiments 200-300. Therefore, for each implementation method in method embodiment 500, reference can be made to the relevant descriptions in the aforementioned method embodiments 200-300, and the same or corresponding technical effects can be achieved. To avoid repetition, they will not be repeated here.

[0154] The model monitoring methods 200-500 provided in the embodiments of the present application may be executed by a model monitoring device. In the embodiments of the present application, the model monitoring device performing the model monitoring methods 200-500 is taken as an example to illustrate the model monitoring device provided in the embodiments of the present application.

[0155] As shown in Figure 6, it is a structural diagram of a model monitoring device 600 provided in an embodiment of the present application. The device 600 includes: a monitoring module 610, which is used to monitor the first AI model according to a predetermined monitoring type; wherein the predetermined monitoring type includes at least one of the following: a first monitoring type, wherein the first monitoring type is to monitor multiple AI models using the same monitoring indicator, wherein the first AI model belongs to the multiple AI models; a second monitoring type, wherein the second monitoring type is to monitor the first AI model using a dedicated monitoring indicator.

[0156] Optionally, the apparatus 600 further includes: a transmission module, configured to receive first information sent by a second device; wherein the first information is used to configure relevant information of the first monitoring type.

[0157] Optionally, the first information includes at least one of the following: a first monitoring indicator, used to indicate a monitoring indicator that the first device needs to send and can reflect the reasoning performance of the first AI model, and the first monitoring indicator corresponds to the first monitoring type; monitoring indicator conversion information, used to convert a second monitoring indicator dedicated to monitoring the first AI model into the first monitoring indicator; the monitoring start condition of the first AI model; the monitoring stop condition of the first AI model; the monitoring window of the first AI model; the monitoring resources of the first AI model, and the monitoring type switching condition, used for the first device to switch between the first monitoring type and the second monitoring type.

[0158] Optionally, the first monitoring indicator includes at least one of the following: a first numerical value, which includes a graded score, and the graded score is a scoring result of the performance of the first AI model; a second numerical value, used to indicate the number of abnormal samples appearing in the monitoring samples; a third numerical value, used to indicate the proportion of abnormal samples appearing in the monitoring samples; a fourth numerical value, used to indicate the number of normal samples appearing in the monitoring samples; a fifth numerical value, used to indicate the proportion of normal samples appearing in the monitoring samples; wherein, the monitoring samples are used to monitor the first AI model.

[0159] Optionally, the monitoring indicator conversion information includes at least one of a monitoring indicator conversion method and auxiliary information required to implement the monitoring indicator conversion.

[0160] Optionally, the monitoring indicator conversion method includes at least one of the following: when the first monitoring indicator includes the first numerical value, mapping the second monitoring indicator to a hierarchical score according to a preset hierarchical score to obtain the first numerical value; when the first monitoring indicator includes the second numerical value or the third numerical value, comparing the second monitoring indicator with the abnormal condition information, and determining the second numerical value or the third numerical value based on the comparison result; when the first monitoring indicator includes the fourth numerical value or the fifth numerical value, comparing the second monitoring indicator with the normal condition information, and determining the fourth numerical value or the fifth numerical value based on the comparison result.

[0161] Optionally, the transmission module is also used to send second information or third information to the second device; wherein, the second information is used to indicate the AI ​​model monitoring capability supported by the first device, and the third information is used to indicate whether the first device has the first monitoring capability, and the first monitoring capability is to support the use of the same monitoring indicators for monitoring multiple AI models.

[0162] Optionally, the AI ​​model monitoring capability includes at least one of the following: a first monitoring capability, which supports monitoring the multiple AI models using the same monitoring indicators; and a second monitoring capability, which supports monitoring different AI models using different monitoring indicators.

[0163] Optionally, the monitoring module 610 is also used to: switch the monitoring type according to at least one of the following: send fourth information to the second device, the fourth information is used to apply to the second device to switch from the third monitoring type to the fourth monitoring type; receive fifth information sent by the second device, the fifth information is used to instruct the first device to switch from the third monitoring type to the fourth monitoring type; switch from the third monitoring type to the fourth monitoring type according to preconfigured monitoring type switching conditions; wherein the third monitoring type is any one of the first monitoring type and the second monitoring type, and the fourth monitoring type is one of the first monitoring type and the second monitoring type except the third monitoring type.

[0164] Optionally, the transmission module is further used to: receive seventh information sent by the second device; and send monitoring results according to the seventh information.

[0165] Optionally, the seventh information includes at least one of the following: the type of the monitoring result, the type of the monitoring result including the monitoring result corresponding to the first monitoring type or the monitoring result corresponding to the second monitoring type, the second monitoring type is to monitor the first AI model using dedicated monitoring indicators; the sending conditions of the monitoring result; the transmission resources of the monitoring result.

[0166] The model monitoring device 600 in the embodiments of the present application can be a communication device, such as a communication device with an operating system, or a component within the communication device, such as an integrated circuit or chip. The communication device can be a terminal or a network-side device. For example, the terminal can include, but is not limited to, the types of terminal 11 listed above, and the network-side device can include, but is not limited to, the types of network-side device 12 listed above, and the embodiments of the present application do not specifically limit this.

[0167] The model monitoring device 600 provided in the embodiment of the present application can implement the various processes implemented in the method embodiments of Figures 2 to 3 and achieve the same technical effects. To avoid repetition, they will not be described here.

[0168] As shown in Figure 7, it is a structural diagram of a model monitoring device 700 provided in an embodiment of the present application. The device 700 includes: a transmission module 710, used to send first information to a first device; wherein, the first information is used to configure relevant information of a first monitoring type, and the first monitoring type is to monitor multiple AI models using the same monitoring indicators.

[0169] Optionally, the first information includes at least one of the following: a first monitoring indicator, used to indicate a monitoring indicator that the first device needs to send and can reflect the inference performance of the first AI model, and the first monitoring indicator corresponds to the first monitoring type; monitoring indicator conversion information, used to convert a second monitoring indicator dedicated to monitoring the first AI model into the first monitoring indicator; the monitoring start condition of the first AI model; the monitoring stop condition of the first AI model; the monitoring window of the first AI model; the monitoring resources of the first AI model; and a monitoring type switching condition, used for the first device to switch between the first monitoring type and the second monitoring type.

[0170] Optionally, the first monitoring indicator includes at least one of the following: a first numerical value, which includes a graded score, and the graded score is a scoring result of the performance of the first AI model; a second numerical value, used to indicate the number of abnormal samples appearing in the monitoring samples; a third numerical value, used to indicate the proportion of abnormal samples appearing in the monitoring samples; a fourth numerical value, used to indicate the number of normal samples appearing in the monitoring samples; a fifth numerical value, used to indicate the proportion of normal samples appearing in the monitoring samples; wherein, the monitoring samples are used to monitor the first AI model.

[0171] Optionally, the monitoring indicator conversion information includes at least one of a monitoring indicator conversion method and auxiliary information required to implement the monitoring indicator conversion.

[0172] Optionally, the monitoring indicator conversion method includes at least one of the following: when the first monitoring indicator includes the first numerical value, mapping the second monitoring indicator to a hierarchical score according to a preset hierarchical score to obtain the first numerical value; when the first monitoring indicator includes the second numerical value or the third numerical value, comparing the second monitoring indicator with the abnormal condition information, and determining the second numerical value or the third numerical value based on the comparison result; when the first monitoring indicator includes the fourth numerical value or the fifth numerical value, comparing the second monitoring indicator with the normal condition information, and determining the fourth numerical value or the fifth numerical value based on the comparison result.

[0173] Optionally, the transmission module 710 is also used to receive second information or third information sent by the first device; wherein, the second information is used to indicate the AI ​​model monitoring capability supported by the first device, and the third information is used to indicate whether the first device has a first monitoring capability, and the first monitoring capability is to support the use of the same monitoring indicators for monitoring multiple AI models.

[0174] Optionally, the AI ​​model monitoring capability includes at least one of the following: a first monitoring capability, which supports monitoring the multiple AI models using the same monitoring indicators; and a second monitoring capability, which supports monitoring different AI models using different monitoring indicators.

[0175] Optionally, the transmission module 710 is also used to receive fourth information sent by the first device, and the fourth information is used to apply to the second device to switch from the third monitoring type to the fourth monitoring type; send fifth information to the first device, and the fifth information is used to instruct the first device to switch from the third monitoring type to the fourth monitoring type; wherein, the third monitoring type is any one of the first monitoring type and the second monitoring type, the fourth monitoring type is one of the first monitoring type and the second monitoring type except the third monitoring type, and the second monitoring type is to monitor the first AI model using dedicated monitoring indicators.

[0176] Optionally, the transmission module 710 is further used to send seventh information to the first device; wherein, the seventh information is used for the first device to send monitoring results.

[0177] Optionally, the seventh information includes at least one of the following: the type of the monitoring result, the type of the monitoring result including the monitoring result corresponding to the first monitoring type or the monitoring result corresponding to the second monitoring type, the second monitoring type is to monitor the first AI model using dedicated monitoring indicators; the sending conditions of the monitoring result; the transmission resources of the monitoring result.

[0178] The model monitoring device 700 in the embodiments of the present application can be a communication device, such as a communication device with an operating system, or a component within the communication device, such as an integrated circuit or chip. The communication device can be a terminal or a network-side device. For example, the terminal can include, but is not limited to, the types of terminal 11 listed above, and the network-side device can include, but is not limited to, the types of network-side device 12 listed above, and the embodiments of the present application do not specifically limit this.

[0179] The model monitoring device 700 provided in the embodiment of the present application can implement each process implemented in the method embodiment of Figure 5 and achieve the same technical effect. To avoid repetition, it will not be described here.

[0180] As shown in Figure 8, an embodiment of the present application further provides a communication device 800, including a processor 801 and a memory 802. The memory 802 stores a program or instruction that can be run on the processor 801. For example, when the communication device 800 is a terminal, the program or instruction, when executed by the processor 801, implements the various steps of the above-mentioned model monitoring method embodiment and can achieve the same technical effect. When the communication device 800 is a network-side device, the program or instruction, when executed by the processor 801, implements the various steps of the above-mentioned model monitoring method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0181] The present application also provides a terminal comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is configured to execute a program or instruction to implement the steps of the method embodiments shown in Figures 2-5. This terminal embodiment corresponds to the aforementioned terminal-side method embodiment, and each implementation process and implementation method of the aforementioned method embodiment is applicable to this terminal embodiment and can achieve the same technical effects. Specifically, Figure 9 is a schematic diagram of the hardware structure of a terminal implementing an embodiment of the present application.

[0182] The terminal 900 includes but is not limited to: a radio frequency unit 901, a network module 902, an audio output unit 903, an input unit 904, a sensor 905, a display unit 906, a user input unit 907, an interface unit 908, a memory 909 and at least some of the components of the processor 910.

[0183] Those skilled in the art will appreciate that the terminal 900 may also include a power supply (such as a battery) to power various components. The power supply may be logically connected to the processor 910 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The terminal structure shown in FIG9 does not limit the terminal. The terminal may include more or fewer components than shown, or may combine certain components, or have different component arrangements, which will not be described in detail here.

[0184] It should be understood that in an embodiment of the present application, the input unit 904 may include a GPU 9041 and a microphone 9042, and the graphics processor 9041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 906 may include a display panel 9061, and the display panel 9061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 907 includes a touch panel 9071 and at least one of other input devices 9072. The touch panel 9071 is also called a touch screen. The touch panel 9071 may include two parts: a touch detection device and a touch controller. Other input devices 9072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and an operating stick, which will not be repeated here.

[0185] In the embodiment of the present application, after receiving downlink data from a network-side device, the RF unit 901 may transmit the data to the processor 910 for processing. Furthermore, the RF unit 901 may send uplink data to the network-side device. Typically, the RF unit 901 includes, but is not limited to, an antenna, an amplifier, a transceiver, a coupler, a low-noise amplifier, a duplexer, and the like.

[0186] The memory 909 can be used to store software programs or instructions and various data. The memory 909 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, applications or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 909 may include a volatile memory or a non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM). The memory 909 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0187] Processor 910 may include one or more processing units. Optionally, processor 910 integrates an application processor and a modem processor. The application processor primarily handles operations related to the operating system, user interface, and application programs, while the modem processor primarily processes wireless communication signals, such as a baseband processor. It is understood that the modem processor may not be integrated into processor 910.

[0188] In one implementation, the processor 910 is used to monitor the first AI model according to a predetermined monitoring type; wherein the predetermined monitoring type includes at least one of the following: a first monitoring type, wherein the first monitoring type is to monitor multiple AI models using the same monitoring indicators, wherein the first AI model belongs to the multiple AI models; a second monitoring type, wherein the second monitoring type is to monitor the first AI model using dedicated monitoring indicators.

[0189] Optionally, the apparatus 600 further includes: a radio frequency unit 901, configured to receive first information sent by a second device; wherein the first information is used to configure relevant information of the first monitoring type.

[0190] Optionally, the first information includes at least one of the following: a first monitoring indicator, used to indicate a monitoring indicator that the first device needs to send and can reflect the inference performance of the first AI model, and the first monitoring indicator corresponds to the first monitoring type; monitoring indicator conversion information, used to convert a second monitoring indicator dedicated to monitoring the first AI model into the first monitoring indicator; the monitoring start condition of the first AI model; the monitoring stop condition of the first AI model; the monitoring window of the first AI model; the monitoring resources of the first AI model; and a monitoring type switching condition, used for the first device to switch between the first monitoring type and the second monitoring type.

[0191] Optionally, the first monitoring indicator includes at least one of the following: a first numerical value, which includes a graded score, and the graded score is a scoring result of the performance of the first AI model; a second numerical value, used to indicate the number of abnormal samples appearing in the monitoring samples; a third numerical value, used to indicate the proportion of abnormal samples appearing in the monitoring samples; a fourth numerical value, used to indicate the number of normal samples appearing in the monitoring samples; a fifth numerical value, used to indicate the proportion of normal samples appearing in the monitoring samples; wherein, the monitoring samples are used to monitor the first AI model.

[0192] Optionally, the monitoring indicator conversion information includes at least one of a monitoring indicator conversion method and auxiliary information required to implement the monitoring indicator conversion.

[0193] Optionally, the monitoring indicator conversion method includes at least one of the following: when the first monitoring indicator includes the first numerical value, mapping the second monitoring indicator to a hierarchical score according to a preset hierarchical score to obtain the first numerical value; when the first monitoring indicator includes the second numerical value or the third numerical value, comparing the second monitoring indicator with the abnormal condition information, and determining the second numerical value or the third numerical value based on the comparison result; when the first monitoring indicator includes the fourth numerical value or the fifth numerical value, comparing the second monitoring indicator with the normal condition information, and determining the fourth numerical value or the fifth numerical value based on the comparison result.

[0194] Optionally, the radio frequency unit 901 is also used to send second information or third information to the second device; wherein, the second information is used to indicate the AI ​​model monitoring capability supported by the first device, and the third information is used to indicate whether the first device has a first monitoring capability, and the first monitoring capability supports monitoring multiple AI models using the same monitoring indicators.

[0195] Optionally, the AI ​​model monitoring capability includes at least one of the following: a first monitoring capability, which supports monitoring the multiple AI models using the same monitoring indicators; and a second monitoring capability, which supports monitoring different AI models using different monitoring indicators.

[0196] Optionally, the processor 910 is further used to: switch the monitoring type according to at least one of the following: send fourth information to the second device, the fourth information being used to apply to the second device for switching from the third monitoring type to the fourth monitoring type; receive fifth information sent by the second device, the fifth information being used to instruct the first device to switch from the third monitoring type to the fourth monitoring type; switch from the third monitoring type to the fourth monitoring type according to a preconfigured monitoring type switching condition; wherein the third monitoring type is any one of the first monitoring type and the second monitoring type, and the fourth monitoring type is one of the first monitoring type and the second monitoring type other than the third monitoring type.

[0197] Optionally, the radio frequency unit 901 is further used to: receive seventh information sent by the second device; and send a monitoring result according to the seventh information.

[0198] Optionally, the seventh information includes at least one of the following: the type of the monitoring result, the type of the monitoring result including the monitoring result corresponding to the first monitoring type or the monitoring result corresponding to the second monitoring type, the second monitoring type is to monitor the first AI model using dedicated monitoring indicators; the sending conditions of the monitoring result; the transmission resources of the monitoring result.

[0199] In another implementation, the radio frequency unit 901 is used to send first information to the first device; wherein, the first information is used to configure relevant information of a first monitoring type, and the first monitoring type is to monitor multiple AI models using the same monitoring indicators.

[0200] Optionally, the first information includes at least one of the following: a first monitoring indicator, used to indicate a monitoring indicator that the first device needs to send and can reflect the inference performance of the first AI model, and the first monitoring indicator corresponds to the first monitoring type; monitoring indicator conversion information, used to convert a second monitoring indicator dedicated to monitoring the first AI model into the first monitoring indicator; the monitoring start condition of the first AI model; the monitoring stop condition of the first AI model; the monitoring window of the first AI model; the monitoring resources of the first AI model; and a monitoring type switching condition, used for the first device to switch between the first monitoring type and the second monitoring type.

[0201] Optionally, the first monitoring indicator includes at least one of the following: a first numerical value, which includes a graded score, and the graded score is a scoring result of the performance of the first AI model; a second numerical value, used to indicate the number of abnormal samples appearing in the monitoring samples; a third numerical value, used to indicate the proportion of abnormal samples appearing in the monitoring samples; a fourth numerical value, used to indicate the number of normal samples appearing in the monitoring samples; a fifth numerical value, used to indicate the proportion of normal samples appearing in the monitoring samples; wherein, the monitoring samples are used to monitor the first AI model.

[0202] Optionally, the monitoring indicator conversion information includes at least one of a monitoring indicator conversion method and auxiliary information required to implement the monitoring indicator conversion.

[0203] Optionally, the monitoring indicator conversion method includes at least one of the following: when the first monitoring indicator includes the first numerical value, mapping the second monitoring indicator to a hierarchical score according to a preset hierarchical score to obtain the first numerical value; when the first monitoring indicator includes the second numerical value or the third numerical value, comparing the second monitoring indicator with the abnormal condition information, and determining the second numerical value or the third numerical value based on the comparison result; when the first monitoring indicator includes the fourth numerical value or the fifth numerical value, comparing the second monitoring indicator with the normal condition information, and determining the fourth numerical value or the fifth numerical value based on the comparison result.

[0204] Optionally, the radio frequency unit 901 is also used to receive second information or third information sent by the first device; wherein, the second information is used to indicate the AI ​​model monitoring capability supported by the first device, and the third information is used to indicate whether the first device has the first monitoring capability, and the first monitoring capability is to support the monitoring of multiple AI models using the same monitoring indicators.

[0205] Optionally, the AI ​​model monitoring capability includes at least one of the following: a first monitoring capability, which supports monitoring the multiple AI models using the same monitoring indicators; and a second monitoring capability, which supports monitoring different AI models using different monitoring indicators.

[0206] Optionally, the radio frequency unit 901 is also used to receive fourth information sent by the first device, and the fourth information is used to apply to the second device to switch from the third monitoring type to the fourth monitoring type; send fifth information to the first device, and the fifth information is used to instruct the first device to switch from the third monitoring type to the fourth monitoring type; wherein, the third monitoring type is any one of the first monitoring type and the second monitoring type, the fourth monitoring type is one of the first monitoring type and the second monitoring type except the third monitoring type, and the second monitoring type is to monitor the first AI model using dedicated monitoring indicators.

[0207] Optionally, the radio frequency unit 901 is further used to send seventh information to the first device; wherein the seventh information is used for the first device to send monitoring results.

[0208] Optionally, the seventh information includes at least one of the following: the type of the monitoring result, the type of the monitoring result including the monitoring result corresponding to the first monitoring type or the monitoring result corresponding to the second monitoring type, the second monitoring type is to monitor the first AI model using dedicated monitoring indicators; the sending conditions of the monitoring result; the transmission resources of the monitoring result.

[0209] It can be understood that the implementation process of each implementation method mentioned in this embodiment can refer to the relevant description of method embodiments 200-500, and achieve the same or corresponding technical effects. To avoid repetition, it will not be repeated here.

[0210] The present application also provides a network-side device, including a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is configured to execute a program or instruction to implement the steps of the method embodiments shown in Figures 2-5. This network-side device embodiment corresponds to the aforementioned network-side device method embodiment, and each implementation process and implementation method of the aforementioned method embodiment is applicable to this network-side device embodiment and can achieve the same technical effects.

[0211] Specifically, an embodiment of the present application also provides a network-side device. As shown in Figure 10, the network-side device 1000 includes: an antenna 1001, a radio frequency device 1002, a baseband device 1003, a processor 1004, and a memory 1005. Antenna 1001 is connected to radio frequency device 1002. In the uplink direction, radio frequency device 1002 receives information via antenna 1001 and sends the received information to baseband device 1003 for processing. In the downlink direction, baseband device 1003 processes the information to be transmitted and sends it to radio frequency device 1002. Radio frequency device 1002 processes the received information and sends it through antenna 1001.

[0212] The method executed by the network-side device in the above embodiment may be implemented in the baseband device 1003 , which includes a baseband processor.

[0213] The baseband device 1003 may, for example, include at least one baseband board, on which multiple chips are arranged, as shown in Figure 10, one of which is, for example, a baseband processor, which is connected to the memory 1005 through a bus interface to call the program in the memory 1005 and execute the network device operations shown in the above method embodiment.

[0214] The network side device may further include a network interface 1006, which is, for example, a Common Public Radio Interface (CPRI).

[0215] Specifically, the network side device 1000 of the embodiment of the present application also includes: instructions or programs stored in the memory 1005 and can be run on the processor 1004. The processor 1004 calls the instructions or programs in the memory 1005 to execute the method of execution of each module shown in Figure 6 or Figure 7, and achieves the same technical effect. To avoid repetition, it will not be repeated here.

[0216] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned model monitoring method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0217] The processor is the processor in the terminal described in the above embodiment. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk. In some examples, the readable storage medium may be a non-transitory readable storage medium.

[0218] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned model monitoring method embodiment and achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0219] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0220] An embodiment of the present application further provides a computer program / program product, which is stored in a storage medium. The computer program / program product is executed by at least one processor to implement the various processes of the above-mentioned model monitoring method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0221] An embodiment of the present application also provides a wireless communication system, including: a first device and a second device, wherein the first device can be used to execute to implement the various processes of the above-mentioned model monitoring method embodiments 200-300, and the second device can be used to execute to implement the various processes of the above-mentioned model monitoring method embodiment 500, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0222] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0223] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of a computer software product plus a necessary general-purpose hardware platform, or of course, by hardware. The computer software product is stored in a storage medium (such as ROM, RAM, magnetic disk, optical disk, etc.) and includes a number of instructions for enabling a terminal or network-side device to execute the methods described in each embodiment of the present application.

[0224] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms of implementation methods without departing from the purpose of this application and the scope of protection of the claims. These implementation methods are all within the protection of this application.

Claims

1. A model monitoring method, wherein: include: The first device monitors the first artificial intelligence AI model according to a predetermined monitoring type; The predetermined monitoring type includes at least one of the following: A first monitoring type, wherein the first monitoring type is to monitor multiple AI models using the same monitoring indicator, wherein the first AI model belongs to the multiple AI models; The second monitoring type is to monitor the first AI model using dedicated monitoring indicators.

2. The method of claim 1, wherein: The method further comprises: receiving first information sent by a second device; The first information is used to configure relevant information of the first monitoring type.

3. The method of claim 2, wherein: The first information includes at least one of the following: A first monitoring indicator, used to indicate a monitoring indicator that the first device needs to send and can reflect the reasoning performance of the first AI model, wherein the first monitoring indicator corresponds to the first monitoring type; Monitoring indicator conversion information, used to convert a second monitoring indicator dedicated to monitoring the first AI model into the first monitoring indicator; Monitoring start conditions of the first AI model; Monitoring stop condition of the first AI model; a monitoring window of the first AI model; Monitoring resources of the first AI model; A monitoring type switching condition is used for the first device to switch between the first monitoring type and the second monitoring type.

4. The method of claim 3, wherein: The first monitoring indicator includes at least one of the following: A first value, the first value comprising a grading score, the grading score being a scoring result of the performance of the first AI model; The second value is used to indicate the number of times abnormal samples appear in the monitoring samples; The third value is used to indicate the proportion of abnormal samples in the monitoring samples; A fourth value is used to indicate the number of times a normal sample appears in the monitoring sample; The fifth value is used to indicate the proportion of normal samples in the monitored samples; Among them, the monitoring samples are used to monitor the first AI model.

5. The method of claim 4, wherein: The monitoring indicator conversion information includes at least one of a monitoring indicator conversion method and auxiliary information required to implement the monitoring indicator conversion.

6. The method of claim 5, wherein: The monitoring indicator conversion method includes at least one of the following: In the case where the first monitoring indicator includes the first value, mapping the second monitoring indicator to a grading score according to a preset grading score to obtain the first value; When the first monitoring indicator includes the second value or the third value, comparing the second monitoring indicator with the abnormal condition information, and determining the second value or the third value according to the comparison result; In the case where the first monitoring indicator includes the fourth value or the fifth value, the second monitoring indicator is compared with the normal condition information, and the fourth value or the fifth value is determined according to the comparison result.

7. The method according to any one of claims 1 to 6, wherein: The method further comprises: sending the second information or the third information to the second device; Among them, the second information is used to indicate the AI ​​model monitoring capability supported by the first device, and the third information is used to indicate whether the first device has the first monitoring capability, and the first monitoring capability supports monitoring multiple AI models using the same monitoring indicators.

8. The method of claim 7, wherein: The AI ​​model monitoring capability includes at least one of the following: A first monitoring capability, wherein the first monitoring capability supports monitoring the multiple AI models using the same monitoring indicators; The second monitoring capability supports the use of different monitoring indicators to monitor different AI models.

9. The method according to any one of claims 1 to 8, wherein: The method further comprises: Switch the monitoring type based on at least one of the following: Sending fourth information to the second device, where the fourth information is used to apply to the second device for switching from the third monitoring type to the fourth monitoring type; receiving fifth information sent by the second device, where the fifth information is used to instruct the first device to switch from the third monitoring type to the fourth monitoring type; According to a preconfigured monitoring type switching condition, switching from the third monitoring type to the fourth monitoring type; The third monitoring type is any one of the first monitoring type and the second monitoring type, and the fourth monitoring type is one of the first monitoring type and the second monitoring type except the third monitoring type.

10. The method according to any one of claims 1 to 9, wherein: The method further comprises: receiving seventh information sent by the second device; The monitoring result is sent according to the seventh information.

11. The method of claim 10, wherein: The seventh information includes at least one of the following: The type of the monitoring result, the type of the monitoring result comprising a monitoring result corresponding to the first monitoring type or a monitoring result corresponding to the second monitoring type; The conditions for sending the monitoring results; The transmission resources of the monitoring results.

12. A model monitoring method, wherein: include: The second device sends the first information to the first device; Among them, the first information is used to configure relevant information of the first monitoring type, and the first monitoring type is to monitor multiple AI models using the same monitoring indicators.

13. The method of claim 12, wherein: The first information includes at least one of the following: A first monitoring indicator, used to indicate a monitoring indicator that the first device needs to send and can reflect the reasoning performance of the first AI model, wherein the first monitoring indicator corresponds to the first monitoring type; Monitoring indicator conversion information, used to convert a second monitoring indicator dedicated to monitoring the first AI model into the first monitoring indicator; Monitoring start conditions of the first AI model; Monitoring stop condition of the first AI model; a monitoring window of the first AI model; Monitoring resources of the first AI model; A monitoring type switching condition is used for the first device to switch between the first monitoring type and the second monitoring type.

14. The method of claim 13, wherein: The first monitoring indicator includes at least one of the following: A first value, the first value comprising a grading score, the grading score being a scoring result of the performance of the first AI model; The second value is used to indicate the number of times abnormal samples appear in the monitoring samples; The third value is used to indicate the proportion of abnormal samples in the monitoring samples; A fourth value is used to indicate the number of times a normal sample appears in the monitoring sample; The fifth value is used to indicate the proportion of normal samples in the monitored samples; Among them, the monitoring samples are used to monitor the first AI model.

15. The method of claim 14, wherein: The monitoring indicator conversion information includes at least one of a monitoring indicator conversion method and auxiliary information required to implement the monitoring indicator conversion.

16. The method of claim 15, wherein: The monitoring indicator conversion method includes at least one of the following: In the case where the first monitoring indicator includes the first value, mapping the second monitoring indicator to a grading score according to a preset grading score to obtain the first value; When the first monitoring indicator includes the second value or the third value, comparing the second monitoring indicator with the abnormal condition information, and determining the second value or the third value according to the comparison result; In the case where the first monitoring indicator includes the fourth value or the fifth value, the second monitoring indicator is compared with the normal condition information, and the fourth value or the fifth value is determined according to the comparison result.

17. The method according to any one of claims 12 to 16, wherein: The method further comprises: receiving second information or third information sent by the first device; Among them, the second information is used to indicate the AI ​​model monitoring capability supported by the first device, and the third information is used to indicate whether the first device has the first monitoring capability, and the first monitoring capability supports monitoring multiple AI models using the same monitoring indicators.

18. The method of claim 17, wherein: The AI ​​model monitoring capability includes at least one of the following: A first monitoring capability, wherein the first monitoring capability supports monitoring the multiple AI models using the same monitoring indicators; The second monitoring capability supports the use of different monitoring indicators to monitor different AI models.

19. The method according to any one of claims 12 to 18, wherein: The method further comprises: receiving fourth information sent by the first device, where the fourth information is used to apply to the second device for switching from the third monitoring type to the fourth monitoring type; Sending fifth information to the first device, where the fifth information is used to instruct the first device to switch from the third monitoring type to the fourth monitoring type; Among them, the third monitoring type is any one of the first monitoring type and the second monitoring type, the fourth monitoring type is one of the first monitoring type and the second monitoring type except the third monitoring type, and the second monitoring type is to monitor the first AI model using dedicated monitoring indicators.

20. The method of any one of claims 12 to 19, wherein: The method further comprises: sending seventh information to the first device; The seventh information is used by the first device to send monitoring results.

21. The method of claim 20, wherein: The seventh information includes at least one of the following: The type of the monitoring result, the type of the monitoring result including a monitoring result corresponding to the first monitoring type or a monitoring result corresponding to a second monitoring type, wherein the second monitoring type is to monitor the first AI model using a dedicated monitoring indicator; The conditions for sending the monitoring results; The transmission resources of the monitoring results.

22. A model monitoring device, wherein: include: A monitoring module, used to monitor the first artificial intelligence AI model according to a predetermined monitoring type; The predetermined monitoring type includes at least one of the following: A first monitoring type, wherein the first monitoring type is to monitor multiple AI models using the same monitoring indicator, wherein the first AI model belongs to the multiple AI models; The second monitoring type is to monitor the first AI model using dedicated monitoring indicators.

23. A model monitoring device, wherein: include: A transmission module, configured to send first information to a first device; Among them, the first information is used to configure relevant information of the first monitoring type, and the first monitoring type is to monitor multiple AI models using the same monitoring indicators.

24. A communication device, wherein: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the method according to any one of claims 1 to 21 are implemented.

25. A readable storage medium, wherein: The readable storage medium stores a program or an instruction, and when the program or the instruction is executed by the processor, the steps as claimed in any one of claims 1 to 21 are implemented.

Images