Model monitoring method and device and communication equipment

Abstract

The invention discloses a model monitoring method and device and communication equipment, and belongs to the technical field of communication, and the model monitoring method comprises the steps that first equipment determines a monitoring result of a first AI model based on a first set; wherein the first set comprises at least one of a first input set and a first output set, the first input set is a part of the input of the first AI model, and the first output set is a part of the output of the first AI model.

Description

Technical Field

[0001] This application belongs to the field of communication technologies, and particularly relates to a model monitoring method, apparatus, and communication device. Background Art

[0002] With the continuous development of communication technologies, Artificial Intelligence (AI) has also been widely applied to various parts of communication systems to improve the performance of communication systems.

[0003] However, when performing AI model monitoring in communication-related technologies to ensure its performance, there are still problems such as high model monitoring overhead, which affects the performance of communication systems. Summary of the Invention

[0004] Embodiments of this application provide a model monitoring method, apparatus, and communication device, which can solve problems such as high model monitoring overhead and improve the performance of communication systems.

[0005] In a first aspect, a model monitoring method is provided, including: a first device determining a monitoring result of a first Artificial Intelligence (AI) model based on a first set; wherein, the first set includes at least one of a first input set and a first output set, the first input set is a part of the inputs of the first AI model, and the first output set is a part of the outputs of the first AI model.

[0006] In a second aspect, a model monitoring method is provided, including at least one of the following: a second device sending first information to a first device, where the first information is used to instruct the first device to perform monitoring of the first Artificial Intelligence (AI) model based on a first set; the second device receiving second information sent by the first device, where the second information is used to instruct the first device to expect to perform monitoring of the first AI model based on the first set; wherein, the first set includes at least one of a first input set and a first output set, the first input set is a part of the inputs of the first AI model, and the first output set is a part of the outputs of the first AI model.

[0007] In a third aspect, a model monitoring apparatus is provided, including: a monitoring module for determining a monitoring result of a first Artificial Intelligence (AI) model based on a first set; wherein, the first set includes at least one of a first input set and a first output set, the first input set is a part of the inputs of the first AI model, and the first output set is a part of the outputs of the first AI model.

[0008] Fourth aspect, a model monitoring device is provided, including: a transmission module for at least one of the following: sending first information to a first device, where the first information is used to instruct the first device to monitor the first artificial intelligence (AI) model based on a first set; receiving second information sent by the first device, where the second information is used to indicate that the first device expects to monitor the first AI model based on the first set; where the first set includes at least one of a first input set and a first output set, the first input set is a part of the inputs of the first AI model, and the first output set is a part of the outputs of the first AI model.

[0009] Fifth aspect, a communication device is provided, which includes a processor and a memory. The memory stores a program or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0010] Sixth aspect, a communication device is provided, including a processor and a communication interface. The communication interface is coupled to the processor, and the processor is used to run a program or instructions to implement the steps of the method described in the first aspect, or to implement the steps of the method described in the second aspect.

[0011] Seventh aspect, a readable storage medium is provided. A program or instructions are stored on the readable storage medium, and when the program or instructions are executed by a processor, the steps of the method described in the first aspect are implemented, or the steps of the method described in the second aspect are implemented.

[0012] Eighth aspect, a wireless communication system is provided, including: a terminal and a network-side device. The terminal can be used to execute the steps of the method described in the first aspect, and the network-side device can be used to execute the steps of the method described in the second aspect.

[0013] Ninth aspect, a chip is provided. The chip includes a processor and a communication interface. The communication interface is coupled to the processor, and the processor is used to run a program or instructions to implement the steps of the method described in the first aspect, or to implement the steps of the method described in the second aspect.

[0014] Tenth aspect, a computer program / program product is provided. The computer program / program product is stored in a storage medium, and the program / program product is executed by at least one processor to implement the steps of the method described in the first aspect, or to implement the steps of the method described in the... aspect.

[0015] In an embodiment of the present application, the first device determines the monitoring result of the first AI model based on at least one of a first input set and a first output set. The first input set is a part of the inputs of the first AI model, and the first output set is a part of the outputs of the first AI model. Thus, the model monitoring overhead can be effectively reduced, the model monitoring efficiency can be improved, and the performance of the communication system can be ensured. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 FIG. is a schematic structural diagram of a wireless communication system provided by an exemplary embodiment of the present application.

[0017] Figure 2 FIG. is one of the schematic flowcharts of a model monitoring method provided by an exemplary embodiment of the present application.

[0018] Figure 3 FIG. is another schematic flowchart of a model monitoring method provided by an exemplary embodiment of the present application.

[0019] Figure 4a FIG. is one of the schematic diagrams of an application scenario of a model monitoring method provided by an exemplary embodiment of the present application.

[0020] Figure 4b FIG. is another schematic diagram of an application scenario of a model monitoring method provided by an exemplary embodiment of the present application.

[0021] Figure 4c FIG. is one of the schematic interaction flowcharts of a model monitoring method provided by an exemplary embodiment of the present application.

[0022] Figure 4d FIG. is another schematic interaction flowchart of a model monitoring method provided by an exemplary embodiment of the present application.

[0023] Figure 4e FIG. is a third schematic interaction flowchart of a model monitoring method provided by an exemplary embodiment of the present application.

[0024] Figure 5 FIG. is a third schematic flowchart of a model monitoring method provided by an exemplary embodiment of the present application.

[0025] Figure 6 FIG. is one of the schematic structural diagrams of a model monitoring device provided by an exemplary embodiment of the present application.

[0026] Figure 7 FIG. is another schematic structural diagram of a model monitoring device provided by an exemplary embodiment of the present application.

[0027] Figure 8 FIG. is a schematic structural diagram of a communication device provided by an exemplary embodiment of the present application.

[0028] Figure 9 It is a schematic structural diagram of a terminal provided by an exemplary embodiment of the present application.

[0029] Figure 10 It is a schematic structural diagram of a network-side device provided by an exemplary embodiment of the present application. Detailed implementation manners

[0030] Next, the technical solutions in the embodiments of the present application will be clearly described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art belong to the scope of protection of the present application.

[0031] The terms "first", "second", etc. in the present application are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that such terms can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first" and "second" are usually of the same type, and the number of objects is not limited. For example, the first object can be one or multiple. In addition, "or" in the present application means at least one of the connected objects. For example, "A or B" covers three scenarios, namely, Scenario 1: including A and not including B; Scenario 2: including B and not including A; Scenario 3: including both A and B. The character " / " generally indicates an "or" relationship between the associated objects before and after.

[0032] The term "indication" in the present application can be either a direct indication (or an explicit indication) or an indirect indication (or an implicit indication). Among them, a direct indication can be understood as that the sender clearly informs the recipient of specific information, operations to be performed, or request results, etc. in the sent indication; an indirect indication can be understood as that the recipient determines the corresponding information according to the indication sent by the sender, or makes a judgment and determines the operations to be performed or request results, etc. according to the judgment result.

[0033] It should be noted that the technology described in the embodiments of this application is not limited to Long Term Evolution (LTE) / LTE-Advanced (LTE-A) systems, but can also be used in other wireless communication systems, such as Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal Frequency Division Multiple Access (OFDMA), Single-carrier Frequency-Division Multiple Access (SC-FDMA), or other systems. The terms "system" and "network" in the embodiments of this application are often used interchangeably, and the described technology can be used in the above-mentioned systems and radio technologies, as well as in other systems and radio technologies. The following description describes the New Radio (NR) system for example purposes, and the NR term is used in most of the following descriptions, but these technologies can also be applied to systems other than the NR system, such as the 6th th Generation (6G) communication system.

[0034] Figure 1A block diagram of a wireless communication system to which embodiments of the present application can be applied is shown. The wireless communication system includes a terminal 11 and a network-side device 12. Among them, the terminal 11 can be a mobile phone, a tablet personal computer, a laptop computer, a notebook computer, a personal digital assistant (PDA), a handheld computer, a netbook, an ultra-mobile personal computer (UMPC), a mobile internet device (MID), an augmented reality (AR), a virtual reality (VR) device, a robot, a wearable device, a flight vehicle, a vehicle user equipment (VUE), a shipborne device, a pedestrian user equipment (PUE), a smart home (home devices with wireless communication functions, such as refrigerators, TVs, washing machines or furniture, etc.), a game console, a personal computer (PC), a teller machine or a self-service machine, etc. Wearable devices include: smart watches, smart bracelets, smart earphones, smart glasses, smart jewelry (smart bracelets, smart bracelets, smart rings, smart necklaces, smart anklets, smart ankle chains, etc.), smart wristbands, smart clothing, etc. Among them, the vehicle user equipment can also be referred to as a vehicle terminal, a vehicle controller, a vehicle module, a vehicle component, a vehicle chip or a vehicle unit, etc. It should be noted that the specific type of the terminal 11 is not limited in the embodiments of the present application. The network-side device 12 can include an access network device or a core network device. Among them, the access network device can also be referred to as a radio access network (RAN) device, a radio access network function or a radio access network unit. The access network device can include a base station, a wireless local area network (WLAN) access point (AP) or a wireless fidelity (WiFi) node, etc.Among them, the base station can be referred to as Node B (NB), Evolved Node B (eNB), the next generation Node B (gNB), New Radio Node B (NR Node B), access point, Relay Base Station (RBS), Serving Base Station (SBS), Base Transceiver Station (BTS), radio base station, radio transceiver, Basic Service Set (BSS), Extended Service Set (ESS), home Node B (HNB), home evolved Node B, Transmission Reception Point (TRP), or some other suitable term in the art. As long as the same technical effect is achieved, the base station is not limited to specific technical terms. It should be noted that in the embodiments of this application, only the base station in the NR system is taken as an example for introduction, and the specific type of the base station is not limited.

[0035] It should be noted that Figure 1 The communication scenario shown in

[0036] a) The first device is Figure 1 the terminal 11 described in Figure 1 and the second device is

[0037] the network-side device 12 shown in Figure 1 b) The first device is Figure 1 the network-side device 12 described in

[0038] and the second device is

[0039] the terminal 11 shown in

[0040] In addition, the AI model mentioned in the context of the present application may also be referred to as an AI unit, an AI structure, etc. Alternatively, the AI model may also refer to a processing unit capable of implementing specific algorithms, formulas, processing flows, capabilities, etc. related to AI. Alternatively, the AI model may also be a processing method, algorithm, function, module, or unit for a specific data set. Alternatively, the AI model may be a processing method, algorithm, function, module, or unit running on AI-related hardware such as a Graphics Processing Unit (GPU), a Neural Network Processing Unit (NPU), a Tensor Processing Unit (TPU), or an Application Specific Integrated Circuit (ASIC). The present application does not make specific limitations in this regard. Optionally, the specific data set may include, but is not limited to, the input or output of the AI model.

[0041] Correspondingly, the subsequent mention of, for example, the first AI model can be described by an AI model identifier. Among them, the AI model identifier can be an AI unit identifier, an AI structure identifier, an AI algorithm identifier, a functionality ID, a physical identifier, a logical identifier, a global identifier, a local identifier, or an identifier of a specific data set associated with the AI model. It can also be an identifier of a specific scenario related to the AI, an environment identifier related to the AI model, a channel feature identifier related to the AI model, an identifier of a device related to the AI model. It can also be an identifier of a function, feature, ability, or module related to the AI. The present application does not make specific limitations in this regard.

[0042] Next, with reference to the accompanying drawings, the technical solutions provided in the embodiments of the present application will be described in detail through some embodiments and their application scenarios.

[0043] As Figure 2 shown, it is a schematic flowchart of a model monitoring method 200 provided by an exemplary embodiment of the present application. The method 200 may, but is not limited to, be executed by a first device, and specifically may be executed by hardware or software installed in the first device. In this embodiment, the method 200 may at least include the following steps.

[0044] S210, the first device determines the monitoring result of the first AI model based on the first set.

[0045] Among them, before the first device monitors the model, it can determine the AI model to be monitored from one or more configured AI models according to the model monitoring period, etc., such as the first AI model. In this embodiment, the first AI model can be used for calculations, predictions, evaluations, etc. of communication data, so as to improve the performance of the communication system.

[0046] Optionally, the first AI model can be, but is not limited to, neural networks, decision trees, support vector machines, Bayesian classifiers, etc. The neural network can be, but is not limited to, deep neural networks, convolutional neural networks, and recurrent neural networks, etc.

[0047] Based on this, in this embodiment, when determining the monitoring result of the first AI model or the model monitoring result, considering that if all the inputs (input) or all the outputs (output) of the first AI model are used to calculate the monitoring result, it will inevitably bring a high model monitoring overhead, such as large computational overhead, large resource overhead for obtaining true value information, large feedback resource overhead, etc. Moreover, for model monitoring scenarios where the performance of the AI model is relatively excellent, etc., using some of the inputs or some of the outputs of the first AI model to calculate the model monitoring result can also obtain good monitoring results. Therefore, in this embodiment, when determining the monitoring result of the first AI model, the monitoring result of the first AI model can be determined based on some of the inputs or some of the outputs of the first AI model, which can achieve the purpose of reducing the model monitoring overhead.

[0048] That is to say, the aforementioned first set can include, but is not limited to, at least one of the first input set and the first output set. Among them, the first input set is a part of the inputs of the first AI model. The first output set is a part of the outputs of the first AI model. Thus, while realizing the monitoring of the first AI model, it can also avoid the problem of large monitoring overhead when using all the inputs or outputs of the first AI model for model monitoring. Among them, the output of the aforementioned first AI model can also be called output information or output set, and the input of the first AI model can also be called input information or input set, which is not limited here.

[0049] Optionally, the first input set and the first output set can be implemented by means such as protocol agreement, high-level configuration, indication by other devices, and independent determination by the first device, etc., which is not limited here.

[0050] Of course, in some embodiments, the first set available for determining the monitoring result of the first AI model may be one or more. Among them, if there are multiple first sets, then the first device may select at least one from the multiple first sets according to pre-configured monitoring trigger conditions, where the selection conditions for different first sets are configured in the monitoring trigger conditions.

[0051] Optionally, the monitoring trigger condition may be an event condition, a periodic condition, etc. Among them, the event condition may be, but is not limited to, whether the communication performance index meets a predetermined threshold. For example, assume that the communication performance index is throughput, and there are 3 first sets configured on the first device, such as set 1, set 2, and set 3. Then, when the throughput is less than or equal to the first threshold and greater than the second threshold, set 1 may be selected; when the throughput is less than or equal to the second threshold and greater than the third threshold, set 2 may be selected; when the throughput is less than the third threshold, set 3 may be selected; the first threshold is greater than the second threshold, and the second threshold is greater than the third threshold.

[0052] The periodic condition may be, but is not limited to, different first sets corresponding to different model monitoring periods. For example, assume that there are 3 first sets configured on the first device, such as set 1, set 2, and set 3, and the model monitoring periods are T1, T2, and T3 respectively. Then, when the model monitoring period is T1, set 1 may be selected; when the model monitoring period is T2, set 2 may be selected; when the model monitoring period is T3, set 3 may be selected; T1 ≤ T2 ≤ T3.

[0053] It should be noted that if two or more first sets are selected for monitoring the first AI model, the first AI model can be monitored based on the two or more first sets simultaneously, or one can be randomly selected from the two or more first sets for monitoring the first AI model in one monitoring, that is, the two or more first sets can be used for two or more times of monitoring the first AI model, and there is no limitation here.

[0054] In this embodiment, the first device determines the monitoring result of the first AI model based on at least one of the first input set and the first output set. The first input set is a part of the input of the first AI model, and the first output set is a part of the output of the first AI model. Thus, while realizing the monitoring of the first AI model, it can effectively reduce the model monitoring overhead, improve the model monitoring efficiency, and ensure the performance of the communication system.

[0055] Such as Figure 3As shown, it is a schematic flowchart of a model monitoring method 300 provided by an exemplary embodiment of the present application. The method 300 may be, but is not limited to, executed by a first device, and specifically may be executed by hardware or software installed in the first device. In this embodiment, the method 300 may at least include the following steps.

[0056] S310, the first device determines the monitoring result of the first AI model based on the first set.

[0057] Wherein, the first set includes at least one of a first input set and a first output set. The first input set is a part of the inputs of the first AI model, and the first output set is a part of the outputs of the first AI model.

[0058] It can be understood that in addition to referring to the relevant descriptions in the method embodiment 200 for the implementation process of S310, as a possible implementation manner, there can also be multiple triggering manners for the first device to determine the monitoring result of the first AI model based on the first set, such as at least one of the following manners 1 - 2.

[0059] Manner 1: The second device sends a first message to the first device. Correspondingly, the first device receives the first message sent by the second device. The first message is used to instruct the first device to monitor the first AI model based on the first set.

[0060] That is to say, the first device can monitor the first AI model according to the instruction of the second device, such as determining the monitoring result of the first AI model based on the first set.

[0061] Optionally, the first message may directly or indirectly indicate the first set. Wherein, if it directly indicates the first set, then the first message may include the first set, that is, all the information of the first set; if it indirectly indicates the first set, then the first device and the second device may pre - agree on the first set corresponding to the first AI model and assign a set identifier to each first set. Then, when the first message includes the identifier of the first set, it can instruct the first device to monitor the first AI model based on the first set.

[0062] Manner 2: The first device sends a second message to the second device, where the second message is used to indicate that the first device expects to monitor the first AI model based on the first set.

[0063] That is, the first device cannot independently monitor the first AI model, but needs to request it from the second device, and the second device determines whether to agree to the first device to monitor the first AI model based on the first set.

[0064] Similar to the first information, the second information can also directly or indirectly indicate the first set. Among them, if it directly indicates the first set, then the first information may include the first set, that is, all the information of the first set; if it indirectly indicates the first set, then the first device and the second device can pre-agree on the first set corresponding to the first AI model and assign a set identifier to each first set. Then, when the first information includes the identifier of the first set, it can indicate that the first device monitors the first AI model based on the first set.

[0065] In some embodiments, when the first device sends the second information to the second device, the second device can also determine whether to allow the first set indicated by the second information to be used for monitoring the first AI model according to the second information, and send the third information to the first device. Correspondingly, the first device receives the third information sent by the second device; wherein, the third information is used to indicate whether to allow the first device to use the first set indicated by the second information to monitor the first AI model.

[0066] For example, if the third information indicates that it is allowed to use the first set indicated by the second information to monitor the first AI model, then the first device performs the step of determining the monitoring result of the first AI model based on the first set. Conversely, the first device can adopt other monitoring schemes to monitor the first AI model.

[0067] It should be noted that in addition to the foregoing methods 1 and 2, the first device can also independently determine to monitor the first AI model based on the first set, and after performing the foregoing steps, notify the second device to monitor the first AI model based on the first set to ensure that the first device and the second device have a consistent understanding of the monitoring process of the first AI model.

[0068] In some embodiments, there can also be multiple implementation manners for the first device to determine the monitoring result of the first AI model based on the first set as described above. For example, at least one of the following methods 1 - 4.

[0069] Method 1: Determine the monitoring result of the first AI model based on the data characteristics or data distribution in the first set.

[0070] For example, assume that the first set is the first input set. Then, the input information in the first input set can be statistically analyzed to calculate the data characteristics or data distribution information of the input information in the first input set, such as mean, mean vector, mean matrix, variance, variance vector, variance matrix, covariance, covariance vector, covariance matrix, maximum value, maximum value vector, maximum value matrix, minimum value, minimum value vector, minimum value matrix, etc. Then, the calculated data characteristics or data distribution information are compared with the applicable distribution information range or usage characteristic information range of the first AI model to obtain the monitoring result of the first AI model. Among them, the first AI model can be a model being executed, such as a model performing inference, an activated model, etc. The applicable distribution information range or applicable characteristic information range of the model refers to the information range corresponding to the model input information involved in training the first AI model.

[0071] It can be understood that the implementation process of the first output set is similar to that of the aforementioned first input set, and will not be elaborated here.

[0072] Method 2: Determine the model performance result of the first AI model based on the first output set, and use the model performance result of the first AI model as the monitoring result of the first AI model.

[0073] In some embodiments, the process by which the first device determines the model performance result of the first AI model based on the first output set may include: obtaining the true value information corresponding to the first output set based on the first resource set, and then determining the model performance result based on the first AI model according to the comparison result between the first output set and the true value information, such as error type metrics, accuracy type metrics, etc.

[0074] Among them, the resources in the first resource set correspond one-to-one with the information in the first output set. Thus, compared with the need to feedback all the inputs or the true value information of the inputs corresponding to the first AI model, in this embodiment, by only feedbacking the first resource set and the true value information corresponding to the first output set, the acquisition or feedback overhead of the true value resources can be reduced.

[0075] Optionally, the first resource set can be implemented by means such as protocol agreement, high-level configuration, and indication from the second device. For example, assume that the first resource set is indicated by the second device. Then, the second device can send the fourth information to the first device. Correspondingly, the first device receives the fourth information sent by the second device; among them, the fourth information is used to apply for and recommend indicating on which resources such as time, frequency, and port the second device sends the true value information. Based on this, in this embodiment, the fourth information can include, but is not limited to, the first resource set.

[0076] In addition, the true value information may be a reference signal or dedicated information dedicated to obtaining the true value information, etc.

[0077] Method 3: The communication system performance result obtained by applying the model output determined by the first output set to the communication system is used as the monitoring result of the first AI model.

[0078] For example, the current communication system performance can be statistically analyzed or calculated according to the first output set, such as throughput, spectral efficiency, signal-to-interference plus noise ratio (SINR), signal-to-noise ratio (SNR), bit error rate, block error rate, packet loss rate, transmission rate (up / down), peak rate (up / down), etc. Then, the current communication system performance obtained by statistical analysis or calculation is compared with a preset threshold to obtain the monitoring result.

[0079] Method 4: The monitoring result of the first AI model is determined according to the comparison result between the first result and the second result, where the first result is determined based on the first set and the first AI model, and the second result is determined based on the first set and other communication algorithms other than the first AI model. The first AI model and the other communication algorithms are used for the calculation, evaluation, or prediction of the same communication function. The other communication algorithms may be communication algorithms implemented based on AI models or non-AI algorithms, etc., which are not limited herein.

[0080] It should be noted that the first result and the second result may be model performance results or communication system performance results. That is, the comparison result may be a comparison result corresponding to the model performance result or the communication system performance result, which is not limited herein.

[0081] In some embodiments, after the first device determines the monitoring result of the first AI model based on the first set, the first device may send fifth information to the second device according to the monitoring result of the first AI model. The fifth information includes at least one of a second set and a second resource set recommended by the first device according to the monitoring result of the first AI model and used for the monitoring of the first AI model.

[0082] Among them, the second set is different from the first set, and the second set may include, but is not limited to, at least one of a second input set and a second output set. The second input set is a part of the input of the first AI model, and the second output set is a part of the output of the first AI model.

[0083] That is to say, the first device can recommend new second sets, second resource sets, etc. for the monitoring results of the first AI model according to the monitoring results of the first AI model, so as to realize the dynamic adjustment of the input set or output set for the monitoring of the first AI model, and further realize the optimization of the monitoring performance of the first AI model and improve the performance of the communication system.

[0084] Correspondingly, after receiving the fifth information, the second device can determine whether to allow the first device to continue to determine the monitoring results of the first AI model based on the second set, or send true value information according to the second resource set, etc.

[0085] In this embodiment, a monitoring solution for an AI model based on a first set is further provided through simple signaling interaction, which can reduce the computational complexity and latency of AI model monitoring, reduce the measurement resource overhead and feedback overhead of true value information, and improve the efficiency of model monitoring.

[0086] In addition, the model monitoring method provided in this application can be applied to communication systems with wireless AI functions such as 5.5G or 6G, but is not limited to this. For this, several common scenarios are provided below.

[0087] Scenario 1: Channel State Information (CSI) prediction scenario based on an AI model

[0088] For a multi-antenna communication transmission system with N transmit antennas, M receive antennas, K resource blocks (RBs), and L subbands, assuming that CSI prediction based on an AI model can predict CSI at T future time points. Then, determining the monitoring results of the first AI model based on the first set may include: selecting the CSI of N' transmit antennas, M' receive antennas, K' RBs, and L' subbands at T' future time points to calculate the monitoring results of the first AI model, where T'≤T, N'≤N, M'≤M, K'≤K, and L'≤L.

[0089] Scenario 2: CSI compression feedback scenario based on an AI model

[0090] CSI compression based on an AI model realizes the compression and reconstruction of multi-dimensional channel information through AI technology, thereby reducing the CSI feedback overhead on the air interface or improving the recovery accuracy of network-side CSI. As Figure 4a shown, its mainstream solution is implemented based on an encoder-decoder neural network structure. The encoder neural network deployed on the terminal side compresses the CSI to generate a bitstream for feedback; the decoder neural network deployed on the network side decodes the received feedback bitstream to reconstruct the CSI.

[0091] Similar to CSI prediction, in the results of CSI compressed feedback, part of the transceiver antennas and subbands can also be selected to calculate the monitoring results of the AI model.

[0092] Scenario 3: Scenario of channel estimation based on the Demodulation Reference Signal (DMRS) implemented by the AI model

[0093] As Figure 4b shown, the channel estimation result at the DMRS resource can be used as the input of the AI model, and the channel estimation on all resources can be used as the output of the AI model to implement channel estimation. Since the correlation relationship of the channels on different transmission resources generally has non-linear characteristics, AI can achieve higher estimation accuracy or reduce the reference signal overhead compared with traditional interpolation methods.

[0094] Then, when determining the monitoring results of the AI model, a part of the output can be selected from the output of the AI model, that is, the channel estimation on all resources, to calculate the monitoring metrics of the AI model.

[0095] Based on the descriptions of the foregoing method embodiments 200-300, the implementation process of the model monitoring scheme provided in this application will be further described exemplarily below in conjunction with Examples 1-3. Among them, it is assumed that the first device is a terminal and the second device is a network-side device.

[0096] Example 1

[0097] S411, as Figure 4c shown, the network-side device sends the sixth information to the terminal, and the A first output sets (A≥1) or B first input sets (B≥1) for the first AI model monitoring are described in the sixth information.

[0098] S412, the terminal selects one first output set from the A first output sets and one first input set from the B first input sets based on the pre-configured monitoring trigger condition, that is, the terminal determines the monitoring results of the first AI model based on the selected one first input set or first output combination.

[0099] S413, the terminal sends the second information to the network-side device, where the second information is used to indicate that the first device expects to monitor the first AI model based on the first set.

[0100] S414, the network-side device sends the third information to the terminal according to the second information, where the third information is used to indicate whether to allow the first device to use the first set indicated by the second information to monitor the first AI model.

[0101] S415. When the third information indicates that it is allowed to use the first set indicated by the second information for monitoring the first AI model, the first device determines the monitoring result of the first AI model based on the first set.

[0102] Example 2

[0103] S421. As Figure 4d shown, the network-side device sends sixth information to the terminal, and A first output sets (A≥1) or B first input sets (B≥1) for monitoring the first AI model are described in the sixth information.

[0104] S422. The network-side device sends second information to the terminal, and the second information is used to indicate that the first device expects to monitor the first AI model based on the first set.

[0105] S423. The first device determines the monitoring result of the first AI model based on the first set.

[0106] S424. The first device sends fifth information to the second device according to the monitoring result of the first AI model. Among them, the fifth information includes at least one of the second set and the second resource set recommended by the first device according to the monitoring result of the first AI model and used for monitoring the first AI model.

[0107] Example 3

[0108] S431. As Figure 4e shown, the network-side device sends sixth information to the terminal, and A first output sets (A≥1) or B first input sets (B≥1) for monitoring the first AI model are described in the sixth information.

[0109] S432. The terminal selects one first output set from the A first output sets and one first input set from the B first input sets based on pre-configured monitoring trigger conditions, that is, the terminal determines the monitoring result of the first AI model based on the selected one first input set or first output combination.

[0110] S433. The terminal sends second information to the network-side device, where the second information is used to indicate that the first device expects to monitor the first AI model based on the first set.

[0111] S434. If the first device determines the model performance result of the first AI model based on the first output set, then the network-side device sends fourth information to the terminal according to the second information, where the fourth information includes a first resource set for obtaining ground-truth information. And the network-side device sends third information to the terminal according to the second information, where the third information is used to indicate whether to allow the first device to use the first set indicated by the second information for monitoring the first AI model.

[0112] S435. In the case where the third information indicates that it is allowed to use the first set indicated by the second information for monitoring the first AI model, the first device obtains ground-truth information based on the first resource set, and determines the monitoring result of the first AI model based on the first set and the ground-truth information.

[0113] It can be understood that the model monitoring methods provided in the foregoing Examples 1-3 may be but are not limited to the foregoing steps, such as may include more or fewer steps than the foregoing. In addition, the implementation process of the model monitoring methods provided in the foregoing Examples 1-3 may refer to the relevant descriptions in the foregoing Method Embodiments 200-300, and achieve the same or corresponding technical effects. To avoid repetition, no limitation is made here.

[0114] As Figure 5 shown, it is a schematic flowchart of a model monitoring method 500 provided by an exemplary embodiment of the present application. The method 500 may be but is not limited to being executed by a second device, and may specifically be executed by hardware or software installed in the second device. In this embodiment, the method 500 may at least include the following steps.

[0115] S510. The second device performs a first operation.

[0116] Wherein, the first operation includes at least one of the following: sending first information to the first device, where the first information is used to instruct the first device to monitor the first artificial intelligence (AI) model based on a first set; receiving second information sent by the first device, where the second information is used to indicate that the first device expects to monitor the first AI model based on the first set; wherein, the first set includes at least one of a first input set and a first output set, the first input set is a part of the inputs of the first AI model, and the first output set is a part of the outputs of the first AI model.

[0117] Optionally, the first information or the second information includes at least one of the following: the first set; an identifier of the first set.

[0118] Optionally, the method further includes: sending third information to the first device; wherein the third information is used to indicate whether the first device is allowed to use the first set indicated by the second information to monitor the first AI model.

[0119] Optionally, the method further includes: sending fourth information to the first device; wherein the fourth information includes a first resource set, and the first resource set is used for the first device to obtain true value information corresponding to the first output set.

[0120] Optionally, the method further includes: receiving fifth information sent by the first device; wherein the fifth information includes at least one of a second set and a second resource set recommended by the first device according to the monitoring result of the first AI model and used for monitoring the first AI model.

[0121] It can be understood that each implementation manner in method embodiment 500 has the same or corresponding technical features as those in the foregoing method embodiments 200-300. Therefore, for each implementation manner in method embodiment 500, reference may be made to the relevant descriptions in the foregoing method embodiments 200-300, and the same or corresponding technical effects can be achieved. To avoid repetition, details are not described herein again.

[0122] In the model monitoring method 200-500 provided in the embodiments of the present application, the execution subject may be a model monitoring device. In the embodiments of the present application, taking the model monitoring device executing the model monitoring method 200-500 as an example, the model monitoring device provided in the embodiments of the present application is described.

[0123] As Figure 6 shown, the following is a schematic structural diagram of a model monitoring device 600 provided in an embodiment of the present application. The device 600 includes: a monitoring module 610, configured to determine a monitoring result of a first AI model based on a first set; wherein the first set includes at least one of a first input set and a first output set, the first input set is a part of the input of the first AI model, and the first output set is a part of the output of the first AI model.

[0124] Optionally, the device further includes a transmission module, configured to perform at least one of the following: receiving first information sent by a second device, wherein the first information is used to instruct the first device to monitor the first AI model based on the first set; sending second information to the second device, wherein the second information is used to instruct the first device to expect to monitor the first AI model based on the first set.

[0125] Optionally, the first information or the second information includes at least one of the following: the first set; an identifier of the first set.

[0126] Optionally, the transmission module is further configured to receive third information sent by the second device when the first device sends second information to the second device; wherein the third information is used to indicate whether the first device is allowed to use the first set indicated by the second information to monitor the first AI model.

[0127] Optionally, the monitoring module 610 determines the monitoring result of the first AI model based on the first set, including any of the following: determining the monitoring result of the first AI model based on the data characteristics or data distribution in the first set; determining the model performance result of the first AI model based on the first output set, and using the model performance result of the first AI model as the monitoring result of the first AI model; using the communication system performance result obtained by applying the model output determined by the first output set to the communication system as the monitoring result of the first AI model; determining the monitoring result of the first AI model according to the comparison result between the first result and the second result, wherein the first result is determined based on the first set and the first AI model, the second result is determined based on the first set and other communication algorithms other than the first AI model, and the first AI model and the other communication algorithms are used for the calculation, evaluation or prediction of the same communication function.

[0128] Optionally, the monitoring module 610 determines the model performance result of the first AI model based on the first output set, including: obtaining the true value information corresponding to the first output set based on the first resource set, where the resources in the first resource set correspond one-to-one to the information in the first output set; determining the model performance result of the first AI model according to the comparison result between the first output set and the true value information.

[0129] Optionally, the transmission module is further configured to receive fourth information sent by the second device; wherein the fourth information includes the first resource set.

[0130] Optionally, if there are multiple first sets, the monitoring module 610 determines the monitoring result of the first AI model based on the first sets, including: selecting at least one of the multiple first sets for monitoring the first AI model according to the pre-configured monitoring trigger condition, where the selection conditions of different first sets are configured in the monitoring trigger condition.

[0131] Optionally, the transmission module is further configured to send fifth information to the second device according to the monitoring result of the first AI model; wherein the fifth information includes at least one of the second set and the second resource set recommended by the first device according to the monitoring result of the first AI model and used for monitoring the first AI model.

[0132] The model monitoring device 600 in the embodiments of the present application may be a communication device, such as a communication device with an operating system, or a component in a communication device, such as an integrated circuit or a chip. The communication device may be a terminal, a network-side device, or other devices other than terminals and network-side devices. Exemplarily, the terminal may include, but is not limited to, the types of the terminal 11 listed above, the network-side device may include, but is not limited to, the types of the network-side device 12 listed above, and other devices may be servers, Network Attached Storage (NAS), etc., which are not specifically limited in the embodiments of the present application.

[0133] The model monitoring device 600 provided in the embodiments of the present application can implement Figures 2 to 3 each process implemented by the method embodiments and achieve the same technical effects. To avoid repetition, it will not be elaborated here.

[0134] As Figure 7 shown, it is a schematic structural diagram of a model monitoring device 700 provided in an embodiment of the present application. The device 700 includes: a transmission module 710, configured to perform at least one of the following: send first information to a first device, where the first information is used to instruct the first device to monitor the first artificial intelligence (AI) model based on a first set; receive second information sent by the first device, where the second information is used to indicate that the first device expects to monitor the first AI model based on the first set; where the first set includes at least one of a first input set and a first output set, the first input set is a part of the inputs of the first AI model, and the first output set is a part of the outputs of the first AI model.

[0135] Optionally, the first information or the second information includes at least one of the following: the first set; an identifier of the first set.

[0136] Optionally, the transmission module 710 is further configured to send third information to the first device; where the third information is used to indicate whether to allow the first device to use the first set indicated by the second information to monitor the first AI model.

[0137] Optionally, the transmission module 710 is further configured to send fourth information to the first device; where the fourth information includes a first resource set, and the first resource set is used for the first device to obtain true value information corresponding to the first output set.

[0138] Optionally, the transmission module 710 is further configured to receive fifth information sent by the first device; wherein, the fifth information includes at least one of a second set and a second resource set recommended by the first device according to the monitoring result of the first AI model and used for monitoring by the first AI model.

[0139] The model monitoring device 700 in the embodiments of the present application may be a communication device, such as a communication device with an operating system, or a component in a communication device, such as an integrated circuit or a chip. The communication device may be a terminal, a network-side device, or other devices other than terminals and network-side devices. Exemplarily, the terminal may include, but is not limited to, the types of the above-listed terminal 11, the network-side device may include, but is not limited to, the types of the above-listed network-side device 12, and other devices may be servers, Network Attached Storage (NAS), etc., which are not specifically limited in the embodiments of the present application.

[0140] The model monitoring device 700 provided by the embodiments of the present application can implement Figure 5 each process implemented by the method embodiments and achieve the same technical effects. To avoid repetition, details are not described herein again.

[0141] As Figure 8 shown, the embodiments of the present application further provide a communication device 800, including a processor 801 and a memory 802. A program or instruction that can run on the processor 801 is stored on the memory 802. For example, when the communication device 800 is a terminal, when the program or instruction is executed by the processor 801, it implements each step of the above model monitoring method embodiment and can achieve the same technical effects. When the communication device 800 is a network-side device, when the program or instruction is executed by the processor 801, it implements each step of the above model monitoring method embodiment and can achieve the same technical effects. To avoid repetition, details are not described herein again.

[0142] The embodiments of the present application further provide a terminal, including a processor and a communication interface. The communication interface is coupled to the processor, and the processor is configured to run a program or instruction to implement the steps in the method embodiments as shown in Figure 2 、 Figure 3 or Figure 5 shown. This terminal embodiment corresponds to the above terminal-side method embodiment. Each implementation process and implementation manner of the above method embodiment can be applied to this terminal embodiment and can achieve the same technical effects. Specifically, Figure 9 is a schematic diagram of the hardware structure of a terminal for implementing the embodiments of the present application.

[0143] The terminal 900 includes, but is not limited to, at least some components such as a radio frequency unit 901, a network module 902, an audio output unit 903, an input unit 904, a sensor 905, a display unit 906, a user input unit 907, an interface unit 908, a memory 909, and a processor 910.

[0144] Those skilled in the art can understand that the terminal 900 may further include a power supply (such as a battery) for supplying power to each component. The power supply can be logically connected to the processor 910 through a power management system, so as to implement functions such as management of charging, discharging, and power consumption management through the power management system. Figure 9 The terminal structure shown does not limit the terminal. The terminal may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0145] It should be understood that in the embodiments of the present application, the input unit 904 may include a graphics processing unit (GPU) 9041 and a microphone 9042. The graphics processor 9041 processes the image data of static pictures or videos obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 906 may include a display panel 9061, and the display panel 9061 may be configured in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit 907 includes at least one of a touch panel 9071 and other input devices 9072. The touch panel 9071 is also called a touch screen. The touch panel 9071 may include two parts: a touch detection device and a touch controller. The other input devices 9072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be elaborated here.

[0146] In the embodiments of the present application, after the radio frequency unit 901 receives downlink data from a network-side device, it can be transmitted to the processor 910 for processing; in addition, the radio frequency unit 901 can send uplink data to the network-side device. Generally, the radio frequency unit 901 includes, but is not limited to, an antenna, an amplifier, a transceiver, a coupler, a low-noise amplifier, a duplexer, etc.

[0147] The memory 909 can be used to store software programs or instructions and various data. The memory 909 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data. Among them, the first storage area may store an operating system, application programs or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 909 may include volatile memory or non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDR SDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM), and a direct rambus random access memory (DRRAM). The memory 909 in the embodiments of the present application includes but is not limited to these and any other suitable types of memory.

[0148] The processor 910 may include one or more processing units; optionally, the processor 910 integrates an application processor and a modem processor. Among them, the application processor mainly processes operations related to the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communication signals, such as a baseband processor. It can be understood that the above modem processor may not be integrated into the processor 910 either.

[0149] Among them, as a possible implementation manner, the processor 910 is used to determine the monitoring result of the first artificial intelligence (AI) model based on a first set; where the first set includes at least one of a first input set and a first output set, the first input set is a part of the inputs of the first AI model, and the first output set is a part of the outputs of the first AI model.

[0150] Optionally, the device further includes a radio frequency unit 901 configured to perform at least one of the following: receiving first information sent by a second device, where the first information is used to instruct the first device to monitor the first AI model based on the first set; sending second information to the second device, where the second information is used to instruct the first device to expect to monitor the first AI model based on the first set.

[0151] Optionally, the first information or the second information includes at least one of the following: the first set; an identifier of the first set.

[0152] Optionally, the radio frequency unit 901 is further configured to receive third information sent by the second device when the first device sends the second information to the second device; where the third information is used to indicate whether to allow the first device to use the first set indicated by the second information to monitor the first AI model.

[0153] Optionally, the processor 910 determines a monitoring result of the first AI model based on a first set, including any one of the following: determining the monitoring result of the first AI model based on data features or data distribution in the first set; determining a model performance result of the first AI model based on the first output set, and using the model performance result of the first AI model as the monitoring result of the first AI model; using a communication system performance result obtained by applying a model output determined by the first output set to a communication system as the monitoring result of the first AI model; determining the monitoring result of the first AI model according to a comparison result between a first result and a second result, where the first result is determined based on the first set and the first AI model, the second result is determined based on the first set and other communication algorithms other than the first AI model, and the first AI model and the other communication algorithms are used for calculation, evaluation, or prediction of the same communication function.

[0154] Optionally, the processor 910 determines a model performance result of the first AI model based on the first output set, including: obtaining true value information corresponding to the first output set based on a first resource set, where resources in the first resource set correspond one-to-one to information in the first output set; determining the model performance result of the first AI model according to a comparison result between the first output set and the true value information.

[0155] Optionally, the radio frequency unit 901 is further configured to receive fourth information sent by the second device; where the first resource set is included in the fourth information.

[0156] Optionally, if there are multiple first sets, the processor 910 determines the monitoring result of the first AI model based on the first sets, including: selecting at least one of the multiple first sets for monitoring the first AI model according to a pre-configured monitoring trigger condition, where the selection conditions for different first sets are configured in the monitoring trigger condition.

[0157] Optionally, the radio frequency unit 901 is further configured to send fifth information to a second device according to the monitoring result of the first AI model; where the fifth information includes at least one of a second set and a second resource set recommended by the first device according to the monitoring result of the first AI model and used for monitoring the first AI model.

[0158] As another possible implementation, the radio frequency unit 901 is configured to perform at least one of the following: send first information to a first device, where the first information is used to instruct the first device to monitor the first artificial intelligence (AI) model based on a first set; receive second information sent by the first device, where the second information is used to instruct the first device to expect to monitor the first AI model based on the first set; where the first set includes at least one of a first input set and a first output set, the first input set is a part of the input of the first AI model, and the first output set is a part of the output of the first AI model.

[0159] Optionally, the first information or the second information includes at least one of the following: the first set; an identifier of the first set.

[0160] Optionally, the radio frequency unit 901 is further configured to send third information to the first device; where the third information is used to indicate whether the first device is allowed to use the first set indicated by the second information to monitor the first AI model.

[0161] Optionally, the radio frequency unit 901 is further configured to send fourth information to the first device; where the fourth information includes a first resource set, and the first resource set is used for the first device to obtain true value information corresponding to the first output set.

[0162] Optionally, the radio frequency unit 901 is further configured to receive fifth information sent by the first device; where the fifth information includes at least one of a second set and a second resource set recommended by the first device according to the monitoring result of the first AI model and used for monitoring the first AI model.

[0163] It can be understood that the implementation processes of the various implementation manners mentioned in this embodiment can refer to the relevant descriptions of Method Embodiments 200-500, and achieve the same or corresponding technical effects. To avoid repetition, they will not be elaborated here.

[0164] An embodiment of the present application further provides a network-side device, including a processor and a communication interface, where the communication interface is coupled to the processor, and the processor is configured to run a program or an instruction to implement the steps of the method embodiment as Figures 2 - 5 shown. This embodiment of the network-side device corresponds to the above-mentioned method embodiment of the network-side device. Each implementation process and implementation manner of the above method embodiment can be applied to this embodiment of the network-side device, and the same technical effects can be achieved.

[0165] Specifically, an embodiment of the present application further provides a network-side device. As Figure 10 shown, the network-side device 1000 includes: an antenna 1001, a radio frequency device 1002, a baseband device 1003, a processor 1004, and a memory 1005. The antenna 1001 is connected to the radio frequency device 1002. In the uplink direction, the radio frequency device 1002 receives information through the antenna 1001 and sends the received information to the baseband device 1003 for processing. In the downlink direction, the baseband device 1003 processes the information to be sent and sends it to the radio frequency device 1002. After processing the received information, the radio frequency device 1002 sends it out through the antenna 1001.

[0166] The method executed by the network-side device in the above embodiments can be implemented in the baseband device 1003, and the baseband device 103 includes a baseband processor.

[0167] The baseband device 1003 may include, for example, at least one baseband board, and a plurality of chips are provided on the baseband board. As Figure 10 shown, one of the chips is, for example, a baseband processor, which is connected to the memory 1005 through a bus interface to call the program in the memory 1005 and execute the operations of the network device shown in the above method embodiments.

[0168] The network-side device may further include a network interface 1006, and this interface is, for example, a Common Public Radio Interface (CPRI).

[0169] Specifically, the network-side device 1000 in the embodiment of the present application further includes: instructions or programs stored on the memory 1005 and executable on the processor 1004. The processor 1004 calls the instructions or programs in the memory 1005 to execute Figure 6 or Figure 7 the methods executed by the respective modules shown, and achieve the same technical effects. To avoid repetition, they will not be elaborated here.

[0170] The embodiment of the present application further provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, it implements each process of the above-mentioned embodiment of the model monitoring method and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0171] Wherein, the processor is the processor in the terminal described in the above embodiment. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk, or an optical disc, etc. In some examples, the readable storage medium may be a non-transitory readable storage medium.

[0172] The embodiment of the present application further provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run a program or instruction to implement each process of the above-mentioned embodiment of the model monitoring method and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0173] It should be understood that the chip mentioned in the embodiment of the present application may also be referred to as a system-on-chip, a system chip, a chip system, or a system-on-chip, etc.

[0174] The embodiment of the present application further provides a computer program / program product, which is stored in a storage medium. The computer program / program product is executed by at least one processor to implement each process of the above-mentioned embodiment of the model monitoring method and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0175] The embodiment of the present application further provides a wireless communication system, including: a first device and a second device. The first device can be used to execute to implement each process of the above-mentioned embodiment 200-300 of the model monitoring method, and the second device can be used to execute to implement each process of the above-mentioned embodiment 500 of the model monitoring method and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0176] It should be noted that in this text, the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the phrase "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising such element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, but may also include performing functions in a substantially simultaneous manner or in the reverse order according to the functions involved. For example, the described methods may be performed in an order different from that described, and various steps may be added, omitted or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0177] From the description of the above embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of computer software products plus the necessary general hardware platforms, and of course, can also be implemented by hardware. The computer software products are stored in storage media (such as ROM, RAM, magnetic disks, optical disks, etc.) and include several instructions for causing a terminal or a network-side device to execute the methods described in the various embodiments of the present application.

[0178] The embodiments of the present application have been described above in conjunction with the accompanying drawings, but the present application is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not restrictive. Under the inspiration of the present application, those of ordinary skill in the art can also make many forms of embodiments without departing from the purpose of the present application and the scope protected by the claims. These embodiments are all within the protection scope of the present application.

Claims

1. A model monitoring method, characterized in that, it includes: A first device determines a monitoring result of a first artificial intelligence (AI) model based on a first set; wherein, the first set includes at least one of a first input set and a first output set, the first input set is a part of the inputs of the first AI model, and the first output set is a part of the outputs of the first AI model.

2. The method according to claim 1, characterized in that, the method further includes at least one of the following: The first device receives first information sent by a second device, wherein the first information is used to instruct the first device to monitor the first AI model based on the first set; The first device sends second information to the second device, wherein the second information is used to indicate that the first device expects to monitor the first AI model based on the first set.

3. The method according to claim 2, characterized in that, the first information or the second information includes at least one of the following: the first set; an identifier of the first set.

4. The method according to claim 2, characterized in that, in the case where the first device sends the second information to the second device, the method further includes: receiving third information sent by the second device; wherein the third information is used to indicate whether to allow the first device to use the first set indicated by the second information to monitor the first AI model.

5. The method according to any one of claims 1-4, characterized in that, determining the monitoring result of the first AI model based on the first set includes at least one of the following: determining the monitoring result of the first AI model based on data characteristics or data distribution in the first set; determining a model performance result of the first AI model based on the first output set, and using the model performance result of the first AI model as the monitoring result of the first AI model; using the communication system performance result obtained by applying the model output determined by the first output set to a communication system as the monitoring result of the first AI model; determining the monitoring result of the first AI model according to the comparison result between a first result and a second result, wherein the first result is determined based on the first set and the first AI model, the second result is determined based on the first set and other communication algorithms other than the first AI model, and the first AI model and the other communication algorithms are used for calculation, evaluation or prediction of the same communication function.

6. The method according to claim 5, characterized in that, determining the model performance result of the first AI model based on the first output set includes: obtaining true value information corresponding to the first output set based on a first resource set, wherein the resources in the first resource set correspond one-to-one with the information in the first output set; determining the model performance result based on the first AI model according to the comparison result between the first output set and the true value information.

7. The method according to claim 6, characterized in that, the method further includes: Receive the fourth information sent by the second device; wherein, the first resource set is included in the fourth information.

8. The method according to any one of claims 1-7, characterized in that if there are multiple first sets, determining the monitoring result of the first AI model based on the first sets includes: selecting at least one of the multiple first sets for monitoring the first AI model according to a pre-configured monitoring trigger condition, wherein the selection conditions of different first sets are configured in the monitoring trigger condition.

9. The method according to any one of claims 1-8, characterized in that the method further includes: sending fifth information to the second device according to the monitoring result of the first AI model; wherein, the fifth information includes at least one of a second set and a second resource set recommended by the first device according to the monitoring result of the first AI model and used for monitoring the first AI model.

10. A model monitoring method, characterized in that including at least one of the following: The second device sends first information to the first device, wherein the first information is used to instruct the first device to monitor the first artificial intelligence (AI) model based on a first set; The second device receives second information sent by the first device, wherein the second information is used to instruct the first device to expect to monitor the first AI model based on the first set; wherein, the first set includes at least one of a first input set and a first output set, the first input set is a part of the input of the first AI model, and the first output set is a part of the output of the first AI model.

11. The method according to claim 10, characterized in that the first information or the second information includes at least one of the following: the first set; the identifier of the first set.

12. The method according to claim 10 or 11, characterized in that the method further includes: sending third information to the first device; wherein, the third information is used to indicate whether the first device is allowed to use the first set indicated by the second information to monitor the first AI model.

13. The method according to any one of claims 10-12, characterized in that the method further includes: sending fourth information to the first device; wherein, the fourth information includes a first resource set, and the first resource set is used for the first device to obtain true value information corresponding to the first output set.

14. The method according to any one of claims 10-13, characterized in that the method further includes: receiving fifth information sent by the first device; wherein, the fifth information includes at least one of a second set and a second resource set recommended by the first device according to the monitoring result of the first AI model and used for monitoring the first AI model.

15. A model monitoring device, characterized in that includes: a monitoring module, configured to determine a monitoring result of a first artificial intelligence (AI) model based on a first set; Wherein, the first set includes at least one of a first input set and a first output set, the first input set is a part of the inputs of the first AI model, and the first output set is a part of the outputs of the first AI model.

16. A model monitoring device, Characterized in that, Comprising: A transmission module for at least one of the following: Sending first information to a first device, where the first information is used to instruct the first device to monitor a first artificial intelligence (AI) model based on a first set; Receiving second information sent by the first device, where the second information is used to indicate that the first device expects to monitor the first AI model based on the first set; Wherein, the first set includes at least one of a first input set and a first output set, the first input set is a part of the inputs of the first AI model, and the first output set is a part of the outputs of the first AI model.

17. A communication device, Characterized in that, Comprising a processor and a memory, the memory stores a program or instructions that can run on the processor, and when the program or instructions are executed by the processor, the steps of the method according to any one of claims 1 to 9 are implemented, or the steps of the method according to any one of claims 10 to 14 are implemented.

18. A readable storage medium, Characterized in that, The readable storage medium stores a program or instructions, and when the program or instructions are executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented, or the steps of the method according to any one of claims 10 to 14 are implemented.

Images