Model monitoring method and apparatus, and communication device

Abstract

The present application relates to the technical field of communications, and discloses a model monitoring method and apparatus, and a communication device. The model monitoring method in embodiments of the present application comprises: a first device performs multi-stage model monitoring on a first AI model, wherein among a plurality of different stages in the multi-stage model monitoring, at least two stages employ different model monitoring modes.

Description

Model monitoring method, device and communication equipment

[0001] Cross-references

[0002] The present invention claims priority to the Chinese patent application filed with the China Patent Office on November 24, 2023, with application number 202311585335X and invention name “Model Monitoring Method, Device and Communication Equipment”. The entire contents of the application are incorporated into the present invention by reference. Technical Field

[0003] The present application belongs to the field of communication technology, and specifically relates to a model monitoring method, device and communication equipment. Background Art

[0004] With the continuous development of communication technology, artificial intelligence (AI) has also been widely used in various parts of communication systems to improve the performance of communication systems.

[0005] Among them, although communication-related technologies provide a variety of different model monitoring solutions to ensure the performance of AI models, for complex and changeable communication systems, there are still problems such as poor robustness of model monitoring results, which makes it impossible to guarantee the performance of communication systems. Summary of the Invention

[0006] The embodiments of the present application provide a model monitoring method, apparatus, and communication equipment, which can improve the robustness of model monitoring results and ensure the performance of the communication system.

[0007] In a first aspect, a model monitoring method is provided, including: a first device performs multi-stage model monitoring on a first AI model; wherein, among the multiple different stages in the multi-stage model monitoring, there are at least two stages that adopt different model monitoring methods.

[0008] In a second aspect, a model monitoring method is provided, including at least one of the following: a second device sends first information to a first device, the first information including the associated configuration of multi-stage model monitoring when the first device performs multi-stage model monitoring on a first AI model; the second device receives second information sent by the first device, the second information being used by the first device to request the first device to perform multi-stage model monitoring on the first AI model; the second device sends third information to the first device, the third information being used to instruct the first device to perform multi-stage model monitoring on the first AI model; wherein, among the multiple different stages in the multi-stage model monitoring, there are at least two stages that adopt different model monitoring methods.

[0009] In a third aspect, a model monitoring device is provided, including: a monitoring module for performing multi-stage model monitoring on a first AI model; wherein, among the multiple different stages in the multi-stage model monitoring, there are at least two stages that adopt different model monitoring methods.

[0010] In a fourth aspect, a model monitoring device is provided, including: a transmission module, used for at least one of the following: sending first information to a first device, the first information including the associated configuration of multi-stage model monitoring when the first device performs multi-stage model monitoring on a first AI model; receiving second information sent by the first device, the second information being used by the first device to request multi-stage model monitoring of the first AI model; sending third information to the first device, the third information being used to instruct the first device to perform multi-stage model monitoring on the first AI model; wherein, among the multiple different stages in the multi-stage model monitoring, there are at least two stages that adopt different model monitoring methods.

[0011] In a fifth aspect, a communication device is provided, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect or the second aspect are implemented.

[0012] In a sixth aspect, a communication device is provided, comprising a processor and a communication interface, wherein the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the steps of the method described in the first aspect, or to implement the steps of the method described in the second aspect.

[0013] In the seventh aspect, a readable storage medium is provided, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented, or the steps of the method described in the second aspect are implemented.

[0014] In an eighth aspect, a wireless communication system is provided, comprising: a first device and a second device, wherein the first device can be used to execute the steps of the method described in the first aspect, and the second device can be used to execute the steps of the method described in the second aspect.

[0015] In the ninth aspect, a chip is provided, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the steps of the method described in the first aspect, or to implement the steps of the method described in the second aspect.

[0016] In the tenth aspect, a computer program / program product is provided, which is stored in a storage medium and is executed by at least one processor to implement the steps of the method described in the first aspect, or to implement the steps of the method described in the second aspect.

[0017] In an embodiment of the present application, the first device performs multi-stage model monitoring on the first AI model, and among the multiple different stages in the multi-stage model monitoring, at least two stages adopt different model monitoring methods, thereby effectively improving the robustness of the monitoring results to adapt to complex and changeable communication systems and ensure the performance of the communication system. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] FIG1 is a schematic structural diagram of a wireless communication system provided by an exemplary embodiment of the present application.

[0019] FIG2 is a flow chart of a model monitoring method according to an exemplary embodiment of the present application.

[0020] FIG3 is a second flow chart of a model monitoring method provided by an exemplary embodiment of the present application.

[0021] FIG4 a is one of the interactive flow diagrams of the model monitoring method provided by an exemplary embodiment of the present application.

[0022] FIG4 b is a second interactive flow diagram of a model monitoring method provided by an exemplary embodiment of the present application.

[0023] FIG4c is a third interactive flow diagram of a model monitoring method provided by an exemplary embodiment of the present application.

[0024] FIG5 is a third flow chart of a model monitoring method provided by an exemplary embodiment of the present application.

[0025] FIG6 is a schematic diagram of a structure of a model monitoring device according to an exemplary embodiment of the present application.

[0026] FIG7 is a second structural diagram of a model monitoring device provided by an exemplary embodiment of the present application.

[0027] FIG8 is a schematic structural diagram of a communication device provided by an exemplary embodiment of the present application.

[0028] FIG9 is a schematic structural diagram of a terminal provided by an exemplary embodiment of the present application.

[0029] FIG10 is a schematic structural diagram of a network-side device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0030] The following will be combined with the accompanying drawings in the embodiments of this application to clearly describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0031] The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same type, and do not limit the number of objects, for example, the first object can be one or more. In addition, "or" in this application represents at least one of the connected objects. For example, "A or B" covers three options, namely, Option 1: including A but not including B; Option 2: including B but not including A; Option 3: including both A and B. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0032] The term "indication" in this application can be either a direct indication (or explicit indication) or an indirect indication (or implicit indication). A direct indication can be understood as the sender explicitly informing the receiver of specific information, the operation to be performed, or the requested result, etc. in the instruction sent; an indirect indication can be understood as the receiver determining the corresponding information based on the instruction sent by the sender, or making a judgment and determining the operation to be performed or the requested result, etc. based on the judgment result.

[0033] It is worth noting that the technology described in the embodiments of the present application is not limited to the Long Term Evolution (LTE) / LTE-Advanced (LTE-A) system, but can also be used in other wireless communication systems, such as Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal Frequency Division Multiple Access (OFDMA), Single-carrier Frequency-Division Multiple Access (SC-FDMA) or other systems. The terms "system" and "network" in the embodiments of the present application are often used interchangeably, and the technology described can be used for the systems and radio technologies mentioned above, as well as for other systems and radio technologies. The following description describes a New Radio (NR) system for illustrative purposes, and NR terminology is used in most of the following description, but these technologies can also be applied to systems other than NR systems, such as 6th generation (6G) systems. th Generation, 6G) communication system.

[0034] FIG1 is a block diagram of a wireless communication system applicable to an embodiment of the present application. The wireless communication system includes a terminal 11 and a network-side device 12. The terminal 11 may be a mobile phone, a tablet computer (Tablet Personal Computer), a laptop computer (Laptop Computer), a notebook computer, a personal digital assistant (PDA), a handheld computer, a netbook, an ultra-mobile personal computer (UMPC), a mobile internet device (MID), an augmented reality (AR), a virtual reality (VR) device, a robot, a wearable device (Wearable Device), an aircraft (Flight Vehicle), a vehicle-mounted device (VUE), a ship-mounted device, a pedestrian user equipment (PUE), a smart home (home appliances with wireless communication capabilities, such as refrigerators, televisions, washing machines, or furniture), a game console, a personal computer (PC), an ATM, or a self-service machine, or other terminal-side devices. Wearable devices include: smart watches, smart bracelets, smart headphones, smart glasses, smart jewelry (smart bracelets, smart bracelets, smart rings, smart necklaces, smart anklets, smart anklets, etc.), smart wristbands, smart clothing, etc. Among them, the vehicle-mounted device can also be called a vehicle-mounted terminal, a vehicle-mounted controller, a vehicle-mounted module, a vehicle-mounted component, a vehicle-mounted chip or a vehicle-mounted unit, etc. It should be noted that the specific type of the terminal 11 is not limited in the embodiment of the present application. The network side device 12 may include an access network device or a core network device, wherein the access network device may also be called a radio access network (Radio Access Network, RAN) device, a radio access network function or a radio access network unit. The access network device may include a base station, a wireless local area network (Wireless Local Area Network, WLAN) access point (Access Point, AS) or a wireless fidelity (Wireless Fidelity, WiFi) node, etc.Among them, the base station can be referred to as Node B (NB), Evolved Node B (eNB), the next generation Node B (gNB), New Radio Node B (NR Node B), access point, Relay Base Station (RBS), Serving Base Station (SBS), Base Transceiver Station (BTS), radio base station, radio transceiver, Basic Service Set (BSS), Extended Service Set (ESS), Home Node B (HNB), Home Evolved Node B (home evolved Node B), Transmission Reception Point (TRP) or other appropriate terms in the field. As long as the same technical effect is achieved, the base station is not limited to specific technical vocabulary. It should be noted that in the embodiment of the present application, only the base station in the NR system is used as an example for introduction, and the specific type of the base station is not limited.

[0035] The communication scenario shown in Figure 1 is one of the application scenarios of the technical solution provided in this application. For example, in this application, the combination of the first device and the second device mentioned later or the application scenario of the technical solution provided in this application can be, but is not limited to, the following.

[0036] a) The first device is the terminal 11 shown in FIG. 1 , and the second device is the network side device 12 shown in FIG. 1 .

[0037] b) The first device is the network side device 12 shown in FIG. 1 , and the second device is the terminal 11 shown in FIG. 1 .

[0038] c) Both the first device and the second device are terminals.

[0039] d) Both the first device and the second device are network-side devices.

[0040] In addition, the AI ​​model mentioned in the context of this application may also be referred to as an AI unit, AI structure, etc., or the AI ​​model may also refer to a processing unit that can implement specific algorithms, formulas, processing procedures, capabilities, etc. related to AI, or the AI ​​model may also be a processing method, algorithm, function, module or unit for a specific data set, or the AI ​​model may be a processing method, algorithm, function, module or unit running on AI-related hardware such as a graphics processing unit (GPU), a neural network processing unit (NPU), a tensor processing unit (TPU), an application-specific integrated circuit (ASIC), etc., and this application does not make specific restrictions on this. Optionally, the specific data set may include but is not limited to the input or output of the AI ​​model.

[0041] Correspondingly, the subsequent AI models such as the first AI model can be described by an AI model identifier. The identifier of the AI ​​model can be an AI unit identifier, an AI structure identifier, an AI algorithm identifier, a functional identifier (functionality ID), a physical identifier, a logical identifier, a global identifier, a local identifier, or an identifier of a specific data set associated with the AI ​​model. It can also be an identifier of a specific scenario related to the AI, an environment identifier related to the AI ​​model, a channel feature identifier related to the AI ​​model, an identifier of a device related to the AI ​​model, or an identifier of a function, feature, capability, or module related to the AI. This application does not specifically limit this.

[0042] The technical solutions provided by the embodiments of the present application are described in detail below through some embodiments and their application scenarios in conjunction with the accompanying drawings.

[0043] Figure 2 is a flow chart of a model monitoring method 200 according to an exemplary embodiment of the present application. This method 200 may be, but is not limited to, executed by a first device, such as a terminal or a network-side device, and may specifically be executed by hardware or software installed in the first device. In this embodiment, the method 200 may include at least the following steps.

[0044] S210: The first device performs multi-stage model monitoring on the first AI model.

[0045] Before performing model monitoring, the first device may determine an AI model to be monitored, such as the first AI model, from among multiple configured AI models based on the model monitoring period, etc. In this embodiment, the first AI model may be used to calculate, predict, and evaluate communication data, thereby improving communication system performance.

[0046] Optionally, the first AI model can be but is limited to a neural network, a decision tree, a support vector machine, a Bayesian classifier, etc., and the neural network can be but is not limited to a deep neural network, a convolutional neural network, a recurrent neural network, etc.

[0047] Based on this, considering that for complex and changeable communication systems, the solution of the AI ​​model monitoring based on a single different model monitoring solution in the related art still has the problem of poor model monitoring robustness. To this end, this embodiment implements monitoring of the first AI model based on a multi-stage model monitoring solution to ensure that the model performance of the first AI model meets the communication requirements. Among them, the aforementioned multi-stage model monitoring solution achieves the purpose of model monitoring by combining a variety of different model monitoring solutions, that is, among the multiple different stages in the multi-stage model monitoring described in this embodiment, there are at least two stages that adopt different model monitoring methods.

[0048] For example, assuming that multi-stage model monitoring includes stage 1, stage 2, stage 3, and stage 4, then different model monitoring methods can be used in stages 1, stage 2, stage 3, and stage 4 respectively, or stages 1 and stage 3 can use the same model monitoring method, and stages 2 and stage 4 can use the same model monitoring method, or stages 1, stage 3, and stage 4 can use the same model monitoring method, and stage 2 can use a model monitoring method, etc., there is no restriction here.

[0049] Optionally, in the aforementioned “among the multiple different stages in the multi-stage model monitoring, there are at least two stages that adopt different model monitoring methods”, the model monitoring method may be, but is not limited to, a monitoring method based on the model input distribution or output distribution, a monitoring method based on the intermediate results obtained by calculating the model output, a monitoring method based on the final performance results, a monitoring method based on the comparison results with other solutions, etc.

[0050] Among them, "monitoring method based on model input distribution or output distribution" can be understood as: statistics on the input information or output information of the AI ​​model to calculate the distribution information of the input information or output information of the AI ​​model, such as mean, mean vector, mean matrix, variance, variance vector, variance matrix, covariance, covariance vector, covariance matrix, maximum value, maximum value vector, maximum value matrix, minimum value, minimum value vector, minimum value matrix, etc. Then, the calculated distribution information is compared with the applicable distribution information range of the AI ​​model to obtain the monitoring result. Among them, the AI ​​model can be an executing model, such as a model undergoing inference, an activated model, etc. The applicable distribution information range of the model refers to the distribution information range corresponding to at least one of the model input information and output information involved in the training of the AI ​​model.

[0051] "Monitoring based on intermediate results calculated from model output" can be understood as comparing the AI ​​model's output information with its corresponding true value information to calculate intermediate results such as the model's output error or accuracy, such as error and accuracy indicators. This intermediate result is then compared with a preset threshold to obtain a monitoring result. If the AI ​​model's output information is a prediction result, the true value information can be understood as the actual result, etc.

[0052] "Monitoring method based on final performance results" can be understood as: statistics or calculation of the current communication system performance, such as throughput, spectrum efficiency, signal to interference plus noise ratio (SINR), signal to noise ratio (SNR), bit error rate, block error rate, packet loss rate, transmission rate (such as uplink / downlink transmission rate), peak rate (such as uplink / downlink peak rate), etc., based on the output information of the AI ​​model, and then comparing the statistically or calculated current communication system performance with the preset threshold to obtain the monitoring results.

[0053] "Monitoring methods based on comparisons with other solutions" can be understood as comparing the intermediate or final performance results obtained based on the AI ​​model with those obtained from other solutions to obtain monitoring results. Other solutions refer to those based on AI models other than the AI ​​model or non-AI communication algorithms.

[0054] It is worth noting that this embodiment can determine different multi-stage model monitoring schemes for different scenarios, which are exemplified below with reference to the following scenarios.

[0055] Scenario 1: The AI ​​model performs well

[0056] In the first stage, monitoring can be performed based on the model input distribution or output distribution. If the monitoring results meet the preset conditions, such as the mean value exceeds the preset mean range, the second stage can monitor the intermediate results (error indicators, accuracy indicators, etc.) obtained based on the model output calculation or monitor based on the final performance results.

[0057] Scenario 2: Poor AI model performance

[0058] In the first stage, monitoring can be performed based on the intermediate results (such as error indicators and accuracy indicators) obtained by calculating the model output or monitoring based on the final performance results. If the monitoring results meet the preset conditions, the second stage can be performed based on the model input distribution or output distribution.

[0059] Scenario 3: The relationship between intermediate results and final performance results is not completely linear.

[0060] In the first stage, the intermediate results (such as error indicators and accuracy indicators) obtained based on the model output calculation can be monitored. If the monitoring results meet the preset conditions, such as the intermediate results are lower than the preset threshold, the second stage can be used to monitor based on the final performance results.

[0061] Scenario 4: Unable to determine the performance of AI-based solutions versus other solutions

[0062] In the first phase, monitoring can be performed based on comparison results with other solutions. If the AI ​​model-based solution performs better than other solutions, monitoring of other solutions can be continued in the second phase. If the AI ​​model-based solution performs worse than other solutions, model adjustments or switching to other solutions can be carried out in the second phase.

[0063] In some embodiments, in order to facilitate the identification of different model monitoring methods and thus facilitate the calling of different model monitoring methods in a multi-stage model monitoring scheme, each of the model monitoring methods can be configured on the first device side through protocol agreement and network side instructions, and different model monitoring methods can be uniquely identified using different identifiers.

[0064] In this embodiment, the first device performs multi-stage model monitoring on the first AI model, and among the multiple different stages in the multi-stage model monitoring, at least two stages adopt different model monitoring methods. This can effectively improve the robustness of the monitoring results to adapt to complex and changeable communication systems and ensure the performance of the communication system.

[0065] FIG3 is a flow chart of a model monitoring method 300 according to an exemplary embodiment of the present application. This method 300 may be, but is not limited to, executed by a first device, specifically hardware or software installed in the first device. In this embodiment, the method 300 may include at least the following steps.

[0066] S310: The first device performs multi-stage model monitoring on the first AI model.

[0067] Among the multiple different stages in the multi-stage model monitoring, there are at least two stages that adopt different model monitoring methods.

[0068] It is understood that, in addition to referring to the relevant description in the aforementioned method embodiment 200, the implementation process of S310 may include, as a possible implementation method, a variety of triggering or selecting methods for the model monitoring methods used at different stages when the first device performs multi-stage model monitoring on the first AI model. For example, in this embodiment, it may include, but is not limited to, at least one of the following methods 1 to 3.

[0069] Method 1: The first device performs multi-stage model monitoring on the first AI model based on the first information, wherein the first information includes the associated configuration of model monitoring at different stages when performing multi-stage model monitoring on the first AI model, such as which model monitoring method to use in different stages, the triggering conditions of each model monitoring method, etc.

[0070] For example, in this embodiment, the first information may include, but is not limited to, at least one of a first trigger condition, a second trigger condition, and a termination condition of the Nth stage model monitoring.

[0071] The first trigger condition is used to trigger model monitoring in the Nth stage, where N is an integer greater than or equal to 1, such as N=1, 2, 3, ... . That is, when the first trigger condition is met, model monitoring in the Nth stage is triggered. The model monitoring method used in the Nth stage can be implemented by protocol agreement or preconfiguration.

[0072] In some embodiments, the first trigger condition may be a trigger period or trigger event information. Optionally, the trigger event information may include, but is not limited to, a communication system throughput or other indicator falling below a preset threshold within a preset time period, or a model monitoring result in the N-1th phase failing to meet predetermined requirements.

[0073] The second triggering condition is used to trigger model monitoring in stage (N+1) based on the model monitoring result in stage (N), where N is an integer greater than or equal to 1, such as N=1, 2, 3, ... . That is, when the model monitoring result in stage (N) meets the second triggering condition, model monitoring in stage (N+1) is triggered. The model monitoring method used in stage (N+1) can be implemented by protocol agreement or preconfiguration.

[0074] The termination condition of the Nth stage model monitoring, wherein the termination condition can be a periodic termination condition or an event information termination condition, etc., and N is an integer greater than or equal to 1, such as N=1, 2, 3,...

[0075] In some implementations, the first information may also include at least one of a third trigger condition, a fourth trigger condition, and a fifth trigger condition, so that the first device can adjust the model according to the model monitoring results, such as continuing to use the first AI model, stopping using the first AI model, using other AI models other than the first AI model, using a non-AI model, etc.

[0076] For example, the third trigger condition is used to trigger model adjustment of the first AI model. That is, the first device can autonomously adjust the model based on the monitoring results of the first AI model, such as stopping use of the first AI model, using an AI model other than the first AI model, using a non-AI model, etc. It is worth noting that the monitoring results of the first AI model mentioned in the context of this embodiment can be the monitoring results of the Nth stage or the complete monitoring results of multi-stage model monitoring, and this is not limited here.

[0077] The fourth trigger condition is used to trigger the transmission of the model adjustment result corresponding to the first AI model. That is, for the first device, if a model adjustment is performed, the model adjustment result can be sent only when the fourth trigger condition is met, thereby avoiding the problems of high signaling overhead and waste of network resources caused by sending all adjustment results to the second device. The model adjustment result includes at least one of stopping use of the first AI model, using an AI model other than the first AI model, and using a non-AI model.

[0078] The fifth trigger condition is used to trigger a model adjustment request, which is used to request model adjustment of the first AI model. In other words, the first device cannot independently adjust the first AI model. Instead, when the fifth trigger condition is met, it requests model adjustment from the second device. The second device determines whether to perform model adjustment, such as stopping use of the first AI model, using an AI model other than the first AI model, or using a non-AI model.

[0079] The aforementioned first information may be implemented by protocol agreement, high-level configuration, network-side configuration, etc. For example, the first information may be, but is not limited to, sent by the second device.

[0080] Method 2: The first device sends a second message to the second device, where the second message is used to request multi-stage model monitoring of the first AI model. In other words, when the first device needs to perform multi-stage model monitoring, it needs to send a request to the second device, and the second device determines whether to perform the multi-stage model monitoring. For example, the second device can send a confirmation message to the first device to indicate whether it agrees with the first device to perform multi-stage model monitoring of the first AI model.

[0081] Optionally, the second information may request only the model monitoring information for the Nth stage, or may simultaneously request the model monitoring information for each stage in a multi-stage model monitoring process. For example, if only the model monitoring information for the Nth stage is requested, the second information may include, but is not limited to, at least one of a model monitoring start request for the Nth stage and a model monitoring termination request for the Nth stage. N is an integer greater than or equal to 1.

[0082] Method 3: The first device receives third information sent by the second device, and the third information is used to instruct the first device to perform multi-stage model monitoring on the first AI model. In other words, the first device needs to perform multi-stage model monitoring according to the instruction of the second device.

[0083] Optionally, the third information may indicate only the model monitoring information for the Nth stage, or may indicate the model monitoring information for each stage of a multi-stage model monitoring process. For example, if only the model monitoring information for the Nth stage is indicated, the third information may include, but is not limited to, at least one of a model monitoring start instruction for the Nth stage and a model monitoring termination instruction for the Nth stage, where N is an integer greater than or equal to 1.

[0084] The third information may be determined by the first device according to the second information, that is, the third information is a response of the second device to the second information. The third information may also be determined autonomously by the second device, which is not limited here.

[0085] In some embodiments, after completing the multi-stage model monitoring, the first device may also send fourth information to the second device; wherein the fourth information may include but is not limited to at least one of a model adjustment result, a model adjustment request, and a model monitoring result of the Nth stage.

[0086] The model adjustment result may be, but is not limited to, determined based on the model monitoring result of stage N. That is, the first device may autonomously adjust the first AI model based on the model monitoring result and transmit the model adjustment result to the second device when the fourth trigger condition is met. The model adjustment result may include, but is not limited to, stopping use of the first AI model, using an AI model other than the first AI model, using a non-AI model, etc.

[0087] The model adjustment request is used to request model adjustment of the first AI model. That is, the first device cannot autonomously adjust the first AI model based on the model monitoring results, but must request the second device to determine whether to adjust the model. The model adjustment includes but is not limited to stopping use of the first AI model, using an AI model other than the first AI model, using a non-AI model, etc.

[0088] The model monitoring results of the Nth stage are used to ensure that the second device and the first device have a consistent understanding of the model performance of the first AI model, and to determine whether to allow or instruct the first device to adjust the first AI model based on the model monitoring results of the Nth stage, such as stopping using the first AI model, using an AI model other than the first AI model, using a non-AI model, etc.

[0089] Based on this, if the second device determines that the first AI model needs to be adjusted based on the model adjustment request or the model monitoring result of the Nth stage, then the fifth information can be sent to the first device. Correspondingly, the first device receives the fifth information sent by the second device and adjusts the model based on the fifth information; wherein, the fifth information includes a model adjustment instruction, and the model adjustment instruction corresponds to the model adjustment request or the model monitoring result of the Nth stage.

[0090] In this embodiment, multi-stage model monitoring is implemented through simple signaling interaction, which can achieve more robust performance than single-stage monitoring and reduce unnecessary model monitoring resource overhead and model switching process.

[0091] In addition, the embodiments of the present application can be applied to but not limited to 5.5G or 6G communication information systems with wireless AI functions.

[0092] Based on the description of the aforementioned method embodiments 200-300, the implementation process of the model monitoring solution provided by this application is further described below in combination with Examples 1-3.

[0093] Example 1

[0094] S411, as shown in Figure 4a, the network side device sends first information to the terminal, where the first information includes associated configurations of model monitoring at different stages when performing multi-stage model monitoring on the first AI model.

[0095] S412: The terminal performs multi-stage model monitoring on the first AI model according to the first information.

[0096] S413: The terminal sends fourth information to the network-side device. The fourth information includes at least one of a model adjustment result, a model adjustment request, and a model monitoring result of the Nth stage.

[0097] S414, the network side device sends fifth information to the terminal according to the fourth information, wherein the fifth information contains a model adjustment instruction, and the model adjustment instruction corresponds to the model adjustment request in the fourth information or the model monitoring result of the Nth stage.

[0098] S415: The terminal adjusts the first AI model according to the fifth information.

[0099] This example 1 provides a multi-stage model monitoring solution based on static configuration. Its specific implementation process can refer to the relevant descriptions in the aforementioned method embodiments 200-300, and achieve the same or corresponding technical effects. To avoid repetition, it will not be repeated here.

[0100] In addition, this Example 1 may include more or fewer steps than the aforementioned S411-S415, which is not limited here.

[0101] Example 2

[0102] S421, as shown in Figure 4b, the network side device sends first information to the terminal, where the first information includes associated configurations of model monitoring at different stages when performing multi-stage model monitoring on the first AI model.

[0103] S422: The terminal sends second information to the network side device, requesting multi-stage model monitoring of the first AI model.

[0104] S423: The network side device sends third information to the terminal according to the second information to instruct the first device to perform multi-stage model monitoring on the first AI model.

[0105] S424: The terminal performs multi-stage model monitoring on the first AI model according to the third information.

[0106] S425: The terminal sends fourth information to the network-side device, where the fourth information includes at least one of a model adjustment result, a model adjustment request, and a model monitoring result of the Nth stage.

[0107] S426, the network side device sends fifth information to the terminal according to the fourth information, wherein the fifth information contains a model adjustment instruction, and the model adjustment instruction corresponds to the model adjustment request in the fourth information or the model monitoring result of the Nth stage.

[0108] S427: The terminal adjusts the first AI model according to the fifth information.

[0109] This Example 2 provides a multi-stage model monitoring solution based on dynamic request implementation. Its specific implementation process can refer to the relevant descriptions in the aforementioned method embodiments 200-300, and achieve the same or corresponding technical effects. To avoid repetition, it will not be repeated here.

[0110] In addition, this Example 2 may include more or fewer steps than the aforementioned S421-S426, which is not limited here.

[0111] Example 3

[0112] S431, as shown in Figure 4c, the network side device sends third information to the terminal to instruct the first device to perform multi-stage model monitoring on the first AI model.

[0113] S432: The terminal performs multi-stage model monitoring on the first AI model according to the third information.

[0114] S433: The terminal sends fourth information to the network side device, where the fourth information includes at least one of a model adjustment result, a model adjustment request, and a model monitoring result of the Nth stage.

[0115] S434, the network side device sends fifth information to the terminal according to the fourth information, wherein the fifth information packet model adjustment instruction corresponds to the model adjustment request in the fourth information or the model monitoring result of the Nth stage.

[0116] S435: The terminal adjusts the first AI model according to the fifth information.

[0117] This Example 3 provides a multi-stage model monitoring solution based on dynamic indication. Its specific implementation process can refer to the relevant descriptions in the aforementioned method embodiments 200-300, and achieve the same or corresponding technical effects. To avoid repetition, it will not be repeated here.

[0118] In addition, this Example 3 may include more or fewer steps than the aforementioned S431-S434, which is not limited here.

[0119] FIG5 is a flow chart of a model monitoring method 500 according to an exemplary embodiment of the present application. This method 500 may be, but is not limited to, executed by a second device, specifically hardware or software installed in the second device. In this embodiment, the method 500 may include at least the following steps.

[0120] S510, the second device performs a first operation, and the first operation includes at least one of the following: sending first information to the first device, the first information including the associated configuration of multi-stage model monitoring when the first device performs multi-stage model monitoring on the first AI model; receiving second information sent by the first device, the second information is used by the first device to request multi-stage model monitoring of the first AI model; sending third information to the first device, the third information is used to instruct the first device to perform multi-stage model monitoring on the first AI model; wherein, among the multiple different stages in the multi-stage model monitoring, there are at least two stages that adopt different model monitoring methods.

[0121] Optionally, the first information includes at least one of the following: a first trigger condition for triggering model monitoring in the Nth stage; a second trigger condition for triggering model monitoring in the N+1th stage based on the model monitoring result in the Nth stage; a termination condition for model monitoring in the Nth stage; a third trigger condition for triggering model adjustment of the first AI model; a fourth trigger condition for triggering the sending of the model adjustment result corresponding to the first AI model; a fifth trigger condition for triggering a model adjustment request, the model adjustment request being used to request model adjustment of the first AI model; wherein the model adjustment or the model adjustment result includes at least one of stopping using the first AI model, using an AI model other than the first AI model, and using a non-AI model.

[0122] Optionally, the second information includes at least one of the following: a model monitoring start request for the Nth stage; a model monitoring termination request for the Nth stage; wherein N is an integer greater than or equal to 1.

[0123] Optionally, the third information includes at least one of the following: a model monitoring start instruction for the Nth stage; a model monitoring termination instruction for the Nth stage; wherein N is an integer greater than or equal to 1.

[0124] Optionally, the method also includes: receiving fourth information sent by the first device; wherein the fourth information includes at least one of the following: a model adjustment result corresponding to the first AI model, wherein the model adjustment result includes at least one of stopping using the first AI model, using other AI models other than the first AI model, and using a non-AI model; a model adjustment request for requesting model adjustment of the first AI model, wherein the model adjustment includes at least one of stopping using the first AI model, using other AI models other than the first AI model, and using a non-AI model; and the model monitoring result of the Nth stage.

[0125] Optionally, the method further includes: sending fifth information to the first device; wherein the fifth information is a model adjustment instruction, and the model adjustment instruction corresponds to the model adjustment request or the model monitoring result of the Nth stage.

[0126] It can be understood that each implementation method in method embodiment 500 has the same or corresponding technical features as the aforementioned method embodiments 200-300. Therefore, for each implementation method in method embodiment 500, reference can be made to the relevant descriptions in the aforementioned method embodiments 200-300, and the same or corresponding technical effects can be achieved. To avoid repetition, they will not be repeated here.

[0127] The model monitoring methods 200-500 provided in the embodiments of the present application may be executed by a model monitoring device. In the embodiments of the present application, the model monitoring device performing the model monitoring methods 200-500 is taken as an example to illustrate the model monitoring device provided in the embodiments of the present application.

[0128] As shown in Figure 6, it is a structural diagram of a model monitoring device 600 provided in an embodiment of the present application. The device 600 includes: a monitoring module 610, which is used to perform multi-stage model monitoring on a first AI model; wherein, among the multiple different stages in the multi-stage model monitoring, there are at least two stages that adopt different model monitoring methods.

[0129] Optionally, the monitoring module 610 performs multi-stage model monitoring on the first AI model, including at least one of the following: performing multi-stage model monitoring on the first AI model according to first information, wherein the first information includes associated configurations of model monitoring at different stages when performing multi-stage model monitoring on the first AI model; sending second information to a second device, wherein the second information is used to request multi-stage model monitoring of the first AI model; and receiving third information sent by the second device, wherein the third information is used to instruct the first device to perform multi-stage model monitoring on the first AI model.

[0130] Optionally, the first information includes at least one of the following: a first trigger condition for triggering model monitoring in the Nth stage; a second trigger condition for triggering model monitoring in the N+1th stage based on the model monitoring result in the Nth stage; and a termination condition for model monitoring in the Nth stage; wherein N is an integer greater than or equal to 1.

[0131] Optionally, the apparatus 600 further includes a transmission module configured to receive the first information sent by the second device.

[0132] Optionally, the first information also includes at least one of the following: a third trigger condition, used to trigger model adjustment of the first AI model; a fourth trigger condition, used to trigger the sending of the model adjustment result corresponding to the first AI model; a fifth trigger condition, used to trigger a model adjustment request, and the model adjustment request is used to request model adjustment of the first AI model; wherein, the model adjustment or the model adjustment result includes at least one of stopping using the first AI model, using other AI models other than the first AI model, and using a non-AI model.

[0133] Optionally, the second information includes at least one of the following: a model monitoring start request for the Nth stage; a model monitoring termination request for the Nth stage; wherein N is an integer greater than or equal to 1.

[0134] Optionally, the third information includes at least one of the following: a model monitoring start instruction for the Nth stage; a model monitoring termination instruction for the Nth stage; wherein N is an integer greater than or equal to 1.

[0135] Optionally, the transmission module is further used to send fourth information to the second device; wherein the fourth information includes at least one of the following: a model adjustment result, which is determined based on the model monitoring result of the Nth stage, and the model adjustment result includes stopping using the first AI model, using other AI models other than the first AI model, and using at least one of a non-AI model; a model adjustment request, used to request model adjustment of the first AI model, wherein the model adjustment includes stopping using the first AI model, using other AI models other than the first AI model, and using at least one of a non-AI model; the model monitoring result of the Nth stage; wherein N is an integer greater than or equal to 1.

[0136] Optionally, the transmission module is further used to receive fifth information sent by the second device; and perform model adjustment according to the fifth information; wherein the fifth information includes a model adjustment instruction, and the model adjustment instruction corresponds to the model adjustment request or the model monitoring result of the Nth stage.

[0137] The model monitoring device 600 in the embodiment of the present application can be a communication device, such as a communication device with an operating system, or a component in the communication device, such as an integrated circuit or chip. The communication device can be a terminal, a network-side device, or other devices other than a terminal or a network-side device. For example, the terminal can include but is not limited to the types of terminal 11 listed above, the network-side device can include but is not limited to the types of network-side device 12 listed above, and other devices can include servers, network attached storage (NAS), etc., which are not specifically limited in the embodiment of the present application.

[0138] The model monitoring device 600 provided in the embodiment of the present application can implement the various processes implemented in the method embodiments of Figures 2 to 3 and achieve the same technical effects. To avoid repetition, they will not be described here.

[0139] As shown in Figure 7, which is a structural diagram of a model monitoring device 700 provided in an embodiment of the present application, the device 700 includes: a transmission module 710, used for at least one of the following: sending first information to a first device, the first information including the associated configuration of multi-stage model monitoring when the first device performs multi-stage model monitoring on the first AI model; receiving second information sent by the first device, the second information being used by the first device to request multi-stage model monitoring of the first AI model; sending third information to the first device, the third information being used to instruct the first device to perform multi-stage model monitoring on the first AI model; wherein, among the multiple different stages in the multi-stage model monitoring, there are at least two stages that adopt different model monitoring methods.

[0140] Optionally, the first information includes at least one of the following: a first trigger condition for triggering model monitoring in the Nth stage; a second trigger condition for triggering model monitoring in the N+1th stage based on the model monitoring result in the Nth stage; a termination condition for model monitoring in the Nth stage; a third trigger condition for triggering model adjustment of the first AI model; a fourth trigger condition for triggering the sending of the model adjustment result corresponding to the first AI model; a fifth trigger condition for triggering a model adjustment request, the model adjustment request being used to request model adjustment of the first AI model; wherein the model adjustment or the model adjustment result includes at least one of stopping using the first AI model, using an AI model other than the first AI model, and using a non-AI model.

[0141] Optionally, the second information includes at least one of the following: a model monitoring start request for the Nth stage; a model monitoring termination request for the Nth stage; wherein N is an integer greater than or equal to 1.

[0142] Optionally, the third information includes at least one of the following: a model monitoring start instruction for the Nth stage; a model monitoring termination instruction for the Nth stage; wherein N is an integer greater than or equal to 1.

[0143] Optionally, the transmission module 710 is also used to receive fourth information sent by the first device; wherein, the fourth information includes at least one of the following: a model adjustment result corresponding to the first AI model, wherein the model adjustment result includes at least one of stopping using the first AI model, using other AI models other than the first AI model, and using a non-AI model; a model adjustment request, used to request model adjustment of the first AI model, wherein the model adjustment includes at least one of stopping using the first AI model, using other AI models other than the first AI model, and using a non-AI model; and the model monitoring result of the Nth stage.

[0144] Optionally, the transmission module 710 is further configured to send fifth information to the first device; wherein the fifth information includes a model adjustment instruction, and the model adjustment instruction corresponds to the model adjustment request or the model monitoring result of the Nth stage.

[0145] The model monitoring device 700 in the embodiment of the present application can be a communication device, such as a communication device with an operating system, or a component in the communication device, such as an integrated circuit or chip. The communication device can be a terminal, a network-side device, or other devices other than a terminal or a network-side device. For example, the terminal can include but is not limited to the types of terminal 11 listed above, the network-side device can include but is not limited to the types of network-side device 12 listed above, and other devices can include servers, network attached storage (NAS), etc., which are not specifically limited in the embodiment of the present application.

[0146] The model monitoring device 700 provided in the embodiment of the present application can implement each process implemented in the method embodiment of Figure 5 and achieve the same technical effect. To avoid repetition, it will not be described here.

[0147] As shown in Figure 8, an embodiment of the present application further provides a communication device 800, including a processor 801 and a memory 802. The memory 802 stores a program or instruction that can be run on the processor 801. For example, when the communication device 800 is a terminal, the program or instruction, when executed by the processor 801, implements the various steps of the above-mentioned model monitoring method embodiment and can achieve the same technical effect. When the communication device 800 is a network-side device, the program or instruction, when executed by the processor 801, implements the various steps of the above-mentioned model monitoring method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0148] The present application also provides a terminal comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is configured to execute a program or instruction to implement the steps of the method embodiments shown in Figures 2-5. This terminal embodiment corresponds to the aforementioned method embodiments 200, 300, or 500. The various implementation processes and implementation methods of the aforementioned method embodiments are applicable to this terminal embodiment and can achieve the same technical effects. Specifically, Figure 9 is a schematic diagram of the hardware structure of a terminal implementing the present application embodiment.

[0149] The terminal 900 includes but is not limited to: a radio frequency unit 901, a network module 902, an audio output unit 903, an input unit 904, a sensor 905, a display unit 906, a user input unit 907, an interface unit 908, a memory 909 and at least some of the components of the processor 910.

[0150] Those skilled in the art will appreciate that the terminal 900 may also include a power supply (such as a battery) to power various components. The power supply may be logically connected to the processor 910 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The terminal structure shown in FIG9 does not limit the terminal. The terminal may include more or fewer components than shown, or may combine certain components, or have different component arrangements, which will not be described in detail here.

[0151] It should be understood that in an embodiment of the present application, the input unit 904 may include a graphics processing unit (GPU) 9041 and a microphone 9042, and the graphics processor 9041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 906 may include a display panel 9061, and the display panel 9061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 907 includes a touch panel 9071 and at least one of other input devices 9072. The touch panel 9071 is also called a touch screen. The touch panel 9071 may include two parts: a touch detection device and a touch controller. Other input devices 9072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and an operating stick, which will not be repeated here.

[0152] In the embodiment of the present application, after receiving downlink data from a network-side device, the RF unit 901 may transmit the data to the processor 910 for processing. Furthermore, the RF unit 901 may send uplink data to the network-side device. Typically, the RF unit 901 includes, but is not limited to, an antenna, an amplifier, a transceiver, a coupler, a low-noise amplifier, a duplexer, and the like.

[0153] The memory 909 can be used to store software programs or instructions and various data. The memory 909 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, applications or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 909 may include a volatile memory or a non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM). The memory 909 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0154] Processor 910 may include one or more processing units. Optionally, processor 910 integrates an application processor and a modem processor. The application processor primarily handles operations related to the operating system, user interface, and application programs, while the modem processor primarily processes wireless communication signals, such as a baseband processor. It is understood that the modem processor may not be integrated into processor 910.

[0155] In one embodiment, the processor 910 is used to perform multi-stage model monitoring on the first AI model; wherein, among the multiple different stages in the multi-stage model monitoring, there are at least two stages that adopt different model monitoring methods.

[0156] Optionally, the processor 910 performs multi-stage model monitoring on the first AI model, including at least one of the following: performing multi-stage model monitoring on the first AI model according to first information, wherein the first information includes associated configurations of model monitoring at different stages when performing multi-stage model monitoring on the first AI model; sending second information to a second device, wherein the second information is used to request multi-stage model monitoring of the first AI model; and receiving third information sent by the second device, wherein the third information is used to instruct the first device to perform multi-stage model monitoring on the first AI model.

[0157] Optionally, the first information includes at least one of the following: a first trigger condition for triggering model monitoring in the Nth stage; a second trigger condition for triggering model monitoring in the N+1th stage based on the model monitoring result in the Nth stage; and a termination condition for model monitoring in the Nth stage; wherein N is an integer greater than or equal to 1.

[0158] Optionally, the radio frequency unit 901 is used to receive the first information sent by the second device.

[0159] Optionally, the first information also includes at least one of the following: a third trigger condition, used to trigger model adjustment of the first AI model; a fourth trigger condition, used to trigger the sending of a model adjustment result corresponding to the first AI model; a fifth trigger condition, used to trigger a model adjustment request, wherein the model adjustment request is used to request model adjustment of the first AI model; wherein the model adjustment or the model adjustment result includes at least one of stopping using the first AI model, using an AI model other than the first AI model, and using a non-AI model.

[0160] Optionally, the second information includes at least one of the following: a model monitoring start request for the Nth stage; a model monitoring termination request for the Nth stage; wherein N is an integer greater than or equal to 1.

[0161] Optionally, the third information includes at least one of the following: a model monitoring start instruction for the Nth stage; a model monitoring termination instruction for the Nth stage; wherein N is an integer greater than or equal to 1.

[0162] Optionally, the RF unit 901 is further used to send fourth information to the second device; wherein the fourth information includes at least one of the following: a model adjustment result, the model adjustment result is determined based on the model monitoring result of the Nth stage, and the model adjustment result includes stopping using the first AI model, using other AI models other than the first AI model, and using at least one of a non-AI model; a model adjustment request, used to request model adjustment of the first AI model, wherein the model adjustment includes stopping using the first AI model, using other AI models other than the first AI model, and using at least one of a non-AI model; the model monitoring result of the Nth stage; wherein N is an integer greater than or equal to 1.

[0163] Optionally, the radio frequency unit 901 is also used to receive fifth information sent by the second device; perform model adjustment according to the fifth information; wherein the fifth information includes a model adjustment instruction, and the model adjustment instruction corresponds to the model adjustment request or the model monitoring result of the Nth stage.

[0164] In another embodiment, the radio frequency unit 901 is used for at least one of the following: sending first information to a first device, the first information including the associated configuration of multi-stage model monitoring when the first device performs multi-stage model monitoring on the first AI model; receiving second information sent by the first device, the second information being used by the first device to request the first device to perform multi-stage model monitoring on the first AI model; sending third information to the first device, the third information being used to instruct the first device to perform multi-stage model monitoring on the first AI model; wherein, among the multiple different stages in the multi-stage model monitoring, there are at least two stages that adopt different model monitoring methods.

[0165] Optionally, the first information includes at least one of the following: a first trigger condition for triggering model monitoring in the Nth stage; a second trigger condition for triggering model monitoring in the N+1th stage based on the model monitoring result in the Nth stage; a termination condition for model monitoring in the Nth stage; a third trigger condition for triggering model adjustment of the first AI model; a fourth trigger condition for triggering the sending of the model adjustment result corresponding to the first AI model; a fifth trigger condition for triggering a model adjustment request, the model adjustment request being used to request model adjustment of the first AI model; wherein the model adjustment or the model adjustment result includes at least one of stopping using the first AI model, using an AI model other than the first AI model, and using a non-AI model.

[0166] Optionally, the second information includes at least one of the following: a model monitoring start request for the Nth stage; a model monitoring termination request for the Nth stage; wherein N is an integer greater than or equal to 1.

[0167] Optionally, the third information includes at least one of the following: a model monitoring start instruction for the Nth stage; a model monitoring termination instruction for the Nth stage; wherein N is an integer greater than or equal to 1.

[0168] Optionally, the RF unit 901 is also used to receive fourth information sent by the first device; wherein the fourth information includes at least one of the following: a model adjustment result corresponding to the first AI model, wherein the model adjustment result includes stopping using the first AI model, using other AI models other than the first AI model, and using at least one of a non-AI model; a model adjustment request, used to request model adjustment of the first AI model, wherein the model adjustment includes stopping using the first AI model, using other AI models other than the first AI model, and using at least one of a non-AI model; and the model monitoring result of the Nth stage.

[0169] Optionally, the radio frequency unit 901 is further used to send fifth information to the first device; wherein the fifth information includes a model adjustment instruction, and the model adjustment instruction corresponds to the model adjustment request or the model monitoring result of the Nth stage.

[0170] It can be understood that the implementation process of each implementation method mentioned in this embodiment can refer to the relevant description of method embodiment 200, 300 or 500, and achieve the same or corresponding technical effect. To avoid repetition, it will not be repeated here.

[0171] The present application also provides a network-side device, including a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is configured to execute a program or instruction to implement the steps of the method embodiments shown in Figures 2-5. This network-side device embodiment corresponds to the aforementioned network-side device method embodiment, and each implementation process and implementation method of the aforementioned method embodiment is applicable to this network-side device embodiment and can achieve the same technical effects.

[0172] Specifically, an embodiment of the present application also provides a network-side device. As shown in Figure 10, the network-side device 1000 includes: an antenna 1001, a radio frequency device 1002, a baseband device 1003, a processor 1004, and a memory 1005. Antenna 1001 is connected to radio frequency device 1002. In the uplink direction, radio frequency device 1002 receives information via antenna 1001 and sends the received information to baseband device 1003 for processing. In the downlink direction, baseband device 1003 processes the information to be transmitted and sends it to radio frequency device 1002. Radio frequency device 1002 processes the received information and sends it through antenna 1001.

[0173] The method executed by the network-side device in the above embodiment may be implemented in the baseband device 1003 , which includes a baseband processor.

[0174] The baseband device 1003 may, for example, include at least one baseband board, on which multiple chips are arranged, as shown in Figure 10, one of which is, for example, a baseband processor, which is connected to the memory 1005 through a bus interface to call the program in the memory 1005 and execute the network device operations shown in the above method embodiment.

[0175] The network side device may further include a network interface 1006, which is, for example, a Common Public Radio Interface (CPRI).

[0176] Specifically, the network side device 1000 of the embodiment of the present application also includes: instructions or programs stored in the memory 1005 and can be run on the processor 1004. The processor 1004 calls the instructions or programs in the memory 1005 to execute the method of execution of each module shown in Figure 6 or Figure 7, and achieves the same technical effect. To avoid repetition, it will not be repeated here.

[0177] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned model monitoring method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0178] The processor is the processor in the terminal described in the above embodiment. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk. In some examples, the readable storage medium may be a non-transitory readable storage medium.

[0179] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned model monitoring method embodiment and achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0180] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0181] An embodiment of the present application further provides a computer program / program product, which is stored in a storage medium. The computer program / program product is executed by at least one processor to implement the various processes of the above-mentioned model monitoring method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0182] An embodiment of the present application also provides a wireless communication system, including: a first device and a second device, wherein the first device can be used to execute to implement the various processes of the above-mentioned model monitoring method embodiments 200-300, and the second device can be used to execute to implement the various processes of the above-mentioned model monitoring method embodiment 500, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0183] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0184] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of a computer software product plus a necessary general-purpose hardware platform, or of course, by hardware. The computer software product is stored in a storage medium (such as ROM, RAM, magnetic disk, optical disk, etc.) and includes a number of instructions for enabling a terminal or network-side device to execute the methods described in each embodiment of the present application.

[0185] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms of implementation methods without departing from the purpose of this application and the scope of protection of the claims. These implementation methods are all within the protection of this application.

Claims

1. A model monitoring method, wherein: include: The first device performs multi-stage model monitoring on the first AI model; Among the multiple different stages in the multi-stage model monitoring, there are at least two stages that adopt different model monitoring methods.

2. The method of claim 1, wherein: The first device performs multi-stage model monitoring on the first AI model, including at least one of the following: The first device performs multi-stage model monitoring on the first AI model according to the first information, wherein the first information includes associated configurations of model monitoring at different stages when the multi-stage model monitoring is performed on the first AI model; The first device sends second information to the second device, where the second information is used to request multi-stage model monitoring of the first AI model; The first device receives third information sent by the second device, where the third information is used to instruct the first device to perform multi-stage model monitoring on the first AI model.

3. The method of claim 2, wherein: The first information includes at least one of the following: The first trigger condition is used to trigger the model monitoring of the Nth stage; The second trigger condition is used to trigger the model monitoring of the N+1 stage according to the model monitoring result of the Nth stage; Termination conditions for model monitoring at stage N; Wherein, N is an integer greater than or equal to 1.

4. The method according to claim 2 or 3, wherein: The method further comprises: Receive the first information sent by the second device.

5. The method according to any one of claims 2 to 4, wherein: The first information also includes at least one of the following: A third trigger condition is used to trigger model adjustment of the first AI model; a fourth trigger condition, used to trigger the sending of a model adjustment result corresponding to the first AI model; a fifth trigger condition, used to trigger a model adjustment request, where the model adjustment request is used to request model adjustment of the first AI model; The model adjustment or the model adjustment result includes at least one of stopping using the first AI model, using other AI models other than the first AI model, and using a non-AI model.

6. The method of claim 2, wherein: The second information includes at least one of the following: Model monitoring start request for stage N; Model monitoring termination request at stage N; Wherein, N is an integer greater than or equal to 1.

7. The method of claim 2, wherein: The third information includes at least one of the following: Model monitoring start instruction for stage N; Model monitoring termination instruction for stage N; Wherein, N is an integer greater than or equal to 1.

8. The method according to any one of claims 2 to 7, wherein: The method further comprises: sending fourth information to the second device; The fourth information includes at least one of the following: a model adjustment result, where the model adjustment result is determined according to the model monitoring result of the Nth stage, and the model adjustment result includes at least one of stopping using the first AI model, using another AI model other than the first AI model, and using a non-AI model; a model adjustment request, used to request model adjustment of the first AI model, wherein the model adjustment includes at least one of stopping using the first AI model, using another AI model other than the first AI model, and using a non-AI model; Model monitoring results at stage N; Wherein, N is an integer greater than or equal to 1.

9. The method of claim 8, wherein: The method further comprises: receiving fifth information sent by the second device; Performing model adjustment according to the fifth information; The fifth information includes a model adjustment instruction, and the model adjustment instruction corresponds to the model adjustment request or the model monitoring result of the Nth stage.

10. A model monitoring method, wherein: Include at least one of the following: The second device sends first information to the first device, where the first information includes an associated configuration of multi-stage model monitoring when the first device performs multi-stage model monitoring on the first AI model; The second device receives second information sent by the first device, where the second information is used by the first device to request the first device to perform multi-stage model monitoring on the first AI model; The second device sends third information to the first device, where the third information is used to instruct the first device to perform multi-stage model monitoring on the first AI model; Among the multiple different stages in the multi-stage model monitoring, there are at least two stages that adopt different model monitoring methods.

11. The method of claim 10, wherein: The first information includes at least one of the following: The first trigger condition is used to trigger the model monitoring of the Nth stage; The second trigger condition is used to trigger the model monitoring of the N+1 stage according to the model monitoring result based on the N stage; Termination conditions for model monitoring at stage N; A third trigger condition is used to trigger model adjustment of the first AI model; a fourth trigger condition, used to trigger the sending of a model adjustment result corresponding to the first AI model; a fifth trigger condition, used to trigger a model adjustment request, where the model adjustment request is used to request model adjustment of the first AI model; The model adjustment or the model adjustment result includes at least one of stopping using the first AI model, using other AI models other than the first AI model, and using a non-AI model.

12. The method of claim 10, wherein: The second information includes at least one of the following: Model monitoring start request for stage N; Model monitoring termination request at stage N; Wherein, N is an integer greater than or equal to 1.

13. The method of claim 10, wherein: The third information includes at least one of the following: Model monitoring start instruction for stage N; Model monitoring termination instruction for stage N; Wherein, N is an integer greater than or equal to 1.

14. The method according to any one of claims 10 to 13, wherein: The method further comprises: receiving fourth information sent by the first device; The fourth information includes at least one of the following: a model adjustment result corresponding to the first AI model, wherein the model adjustment result includes at least one of stopping using the first AI model, using an AI model other than the first AI model, and using a non-AI model; a model adjustment request, used to request model adjustment of the first AI model, wherein the model adjustment includes at least one of stopping using the first AI model, using another AI model other than the first AI model, and using a non-AI model; Model monitoring results at stage N.

15. The method of claim 14, wherein: The method further comprises: sending fifth information to the first device; The fifth information includes a model adjustment instruction, and the model adjustment instruction corresponds to the model adjustment request or the model monitoring result of the Nth stage.

16. A model monitoring device, wherein: include: A monitoring module, used for performing multi-stage model monitoring on the first AI model; Among the multiple different stages in the multi-stage model monitoring, there are at least two stages that adopt different model monitoring methods.

17. A model monitoring device, wherein: include: A transport module for at least one of the following: Sending first information to the first device, where the first information includes an associated configuration of multi-stage model monitoring when the first device performs multi-stage model monitoring on the first AI model; receiving second information sent by the first device, where the second information is used by the first device to request multi-stage model monitoring of the first AI model; Sending third information to the first device, where the third information is used to instruct the first device to perform multi-stage model monitoring on the first AI model; Among the multiple different stages in the multi-stage model monitoring, there are at least two stages using different models. Monitoring method.

18. A communication device, wherein: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the method according to any one of claims 1 to 9 are implemented, or the steps of the method according to any one of claims 10 to 15 are implemented.

19. A readable storage medium, wherein: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented, or the steps of the method according to any one of claims 10 to 15 are implemented.

Images