Model performance monitoring method and device, communication equipment and storage medium

Abstract

The invention relates to a model performance monitoring method and device, communication equipment and a storage medium. The method comprises the following steps: receiving a first signaling sent by first equipment, wherein the first signaling comprises configuration information; according to the configuration information, a target object is monitored, and the target object comprises at least one of a second device, an artificial intelligence or machine learning AI / ML model and a second device side function. By the adoption of the method, the second device can monitor the performance of the AI / ML model according to the configuration information of the first device, corresponding measures are taken in time when the performance of the AI / ML model is reduced, and it is ensured that the performance of the AI / ML model is stable.

Description

Technical Field

[0001] The present application relates to the field of wireless communication technology, and in particular to a model performance monitoring method, apparatus, communication equipment and storage medium. Background Art

[0002] With the progress of wireless communications, 5G-A has started research on intelligent wireless air interface technology. The 6G wireless air interface will adopt artificial intelligence (AI) endogenous design. The integration of AI and communication technology will affect the spectrum efficiency, user experience rate, AI-related indicators, latency, positioning accuracy, security and privacy in the key performance indicator system. Endogenous AI will become a necessary and important technical feature to meet and improve multiple performance indicators in future mobile communication systems.

[0003] However, artificial intelligence or machine learning (AI / ML) technologies represented by deep learning have poor interpretability and are often subject to uncertainty and unpredictability when used in mobile communication systems. This results in unstable performance of AI / ML models in actual network deployment, which in turn affects the overall performance of mobile communication systems.

[0004] Therefore, there is a problem of unstable AI / ML model performance in current wireless communication technology. Summary of the invention

[0005] The embodiments of the present application provide a model performance monitoring method, apparatus, communication device, storage medium, and computer program product, which can realize performance monitoring of AI / ML models and ensure stable performance of AI / ML models.

[0006] A model performance monitoring method, the method being applied to a second device, comprising:

[0007] receiving a first signaling sent by a first device, where the first signaling includes configuration information;

[0008] According to the configuration information, a target object is monitored, where the target object includes at least one of a second device, an artificial intelligence or machine learning AI / ML model, and a second device side function.

[0009] In one of the embodiments, the first device includes at least one of a core network network element, a network management operation and maintenance, a base station, a network device, an artificial intelligence layer, and an artificial intelligence management and orchestration layer; the second device includes at least one of an eNB, a gNB, a main control board, an intelligent computing board, a computing power board, a baseband board, a wireless access network element, a user equipment, a wireless access network artificial intelligence layer, a wireless artificial intelligence management layer, a wireless access network artificial intelligence management unit, a location management function network element LMF, and a sending and receiving point TRP.

[0010] In one of the embodiments, the first signaling includes at least one of management plane message, management plane signaling, control plane message, control plane signaling, intelligent plane message, intelligent plane signaling, data plane message, data plane signaling, computing plane message, computing plane signaling, user plane message, user plane signaling, system message, non-access layer NAS signaling, access layer AS signaling, dedicated configuration message, dedicated configuration signaling, radio resource control RRC signaling, high-level protocol signaling, media access control control unit MAC CE, access control layer signaling, open interface protocol signaling, downlink control information DCI, physical layer signaling, layer 1 L1 signaling, layer 2 L2 signaling, and layer 3 L3 signaling.

[0011] In one of the embodiments, the configuration information includes whether to enable performance monitoring, performance measurement reporting configuration, and at least one of performance monitoring configuration; the performance measurement reporting configuration includes at least one of reporting identification, measurement reporting type, measurement reporting period, model identification, model function, feature identification, and feature group identification; the performance monitoring configuration includes at least one of performance monitoring period, performance monitoring threshold, model accuracy requirement, performance monitoring reporting resource configuration, and fallback configuration.

[0012] In one embodiment, the method further comprises:

[0013] Reporting monitoring results to the first device, wherein the reporting includes at least one of periodic reporting, non-periodic reporting, and semi-continuous reporting.

[0014] In one embodiment, the method further comprises at least one of the following:

[0015] According to the monitoring result, performing at least one of management, activation, deactivation, rollback, update, fine-tuning, retraining, continued use, and maintenance on the target object;

[0016] According to the monitoring result and the configuration information, perform at least one of management, activation, deactivation, rollback, update, fine-tuning, retraining, continued use, and maintenance on the target object;

[0017] Receive a second signaling sent by the first device, and based on the second signaling, perform at least one of management, activation, deactivation, rollback, update, fine-tuning, retraining, continued use, and maintenance on the target object.

[0018] In one embodiment, the method further comprises at least one of the following:

[0019] When a first condition is met, performing at least one of deactivation, rollback, update, fine-tuning, and retraining on the target object, wherein the first condition includes that the monitoring result is lower than the monitoring threshold or the accuracy of the AI / ML model is lower than the model accuracy requirement;

[0020] When the second condition is met, the target object is replaced with a new target object, and the second condition includes at least one of the following: the monitoring result of the new target object is higher than the monitoring result of the target object; the monitoring result of the new target object is higher than the monitoring threshold; the monitoring result of the new target object has a specified number of results higher than the monitoring result of the target object within a specified time window.

[0021] In one embodiment, the method further comprises at least one of the following:

[0022] When a first event is triggered, starting or resetting a timer; the first event includes a monitoring result being lower than a monitoring threshold;

[0023] Within the timing range of the timer, if the number of occurrences of the first event exceeds a preset number of times or the duration of the occurrence of the first event exceeds a preset duration, performing at least one of deactivation, rollback, update, fine-tuning, and retraining on the target object;

[0024] Within the timing range of the timer, if the number of occurrences of the first event does not exceed the preset number of times or the duration of the occurrence of the first event does not exceed the preset duration, continue to use the target object and / or reset the timer.

[0025] In one embodiment, the method further comprises at least one of the following:

[0026] When a first event is triggered, starting or resetting the monitoring window; the first event includes a monitoring result being lower than a monitoring threshold;

[0027] Within the monitoring window, if the number of occurrences of the first event exceeds a preset number of times or the duration of the occurrence of the first event exceeds a preset duration, performing at least one of deactivation, rollback, update, fine-tuning, and retraining on the target object;

[0028] Within the monitoring window, if the number of occurrences of the first event does not exceed a preset number of times or the duration of the occurrence of the first event does not exceed a preset duration, continue to use the target object and / or reset the monitoring window.

[0029] In one embodiment, the method further comprises:

[0030] When at least one of rollback, update, fine-tuning, and retraining is not performed on the target object within a preset period, at least one of the following is performed:

[0031] Reset the timer or monitoring window;

[0032] receiving the first device configuration, the configuration including a second event-related configuration, and performing at least one of rollback, update, fine-tune, and retraining on the target object when a second event is triggered according to the second event-related configuration;

[0033] The first device configuration is received, where the configuration includes a third event related configuration, and according to the third event related configuration, when a third event is triggered, a monitoring result is reported to the first device.

[0034] A model performance monitoring method, the method being applied to a first device, comprising:

[0035] A first signaling including configuration information is sent to a second device so that the second device monitors a target object according to the configuration information, wherein the target object includes at least one of a second device, an artificial intelligence or machine learning AI / ML model, and a second device side function.

[0036] A model performance monitoring device, applied to a second device, comprising:

[0037] A receiving module, configured to receive a first signaling sent by a first device, wherein the first signaling includes configuration information;

[0038] A monitoring module is used to monitor the target object according to the configuration information, and the target object includes at least one of a second device, an artificial intelligence or machine learning AI / ML model, and a second device side function.

[0039] A model performance monitoring device, applied to a first device, comprising:

[0040] A sending module is used to send a first signaling containing configuration information to a second device, so that the second device can monitor a target object according to the configuration information. The target object includes at least one of a second device, an artificial intelligence or machine learning AI / ML model, and a second device side function.

[0041] A communication device, comprising: a receiver and a processor;

[0042] The receiver is configured to receive a first signaling sent by a first device, where the first signaling includes configuration information;

[0043] The processor is used to monitor the target object according to the configuration information, and the target object includes at least one of a second device, an artificial intelligence or machine learning AI / ML model, and a second device side function.

[0044] A communication device, comprising: a transmitter;

[0045] The transmitter is used to send a first signaling containing configuration information to a second device, so that the second device monitors a target object according to the configuration information. The target object includes at least one of a second device, an artificial intelligence or machine learning AI / ML model, and a second device side function.

[0046] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0047] receiving a first signaling sent by a first device, where the first signaling includes configuration information;

[0048] According to the configuration information, a target object is monitored, where the target object includes at least one of a second device, an artificial intelligence or machine learning AI / ML model, and a second device side function.

[0049] A computer program product includes a computer program, wherein when the computer program is executed by a processor, the model performance monitoring method provided in the embodiment of the present application is implemented, and the method may be:

[0050] receiving a first signaling sent by a first device, where the first signaling includes configuration information;

[0051] According to the configuration information, a target object is monitored, where the target object includes at least one of a second device, an artificial intelligence or machine learning AI / ML model, and a second device side function.

[0052] The above-mentioned model performance monitoring method, apparatus, communication equipment, storage medium and computer program product receive a first signaling sent by a first device, the first signaling includes configuration information, and monitors the target object according to the configuration information. The target object includes at least one of a second device, an artificial intelligence or machine learning AI / ML model, and a second device side function; the second device can monitor the AI / ML model performance according to the configuration information of the first device, and take corresponding measures in time when the AI / ML model performance deteriorates, so as to ensure the stability of the AI / ML model performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] FIG1 is an application environment diagram of a model performance monitoring method in one embodiment;

[0054] FIG2 is a schematic diagram of a flow chart of a model performance monitoring method in one embodiment;

[0055] FIG3 is an interactive flow chart of a model performance monitoring method in one embodiment;

[0056] FIG. 4 is a diagram showing the internal structure of a communication device in one embodiment. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] Fig. 1 is a schematic diagram of an application scenario of a model performance monitoring method provided in an embodiment of the present application. As shown in Fig. 1, the scenario includes a first device 102 and a second device 104. The first device 102 and the second device 104 perform data transmission via a network.

[0059] Among them, the first device 102 may be, but is not limited to, a core network element, an operation administration and maintenance (OAM), a base station, a network device, an artificial intelligence layer (AI Layer), an artificial intelligence management and orchestration layer, etc. The second device 104 may be, but is not limited to, an evolutionary Node B (eNB), the next generation Node B (gNB), a main control board, an intelligent computing board, a computing power board, a baseband board, a radio access network (RAN) element, a user equipment (UE), a radio access network artificial intelligence layer (RAN AI Layer), a wireless artificial intelligence management layer, a radio access network artificial intelligence management unit, a location management function (LMF) element, a transmission and receiving point (TRP), etc.

[0060] Among them, the base station can be a base station (Base Transceiver Station, BTS) in Global System of Mobile communication (GSM) or Code Division Multiple Access (CDMA), or a base station (NodeB, NB) in Wideband Code Division Multiple Access (WCDMA), or an evolved base station in LTE, or a relay station or access point, or a base station in a 5G network, etc., which is not limited here.

[0061] User equipment can be a wireless terminal, which can be a device that provides voice and / or other business data connectivity to users, or a handheld device with wireless connection function, or other processing equipment connected to a wireless modem. A wireless terminal can communicate with one or more core networks via a wireless access network. A wireless terminal can be a mobile terminal, such as a mobile phone (or "cellular" phone) and a computer with a mobile terminal, for example, a portable, pocket-sized, handheld, computer-built-in or vehicle-mounted mobile device, which exchanges language and / or data with a wireless access network. A wireless terminal can also be called a system, a subscriber unit, a subscriber station, a mobile station, a mobile station, a remote station, a remote terminal, an access terminal, a user terminal, or a user agent, without limitation here.

[0062] The network equipment may be, but is not limited to, various repeaters, routers, gateways, etc., which are not limited here. The artificial intelligence layer and the artificial intelligence management and orchestration layer may be various structures with artificial intelligence functions or management functions based on artificial intelligence. The radio access network artificial intelligence layer (RAN AI Layer) and the wireless artificial intelligence management layer (AI Layer) may be structures with artificial intelligence functions or management functions based on artificial intelligence in the radio access network. The radio access network artificial intelligence management unit may be a unit module with management functions based on artificial intelligence in the radio access network.

[0063] The following is a detailed description of the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. In addition, it should be pointed out that in the method and device of the present application, it is obvious that each component or each step can be decomposed and / or recombined, and these decompositions and / or recombinations should be regarded as equivalent schemes of the present application, and the steps of performing the above-mentioned series of processing can naturally be performed in chronological order according to the order of description, but it is not necessary to perform them in chronological order, and some steps can be performed in parallel or independently of each other. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0064] In one embodiment, as shown in FIG. 2 , a model performance monitoring method is provided, which is described by taking the method applied to the second device 104 in FIG. 1 as an example, and includes the following steps:

[0065] Step S202: Receive a first signaling sent by a first device, where the first signaling includes configuration information.

[0066] The first signaling may be a signaling message instructing the second device to monitor the performance of the AI / ML model. The configuration information may be configuration information related to the AI / ML model performance monitoring.

[0067] In a specific implementation, the first device may send a first signaling to the second device, where the first signaling includes configuration information, and the second device receives the first signaling including the configuration information.

[0068] For example, when the first device is a base station and the second device is a terminal, an AI / ML model may be configured on the terminal. In order to enable the terminal to monitor the performance of the AI / ML model in real time, the base station may send a first signaling containing configuration information to the terminal.

[0069] Step S204: monitor the target object according to the configuration information, where the target object includes at least one of a second device, an artificial intelligence or machine learning AI / ML model, and a second device side function.

[0070] The target object may be a monitored object.

[0071] In a specific implementation, the second device may monitor the second device, the AI / ML model, the second device side function, etc. according to the configuration information in the first signaling.

[0072] For example, the terminal can obtain the configuration information in the first signaling and monitor the performance of its own AI / ML model based on the configuration information.

[0073] It can be understood that AI / ML model performance monitoring (Model Monitoring) can be equivalent to monitoring the function or performance of the second device (Functionality Monitoring, UE-side Functionality Monitoring), and can also be equivalent to monitoring AI / ML-specific features or exclusive technologies, AI / ML feature groups or technology groups (AI / ML-specific Feature Group); wherein the function can be equivalent to at least one of supported functions (Supported Functionalities), available functions (Applicable Functionalities), and activated functions (Activated Functionalities).

[0074] It can be understood that the monitoring in the embodiments of the present application can be equivalent to detection, including but not limited to performance detection, function detection, etc.

[0075] The above-mentioned model performance monitoring method receives a first signaling sent by a first device, the first signaling includes configuration information, and monitors the target object according to the configuration information. The target object includes at least one of a second device, an artificial intelligence or machine learning AI / ML model, and a second device side function; the second device can monitor the AI / ML model performance according to the configuration information of the first device, and take corresponding measures in time when the AI / ML model performance deteriorates, so as to ensure the stability of the AI / ML model performance.

[0076] In one embodiment, the first device includes at least one of a core network network element, a network management operation management and maintenance, a base station, a network device, an artificial intelligence layer, and an artificial intelligence management orchestration layer; the second device includes at least one of an eNB, a gNB, a main control board, an intelligent computing board, a computing power board, a baseband board, a wireless access network element, a user equipment, a wireless access network artificial intelligence layer, a wireless artificial intelligence management layer, a wireless access network artificial intelligence management unit, a location management function network element LMF, and a sending and receiving point TRP.

[0077] In the specific implementation, the first device can be but is not limited to core network elements, network management operation and maintenance, base stations, network equipment, artificial intelligence layer, artificial intelligence management orchestration layer, etc. The second device can be but is not limited to eNB, gNB, main control board, intelligent computing board, computing power board, baseband board, wireless access network element, user equipment, wireless access network artificial intelligence layer, wireless artificial intelligence management layer, wireless access network artificial intelligence management unit, location management function network element LMF, sending and receiving point TRP, etc.

[0078] It can be understood that the first device can be a device that sends signaling to the second device. For example, when the first device is a network management system, the second device can be a base station; when the first device is a base station, the second device can be a user device; when the first device is a core network element, the second device can be a base station or a user; when the first device is a network management system, the second device can be a main control board, a computing power board, an intelligent computing board, a baseband board or a wireless access network element; when the first device is an artificial intelligence layer or an artificial intelligence management and orchestration layer, the second device can be a wireless access network artificial intelligence layer, a wireless artificial intelligence management layer or a wireless access network artificial intelligence management unit.

[0079] In this embodiment, by setting up the first device and the second device, the second device can obtain configuration information from the first device, and monitor the AI / ML model performance according to the configuration information to ensure the AI / ML model performance.

[0080] In one embodiment, the first signaling includes at least one of management plane message, management plane signaling, control plane message, control plane signaling, intelligent plane message, intelligent plane signaling, data plane message, data plane signaling, computing plane message, computing plane signaling, user plane message, user plane signaling, system message, non-access layer NAS signaling, access layer AS signaling, dedicated configuration message, dedicated configuration signaling, radio resource control RRC signaling, high-level protocol signaling, media access control control unit MAC CE, access control layer signaling, open interface protocol signaling, downlink control information DCI, physical layer signaling, layer 1 L1 signaling, layer 2 L2 signaling, and layer 3 L3 signaling.

[0081] Among them, the intelligent plane message and the intelligent plane signaling can be messages and signaling related to artificial intelligence, respectively. The computing plane message and the computing plane signaling can be messages and signaling related to artificial intelligence computing, respectively. The dedicated configuration message and the dedicated configuration signaling can be messages and signaling dedicated to artificial intelligence, respectively.

[0082] In a specific implementation, the first signaling sent by the first device to the second device may be, but is not limited to, a management plane message / signaling, a control plane message / signaling, an intelligent plane message / signaling, a data plane message / signaling, a computing plane message / signaling, a user plane message / signaling, a system message, a non-access stratum (NAS) signaling, an access stratum (AS) signaling, a dedicated configuration message, a dedicated configuration signaling, a radio resource control (RRC) signaling, a high-level protocol signaling, a media access control control unit (MACCE), an access control layer signaling, an open interface protocol signaling, downlink control information (DCI), a physical layer signaling, a layer 1 (L1) signaling, a layer 2 (L2) signaling, a layer 3 (L4) signaling, etc.

[0083] In this embodiment, by setting the first signaling, the first device can send configuration information to the second device to instruct the second device to monitor the performance of the AI / ML model.

[0084] In one embodiment, the configuration information includes whether to enable performance monitoring, performance measurement reporting configuration, and at least one of the performance monitoring configuration; the performance measurement reporting configuration includes at least one of the reporting identifier, measurement reporting type, measurement reporting period, model identifier, model function, feature identifier, and feature group identifier; the performance monitoring configuration includes at least one of the performance monitoring period, performance monitoring threshold, model accuracy requirement, performance monitoring reporting resource configuration, and fallback configuration.

[0085] Whether to enable performance monitoring refers to whether to enable the AI / ML model performance monitoring function. The performance measurement reporting configuration may be configuration information for reporting performance monitoring results. The performance monitoring configuration may be configuration information indicating monitoring of AI / ML model performance.

[0086] In a specific implementation, the configuration information may include at least one of whether to enable performance monitoring, performance measurement reporting configuration, and performance monitoring configuration.

[0087] Among them, whether to enable performance monitoring may be to indicate whether the second device enables the AI / ML model performance monitoring function.

[0088] The performance measurement reporting configuration may include, but is not limited to, a reporting identifier (an identifier of the reported measurement result), a measurement reporting type (periodic, semi-continuous or non-periodic), a measurement reporting period (a period for reporting the measurement result), a model identifier (Functionality / model ID), a model function, a feature identifier, a feature group identifier, etc. Among them, the model identifier is used to indicate the applicable scenario or function of the AI / ML model, such as enhanced wireless air interface channel state information (CSI) feedback, beam management, enhanced indoor positioning accuracy, RRM prediction, mobility enhancement, wireless slicing, coverage and capacity enhancement, load balancing, network energy saving, etc. The model identifier may be applicable to a single use case or feature, or to a group of use cases or feature groups. For example, a UE may have a single or multiple AI / ML models corresponding to a model identifier. For a use case (e.g., beam management), there may be a single or multiple model identifiers, that is, the model identifier may be used to configure one or more AI / ML models used in the performance detection reporting configuration.

[0089] Performance monitoring configuration may include, but is not limited to, performance monitoring period, performance monitoring threshold, model accuracy requirements (threshold of AI / ML model accuracy, etc.), performance monitoring reporting resource configuration (configuration of resources for reporting measurement results), fallback configuration (configuration related to AI / ML model fallback), etc.

[0090] In this embodiment, by setting the configuration information, the second device can monitor the AI / ML model performance according to the configuration information, thereby improving the efficiency and reliability of model performance monitoring.

[0091] In one embodiment, the above-mentioned model performance monitoring method may further specifically include: reporting the monitoring result to the first device, wherein the reporting includes at least one of periodic reporting, non-periodic reporting, and semi-continuous reporting.

[0092] The monitoring result may be a result obtained by performing performance monitoring on the AI / ML model.

[0093] In a specific implementation, the second device may report the monitoring result to the first device, and the manner of reporting the monitoring result includes but is not limited to periodic reporting, non-periodic reporting, semi-continuous reporting, and the like.

[0094] In this embodiment, by reporting the monitoring results to the first device, the first device can obtain the performance monitoring results of the AI / ML model of the second device, so as to facilitate scheduling according to the performance monitoring results.

[0095] In one embodiment, the above-mentioned model performance monitoring method may further specifically include at least one of the following: according to the monitoring results, performing at least one of management, activation, deactivation, rollback, update, fine-tuning, retraining, continued use, and maintenance on the target object; according to the monitoring results and configuration information, performing at least one of management, activation, deactivation, rollback, update, fine-tuning, retraining, continued use, and maintenance on the target object; receiving a second signaling sent by the first device, and according to the second signaling, performing at least one of management, activation, deactivation, rollback, update, fine-tuning, retraining, continued use, and maintenance on the target object.

[0096] The second signaling may be a signaling message instructing the second device to manage the AI / ML model.

[0097] Among them, update refers to replacing a new AI / ML model. Fine-tuning refers to adjusting the parameters of the current AI / ML model. Retraining refers to retraining the current AI / ML model. Continue to use refers to continuing to use the current AI / ML model. Keep means not doing anything. Fallback refers to switching back to a traditional solution or a non-AI solution. It is understandable that fallback can be equivalent to deactivation or functionality switching.

[0098] In a specific implementation, after obtaining the monitoring results of the AI / ML model, the second device may manage, activate, deactivate, rollback, update, fine-tune, retrain, continue to use or maintain operations on the second device, the AI / ML model, the second device side function, etc. according to the monitoring results; the second device may also manage, activate, deactivate, rollback, update, fine-tune, retrain, continue to use or maintain operations on the second device, the AI / ML model, the second device side function, etc. according to the monitoring results and the acquired configuration information; in addition, the second device may also report the monitoring results to the first device, the first device makes a decision based on the reported monitoring results and generates a second signaling, and returns the second signaling to the second device, and the second device manages, activates, deactivates, rolls back, updates, fine-tunes, retrains, continues to use or maintains operations on the second device, the AI / ML model, the second device side function, etc. according to the received second signaling.

[0099] In this embodiment, at least one of management, activation, deactivation, rollback, update, fine-tuning, retraining, continued use, and maintenance is performed on the target object according to the monitoring results; at least one of management, activation, deactivation, rollback, update, fine-tuning, retraining, continued use, and maintenance is performed on the target object according to the monitoring results and configuration information; a second signaling sent by the first device is received; and at least one of management, activation, deactivation, rollback, update, fine-tuning, retraining, continued use, and maintenance is performed on the target object according to the second signaling. According to the performance monitoring results of the AI / ML model, corresponding processing measures can be taken on the AI / ML model to ensure model performance.

[0100] In one embodiment, the above-mentioned model performance monitoring method may further specifically include at least one of the following: when a first condition is met, at least one of deactivation, rollback, update, fine-tuning, and retraining is performed on the target object, and the first condition includes that the monitoring result is lower than the monitoring threshold or the accuracy of the AI / ML model is lower than the model accuracy requirement; when a second condition is met, the target object is replaced with a new target object, and the second condition includes at least one of the following: the monitoring result of the new target object is higher than the monitoring result of the target object; the monitoring result of the new target object is higher than the monitoring threshold; the monitoring result of the new target object has a specified number of monitoring results that are higher than the monitoring result of the target object within a specified time window.

[0101] Among them, the first condition may be that the monitoring result is lower than a preset monitoring threshold, or the accuracy of the AI / ML model is lower than a preset accuracy threshold. It is understandable that the monitoring results in the embodiments of the present application include but are not limited to model accuracy, model performance, accuracy, performance results, etc., and the monitoring threshold includes but is not limited to a preset threshold, performance threshold, accuracy threshold, AI / ML model accuracy threshold, model performance threshold, etc.

[0102] Among them, the second condition may be that the monitoring result of the new model is higher than the monitoring result of the original model, or the monitoring result of the new model is higher than the monitoring result of the original model and the monitoring result of the new model is higher than the monitoring result of the original model for more than a specified number of times within a specified time window.

[0103] Among them, replacement includes but is not limited to change, switch, deployment, etc.

[0104] In a specific implementation, the second device can deactivate, roll back, update, fine-tune or retrain the current second device, AI / ML model, second device-side function, etc. when the performance monitoring result is lower than the monitoring threshold (the model accuracy is lower than the model accuracy requirement). The second device can also replace the current second device, AI / ML model, second device-side function with a new second device, AI / ML model, second device-side function when the performance monitoring result of the new second device, AI / ML model, second device-side function is higher than the current performance monitoring result, or when the performance monitoring result of the new second device, AI / ML model, second device-side function is higher than the current performance monitoring result and the monitoring threshold, or when the performance monitoring result of the new second device, AI / ML model, second device-side function is continuously / satisfies a certain number of times higher than the monitoring threshold within a specified time window.

[0105] In this embodiment, by performing at least one of deactivation, rollback, update, fine-tuning, and retraining on the target object when the first condition is met, and replacing the target object with a new target object when the second condition is met, corresponding processing can be performed on the AI / ML model based on the monitoring results and preset conditions to ensure model performance.

[0106] In one embodiment, the above-mentioned model performance monitoring method may further specifically include at least one of the following items: when a first event is triggered, starting or resetting the timer; the first event includes a monitoring result being lower than a monitoring threshold; within the timing range of the timer, if the number of occurrences of the first event exceeds a preset number of times or the duration of the occurrence of the first event exceeds a preset duration, performing at least one of deactivation, rollback, update, fine-tuning, and retraining on the target object; within the timing range of the timer, if the number of occurrences of the first event does not exceed the preset number of times or the duration of the occurrence of the first event does not exceed the preset duration, continue to use the target object and / or reset the timer.

[0107] The first event may be that the monitoring result is lower than a preset monitoring threshold, wherein the monitoring result may be the accuracy of the AI / ML model, and the monitoring threshold may be the threshold of the AI / ML model accuracy.

[0108] In a specific implementation, the second device can start or reset the timer when the performance monitoring result is lower than the monitoring threshold (the model accuracy is lower than the model accuracy requirement). Before the timer times out, the number of times or duration that the performance monitoring result is lower than the monitoring threshold is counted. If the number exceeds the preset number or the duration exceeds the preset duration, deactivation, rollback, update, fine-tuning, retraining and other operations are performed on the second device, AI / ML model, second device-side function, etc. Otherwise, if the number does not exceed the preset number or the duration does not exceed the preset duration, the current second device, AI / ML model, second device-side function, etc. can continue to be used, and the timer can also be reset.

[0109] In this embodiment, when the first event is triggered, the timer is started or reset. Within the timing range of the timer, if the number of occurrences of the first event exceeds the preset number of times or the duration of the first event exceeds the preset duration, at least one of deactivation, rollback, update, fine-tuning, and retraining is performed on the target object. Within the timing range of the timer, if the number of occurrences of the first event does not exceed the preset number of times or the duration of the first event does not exceed the preset duration, the target object continues to be used and / or the timer is reset. Deactivation, rollback, update, fine-tuning, retraining, continued use, and other operations of the AI / ML model can be triggered based on events to ensure model performance.

[0110] In one embodiment, the above-mentioned model performance monitoring method may further specifically include at least one of the following: when a first event is triggered, starting or resetting the monitoring window; the first event includes a monitoring result being lower than a monitoring threshold; within the monitoring window, if the number of occurrences of the first event exceeds a preset number of times or the duration of the occurrence of the first event exceeds a preset duration, performing at least one of deactivation, rollback, update, fine-tuning, and retraining on the target object; within the monitoring window, if the number of occurrences of the first event does not exceed a preset number of times or the duration of the occurrence of the first event does not exceed a preset duration, continuing to use the target object and / or resetting the monitoring window.

[0111] The monitoring window may be a time window or a time length for monitoring model performance.

[0112] In a specific implementation, the second device can start or reset the monitoring window when the performance monitoring result is lower than the monitoring threshold (the model accuracy is lower than the model accuracy requirement). Before the end of the monitoring window, the number of times or duration that the performance monitoring result is lower than the monitoring threshold is counted. If the number exceeds the preset number or the duration exceeds the preset duration, deactivation, rollback, update, fine-tuning, retraining and other operations are performed on the second device, AI / ML model, second device-side functions, etc. Otherwise, if the number does not exceed the preset number or the duration does not exceed the preset duration, the current second device, AI / ML model, second device-side functions, etc. can continue to be used, and the monitoring window can also be reset.

[0113] It is understandable that at least one of the timer, preset number of times, preset duration, and monitoring window in the embodiment of the present application may be predefined or included in the configuration information of the first signaling. Among them, the preset duration may be the window length or the timer length, and the preset number of times may be a threshold, a preset number, or a preset value.

[0114] In this embodiment, when the first event is triggered, the monitoring window is started or reset. Within the monitoring window, if the number of occurrences of the first event exceeds a preset number of times or the duration of the first event exceeds a preset duration, at least one of deactivation, rollback, update, fine-tuning, and retraining is performed on the target object. Within the monitoring window, if the number of occurrences of the first event does not exceed the preset number of times or the duration of the first event does not exceed the preset duration, the target object continues to be used and / or the monitoring window is reset. Deactivation, rollback, update, fine-tuning, retraining, continued use, and other operations of the AI / ML model can be triggered based on events to ensure model performance.

[0115] In one embodiment, the above-mentioned model performance monitoring method may further specifically include: when at least one of rollback, update, fine-tuning, and retraining is not performed on the target object within a preset period, performing at least one of the following: resetting the timer or the monitoring window; receiving a first device configuration, the configuration including a second event-related configuration, and according to the second event-related configuration, when the second event is triggered, performing at least one of rollback, update, fine-tuning, and retraining on the target object; receiving a first device configuration, the configuration including a third event-related configuration, and according to the third event-related configuration, when the third event is triggered, reporting the monitoring results to the first device.

[0116] Among them, the preset period may be a period for starting model performance monitoring. The second event-related configuration may be configuration information related to the second event. The second event may be an event that triggers model rollback, update, fine-tuning, or retraining. The third event-related configuration may be configuration information related to the third event. The third event may be an event that triggers reporting of model performance monitoring results.

[0117] In a specific implementation, the second device can start model performance monitoring according to a preset period. When no rollback, update, fine-tuning, retraining or other operations are performed on the target object within a preset period, the second device can reset the timer or monitoring window; the second device can also receive configuration information of the first device, and the configuration information can include a second event-related configuration that triggers a second event. According to the second event-related configuration, when the second event is triggered, the second device can perform rollback, update, fine-tuning, retraining or other operations on the target object; the configuration information of the first device can also include a third event-related configuration that triggers a third event. According to the third event-related configuration, when the third event is triggered, the second device can report the model performance monitoring results to the first device.

[0118] It is understandable that the second event-related configuration can be implemented by the first device starting or resetting the timer or monitoring window when the monitoring result is lower than the monitoring threshold, and the configuration is determined according to the number of times or duration that the monitoring result is lower than the monitoring threshold within the timer timing range or the monitoring window. The third event-related configuration can be implemented by configuring the first device to detect the event. For example, the first device can be configured to report the specified monitoring result when the second device obtains the specified monitoring result. Since the aforementioned embodiments have been described in detail, they will not be repeated here.

[0119] In this embodiment, when at least one of rollback, update, fine-tuning, and retraining is not performed on the target object within a preset period, at least one of the following is performed: resetting a timer or a monitoring window; receiving a first device configuration, the configuration including a second event-related configuration, and according to the second event-related configuration, when the second event is triggered, performing at least one of rollback, update, fine-tuning, and retraining on the target object; receiving a first device configuration, the configuration including a third event-related configuration, and according to the third event-related configuration, when the third event is triggered, reporting the monitoring result to the first device, so that the second device can trigger the rollback, update, fine-tuning, retraining of the model and reporting of the monitoring result according to the first device configuration, thereby saving processing resources of the second device.

[0120] In order to facilitate those skilled in the art to have a deeper understanding of the embodiments of the present application, a specific example will be described below.

[0121] The present application proposes a method for monitoring the performance of an AI / ML model by deploying an AI model on the wireless side, which is used to ensure the performance monitoring of the AI ​​model in actual network deployment, ensure the stability of network system performance, and improve user perception. In one embodiment, the method includes the following steps:

[0122] Step S401: The second device receives a signaling from the first device, where the signaling includes model performance monitoring configuration information, where the model performance monitoring configuration information is used by the second device to perform AI / ML model performance monitoring;

[0123] Step S402, the first device is at least one of the following: core network element, network management OAM, base station, AI Layer, AI management orchestration layer; the second device is at least one of the following: base station, main control board, intelligent computing board, computing power board, baseband board, wireless access network element, user equipment UE, RAN AI Layer, wireless AI management layer, RAN AI management unit;

[0124] For example, when the first device is a network management system, the second device is a base station; when the first device is a base station, the second device is a user device; when the first device is a core network element, the second device is a base station or a user; when the first device is a network management system, the second device is a main control board / computing board / intelligent computing board / baseband board / wireless access network element; when the first device is an AI Layer / AI management orchestration layer, the second device is a RAN AI Layer / wireless AI management layer / RAN AI management unit;

[0125] Step S403, the signaling includes: at least one of management plane message / signaling, control plane message / signaling, intelligence plane message / signaling, data plane message / signaling, computing plane message / signaling, system message, NAS / AS signaling, dedicated configuration message / signaling, RRC / high-level protocol signaling, MAC CE, access control layer signaling, open interface protocol signaling, and DCI / physical layer signaling;

[0126] Step S404: The second device receives a signal from the first device, where the signal includes configuration information for performance detection of the AI ​​model, and the configuration information includes at least one of the following information:

[0127] 1. Whether to enable AI / ML model performance monitoring;

[0128] 2. AI / ML model performance measurement reporting configuration, including at least one of the following: reporting identifier / ID, measurement reporting type (periodic / semi-continuous / non-periodic), measurement reporting period, model identifier (functionality / model ID);

[0129] The model identifier is used to indicate the applicable scenario or function of the AI / ML model, such as: wireless air interface CSI feedback enhancement, beam management, indoor positioning accuracy enhancement, RRM prediction, mobility enhancement, wireless slicing, coverage and capacity enhancement, load balancing, and network energy saving;

[0130] Model identification can apply to a single use case or to a group of use cases;

[0131] For example, a UE may have a single or multiple AI / ML models corresponding to a model identifier (functionality / modelID);

[0132] For a use case, such as beam management, there can be single or multiple model IDs (functionality / model ID);

[0133] That is, the performance detection related configuration may use the model identification configuration to configure one or more models used by the performance detection configuration;

[0134] 3. Model performance monitoring configuration, including at least one of the following: performance monitoring cycle, performance monitoring threshold, model accuracy requirements, performance monitoring reporting resource configuration, and fallback configuration;

[0135] Step S405, the second device performs a model performance detection process according to the configuration information;

[0136] Step S406: Optionally, the second device reports the performance detection result, where the reporting includes at least one of periodic reporting, aperiodic reporting, and semi-continuous reporting;

[0137] Step S407: Execute performance detection process

[0138] Option 1, the second device performs at least one of model update, model replacement, rollback, and model fine-tuning according to the performance detection result;

[0139] Option 2, the second device performs at least one of model update, model replacement, rollback, and model fine-tuning according to the performance detection result and the configuration information;

[0140] Option 3, the first device makes a decision according to the performance detection report; the second device receives the signaling of the first device, and performs at least one of model update, model replacement, rollback, and model fine-tuning according to the signaling;

[0141] Step S408, executing a rollback process based on conditional triggering: defining a model rollback triggering condition, executing model rollback / update / replacement according to the model performance monitoring configuration, executed by the first device or the second device, and supporting at least one of the following:

[0142] When the model accuracy is lower than the performance monitoring threshold / model accuracy requirement, deactivate the current AI / ML model or fall back to the traditional / non-AI solution;

[0143] When the accuracy of the newly constructed model is higher than that of the existing model, or when the accuracy of the newly constructed model is higher than that of the existing model and the accuracy meets the threshold, or when the accuracy of the newly constructed model is higher than that of the existing model for a certain number of times within the time window timer length, the model is replaced and the existing model is replaced with the newly constructed model;

[0144] When the model accuracy is worse than the performance monitoring threshold / model accuracy requirement, the model update process is initiated to perform data collection / model fine-tuning / retraining;

[0145] Step S409, executing the rollback process based on event triggering: defining the model rollback triggering event, and executing the model rollback / update / replacement according to the model performance monitoring configuration, for example:

[0146] 1. Define model rollback events A1 / A2 / A3, model update events B1 / B2 / B3, and model replacement events C1 / C2 / C3. Note: the event letters can be replaced by any English letters, strings, words, or numbers and their combinations, for example: Event_A1, and the number of events is not limited to 3, but can be several;

[0147] 2. When the model accuracy is worse than the performance monitoring threshold / model accuracy requirement, start / reset the timer / monitoring window. When the number of times the model accuracy is worse than the performance detection threshold / model accuracy requirement exceeds M or the duration exceeds L before the timer times out / the sliding window ends, where M≥1 and L≥0, the current AI / ML model is deactivated or rolled back to the traditional / non-AI solution; otherwise, the model continues to be used and the timer is reset.

[0148] For example, if the positioning enhancement setting performance detection threshold / model accuracy requires the positioning accuracy threshold to be 30 cm, and before the timer expires after 30 minutes, if the model accuracy (model output performance) is greater than (worse than) the performance detection threshold requirement for more than M=3 times, the current AI / ML positioning model is deactivated;

[0149] 3. When the model accuracy is worse than the performance monitoring threshold / model accuracy requirement, start / reset the timer / monitoring window. When the number of times the model accuracy is worse than the performance detection threshold / model accuracy requirement exceeds M or the duration exceeds L before the timer times out / the sliding window ends, where M≥1 and L≥0, start the model reconstruction process, or start the model reconstruction process while falling back to the traditional / non-AI solution, and reset / stop the timer; otherwise, continue to use the model and reset / stop the timer.

[0150] For example, the beam prediction accuracy is defined as 80% correct. When the model accuracy is worse than the performance detection threshold (80% correct) for more than 20 seconds, the model reconstruction process is started and the traditional beam management solution is reverted.

[0151] 4. Start model performance detection according to a predefined period W. If there is no model update / replacement / rollback within a period, perform at least one of the following:

[0152] Reset the above timer;

[0153] According to the events configured by the first device (for example, detecting events A2 and C1 according to the configuration of the base station), when the event triggering condition is met, executing model update / replacement / rollback;

[0154] According to the events configured by the first device (for example, according to the base station configuration detection events A1 and B3), when the event triggering condition is met, performance monitoring measurement reporting is performed.

[0155] The above-mentioned method for wireless side AI / ML model performance monitoring defines that the second device receives signaling from the first device, and the signaling includes model performance detection configuration information, and the model performance monitoring configuration information is used for the second device to perform AI / ML model performance monitoring; the first device is defined as at least one of the following: core network element, network management OAM, base station, AILayer, AI management orchestration layer; the second device is at least one of the following: base station, main control board, intelligent computing board, computing power board, baseband board, wireless access network element, user equipment UE, RAN AI Layer, wireless AI management layer, RAN AI management unit; define that the signaling includes: management plane message / signaling, control plane message / signaling, intelligence plane message / signaling, data plane message / signaling, computing plane message / signaling, system message, NAS / AS signaling, dedicated configuration message / signaling, RRC / high-level protocol signaling, MACCE, access control layer signaling, open interface protocol signaling, DCI / physical layer signaling; define that the second device receives the first device signaling, and the signaling includes configuration information for performance detection of the AI ​​model, and the specific content of the configuration information; define (optionally) that the second device reports / reports the performance detection result, and the report includes: at least one of periodic reporting, non-periodic reporting, and semi-continuous reporting; define that the performance detection processing process includes automatic execution by the second device, execution by the second device according to the configuration, and execution by the first device; define that the performance detection combing process includes a fallback process based on conditional triggering, define the model fallback triggering condition, and execute the model fallback / update / replacement according to the model performance monitoring configuration, which is executed by the first device or the second device; define the fallback process based on event triggering.

[0156] This embodiment proposes a method suitable for monitoring the performance of AI / ML models on the wireless side, which can implement signaling configuration for monitoring the performance of AI / ML models between the wireless network and the base station, and between the base station and the terminal on the RAN side, so as to enable the network to monitor, manage and control the performance of the AI ​​model, and enable AI to inherently improve the performance of the wireless communication system.

[0157] In one embodiment, another model performance monitoring method is provided, which is illustrated by taking the first device in Figure 1 as an example. The method includes: sending a first signaling containing configuration information to a second device, so that the second device monitors a target object according to the configuration information, and the target object includes a second device, an artificial intelligence or machine learning AI / ML model, and at least one of a second device side function.

[0158] In a specific implementation, the first device may send a first signaling to the second device, where the first signaling includes configuration information. The second device may perform performance monitoring on the second device, the AI / ML model, the second device side function, etc. based on the received configuration information.

[0159] Since the specific processing process of the first device has been described in detail in the above embodiments, it will not be repeated here.

[0160] In this embodiment, a first signaling including configuration information is sent to a second device so that the second device can monitor a target object according to the configuration information. The target object includes a second device, an artificial intelligence or machine learning AI / ML model, and at least one of a second device side function. The second device can monitor the performance of the AI / ML model according to the configuration information of the first device, and take corresponding measures in a timely manner when the performance of the AI / ML model deteriorates, so as to ensure stable performance of the AI / ML model.

[0161] In one embodiment, as shown in FIG3 , an interactive flow chart of a model performance monitoring method is provided, including:

[0162] Step S302: The first device sends a first signaling to the second device, where the first signaling includes configuration information;

[0163] Step S304: The second device receives the first signaling and obtains configuration information in the first signaling;

[0164] Step S306: The second device monitors the target object according to the configuration information, where the target object includes at least one of the second device, an artificial intelligence or machine learning AI / ML model, and a second device-side function;

[0165] Step S308: The second device reports the monitoring result to the first device.

[0166] In a specific implementation, the first device may send a first signaling to the second device, where the first signaling includes configuration information. The second device may perform performance monitoring on the second device, the AI / ML model, the second device side functions, etc. based on the received configuration information, and report the obtained monitoring results to the first device.

[0167] Since the specific processing procedures of the first device and the second device have been described in detail in the aforementioned embodiments, they will not be repeated here.

[0168] In this embodiment, a first signaling is sent from a first device to a second device, the first signaling includes configuration information, the second device receives the first signaling, obtains the configuration information in the first signaling, and monitors the target object according to the configuration information; the second device can monitor the performance of the AI / ML model according to the configuration information of the first device, and take corresponding measures in time when the performance of the AI / ML model deteriorates, so as to ensure the stability of the performance of the AI / ML model.

[0169] It should be understood that, although the various steps in the flowcharts of Figures 2-3 are displayed in sequence according to the indications of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in Figures 2-3 may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the steps or stages in other steps.

[0170] In one embodiment, a model performance monitoring device is provided, which is applied to a second device and includes:

[0171] A receiving module, configured to receive a first signaling sent by a first device, wherein the first signaling includes configuration information;

[0172] A monitoring module is used to monitor the target object according to the configuration information, and the target object includes at least one of a second device, an artificial intelligence or machine learning AI / ML model, and a second device side function.

[0173] In one embodiment, the first device includes at least one of a core network network element, a network management operation and maintenance, a base station, a network device, an artificial intelligence layer, and an artificial intelligence management and orchestration layer; the second device includes at least one of an eNB, a gNB, a main control board, an intelligent computing board, a computing power board, a baseband board, a wireless access network element, a user equipment, a wireless access network artificial intelligence layer, a wireless artificial intelligence management layer, a wireless access network artificial intelligence management unit, a location management function network element LMF, and a sending and receiving point TRP.

[0174] In one embodiment, the first signaling includes at least one of management plane message, management plane signaling, control plane message, control plane signaling, intelligent plane message, intelligent plane signaling, data plane message, data plane signaling, computing plane message, computing plane signaling, user plane message, user plane signaling, system message, non-access layer NAS signaling, access layer AS signaling, dedicated configuration message, dedicated configuration signaling, radio resource control RRC signaling, high-level protocol signaling, media access control control unit MAC CE, access control layer signaling, open interface protocol signaling, downlink control information DCI, physical layer signaling, layer 1 L1 signaling, layer 2 L2 signaling, and layer 3 L3 signaling.

[0175] In one embodiment, the configuration information includes whether to enable performance monitoring, performance measurement reporting configuration, and at least one of performance monitoring configuration; the performance measurement reporting configuration includes at least one of reporting identification, measurement reporting type, measurement reporting period, model identification, model function, feature identification, and feature group identification; the performance monitoring configuration includes at least one of performance monitoring period, performance monitoring threshold, model accuracy requirement, performance monitoring reporting resource configuration, and fallback configuration.

[0176] In one embodiment, the above-mentioned model performance monitoring device further includes a reporting module, which is used to report monitoring results to the first device, and the reporting includes at least one of periodic reporting, non-periodic reporting, and semi-continuous reporting.

[0177] In one embodiment, the above-mentioned model performance monitoring device also includes a processing module for executing at least one of the following: executing at least one of management, activation, deactivation, rollback, update, fine-tuning, retraining, continued use, and maintenance on the target object according to the monitoring results; executing at least one of management, activation, deactivation, rollback, update, fine-tuning, retraining, continued use, and maintenance on the target object according to the monitoring results and the configuration information; receiving a second signaling sent by the first device, and executing at least one of management, activation, deactivation, rollback, update, fine-tuning, retraining, continued use, and maintenance on the target object according to the second signaling.

[0178] In one embodiment, the above-mentioned model performance monitoring device also includes a conditional trigger module, which is used to execute at least one of the following: when a first condition is met, at least one of deactivation, rollback, update, fine-tuning, and retraining is performed on the target object, and the first condition includes that the monitoring result is lower than the monitoring threshold or the accuracy of the AI / ML model is lower than the model accuracy requirement; when a second condition is met, the target object is replaced with a new target object, and the second condition includes at least one of the following: the monitoring result of the new target object is higher than the monitoring result of the target object; the monitoring result of the new target object is higher than the monitoring threshold; the monitoring result of the new target object has a specified number of monitoring results that are higher than the monitoring result of the target object within a specified time window.

[0179] In one embodiment, the above-mentioned model performance monitoring device also includes a first event triggering module, which is used to execute at least one of the following: when the first event is triggered, starting or resetting the timer; the first event includes a monitoring result being lower than a monitoring threshold; within the timing range of the timer, if the number of occurrences of the first event exceeds a preset number of times or the duration of the occurrence of the first event exceeds a preset duration, performing at least one of deactivation, rollback, update, fine-tuning, and retraining on the target object; within the timing range of the timer, if the number of occurrences of the first event does not exceed the preset number of times or the duration of the occurrence of the first event does not exceed the preset duration, continue to use the target object and / or reset the timer.

[0180] In one embodiment, the above-mentioned model performance monitoring device also includes a second event triggering module, which is used to execute at least one of the following: when a first event is triggered, starting or resetting the monitoring window; the first event includes a monitoring result being lower than a monitoring threshold; within the monitoring window, if the number of occurrences of the first event exceeds a preset number of times or the duration of the occurrence of the first event exceeds a preset duration, performing at least one of deactivation, rollback, update, fine-tuning, and retraining on the target object; within the monitoring window, if the number of occurrences of the first event does not exceed a preset number of times or the duration of the occurrence of the first event does not exceed a preset duration, continue to use the target object and / or reset the monitoring window.

[0181] In one embodiment, the above-mentioned model performance monitoring device also includes a third event triggering module, which is used to perform at least one of the following when at least one of rollback, update, fine-tuning, and retraining is not performed on the target object within a preset period: reset the timer or monitoring window; receive the first device configuration, the configuration includes a second event-related configuration, and according to the second event-related configuration, when the second event is triggered, perform at least one of rollback, update, fine-tuning, and retraining on the target object; receive the first device configuration, the configuration includes a third event-related configuration, and according to the third event-related configuration, when the third event is triggered, report the monitoring results to the first device.

[0182] In one embodiment, a model performance monitoring device is provided, which is applied to a first device and includes:

[0183] A sending module is used to send a first signaling containing configuration information to a second device, so that the second device can monitor a target object according to the configuration information. The target object includes at least one of a second device, an artificial intelligence or machine learning AI / ML model, and a second device side function.

[0184] For the specific definition of the model performance monitoring device, please refer to the definition of the model performance monitoring method above, which will not be repeated here. The various modules in the above-mentioned model performance monitoring device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0185] In one embodiment, a communication device is provided, see Figure 4. Figure 4 is a schematic diagram of the structure of a terminal device provided in an embodiment of the present application. The terminal device 700 shown in Figure 4 includes: at least one processor 701, a memory 702, at least one network interface 704 and a user interface 703. The various components in the terminal device 700 are coupled together through a bus system 705. It can be understood that the bus system 705 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 705 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, various buses are marked as bus systems 705 in Figure 1. In addition, in an embodiment of the present application, a transceiver 706 is also included. The transceiver can be a plurality of components, that is, a transmitter and a receiver, providing a unit for communicating with various other devices on a transmission medium.

[0186] The user interface 703 may include a display, a keyboard, or a pointing device (eg, a mouse, a trackball, a touch pad, or a touch screen).

[0187] It can be understood that the memory 702 in the embodiment of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DRRAM). The memory 702 of the system and method described in the embodiments of the present application is intended to include but is not limited to these and any other suitable types of memory.

[0188] In some implementations, the memory 702 stores the following elements, executable modules or data structures, or a subset thereof, or an extended set thereof: an operating system 7021 and an application 7022 .

[0189] The operating system 7021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application 7022 includes various application programs, such as a media player (MediaPlayer), a browser (Browser), etc., which are used to implement various application services. The program for implementing the method of the embodiment of the present application can be included in the application 7022.

[0190] In an embodiment of the present application, by calling the program or instructions stored in the memory 702, specifically, it can be the program or instructions stored in the application 7022, wherein the receiver is used to receive a first signaling sent by a first device, and the first signaling includes configuration information; the processor is used to monitor the target object according to the configuration information, and the target object includes at least one of a second device, an artificial intelligence or machine learning AI / ML model, and a second device side function.

[0191] Part or all of the methods disclosed in the above embodiments of the present application may also be applied to the processor 701, or implemented by the processor 701, or implemented by the processor 701 in cooperation with other components (such as a transceiver). The processor 701 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method may be completed by an integrated logic circuit of hardware or software instructions in the processor 701. The above processor 701 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 702, and the processor 701 reads the information in the memory 702 and completes the steps of the above method in combination with its hardware.

[0192] It is understood that the embodiments described in the embodiments of the present application can be implemented by hardware, software, firmware, middleware, microcode or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (Application Specific Integrated Circuits, ASIC), digital signal processors (Digital Signal Processing, DSP), digital signal processing devices (DSP Device, DSPD), programmable logic devices (Programmable Logic Device, PLD), field programmable gate arrays (Field-Programmable Gate Array, FPGA), general processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application or a combination thereof.

[0193] For software implementation, the technology described in the embodiments of the present application can be implemented by a module (such as a process, function, etc.) that performs the functions described in the embodiments of the present application. The software code can be stored in a memory and executed by the processor 701. The memory can be implemented in the processor 701 or outside the processor 701.

[0194] In one embodiment, the first device includes at least one of a core network network element, a network management operation and maintenance, a base station, a network device, an artificial intelligence layer, and an artificial intelligence management and orchestration layer; the second device includes at least one of an eNB, a gNB, a main control board, an intelligent computing board, a computing power board, a baseband board, a wireless access network element, a user equipment, a wireless access network artificial intelligence layer, a wireless artificial intelligence management layer, a wireless access network artificial intelligence management unit, a location management function network element LMF, and a sending and receiving point TRP.

[0195] In one embodiment, the first signaling includes at least one of management plane message, management plane signaling, control plane message, control plane signaling, intelligent plane message, intelligent plane signaling, data plane message, data plane signaling, computing plane message, computing plane signaling, user plane message, user plane signaling, system message, non-access layer NAS signaling, access layer AS signaling, dedicated configuration message, dedicated configuration signaling, radio resource control RRC signaling, high-level protocol signaling, media access control control unit MAC CE, access control layer signaling, open interface protocol signaling, downlink control information DCI, physical layer signaling, layer 1 L1 signaling, layer 2 L2 signaling, and layer 3 L3 signaling.

[0196] In one embodiment, the configuration information includes whether to enable performance monitoring, performance measurement reporting configuration, and at least one of performance monitoring configuration; the performance measurement reporting configuration includes at least one of reporting identification, measurement reporting type, measurement reporting period, model identification, model function, feature identification, and feature group identification; the performance monitoring configuration includes at least one of performance monitoring period, performance monitoring threshold, model accuracy requirement, performance monitoring reporting resource configuration, and fallback configuration.

[0197] In one embodiment, the transmitter is specifically used to report the monitoring result to the first device, and the reporting includes at least one of periodic reporting, non-periodic reporting, and semi-continuous reporting.

[0198] In one embodiment, the processor is further used to perform at least one of management, activation, deactivation, rollback, update, fine-tuning, retraining, continued use, and maintenance on the target object based on the monitoring results; perform at least one of management, activation, deactivation, rollback, update, fine-tuning, retraining, continued use, and maintenance on the target object based on the monitoring results and the configuration information; receive a second signaling sent by the first device, and perform at least one of management, activation, deactivation, rollback, update, fine-tuning, retraining, continued use, and maintenance on the target object based on the second signaling.

[0199] In one embodiment, the processor is further used to perform at least one of deactivation, rollback, update, fine-tuning, and retraining on the target object when a first condition is met, and the first condition includes that the monitoring result is lower than a monitoring threshold or the accuracy of the AI / ML model is lower than the model accuracy requirement; when a second condition is met, the target object is replaced with a new target object, and the second condition includes at least one of the following: the monitoring result of the new target object is higher than the monitoring result of the target object; the monitoring result of the new target object is higher than the monitoring threshold; the monitoring result of the new target object has a specified number of monitoring results that are higher than the monitoring result of the target object within a specified time window.

[0200] In one embodiment, the processor is further used to start or reset the timer when a first event is triggered; the first event includes a monitoring result being lower than a monitoring threshold; within the timing range of the timer, if the number of occurrences of the first event exceeds a preset number of times or the duration of the occurrence of the first event exceeds a preset duration, at least one of deactivation, rollback, update, fine-tuning, and retraining is performed on the target object; within the timing range of the timer, if the number of occurrences of the first event does not exceed the preset number of times or the duration of the occurrence of the first event does not exceed the preset duration, continue to use the target object and / or reset the timer.

[0201] In one embodiment, the processor is further used to start or reset the monitoring window when a first event is triggered; the first event includes a monitoring result being lower than a monitoring threshold; within the monitoring window, if the number of occurrences of the first event exceeds a preset number of times or the duration of the occurrence of the first event exceeds a preset duration, at least one of deactivation, rollback, update, fine-tuning, and retraining is performed on the target object; within the monitoring window, if the number of occurrences of the first event does not exceed a preset number of times or the duration of the occurrence of the first event does not exceed a preset duration, continue to use the target object and / or reset the monitoring window.

[0202] In one embodiment, the processor is further used to execute at least one of the following when at least one of rollback, update, fine-tuning, and retraining is not performed on the target object within a preset period: resetting a timer or a monitoring window; receiving the first device configuration, the configuration including a second event-related configuration, and according to the second event-related configuration, when a second event is triggered, executing at least one of rollback, update, fine-tuning, and retraining on the target object; receiving the first device configuration, the configuration including a third event-related configuration, and according to the third event-related configuration, when a third event is triggered, reporting monitoring results to the first device.

[0203] In one embodiment, another communication device is provided, in which a transmitter is used to send a first signaling containing configuration information to a second device, so that the second device monitors a target object according to the configuration information, and the target object includes at least one of a second device, an artificial intelligence or machine learning AI / ML model, and a second device side function.

[0204] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0205] The embodiments of the present application also provide a computer program product including instructions, which, when executed on a computer, enables the computer to execute the steps in the above-mentioned method embodiments.

[0206] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0207] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0208] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A model performance monitoring method, characterized in that: The method is applied to a second device, comprising: receiving a first signaling sent by a first device, where the first signaling includes configuration information; According to the configuration information, a target object is monitored, where the target object includes at least one of a second device, an artificial intelligence or machine learning AI / ML model, and a second device side function.

2. The method according to claim 1, characterized in that The first device includes at least one of core network network elements, network management operation and maintenance, base stations, network equipment, artificial intelligence layer, and artificial intelligence management and orchestration layer; the second device includes at least one of eNB, gNB, main control board, intelligent computing board, computing power board, baseband board, wireless access network network elements, user equipment, wireless access network artificial intelligence layer, wireless artificial intelligence management layer, wireless access network artificial intelligence management unit, location management function network element LMF, and sending and receiving point TRP.

3. The method according to claim 1, characterized in that The first signaling includes at least one of management plane message, management plane signaling, control plane message, control plane signaling, intelligent plane message, intelligent plane signaling, data plane message, data plane signaling, computing plane message, computing plane signaling, user plane message, user plane signaling, system message, non-access layer NAS signaling, access layer AS signaling, dedicated configuration message, dedicated configuration signaling, radio resource control RRC signaling, high-level protocol signaling, media access control control unit MAC CE, access control layer signaling, open interface protocol signaling, downlink control information DCI, physical layer signaling, layer 1 L1 signaling, layer 2 L2 signaling, and layer 3 L3 signaling.

4. The method according to claim 1, characterized in that The configuration information includes whether to enable performance monitoring, performance measurement reporting configuration, and at least one of performance monitoring configuration; the performance measurement reporting configuration includes at least one of reporting identification, measurement reporting type, measurement reporting period, model identification, model function, feature identification, and feature group identification; the performance monitoring configuration includes at least one of performance monitoring period, performance monitoring threshold, model accuracy requirement, performance monitoring reporting resource configuration, and fallback configuration.

5. The method according to claim 1, characterized in that The method further comprises: Reporting monitoring results to the first device, wherein the reporting includes at least one of periodic reporting, non-periodic reporting, and semi-continuous reporting.

6. The method according to claim 1, characterized in that The method further comprises at least one of the following: According to the monitoring result, performing at least one of management, activation, deactivation, rollback, update, fine-tuning, retraining, continued use, and maintenance on the target object; According to the monitoring result and the configuration information, perform at least one of management, activation, deactivation, rollback, update, fine-tuning, retraining, continued use, and maintenance on the target object; Receive a second signaling sent by the first device, and based on the second signaling, perform at least one of management, activation, deactivation, rollback, update, fine-tuning, retraining, continued use, and maintenance on the target object.

7. The method according to claim 1, characterized in that The method further comprises at least one of the following: When a first condition is met, performing at least one of deactivation, rollback, update, fine-tuning, and retraining on the target object, wherein the first condition includes that the monitoring result is lower than the monitoring threshold or the accuracy of the AI / ML model is lower than the model accuracy requirement; When the second condition is met, the target object is replaced with a new target object, and the second condition includes at least one of the following: the monitoring result of the new target object is higher than the monitoring result of the target object; the monitoring result of the new target object is higher than the monitoring threshold; the monitoring result of the new target object has a specified number of results higher than the monitoring result of the target object within a specified time window.

8. The method according to claim 1, characterized in that The method further comprises at least one of the following: When a first event is triggered, starting or resetting a timer; the first event includes a monitoring result being lower than a monitoring threshold; Within the timing range of the timer, if the number of occurrences of the first event exceeds a preset number of times or the duration of the occurrence of the first event exceeds a preset duration, performing at least one of deactivation, rollback, update, fine-tuning, and retraining on the target object; Within the timing range of the timer, if the number of occurrences of the first event does not exceed the preset number of times or the duration of the occurrence of the first event does not exceed the preset duration, continue to use the target object and / or reset the timer.

9. The method according to claim 1, characterized in that: The method further comprises at least one of the following: When a first event is triggered, starting or resetting the monitoring window; the first event includes a monitoring result being lower than a monitoring threshold; Within the monitoring window, if the number of occurrences of the first event exceeds a preset number of times or the duration of the occurrence of the first event exceeds a preset duration, performing at least one of deactivation, rollback, update, fine-tuning, and retraining on the target object; Within the monitoring window, if the number of occurrences of the first event does not exceed a preset number of times or the duration of the occurrence of the first event does not exceed a preset duration, continue to use the target object and / or reset the monitoring window.

10. The method according to claim 1, characterized in that The method further comprises: When at least one of rollback, update, fine-tuning, and retraining is not performed on the target object within a preset period, at least one of the following is performed: Reset the timer or monitoring window; receiving the first device configuration, the configuration including a second event-related configuration, and performing at least one of rollback, update, fine-tune, and retraining on the target object according to the second event-related configuration when a second event is triggered; The first device configuration is received, where the configuration includes a third event related configuration, and according to the third event related configuration, when a third event is triggered, a monitoring result is reported to the first device.

11. A model performance monitoring method, characterized in that: The method is applied to a first device, comprising: A first signaling including configuration information is sent to a second device so that the second device monitors a target object according to the configuration information, wherein the target object includes at least one of a second device, an artificial intelligence or machine learning AI / ML model, and a second device side function.

12. A model performance monitoring device, characterized in that: The apparatus is applied to a second device, comprising: A receiving module, configured to receive a first signaling sent by a first device, wherein the first signaling includes configuration information; A monitoring module is used to monitor the target object according to the configuration information, and the target object includes at least one of a second device, an artificial intelligence or machine learning AI / ML model, and a second device side function.

13. A model performance monitoring device, characterized in that: The device is applied to a first device and includes: A sending module is used to send a first signaling containing configuration information to a second device, so that the second device can monitor a target object according to the configuration information. The target object includes at least one of a second device, an artificial intelligence or machine learning AI / ML model, and a second device side function.

14. A communication device, characterized in that: include: Receiver and processor; The receiver is configured to receive a first signaling sent by a first device, where the first signaling includes configuration information; The processor is used to monitor the target object according to the configuration information, and the target object includes at least one of a second device, an artificial intelligence or machine learning AI / ML model, and a second device side function.

15. A communication device, characterized in that: include: Transmitter; The transmitter is used to send a first signaling containing configuration information to a second device, so that the second device monitors a target object according to the configuration information. The target object includes at least one of a second device, an artificial intelligence or machine learning AI / ML model, and a second device side function.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 or 11 are implemented.

17. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 or 11 are implemented.

Images