Perception model performance monitoring method and system, SF entity and NWDAF entity

Abstract

The invention provides a perception model performance monitoring method and system, an SF entity and an NWDAF entity, and relates to the field of mobile communication. The method comprises the following steps: an SF entity receives a perception model trained by an NWDAF entity; based on the perception data, utilizing a perception model to obtain a perception result; and monitoring the performance of the perception model based on the perception result.

Description

Technical Field

[0001] The present disclosure relates to the field of mobile communications, and in particular to a perception model performance monitoring method, system, SF (Sensing Function) entity and NWDAF (Network Data Analytics Function) entity. Background Art

[0002] With the rapid development of network technology, the scale and complexity of the network are constantly increasing, and the demand for high-speed transmission, low latency and high reliability is also growing. In this context, the integration of communication and perception technology (Integrated sensing and communication) is gradually becoming the focus of technological innovation. The core of this fusion technology is to complete the sensing and communication tasks simultaneously through RF (Radio Frequency) signals, without adding additional hardware costs, to achieve real-time monitoring of the environment and rapid transmission of data. Intelligent perception technology based on AI (Artificial Intelligence) / ML (Machine Learning) has become the key to improving network performance. This technology can significantly improve the intelligence level and operational efficiency of the network. Real-time performance monitoring of the perception model can promptly detect performance bottlenecks or declines, so as to quickly respond and make adjustments to ensure service quality. Summary of the invention

[0003] A technical problem to be solved by the present disclosure is to provide a perception model performance monitoring method, system, SF entity and NWDAF entity, which can realize the performance monitoring of the perception model and improve the performance of intelligent perception and the level of network intelligence.

[0004] According to one aspect of the present disclosure, a perception model performance monitoring method is proposed, which is executed by a perception function SF entity, including: receiving a perception model trained by a network data analysis function NWDAF entity; using the perception model based on the perception data to obtain a perception result; and monitoring the performance of the perception model based on the perception result.

[0005] In some embodiments, a NWDAF entity supporting the perception model is discovered based on the perception model training capability identifier and the first model interoperability information.

[0006] In some embodiments, the NWDAF entity is discovered based on the perception model training capability identifier, the first model interoperability information, and the model filtering information.

[0007] In some embodiments, a registration request is sent to a network storage function NRF entity, where the registration request includes second model interoperability information and a support identifier based on intelligent perception and model performance monitoring.

[0008] In some embodiments, monitoring the performance of the perception model includes analyzing the accuracy of the perception results.

[0009] In some embodiments, when it is determined that verification data needs to be collected, the verification data is collected; and based on the verification data, the accuracy of the perception result is analyzed.

[0010] In some embodiments, when the accuracy of the perception result does not meet a preset standard, the perception service user is instructed to stop using the output result of the perception model.

[0011] In some embodiments, the accuracy of the perception results is sent to the NWDAF entity so that the NWDAF entity determines whether to retrain the perception model.

[0012] In some embodiments, an accuracy analysis subscription request sent by a user of accuracy analysis is received; and the accuracy of the perception result is sent to the user of accuracy analysis so that the user of accuracy analysis determines whether to modify the perception service request.

[0013] In some embodiments, the verification accuracy analyzes the parameters carried in the subscription request; and in the case where the parameter verification passes, the accuracy of the perception result is analyzed.

[0014] In some embodiments, when the user of the perception service and the user of the accuracy analysis are the same network element, the perception result and the accuracy of the perception result are sent to the user of the perception service at the same time.

[0015] According to another aspect of the present disclosure, a perception model performance monitoring method is also proposed, which is executed by a network data analysis function NWDAF entity, including: receiving a perception model subscription request sent by a perception function SF entity; training a perception model based on perception data; and sending the perception model to the SF entity so that the SF entity monitors the performance of the perception model.

[0016] In some embodiments, a registration request is sent to a network storage function NRF entity, where the registration request includes a perception model training capability identifier and first model interoperability information.

[0017] In some embodiments, the registration request also includes model filtering information.

[0018] In some embodiments, an accuracy analysis subscription request is sent to a SF entity; the accuracy of the perception result sent by the SF entity is received; and based on the accuracy of the perception result, it is determined whether to retrain the perception model.

[0019] According to another aspect of the present disclosure, a perception function SF entity is also proposed, including: a model receiving module, configured to receive a perception model trained by a network data analysis function NWDAF entity; a reasoning module, configured to obtain a perception result by using the perception model based on the perception data; and a performance monitoring module, configured to monitor the performance of the perception model based on the perception result.

[0020] According to another aspect of the present disclosure, a perception function SF entity is also proposed, including: a memory; and a processor coupled to the memory, wherein the processor is configured to execute the above-mentioned perception model performance monitoring method based on instructions stored in the memory.

[0021] According to another aspect of the present disclosure, a network data analysis function NWDAF entity is proposed, including: a request receiving module, configured to receive a perception model subscription request sent by a perception function SF entity; a training module, configured to train a perception model based on perception data; and a model sending module, configured to send the perception model to the SF entity, so that the SF entity can monitor the performance of the perception model.

[0022] According to another aspect of the present disclosure, a network data analysis function NWDAF entity is also proposed, including: a memory; and a processor coupled to the memory, the processor being configured to execute the above-mentioned perception model performance monitoring method based on instructions stored in the memory.

[0023] According to another aspect of the present disclosure, a perception model performance monitoring system is also proposed, including: the above-mentioned perception function SF entity; and the above-mentioned network data analysis function NWDAF entity.

[0024] According to another aspect of the present disclosure, a computer-readable storage medium is also proposed, on which computer instructions are stored, and when the instructions are executed by a processor, the above-mentioned perception model performance monitoring method is implemented.

[0025] According to another aspect of the present disclosure, a computer program product is also proposed, including computer instructions, which implement the above-mentioned perception model performance monitoring method when executed by a processor.

[0026] In the disclosed embodiment, the SF entity performs perception result inference on the perception model trained by the NWDAF entity in order to monitor the performance of the perception model, thereby ensuring that the perception model maintains optimal performance and accuracy in actual applications, and timely adapts to environmental changes and the evolution of data characteristics, thereby providing more accurate and reliable intelligent perception services.

[0027] Other features and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0029] The present disclosure may be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0030] Figure 1 It is a flowchart of some embodiments of the perception model performance monitoring method disclosed in the present invention;

[0031] Figure 2 Schematic diagram of the flow chart of other embodiments of the perception model performance monitoring method disclosed in the present invention;

[0032] Figure 3 Schematic diagram of the flow chart of other embodiments of the perception model performance monitoring method disclosed in the present invention;

[0033] Figure 4 Schematic diagram of the flow chart of other embodiments of the perception model performance monitoring method disclosed in the present invention;

[0034] Figure 5 Schematic diagram of the flow chart of other embodiments of the perception model performance monitoring method disclosed in the present invention;

[0035] Figure 6 It is a schematic diagram of the structure of some embodiments of the SF entity disclosed in the present invention;

[0036] Figure 7 Schematic diagrams of structures of other embodiments of the SF entity disclosed in the present invention;

[0037] Figure 8 It is a schematic diagram of the structure of some embodiments of the NWDAF entity disclosed in the present invention;

[0038] Fig. 9 It is a schematic diagram of the structure of some embodiments of the NWDAF entity disclosed in the present invention;

[0039] Fig.10 It is a schematic diagram of the structure of some embodiments of the NWDAF entity disclosed in the present invention. DETAILED DESCRIPTION

[0040] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless otherwise specifically stated.

[0041] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0042] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0043] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0044] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0045] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0046] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0047] In the construction and development of communication-aware fusion networks, SF plays a vital role. For example, it is responsible for collecting and analyzing environmental perception information. In addition, the intelligent unit NWDAF introduced by 3GPP in the 5G (5th Generation Mobile Networks) R15 version acts as the core engine for data analysis in the core network. The functions of NWDAF cover data collection, machine learning model training, logical reasoning, intelligent prediction, etc. The analysis results it outputs provide strong support for network operations, management decisions and application strategy formulation. With the continuous advancement of network technology, network intelligence has become an irreversible development direction. By integrating the environmental data collected by SF with the deep analysis capabilities of NWDAF, we can not only achieve more comprehensive data fusion and intelligent analysis, but also significantly improve the network's operating efficiency and the user's overall service experience.

[0048] In the field of intelligent perception, the performance stability of intelligent perception is crucial to ensure data accuracy and service quality. Due to the inconsistency of data quality and the limitation of model generalization ability, intelligent perception may encounter performance fluctuation problems in practical applications. These problems may arise from a variety of factors, including but not limited to noise in data collection, environmental changes, model obsolescence or improper parameter settings. Performance monitoring of perception models is an issue that needs to be addressed.

[0049] The present disclosure provides a perception model performance monitoring method, which realizes the performance monitoring of the perception model through the collaboration of the SF entity and the NWDAF entity, thereby improving the performance of intelligent perception and the level of network intelligence.

[0050] Figure 1 It is a flowchart of some embodiments of the perception model performance monitoring method disclosed in the present invention. The embodiment is executed by the SF entity and includes steps S11-S13.

[0051] In step S11, the perception model trained by the NWDAF entity is received.

[0052] In some embodiments, the NWDAF entity is divided into two logical functions, namely AnLF (Analytics Logical Function) and MTLF (Model Training Logical Function), AnLF is responsible for reasoning, and MTLF is responsible for model training. In this embodiment, SF can be combined with AnLF responsible for model reasoning or embedded with a module responsible for reasoning calculation, so as to realize the reasoning calculation and performance monitoring of perception data processing and final perception results.

[0053] The SF entity receives the perception request from the perception service user. If it decides to use an AI / ML-based solution for perception, it will discover the NWDAF entity with MTLF that supports AI / ML synaesthesia model subscription and sharing, and subscribe to the perception model for AI / ML perception from the NWDAF entity with MTLF. After the MTLF of the NWDAF entity collects the perception data, it trains the perception model and sends the trained perception model to the SF entity.

[0054] The perception service user is, for example, UE (User Equipment), AF (Application Function), Consumer NF (Network Function), Consumer NF is, for example, AMF (Access and Mobility Management Function), SMF (Session Management Function), etc.

[0055] The perception request includes, for example, service type (e.g., dynamic map, vehicle tracking, etc.), service requirements (e.g., perception resolution, perception accuracy, duration, target area information, latency, etc.), and perception node information (e.g., UE information).

[0056] In step S12, a perception result is obtained based on the perception data using a perception model.

[0057] After receiving the model trained by the NWDAF entity containing MTLF, the SF entity collects the perception data for perception reasoning, and uses the perception data to perform result reasoning based on the perception model to obtain the perception result. The SF entity can feed back the perception result obtained by reasoning to the perception service user.

[0058] In step S13, based on the perception results, the performance of the perception model is monitored.

[0059] For example, analyzing the accuracy of perception results, the accuracy of perception results reflects the performance of the perception model.

[0060] In the above embodiment, the SF entity performs perception result inference on the perception model trained by the NWDAF entity in order to monitor the performance of the perception model, thereby ensuring that the perception model maintains optimal performance and accuracy in actual applications, and timely adapts to environmental changes and the evolution of data characteristics, thereby providing more accurate and reliable intelligent perception services.

[0061] Figure 2 Schematic diagram of the flow chart of some other embodiments of the perception model performance monitoring method disclosed in the present invention, which is executed by SF and includes steps S21-S26.

[0062] In step S21, the NWDAF entity supporting the perception model is discovered according to the perception model training capability identifier and the first model interoperability information.

[0063] In some embodiments, the NWDAF entity sends a registration message to the NRF (Network Repository Function) entity, and the registration message includes a perception model training capability identifier and a first model interoperability information. The perception model training capability identifier is used to determine whether the NWDAF entity supports AI / ML-based perception model training. The first model interoperability information is used to determine whether the model can be shared between the SF entity and the NWDAF entity. The first model interoperability information includes, for example, a list of manufacturers of ML models shared by the NWDAF entity and the format and operating environment of the ML model. The SF entity discovers the NWDAF entity containing MTLF that supports AI / ML synaesthesia model subscription and sharing through NRF.

[0064] In some embodiments, a NWDAF entity supporting the perception model is discovered based on the perception model training capability identifier, the first model interoperability information, and the model filtering information.

[0065] In some embodiments, the NWDAF entity sends a registration message to the NRF entity, the registration message including a perception model training capability identifier, first model interoperability information, and model filtering information. The model filtering information can be used for model selection to obtain a model corresponding to the model subscription request. The model filtering information includes, for example, slice information, target area, etc.

[0066] In step S22, a perception model for AI / ML perception is subscribed from the NWDAF entity including the MTLF.

[0067] For example, SF subscribes to ML models for AI / ML perception from NWDAF with MTLF through the Nnwdaf_MLModelProvision_Subscribe service.

[0068] In step S23, the perception model trained by the NWDAF entity including the MTLF is received.

[0069] In step S24, the perception model is used to perform perception reasoning based on the perception data to obtain a perception result, which can be fed back to the perception service user.

[0070] In some embodiments, the sensing service user sends sensing node information, and the SF entity collects sensing data based on the sensing node information. The sensing node information includes, for example, identification information of the sensing node, network information location information, and the like.

[0071] In step S25, if it is determined that verification data needs to be collected, the verification data is collected.

[0072] In some embodiments, an accuracy analysis subscription request sent by a user of the accuracy analysis is received to evaluate the accuracy of the perception result. If the SF entity does not contain enough necessary data, verification data needs to be collected. For example, data collection is performed according to the analysis ID (Identifier). The analysis ID represents a specific perception task, for example, the perception task is intrusion monitoring, positioning, etc., so that the SF entity collects the corresponding verification data.

[0073] Users of analysis accuracy are, for example, MTLF or perception service users.

[0074] If it is not necessary to collect verification data, step S26 can be directly executed.

[0075] In step S26, the accuracy of the perception result is analyzed based on the verification data.

[0076] The accuracy of the perception result includes, for example, the correct rate and false alarm rate of target detection, or the perception delay, the environmental adaptability of the perception model, etc.

[0077] In the above embodiment, the SF entity is able to collect and process environmental perception data in real time, thereby enabling reasoning and performance monitoring of the perception model, which helps to optimize network resource allocation and thus improve the overall efficiency of the network.

[0078] In some embodiments, when the accuracy of the perception result does not meet a preset standard, the SF entity instructs the perception service user to stop using the output result of the perception model.

[0079] For example, when the SF entity returns the inferred perception results to the perception service user, but determines that the performance of the perception model is poor, the SF entity can instruct the perception service user to stop using the output results of the perception model, thereby preventing the erroneous perception results from being further used.

[0080] In some embodiments, the SF entity sends the accuracy of the perception results to the NWDAF entity so that the NWDAF entity determines whether to retrain the perception model.

[0081] For example, the NWDAF entity is a user of analysis accuracy, and the MTLF in the NWDAF entity can decide whether to retrain the model based on new analysis accuracy information. For example, if the accuracy of the perception result meets the preset standard, the perception model is not updated. If the accuracy of the perception result does not meet the preset standard, the perception model is retrained to improve the accuracy of the perception model to adapt to environmental changes and the evolution of data characteristics. Retraining the perception model, for example, is to recollect model training data and retrain the perception model, or it can be to replace the perception model and retrain the replaced perception model.

[0082] In some embodiments, the accuracy of the perception result is sent to a user who analyzes the accuracy so that the user who analyzes the accuracy determines whether to modify the perception service request.

[0083] For example, if the accuracy of the perception result meets the preset standard, it means that the perception model can provide accurate perception services, so the user who analyzes the accuracy can continue to subscribe to the perception service. If the accuracy of the perception result does not meet the preset standard, it means that the perception model cannot provide accurate perception services, so the user who analyzes the accuracy can modify the subscription perception service.

[0084] In some embodiments, when the user of the perception service and the user of the accuracy analysis are the same network element, the perception result and the accuracy of the perception result are sent to the user of the perception service at the same time.

[0085] For example, if the SF entity receives a subscription request for analyzing the accuracy of a perception model from a user of analysis accuracy, if it is determined that there is no need to collect new verification data, and the user of analysis accuracy and the perception service user are the same network element, the perception results and the accuracy of the perception results can be sent to the perception service user at the same time, thereby simplifying the interaction process.

[0086] In some embodiments, the SF entity verifies the accuracy of the parameters carried in the subscription request; and if the parameter verification passes, analyzes the accuracy of the perception result.

[0087] For example, when the SF entity receives a subscription request for analyzing the accuracy of a perception model from a user of analysis accuracy, it will first verify the parameters in the analysis accuracy subscription request. The parameters include, for example, analysis ID, model ID, monitoring time, result reporting target, etc. The SF entity checks the analysis ID and model ID to determine whether it is the perception task for which it is responsible for reasoning. If so, it starts the analysis accuracy monitoring of the perception model.

[0088] In some embodiments, the SF entity sends a registration request to the NRF entity, and the registration request includes the second model interoperability information and a support identifier based on intelligent perception and model performance monitoring. The second model interoperability information is used to determine whether the model can be shared between the SF entity and the NWDAF entity, for example, including the manufacturer ID of the SF and the format and operating environment of the ML model supported by the SF. The support identifier based on intelligent perception and model performance monitoring is used to determine whether the SF entity supports intelligent perception and model performance monitoring. Users of perception services or users who analyze accuracy can find SF entities that meet the requirements through the second model interoperability information and the support identifier based on intelligent perception and model performance monitoring.

[0089] Figure 3 1 is a flowchart of some other embodiments of the perception model performance monitoring method disclosed in the present invention. The embodiment is executed by the NWDAF entity and includes steps S31-S33.

[0090] In step S31, a perception model subscription request sent by a SF entity is received.

[0091] In some embodiments, the SF entity discovers a NWDAF entity that supports the perception model based on the perception model training capability identifier and the first model interoperability information, and sends a perception model subscription request to the NWDAF entity.

[0092] The NWDAF entity sends a registration request to the NRF entity, and the registration request includes a perception model training capability identifier and a first model interoperability information. The perception model training capability identifier is used to determine whether the NWDAF entity supports AI / ML-based perception model training. The first model interoperability information is used to determine whether models can be shared between the SF entity and the NWDAF entity. The first model interoperability information includes, for example, a list of manufacturers of shared ML models and the format and operating environment of the ML model. The SF entity discovers the NWDAF entity containing MTLF that supports AI / ML synaesthesia model subscription and sharing through NRF.

[0093] In other embodiments, the SF entity discovers a NWDAF entity that supports the perception model based on the perception model training capability identifier, the first model interoperability information, and the model filtering information, and sends a perception model subscription request to the NWDAF entity.

[0094] The NWDAF entity sends a registration request to the NRF entity, and the registration request includes a perception model training capability identifier, first model interoperability information, and model filtering information. The model filtering information can be used for model selection to obtain a model corresponding to the model subscription request. The model filtering information includes, for example, slice information, target area, etc.

[0095] In step S32, a perception model is trained based on the perception data.

[0096] The NWDAF entity collects necessary perception data and uses it to train AI / ML perception models to adapt to specific perception tasks.

[0097] In step S33, the perception model is sent to the SF entity so that the SF entity performs performance monitoring on the perception model.

[0098] After training the perception model, the NWDAF entity sends the perception model to the SF entity, which uses the perception model based on the perception data to obtain the perception results and monitors the performance of the perception model based on the perception results.

[0099] For example, after receiving the model trained by the NWDAF entity containing MTLF, the SF entity collects the perception data for perception reasoning, and uses the perception data to perform result reasoning based on the perception model to obtain the perception result. Then, the accuracy of the perception result is analyzed, and the accuracy of the perception result reflects the performance of the perception model.

[0100] In the above embodiment, the NWDAF entity trains the perception model through interaction with the SF entity, and sends the trained perception model to the SF entity so that the SF entity can monitor the performance of the perception model, thereby ensuring that the perception model maintains optimal performance and accuracy in actual applications, and adapts to environmental changes and the evolution of data characteristics in a timely manner, thereby providing more accurate and reliable intelligent perception services.

[0101] Figure 4 It is a flowchart diagram of other embodiments of the perception model performance monitoring method disclosed in the present invention, which is composed of an NWDAF entity and includes steps S41-S43.

[0102] In step S41, an accuracy analysis subscription request is sent to the SF entity.

[0103] For example, the NWDAF entity subscribes to the analysis accuracy information of the AI / ML perception model from the SF entity that supports model performance monitoring in order to evaluate the accuracy of the perception results.

[0104] In step S42, the accuracy of the sensing result sent by the SF entity is received.

[0105] In step S43, based on the accuracy of the perception result, it is determined whether to retrain the perception model.

[0106] For example, if the accuracy of the perception result meets the preset standard, the NWDAF entity does not update the perception model. If the accuracy of the perception result does not meet the preset standard, the NWDAF entity retrains the perception model to improve the accuracy of the perception model to adapt to environmental changes and the evolution of data characteristics. Retraining the perception model, for example, is to recollect model training data and retrain the perception model, or it can be to replace the perception model and retrain the replaced perception model.

[0107] In the above embodiment, the NWDAF entity interacts with the SF entity, and the intelligent perception performance can be improved through monitoring and feedback.

[0108] The following will introduce the collaboration process between the SF entity and the NWDAF entity of the present disclosure by taking a specific embodiment as an example. Figure 5 As shown, Figure 5 It is a flowchart diagram of other embodiments of the perception model performance monitoring method disclosed in the present invention, including steps S51-S512.

[0109] In step S51, the SF entity and the SF entity register with the NRF entity respectively.

[0110] For example, the NWDAF entity and the SF entity etc. register with the NRF through the Nnrf_NFManagement_NFRegister service respectively.

[0111] When the NWDAF entity containing MTLF registers with the NRF entity, a new synaesthesia model training capability identifier, new ML model interoperability information, and ML model filtering information are added. The ML model interoperability information includes a list of manufacturers that can share ML models with the NWDAF entity and the format and operating environment of the ML model, ensuring smooth collaboration and model subscription sharing between network elements.

[0112] When registering SF, a new identifier for its support for AI / ML-based intelligent perception and model performance monitoring is added, and a new mandatory parameter ML model interoperability information is added. The ML model interoperability information includes the manufacturer ID of the SF entity and the format and operating environment of the ML model supported by the SF entity, to ensure the smooth implementation of intelligent perception and perception model performance monitoring.

[0113] In step S52, the awareness service user sends an awareness request to the SF entity.

[0114] For example, the UE or AF sends a perception request to the SF entity, which includes the service type (such as dynamic map, vehicle tracking, etc.), service requirements (perception resolution, perception accuracy, duration, target area information, latency, etc.) and designated perception node information (such as UE information).

[0115] In step S53, the SF entity discovers the NWDAF entity with MTLF that supports AI / ML-aware model subscription and sharing, and subscribes to the ML model for AI / ML-awareness from the NWDAF entity with MTLF.

[0116] For example, if the SF entity decides to use an AI / ML-based approach for perception, the NWDAF entity containing MTLF that supports subscription and sharing of AI / ML synaesthesia models is discovered through the NRF entity.

[0117] The SF entity can decide whether to use traditional algorithms or perception model-based methods to calculate perception reasoning results based on built-in rules. The built-in rules are defined by the manufacturer. The reference basis for judgment includes parameters carried in the perception request, such as performance requirements for perception reasoning results, perceived environment (such as indoor and outdoor), and perception scenario (such as environmental intrusion monitoring and moving object tracking).

[0118] The SF entity subscribes to the ML model for AI / ML perception from the NWDAF entity with MTLF through the Nnwdaf_MLModelProvision_Subscribe service.

[0119] In step S54, the NWDAF entity collects perception data and trains the ML model.

[0120] The MTLF of the NWDAF entity collects necessary perception data, which can come from SF, UE, AMF or RAN (Radio Access Network), etc. These perception data are used to train ML models to adapt to specific perception tasks.

[0121] In step S55, the NWDAF entity sends the trained ML model to the SF entity.

[0122] For example, after MTLF completes ML model training, it sends the subscribed ML model information to the SF entity through the Nnwdaf_MLModelProvision_Notify service.

[0123] In step S56, after receiving the ML model trained by the NWDAF entity containing MTLF, the SF entity collects data for perception reasoning and performs perception result reasoning.

[0124] In step S57, the SF entity returns the inferred perception result to the perception service user, such as UE, NF (AMF, SMF), or AF.

[0125] In step S58, the user of analysis accuracy subscribes to the analysis accuracy information of the ML model from the SF entity that supports model performance monitoring.

[0126] For example, MTLF or perception service users subscribe to the analytical accuracy information of the ML model from the SF entity that supports model performance monitoring through the Nnwdaf_AnalyticsSubscription_Subscribe service, so as to evaluate the accuracy of the perception results.

[0127] In step S59, the SF entity verifies the parameters of the analysis accuracy request message and starts accuracy monitoring.

[0128] After receiving the subscription request, the SF entity verifies the parameters of the analysis accuracy request information, such as analysis ID, model ID, monitoring time, result reporting target, etc. If the SF entity does not have enough data, step S510 is executed. If new data does not need to be collected, and the user of the analysis accuracy and the perception service user are the same network element, the analysis accuracy information can be returned to the perception service user together with the inference result in step S67.

[0129] In step S510, the SF entity performs data collection according to the analysis ID and generates an inference result.

[0130] In step S511, the SF entity returns the accuracy of the inference result to the user who analyzed the accuracy.

[0131] For example, SF returns the monitored analysis accuracy information to MTLF or perception service users through Nnwdaf_AnalyticsSubscription_Subscribe. MTLF can decide whether to retrain the model based on the new analysis accuracy information, and perception service users can decide whether to modify the perception service subscription.

[0132] In step S512, if the monitored analysis accuracy does not meet the preset standard, the SF entity instructs to stop using the output of the current ML model to prevent the erroneous perception results from being further used.

[0133] In the above embodiment, the SF entity collaborates with the NWDAF entity including the MTLF, the NWDAF entity performs model training, and the SF entity performs reasoning and model performance monitoring. The life cycle management of the model can be realized, including model training, reasoning, monitoring and optimization, which helps to maintain the accuracy and timeliness of the model, thereby improving the performance of intelligent perception and the level of network intelligence.

[0134] Figure 6 The schematic diagram of the structure of some embodiments of the SF entity disclosed in the present invention is mainly responsible for the reasoning calculation and performance monitoring of the perception data processing and the final perception result. The SF entity can be combined with the AnLF responsible for model reasoning or embed the module responsible for reasoning calculation, so that the SF entity can reason and calculate the perception result according to the AI / ML model trained by the MTLF and the synaesthesia data collected by itself, and perform model performance monitoring. The SF entity includes a model receiving module 610, a reasoning module 620 and a performance monitoring module 630.

[0135] The model receiving module 610 is configured to receive the perception model trained by the NWDAF entity, for example, Figure 1 Shown is step S11.

[0136] In some embodiments, the SF entity discovers the NWDAF entity that supports the perception model based on the perception model training capability identifier and the first model interoperability information, and subscribes to the perception model for AI / ML perception from the NWDAF entity. After the MTLF of the NWDAF entity collects the perception data, it trains the perception model and sends the trained perception model to the SF entity.

[0137] For another example, the SF entity discovers the NWDAF entity based on the perception model training capability identifier, the first model interoperability information, and the model filtering information, and subscribes to the perception model for AI / ML perception from the NWDAF entity.

[0138] The reasoning module 620 is configured to use the perception model based on the perception data to obtain a perception result, such as executing Figure 1 Step S12 is shown.

[0139] The performance monitoring module 630 is configured to monitor the performance of the perception model based on the perception results.

[0140] For example, analyzing the accuracy of perception results, the accuracy of perception results reflects the performance of the perception model.

[0141] In some embodiments, the performance monitoring module 630 is further configured to collect verification data when it is determined that verification data needs to be collected; and analyze the accuracy of the perception result based on the verification data.

[0142] In some embodiments, the performance monitoring module 630 is further configured to instruct the perception service user to stop using the output results of the perception model when the accuracy of the perception results does not meet a preset standard.

[0143] In some embodiments, the performance monitoring module 630 is further configured to send the accuracy of the perception result to the NWDAF entity so that the NWDAF entity determines whether to retrain the perception model.

[0144] In some embodiments, the performance monitoring module 630 is also configured to receive an accuracy analysis subscription request sent by a user who analyzes accuracy, and then analyze the accuracy of the perception results; the accuracy of the perception results is sent to the user who analyzes accuracy so that the user who analyzes accuracy can determine whether to modify the perception service request.

[0145] In some embodiments, the performance monitoring module 630 is further configured to verify the accuracy of the parameters carried in the analysis subscription request; and if the parameter verification passes, analyze the accuracy of the perception result.

[0146] In some embodiments, the performance monitoring module 630 is further configured to send the perception result and the accuracy of the perception result to the perception service user at the same time when the perception service user and the user analyzing the accuracy are the same network element.

[0147] In the above embodiment, the SF entity performs perception result inference on the perception model trained by the NWDAF entity in order to monitor the performance of the perception model, thereby ensuring that the perception model maintains optimal performance and accuracy in actual applications, and timely adapts to environmental changes and the evolution of data characteristics, thereby providing more accurate and reliable intelligent perception services.

[0148] In some embodiments, the SF entity also includes a first registration module (not shown), which is configured to send a registration request to the NRF entity, wherein the registration request includes second model interoperability information and a support identifier based on intelligent perception and model performance monitoring, so that perception service users and analysis accuracy information users can find SFs that meet the requirements.

[0149] Figure 7 Schematic diagram of the structure of other embodiments of the SF entity of the present disclosure, the SF entity 700 includes a memory 710 and a processor 720. Among them: the memory 710 can be a disk, a flash memory or any other non-volatile storage medium. The memory is used to store the instructions in the above embodiments. The processor 720 is coupled to the memory 710 and can be implemented as one or more integrated circuits, such as a microprocessor or a microcontroller. The processor 720 is used to execute the instructions stored in the memory.

[0150] In some embodiments, the processor 720 is coupled to the memory 710 via a BUS 730. The SF entity 700 can also be connected to an external storage device 750 via a storage interface 740 to call external data, and can also be connected to a network or another computer system (not shown) via a network interface 760. No further detailed description will be given here.

[0151] Figure 8 8 is a schematic diagram of the structure of some embodiments of the NWDAF entity of the present disclosure, wherein the NWDAF entity includes the MTLF logic function, that is, is responsible for the training of the model. Specifically, the NWDAF entity includes a request receiving module 810, a training module 820 and a model sending module 830.

[0152] The request receiving module 810 is configured to receive a perception model subscription request sent by a SF entity.

[0153] In some embodiments, the SF entity discovers a NWDAF entity that supports the perception model based on the perception model training capability identifier and the first model interoperability information, and sends a perception model subscription request to the NWDAF entity.

[0154] The NWDAF entity also includes a second registration unit (not shown), which is configured to send a registration request to the NRF entity, where the registration request includes a perception model training capability identifier and first model interoperability information.

[0155] In other embodiments, the SF entity discovers a NWDAF entity that supports the perception model based on the perception model training capability identifier, the first model interoperability information, and the model filtering information, and sends a perception model subscription request to the NWDAF entity.

[0156] The second registration unit is also configured to send a registration request to the NRF entity, where the registration request includes a perception model training capability identifier, first model interoperability information, and model filtering information.

[0157] The training module 820 is configured to train a perception model based on the perception data.

[0158] The NWDAF entity collects necessary perception data and uses it to train AI / ML perception models to adapt to specific perception tasks.

[0159] The model sending module 830 is configured to send the perception model to the SF entity so that the SF entity performs performance monitoring on the perception model.

[0160] After training the perception model, the NWDAF entity sends the perception model to the SF entity, which uses the perception model based on the perception data to obtain the perception results and monitors the performance of the perception model based on the perception results.

[0161] In the above embodiment, the NWDAF entity trains the perception model through interaction with the SF entity, and sends the trained perception model to the SF entity so that the SF entity can monitor the performance of the perception model, thereby ensuring that the perception model maintains optimal performance and accuracy in actual applications, and adapts to environmental changes and the evolution of data characteristics in a timely manner, thereby providing more accurate and reliable intelligent perception services.

[0162] In other embodiments, Fig. 9 As shown, the NWDAF entity also includes a request sending module 910 , a performance receiving module 920 and a determination module 930 .

[0163] The request sending module 910 is configured to send an accuracy analysis subscription request to the SF entity.

[0164] For example, the request sending module 910 subscribes to the analysis accuracy information of the AI / ML perception model from the SF entity that supports model performance monitoring so as to evaluate the accuracy of the perception results.

[0165] The performance receiving module 920 is configured to receive the accuracy of the perception result sent by the SF entity.

[0166] The determination module 930 is configured to determine whether to retrain the perception model based on the accuracy of the perception result.

[0167] For example, if the accuracy of the perception result meets the preset standard, the NWDAF entity does not update the perception model. If the accuracy of the perception result does not meet the preset standard, the NWDAF entity retrains the perception model to improve the accuracy of the perception model to adapt to environmental changes and the evolution of data characteristics. Retraining the perception model, for example, is to recollect model training data and retrain the perception model, or it can be to replace the perception model and retrain the replaced perception model.

[0168] In the above embodiment, the NWDAF entity interacts with the SF entity, and the intelligent perception performance can be improved through monitoring and feedback.

[0169] Fig.10 1 is a schematic diagram of the structure of some embodiments of the NWDAF entity of the present disclosure. The NWDAF entity 1000 includes a memory 1010 and a processor 1020. Among them: the memory 1010 can be a disk, a flash memory or any other non-volatile storage medium. The memory is used to store the instructions in the above embodiments. The processor 1020 is coupled to the memory 1010 and can be implemented as one or more integrated circuits, such as a microprocessor or a microcontroller. The processor 1020 is used to execute the instructions stored in the memory.

[0170] In some embodiments, the processor 1020 is coupled to the memory 1010 via a BUS 1030. The NWDAF entity 1000 can also be connected to an external storage device 1050 via a storage interface 1040 to call external data, and can also be connected to a network or another computer system (not shown) via a network interface 1060. No further detailed description will be given here.

[0171] In other embodiments of the present disclosure, a perception model performance monitoring system is protected. The perception model performance monitoring system includes the SF entity and the NWDAF entity in the above embodiments. Through the collaboration of the SF entity and the NWDAF entity, the performance of intelligent perception and the level of network intelligence can be improved.

[0172] In other embodiments, a computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the steps of the method in the above-mentioned embodiment. It should be understood by those skilled in the art that the embodiments of the present disclosure may be provided as methods, devices, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable non-transient storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0173] In some embodiments, a computer program product is protected, including a computer program or an instruction, which implements the above method when the computer program or instruction is executed by a processor. The computer program product includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by the CPU, the above functions defined in the method of the embodiment of the present disclosure are executed.

[0174] The present disclosure is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram and the combination of processes and / or blocks in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0175] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0176] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0177] So far, the present disclosure has been described in detail. In order to avoid obscuring the concept of the present disclosure, some details known in the art are not described. Based on the above description, those skilled in the art can fully understand how to implement the technical solution disclosed here.

[0178] The method and apparatus of the present disclosure may be implemented in many ways. For example, the method and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

[0179] Although some specific embodiments of the present disclosure have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present disclosure. It should be understood by those skilled in the art that the above embodiments may be modified without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. A method for monitoring performance of a perception model, performed by a perception function SF entity, comprising: Receive the perception model trained by the network data analysis function NWDAF entity; Obtaining a perception result based on the perception data by using the perception model; as well as Based on the perception results, the performance of the perception model is monitored.

2. The method for monitoring performance of a perception model according to claim 1, further comprising: According to the perception model training capability identifier and the first model interoperability information, an NWDAF entity supporting the perception model is discovered.

3. The method for monitoring performance of a perception model according to claim 2, further comprising: The NWDAF entity is discovered according to the perception model training capability identifier, the first model interoperability information, and the model filtering information.

4. The method for monitoring performance of a perception model according to claim 1, further comprising: A registration request is sent to a network storage function NRF entity, where the registration request includes the second model interoperability information and a support identifier based on intelligent perception and model performance monitoring.

5. The method for monitoring the performance of a perception model according to any one of claims 1 to 4, wherein: Monitoring the performance of the perception model includes: The accuracy of the perception results is analyzed.

6. The method for monitoring performance of a perception model according to claim 5, further comprising: Collect verification data when it is determined that verification data needs to be collected; as well as Based on the verification data, the accuracy of the perception result is analyzed.

7. The method for monitoring performance of a perception model according to claim 5, further comprising: When the accuracy of the perception result does not meet a preset standard, the perception service user is instructed to stop using the output result of the perception model.

8. The method for monitoring performance of a perception model according to claim 5, further comprising: The accuracy of the perception result is sent to the NWDAF entity so that the NWDAF entity determines whether to retrain the perception model.

9. The method for monitoring performance of a perception model according to claim 5, further comprising: Receiving accuracy analysis subscription requests sent by users who want to analyze accuracy; as well as The accuracy of the perception result is sent to the user of the analysis accuracy so that the user of the analysis accuracy determines whether to modify the perception service request.

10. The method for monitoring performance of a perception model according to claim 9, further comprising: Verifying the accuracy of the analysis of parameters carried in the subscription request; as well as When the parameter verification passes, the accuracy of the perception result is analyzed.

11. The method for monitoring performance of a perception model according to claim 8, wherein: In the case that the perception service user and the user of the accuracy analysis are the same network element, the perception result and the accuracy of the perception result are sent to the perception service user at the same time.

12. A method for monitoring performance of a perception model, performed by a network data analysis function NWDAF entity, comprising: Receive a perception model subscription request sent by a perception function SF entity; Training a perception model based on perception data; as well as The perception model is sent to the SF entity so that the SF entity performs performance monitoring on the perception model.

13. The method for monitoring performance of a perception model according to claim 12, further comprising: A registration request is sent to a network storage function NRF entity, wherein the registration request includes a perception model training capability identifier and first model interoperability information.

14. The method for monitoring performance of a perception model according to claim 13, wherein: The registration request also includes model filtering information.

15. The method for monitoring performance of a perception model according to any one of claims 12 to 14, further comprising: Sending an accuracy analysis subscription request to the SF entity; receiving the accuracy of the sensing result sent by the SF entity; as well as Based on the accuracy of the perception result, it is determined whether to retrain the perception model.

16. A perception function SF entity, comprising: A model receiving module configured to receive a perception model trained by a network data analysis function NWDAF entity; A reasoning module, configured to obtain a perception result based on the perception data using the perception model; as well as The performance monitoring module is configured to monitor the performance of the perception model based on the perception result.

17. A perception function SF entity, comprising: Memory; and A processor coupled to the memory, the processor being configured to execute the perception model performance monitoring method according to any one of claims 1 to 11 based on instructions stored in the memory.

18. A network data analysis function NWDAF entity, comprising: A request receiving module, configured to receive a perception model subscription request sent by a perception function SF entity; a training module configured to train a perception model based on the perception data; as well as The model sending module is configured to send the perception model to the SF entity so that the SF entity performs performance monitoring on the perception model.

19. A network data analysis function NWDAF entity, comprising: Memory; and A processor coupled to the memory, the processor being configured to execute the perception model performance monitoring method according to any one of claims 12 to 15 based on instructions stored in the memory.

20. A perception model performance monitoring system, comprising: The sensing function SF entity according to claim 16 or 17; as well as The network data analysis function NWDAF entity as described in claim 18 or 19.

21. A computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the perception model performance monitoring method described in any one of claims 1 to 15.

22. A computer program product, comprising computer instructions, which, when executed by a processor, implement the perception model performance monitoring method described in any one of claims 1 to 15.

Images