Model monitoring method and device, monitoring entity and storage medium

Abstract

The embodiment of the invention provides a model monitoring method and device, a monitoring entity and a storage medium, relates to the technical field of networks, is applied to the monitoring entity, and comprises the following steps: obtaining first measurement data of a first reference signal between a first entity and a second entity in a current monitoring period, the first entity and the second entity are located in a service area of the data processing model, and the first measurement data are input data of the data processing model; obtaining training data of the data processing model; determining a first cluster to which the first measurement data belongs and a second cluster to which the training data belongs; and determining a monitoring result of the data processing model according to the Euclidean distance between the central point of the first cluster and the central point of the second cluster and a preset monitoring threshold. By applying the technical scheme provided by the embodiment of the invention, the data processing precision of the data processing model in actual production can be improved.

Description

Model monitoring method, device, monitoring entity and storage medium Technical Field

[0001] The present application relates to the field of network technology, and in particular to a model monitoring method, device, monitoring entity and storage medium. Background Art

[0002] On the Internet, data processing models based on artificial intelligence / machine learning (AI / ML) are often used to process the measured values ​​of reference signals. For example, a positioning model based on AI / ML is used to measure and process the measured values ​​of positioning reference signals to obtain the predicted position of the user equipment (UE) and complete positioning.

[0003] The data processing model can improve the data processing efficiency. However, there are differences between the input data of the data processing model obtained in actual production (i.e., the measurement value of the reference signal) and the training data of the data processing model, which leads to a decrease in the data processing accuracy of the data processing model in actual production.

[0004] Summary of the Invention

[0005] The purpose of the embodiments of the present application is to provide a model monitoring method, device, monitoring entity, and storage medium to improve the data processing accuracy of the data processing model in actual production. The specific technical solution is as follows:

[0006] In a first aspect, an embodiment of the present application provides a model monitoring method, applied to a monitoring entity, the method comprising:

[0007] Acquire first measurement data of a first reference signal between a first entity and a second entity in a current monitoring period, where the first entity and the second entity are located in a service area of ​​a data processing model, and the first measurement data is input data of the data processing model;

[0008] Obtaining training data for the data processing model;

[0009] determining a first cluster to which the first measurement data belongs and a second cluster to which the training data belongs;

[0010] A monitoring result of the data processing model is determined according to the Euclidean distance between the center point of the first cluster and the center point of the second cluster and a preset monitoring threshold.

[0011] In a second aspect, an embodiment of the present application provides a model monitoring device, which is applied to a monitoring entity, and the device includes:

[0012] a first acquisition module, configured to acquire first measurement data of a first reference signal between a first entity and a second entity in a current monitoring period, the first entity and the second entity being located in a service area of ​​a data processing model, the first measurement data being input data of the data processing model;

[0013] A second acquisition module is used to obtain training data of the data processing model;

[0014] a first determining module, configured to determine a first cluster to which the first measurement data belongs and a second cluster to which the training data belongs;

[0015] The second determining module is configured to determine a monitoring result of the data processing model according to a Euclidean distance between a center point of the first cluster and a center point of the second cluster and a preset monitoring threshold.

[0016] In a third aspect, an embodiment of the present application provides a monitoring entity, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0017] The memory is used to store computer programs;

[0018] The processor is configured to implement any of the above-described method steps when executing the program stored in the memory.

[0019] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the method steps described above is implemented.

[0020] Beneficial effects of the embodiments of the present application:

[0021] In the technical solution provided by the embodiment of the present application, the monitoring entity clusters the input data (i.e., measurement data) of the data processing model collected in actual production and the training data of the data processing model into two clusters, namely, the first cluster to which the measurement data belongs and the second cluster to which the training data belongs, and compares the Euclidean distance between the center point of the first cluster and the center point of the second cluster with the preset monitoring threshold value, so as to determine whether there is a difference between the input data of the data processing model obtained in actual production and the training data of the data processing model, and obtain the corresponding monitoring result. If there is a difference between the input data of the data processing model obtained in actual production and the training data of the data processing model, it means that the environmental data has been offset and the data processing model cannot accurately process the input data in actual production. By using the above monitoring results, when there is a difference between the input data of the data processing model obtained in actual production and the training data of the data processing model, the data processing model can be retrained in time, thereby improving the data processing accuracy of the data processing model in actual production.

[0022] Of course, it is not necessary to achieve all the advantages described above at the same time when implementing any product or method of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings described herein are used to provide further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application.

[0024] Figure 1(a) is a schematic diagram of the first structure of the AI / ML-assisted positioning framework;

[0025] Figure 1(b) is a schematic diagram of the second structure of the AI / ML-assisted positioning framework;

[0026] Figure 2 is a schematic diagram of the structure of the AI / ML direct positioning framework;

[0027] FIG3 is a schematic diagram of a first flow chart of a model monitoring method provided in an embodiment of the present application;

[0028] FIG4 is a schematic diagram of measurement configuration information provided in an embodiment of the present application;

[0029] FIG5 is a detailed schematic diagram of step S33 provided in an embodiment of the present application;

[0030] FIG6 is a detailed schematic diagram of step S34 provided in an embodiment of the present application;

[0031] FIG7 is a flow chart of a model updating method provided in an embodiment of the present application;

[0032] FIG8 is a flow chart of a model monitoring capability request / providing process according to an embodiment of the present application;

[0033] FIG9 is a schematic diagram of a second flow chart of the model monitoring method provided in an embodiment of the present application;

[0034] Figures 10-16 are schematic diagrams of a model monitoring scenario provided in an embodiment of the present application;

[0035] Figures 17 to 19 are schematic diagrams of the distribution of measurement data after dimensionality reduction using the t-SNE algorithm provided in an embodiment of the present application;

[0036] FIG20 is a schematic structural diagram of a model monitoring device provided in an embodiment of the present application;

[0037] Figure 21 is a structural diagram of a monitoring entity provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] To make the objectives, technical solutions, and advantages of this application more clearly understood, the present application is further described below with reference to the accompanying drawings and examples. It is apparent that the described examples are only a portion of the embodiments of this application, and not all of them. All other embodiments derived by persons of ordinary skill in the art based on the examples in this application are intended to fall within the scope of protection of this application.

[0039] On the Internet, AI / ML-based data processing models are often used to process the measurement values ​​of reference signals. For example, the positioning reference signals are measured using AI / ML-based positioning models to obtain the measurement values, which are then processed to obtain the predicted position of the UE and complete positioning.

[0040] Take the use of AI / ML models for positioning enhancement in 5G NR (5th Generation Mobile Communication Technology New Radio) positioning as an example. In 5G NR positioning, the positioning model is an AI / ML model, i.e., a data processing model. The measurement values ​​input to the positioning model are the measurement data, and the predicted position output by the positioning model is the processing result of the data processing model. In 5G NR positioning, the UE or gNB (gNodeB, 5G base station) measures the PRS (Positioning Reference Signal) or SRS (Sounding Reference Signal) to obtain channel-related information such as the CIR (Channel Impulse Response) and PDP (Power Delay Profile). The positioning model outputs a position prediction for the UE based on the measured values, thereby implementing the 5G NR positioning function.

[0041] According to the 3GPP TSG RAN1 (3rd Generation Partnership Project Technical Specification Group Radio Access Network Work Group 1) standardization discussion on 5G NR AI / ML positioning, the positioning model framework is mainly divided into the following two categories.

[0042] 1) AI / ML-assisted positioning framework.

[0043] The AI / ML-assisted positioning framework is shown in Figures 1(a) and 1(b). In AI / ML-assisted positioning, AI / ML is combined with traditional NR positioning methods such as TDOA (Time Difference of Arrival). Measurements such as CIR are input into the positioning model, as shown in Figures 1(a) and 1(b). The AI / ML model then outputs intermediate quantities such as TOA (Time of Arrival) during the positioning process, as shown in Figures 1(a) and 1(b). These intermediate quantities are then input into the LMF (Location Management Function), which performs UE positioning calculations based on the principles of traditional NR positioning methods to obtain the UE's predicted position, the positioning result.

[0044] 2) AI / ML direct positioning framework.

[0045] The AI / ML direct positioning framework is shown in Figure 2. In AI / ML direct positioning, the positioning model receives inputs such as CIR and other measurement values. As shown in Figure 2, measurement value 1 minus measurement value N is input into the AI / ML model, and the positioning model directly outputs the predicted position of the UE, that is, the positioning result.

[0046] The data processing model uses a specific dataset collected at a specific moment as the training set. When the data processing model is deployed in production, there are often differences between the original data included in the training set and the dynamic measurement data in the production environment. This difference may cause the performance of the data processing model to gradually degrade over time. To address this issue, the entity needs to monitor the data processing model while it is working. If the data processing model is detected to be unavailable, the data processing model must be retrained and its network parameters must be updated to prevent the accuracy of the data processing model's processing results from decreasing over time, just as the positioning accuracy of the positioning model mentioned above is prevented from decreasing over time.

[0047] The typical data processing model generation process includes: Step 1: Data Collection; Step 2: Model Training and Testing. Therefore, at the functional level, model monitoring can be designed around these steps, such as detecting changes in input data distribution and model concept drift.

[0048] 1) Detection of changes in input data distribution. When a data processing model receives new measurement data that is significantly different from the training set, the performance of the data processing model may degrade. Therefore, it is crucial to provide early warning of changes in the characteristics of the data processing model and the data distribution predicted by the data processing model. Model monitoring design can monitor changes in input data distribution. When the input data distribution during data processing model deployment is significantly different from the data distribution of the data processing model's training set, it can be indirectly determined that the performance of the data processing model has degraded, and further operations such as updating the data processing model can be performed to improve the performance of the data processing model.

[0049] 2) Model drift detection. When a data processing model is applied to production, the inherent characteristics of the production environment may evolve over time, which causes the output of the data processing model to deviate from the actual situation of the production environment, thereby causing the performance of the data processing model to degrade. Therefore, it is necessary to continuously monitor the effectiveness of the data processing model. If the true value of the data processing model output can be obtained through other means, the model monitoring design can use the collected real input data and true values ​​as a test set during the deployment of the data processing model to test the data processing model and directly measure the performance of the data processing model to determine whether the performance of the data processing model has degraded, and perform subsequent data processing model updates and other operations.

[0050] Taking the scenario of using AI / ML models for positioning enhancement in 5G NR positioning as an example, two major types of model monitoring methods are proposed to address the model monitoring problem. These two types of methods are:

[0051] 1) Model monitoring method based on true value labels (or label estimates). In this method, the monitoring entity can obtain the true UE location corresponding to the measured value. For example, using a known PRU (Positioning Reference Unit) or UE feedback of the true location, or using other positioning methods to generate a more accurate positioning result. The true value labels are used to directly test and evaluate the positioning model to determine whether the positioning model performance has degraded.

[0052] 2) Model monitoring method without labels: In this method, the monitoring entity only relies on the statistical features of the model input data or output data to monitor the positioning model.

[0053] With the development of semi-supervised learning, a technology called adversarial verification has been widely adopted to solve the problem of model overfitting caused by the inconsistent distribution of training and test sets used by data processing models.

[0054] In adversarial validation, the original labels of the training and test data used by the data processing model are deleted, and the training and test data are relabeled, such that all training data is labeled 0 and all test data is labeled 1. The relabeled training and test data are then merged into a single dataset, and new training and test sets are generated based on this dataset. A binary classifier is then trained using the new training data, and its performance is tested on the new test data. If the classifier can effectively distinguish between the original training data and the original test data in the new test data, then the data distributions of the original training and test sets are considered to be significantly different. If the classifier cannot accurately distinguish between the original training data and the original test data in the new test data, then the data distributions of the original training and test sets are considered to be similar.

[0055] Using this adversarial verification method can also help solve the problem of model monitoring that does not use true value labels. In model monitoring, the dataset used during data processing model training can be regarded as the original training set, and the real data of the production environment obtained during model deployment can be regarded as the original test set in adversarial verification. Then, the principle of adversarial verification can be used to effectively determine whether there is a difference between the data used during model training and the real data of the environment, thereby effectively detecting changes in the input data distribution of the data processing model.

[0056] Adversarial verification, a semi-supervised machine learning method, is used in 5G NR model monitoring scenarios. Each time a monitoring entity needs to obtain model monitoring results, it must annotate the data processing model's training set and real-world measurement data, train a binary classifier, and evaluate the performance of this binary classifier to determine whether there are any changes in the data distribution. Once the monitoring entity generates monitoring results, these changes cannot be reused, requiring the next model monitoring attempt to retrain a new binary classifier based on newly collected real-world measurement data. Therefore, in practical applications, model monitoring methods based on adversarial verification can waste computing resources and time due to the complexity of the classifier.

[0057] In order to improve the data processing accuracy of the data processing model in actual production, an embodiment of the present application provides a model monitoring method, which is applied to a monitoring entity. The monitoring entity can be a base station (gNB), a terminal (UE) or a management entity, or other electronic devices capable of model monitoring, without limitation. The management entity can be an LMF end or other device with management functions. The model monitoring method provided in the embodiment of the present application can be applied to scenarios where AI / ML is used for positioning enhancement in 5G NR positioning. In this case, the data processing model is a positioning model and the management entity is an LMF end.

[0058] In the model monitoring method provided in the embodiment of the present application, the monitoring entity maps the input data (i.e., measurement data) of the data processing model collected in actual production and the training data of the data processing model to a low-dimensional space through dimensionality reduction processing and clusters them into two clusters, namely the first cluster to which the measurement data belongs and the second cluster to which the training data belongs. In this way, the relative relationship between the high-dimensional data can be retained as much as possible and it can be visualized and classified in the low-dimensional space. Based on this, the Euclidean distance between the center point of the first cluster and the center point of the second cluster is compared with the preset monitoring threshold to determine whether there is a difference between the input data of the data processing model obtained in actual production and the training data of the data processing model, and obtain the corresponding monitoring results. If there is a difference between the input data of the data processing model obtained in actual production and the training data of the data processing model, it means that the environmental data has shifted and the data processing model cannot accurately process the input data in actual production. Using the above monitoring results, when there is a difference between the input data of the data processing model obtained in actual production and the training data of the data processing model, the data processing model can be retrained in time, thereby improving the data processing accuracy of the data processing model in actual production.

[0059] The model monitoring method provided in the embodiments of the present application is described in detail below through specific examples.

[0060] Refer to Figure 3, which is a first flow chart of the model monitoring method provided in an embodiment of the present application, which is applied to a monitoring entity. The method includes the following steps.

[0061] Step S31 : obtaining first measurement data of a first reference signal between a first entity and a second entity in a current monitoring period, wherein the first entity and the second entity are located in a service area of ​​a data processing model, and the first measurement data is input data of the data processing model.

[0062] Step S32: Obtain training data for the data processing model.

[0063] Step S33: Determine the first cluster to which the first measurement data belongs and the second cluster to which the training data belongs.

[0064] Step S34 : determining a monitoring result of the data processing model according to the Euclidean distance between the center point of the first cluster and the center point of the second cluster and a preset monitoring threshold.

[0065] In the technical solution provided by the embodiment of the present application, the monitoring entity clusters the input data (i.e., measurement data) of the data processing model collected in actual production and the training data of the data processing model into two clusters, namely, the first cluster to which the measurement data belongs and the second cluster to which the training data belongs, and compares the Euclidean distance between the center point of the first cluster and the center point of the second cluster with the preset monitoring threshold value, so as to determine whether there is a difference between the input data of the data processing model obtained in actual production and the training data of the data processing model, and obtain the corresponding monitoring result. If there is a difference between the input data of the data processing model obtained in actual production and the training data of the data processing model, it means that the environmental data has been offset and the data processing model cannot accurately process the input data in actual production. By using the above monitoring results, when there is a difference between the input data of the data processing model obtained in actual production and the training data of the data processing model, the data processing model can be retrained in time, thereby improving the data processing accuracy of the data processing model in actual production.

[0066] In addition, in the technical solution provided in the embodiment of the present application, there is no need to train a classifier each time the model is monitored, which to a certain extent alleviates the problem of waste of computing resources and time in existing semi-supervised learning.

[0067] In step S31 above, the data processing model can be deployed on the monitoring entity or on another entity, such as the fifth entity. The data processing model can be a positioning model or other model. The following description uses the positioning model as an example and does not serve as a limitation. The monitoring period is the period during which the monitoring entity monitors the data processing model once. The duration of the monitoring period can be set according to actual conditions. The service area of ​​the data processing model (hereinafter referred to as the service area) can be one or multiple adjacent cells.

[0068] The first entity can be a physical device such as a UE or gNB within the service area, and the second entity can also be a physical device such as a UE or gNB within the service area. The first entity and the second entity are entities that mutually transmit reference signals. For example, when the first entity is a UE, the second entity is a gNB, and when the first entity is a gNB, the second entity is a UE. The first reference signal is an uplink reference signal and / or downlink reference signal transmitted between the first and second entities, such as an SRS, a Channel State Information-Reference Signal (CSI-RS), a PRS, a Synchronization Signal Block (SSB), a Demodulation Reference Signal (DMRS), a Phase Tracking Reference Signal (PTRS), etc. The first measurement data is data obtained by measuring the first reference signal, such as a CIR, a PDP, etc. Both the first and second entities can serve as entities that measure the first reference signal.

[0069] When a monitoring cycle arrives, the arrived monitoring cycle becomes the current monitoring cycle. The monitoring entity may obtain a first reference signal within the service area during the current monitoring cycle, measure the first reference signal, and obtain first measurement data. Alternatively, another entity may measure the first reference signal to obtain the first measurement data, and the monitoring entity may obtain the first measurement data directly from the other entity. Here, the first measurement data obtained by the monitoring entity is actual measurement data from a production environment.

[0070] In an embodiment of the present application, when performing model monitoring, the monitoring entity may obtain first measurement data of a first reference signal from the first entity or the second entity. The first measurement data does not carry a true value label. For example, when the data processing model is a positioning model, the true positions of the first entity and the second entity are unknown.

[0071] In the embodiment of the present application, the monitoring entity can be the first entity or the second entity, or can be the management entity. According to the location of the monitoring entity, there are two situations.

[0072] 1) When the monitoring entity is the first entity or the second entity, taking the monitoring entity being the first entity as an example, the monitoring entity can implement the above step S31 in the following three ways.

[0073] a. Receive a first reference signal sent by a second entity in a current monitoring period; measure the first reference signal to obtain a first measurement result, where the first measurement result includes first measurement data of the first reference signal.

[0074] In an embodiment of the present application, the first reference signal can be a reference signal broadcast by the second entity to the first entity, or can be a reference signal unicast by the second entity to the first entity. The first entity is responsible for measuring the reference signal. During the current monitoring period, the second entity sends the first reference signal to the first entity; the first entity (i.e., the monitoring entity) receives the first reference signal sent by the second entity and measures the first reference signal to obtain a first measurement result including first measurement data.

[0075] In this embodiment of the present application, one of the first entity and the second entity is a base station and the other is a terminal. When the first entity is a terminal and the second entity is a base station, the first reference signal may be a PRS, CSI-RS, SRS, SSB, DMRS, or PTRS; when the first entity is a base station and the second entity is a terminal, the first reference signal may be an SRS.

[0076] In some embodiments, the first reference signal is a reference signal sent or received by a terminal based on first configuration information sent by a base station. The first configuration information indicates the time-frequency resources occupied by the first reference signal. To ensure accurate identification and measurement of the first reference signal, the base station sends the first configuration information to the terminal; and the terminal sends or receives the first reference signal based on the first configuration information.

[0077] In some embodiments, the first configuration information is configuration information sent by the base station to the terminal in response to a first request sent by a third entity, where the first request instructs the base station to send the first configuration information to the terminal. To accurately control the sending of the first configuration information, upon entering a monitoring cycle, the third entity (e.g., a management entity) sends the first request to the base station; the base station sends the first configuration information to the terminal in response to the first request sent by the third entity, and the terminal then sends or receives the first reference signal in response to the first configuration information.

[0078] The first measurement data may be measurement data of the first reference signal obtained by the monitoring entity based on second configuration information issued by the third entity. Specifically, the third entity sends the second configuration information to the second entity, where the second configuration information specifies the relevant configuration for model monitoring. The second entity obtains the first measurement data based on the second configuration information and completes model monitoring.

[0079] The second configuration information may include at least one of the following: monitoring cycle information, measurement configuration information, monitoring algorithm, and preset monitoring threshold. The measurement configuration information, monitoring algorithm, and preset monitoring threshold may be represented by a number of bits.

[0080] a1) Monitoring cycle information. Monitoring cycle information can include a cycle unit and a number of bits. The cycle unit can be seconds, minutes, hours, etc. The cycle unit and the number of bits together represent the monitoring cycle length, which can be determined based on actual system requirements and algorithm time consumption. For example, when the monitoring cycle is in hours, 5 bits can be used to represent a monitoring cycle length of 1 to 24 hours. To save bits, different cycle options can also be agreed upon. For example, when the cycle unit is times / day, 2 bits can be used to represent four monitoring cycle options. The cycle options can be defined as: {00: 1 time / day (i.e., 24-hour cycle); 01: 4 times / day (i.e., 6-hour cycle); 10: 8 times / day (i.e., 3-hour cycle); 11: 24 times / day (i.e., 1-hour cycle)}. In other words, a bit number of 00 indicates a 24-hour monitoring cycle; a bit number of 01 indicates a 6-hour monitoring cycle, and so on. The monitoring entity can determine the monitoring cycle for model monitoring using the cycle unit and the number of bits.

[0081] a2) Measurement configuration information. Measurement configuration information may include the measurement cycle length, measurement time slice length, and measurement frequency. The measurement cycle length is the measurement data collection period for model monitoring, the measurement time slice length is the continuous time length of each data collection measurement, and the measurement frequency is the total number of measurements in this data collection. In this embodiment of the present application, the measurement cycle length is greater than the measurement time slice length. The relationship between the two is shown in Figure 4.

[0082] In an embodiment of the present application, a parameter set can be used to represent different measurement configuration sets, which can be set specifically according to actual needs. For example, 2 bits are used to represent four parameter selections: {00: (measurement cycle length: 512; slice length: 256; measurement frequency: 4 times); 01: (measurement cycle length: 256; slice length: 128; measurement frequency: 4 times); 10: (measurement cycle length: 512; slice length: 256; measurement frequency: 8 times); 11: (measurement cycle length: 256; slice length: 128; measurement frequency: 8 times)}, that is, when the number of bits corresponding to the configuration information is 00, it means that the data collection cycle is 512, the continuous time length of each collection is 256, the number of collections is 4 times, and so on. The first entity obtains the first measurement data according to the measurement configuration information.

[0083] a3) Monitoring algorithm. Considering that the model monitoring method can support different monitoring algorithms, it is necessary to specify the algorithm type used by the monitoring entity. The number of bits used is determined based on the total number of optional algorithms, and the correspondence between the number of bits and the monitoring algorithm is preset. For example, 2 bits are used to represent 4 monitoring algorithms: {00: t-SNE-based monitoring algorithm; 01: adversarial verification-based model monitoring algorithm; 10: KS test-based model monitoring algorithm; 11: autoencoder-based model monitoring algorithm}. For example, when the number of bits corresponding to the monitoring algorithm is 00, the monitoring entity uses the t-SNE-based monitoring algorithm for model monitoring.

[0084] a4) Preset monitoring threshold. Since different monitoring algorithms are used for model monitoring, the output types are different. Therefore, it is necessary to specify the monitoring threshold or related data for the threshold auxiliary calculation in the current situation according to the monitoring algorithm, and determine the number of bits used according to the threshold type or auxiliary data type of different monitoring algorithms. For example, in the monitoring algorithm based on t-SNE designed in the embodiment of the present application, the ratio of the Euclidean distance of the center points of the two clusters to the sum of the radii of the two clusters is designed as the discriminant, then the ten ratio options [0.1, 0.2, ... 0.9, 1] can be represented by 4 bits as the threshold for determining data drift: {0000: 0.1; 0001: 0.2; ... 1000: 0.9; 1001: 1}. For example, when the number of bits corresponding to the preset monitoring threshold is 0000, the preset monitoring threshold is 0.1, which means that when the ratio of the Euclidean distance of the center points of the two clusters to the sum of the radii of the two clusters is greater than 0.1, it can be determined as data drift.

[0085] b. Sending a first reference signal to the second entity within a current monitoring period; and receiving a second measurement result sent by the second entity, where the second measurement result includes first measurement data of the first reference signal.

[0086] In the embodiment of the present application, the first reference signal may be a reference signal broadcast by the first entity to the second entity, or may be a reference signal unicast by the first entity to the second entity. The second entity is responsible for measuring the reference signal.

[0087] During the current monitoring period, the first entity (i.e., the monitoring entity) sends a first reference signal to the second entity; the second entity receives the first reference signal, measures the first reference signal, obtains a second measurement result including the first measurement data, and sends the second measurement result to the monitoring entity; the monitoring entity obtains the first measurement data by receiving the second measurement result.

[0088] In the embodiment of the present application, one of the first entity and the second entity is a base station and the other entity is a terminal. When the first entity is a terminal and the second entity is a base station, the first reference signal is an SRS; when the first entity is a base station and the second entity is a terminal, the first reference signal can be a PRS, CSI-RS, SRS, SSB, DMRS, or PTRS, etc.

[0089] In some embodiments, the first reference signal is a reference signal sent or received by a terminal based on first configuration information sent by a base station. The first configuration information indicates the time-frequency resources occupied by the first reference signal. To ensure accurate identification and measurement of the first reference signal, the base station sends the first configuration information to the terminal; and the terminal sends or receives the first reference signal based on the first configuration information.

[0090] In some embodiments, the first configuration information is configuration information sent by the base station to the terminal in response to a first request sent by a third entity, where the first request instructs the base station to send the first configuration information to the terminal. To accurately control the sending of the first configuration information, upon entering a monitoring cycle, the third entity (e.g., a management entity) sends the first request to the base station; the base station sends the first configuration information to the terminal in response to the first request sent by the third entity, and the terminal then sends or receives the first reference signal in response to the first configuration information.

[0091] The first measurement data may be measurement data of the first reference signal obtained by the monitoring entity based on second configuration information issued by the third entity. Specifically, the third entity (management entity) sends second configuration information to the first entity, where the second configuration information specifies the relevant configuration for model monitoring. The first entity obtains the first measurement data from the second entity based on the second configuration information, completing model monitoring. The specific form of the second configuration information can be found in the description in section a above.

[0092] c. Send a second request to a third entity, where the third entity stores first measurement data of a first reference signal between the first entity and the second entity. The second request instructs the third entity to send the first measurement data to the monitoring entity; and receive a third measurement result corresponding to the first request sent by the third entity, where the third measurement result includes the first measurement data of the first reference signal between the first entity and the second entity.

[0093] In the implementation of this application, the first entity may be a terminal or a base station, and the third entity may be a management entity, such as an LMF end. The second request is a request for a measurement result. The third entity (such as the LMF end) stores existing measurement data. During the current monitoring cycle, the first entity (i.e., the monitoring entity) directly sends a second request to the LMF end, requesting to obtain the first measurement data already in the LMF end; the LMF end sends a third measurement result including the first measurement data to the first entity according to the second request; the monitoring entity obtains the first measurement data by receiving the third measurement result. In the embodiment of the present application, considering that there may be existing measurement data on the LMF end, it is convenient to quickly obtain measurement data.

[0094] The first measurement data may be measurement data of the first reference signal obtained by the monitoring entity based on second configuration information issued by the third entity. Specifically, the third entity (management entity) sends second configuration information to the first entity, where the second configuration information specifies the relevant configuration for model monitoring. The first entity obtains the first measurement data from the third entity based on the second configuration information, completing model monitoring. The specific form of the second configuration information can be found in the description in section a above.

[0095] The model monitoring method when the monitoring entity is the second entity is similar to the model monitoring method when the monitoring entity is the first entity. Please refer to the description of the above situations ac, and will not be repeated here.

[0096] 2) When the monitoring entity is a third entity, the monitoring entity may implement the above step S31 by the following steps: obtaining first measurement data of the first reference signal between the first entity and the second entity in the current monitoring period from the first target entity, where the first target entity is the entity between the first entity and the second entity that measures the first reference signal.

[0097] In an embodiment of the present application, the third entity may be a management entity, such as an LMF end; when the first entity is a terminal, the second entity is a base station, the first target entity is a terminal, and the first reference signal may be PRS, CSI-RS, SRS, SSB, DMRS, or PTRS, etc.; when the first entity is a base station, the second entity is a terminal, and the first target entity is a base station, the first reference signal is SRS.

[0098] During a current monitoring period, a first reference signal is transmitted between a first entity and a second entity. One of the first entity and the second entity measures the first reference signal to obtain first measurement data. The entity performing the measurement is the first target entity. The monitoring entity (i.e., the third entity) obtains the first measurement data obtained by measurement from the first target entity.

[0099] In this embodiment of the present application, the third entity stores second configuration information. The monitoring entity (i.e., the third entity) obtains first measurement data of the first reference signal based on the stored second configuration information. The specific form of the second configuration information can be found in the description in a above.

[0100] In the technical solution provided in the embodiment of the present application, the monitoring entity obtains first measurement data based on the first reference signal between the first entity and the second entity. The first measurement data is the data that needs to be input into the data processing model in actual production, reflecting the data distribution in actual production within the current monitoring cycle, and facilitating accurate model monitoring without true value labels.

[0101] In step S32 above, the training data refers to data in the training set of the data processing model, such as CIR and PDP. The monitoring entity obtains the training data when training the data processing model. If the data processing model is generated on the monitoring entity side, the monitoring entity can obtain the training data directly from the monitoring entity itself. The monitoring entity can also receive the training data from other entities containing training data, such as the entity that generated the data processing model or the LMF. The method for obtaining the training data is not limited herein.

[0102] The monitoring entity stores the training data of the currently deployed data processing model and the real measurement data during the deployment of the data processing model through the above steps S31 and S32, and collects the data required for model monitoring. The execution order of the above steps S31 and S32 is not limited.

[0103] In the above step S33, the monitoring entity may use a monitoring algorithm such as t-SNE (t-Distributed Stochastic Neighbor Embedding) to process the collected first measurement data and training data to obtain clusters corresponding to the first measurement data and the training data, i.e., a first cluster and a second cluster.

[0104] In step S34, the preset monitoring threshold is a parameter used to evaluate the monitoring results and is used to determine whether the data processing model is usable. The preset monitoring threshold can be set based on actual circumstances. The monitoring entity calculates the Euclidean distance between the center point of the first cluster and the center point of the second cluster and determines the monitoring result of the data processing model based on the relationship between this Euclidean distance and the preset monitoring threshold. For example, if the Euclidean distance is greater than the preset monitoring threshold, the monitoring result indicates that the data processing model is unusable; if the Euclidean distance is less than or equal to the preset monitoring threshold, the monitoring result indicates that the data processing model is usable.

[0105] In some embodiments, referring to FIG5 , which is a detailed schematic diagram of the above-mentioned step S33 provided in an embodiment of the present application, the above-mentioned step S33 may include the following steps.

[0106] Step S51 : performing dimensionality reduction processing on the first measurement data and the training data according to a preset monitoring algorithm to obtain dimensionality reduced data.

[0107] Step S52 : clustering the dimension-reduced data to obtain a first cluster to which the first measurement data belongs and a second cluster to which the training data belongs.

[0108] In the technical solution provided by the embodiments of the present application, for high-dimensional first measurement data and training data, the monitoring entity can reduce the dimensionality of the data using a preset monitoring algorithm before clustering the data. After mapping the high-dimensional measurement data and training data to a low-dimensional space, the data processing load of the model monitoring method is reduced, thereby improving the efficiency of model monitoring. At the same time, the relative relationship between the measurement data and the training data is preserved, ensuring the accuracy of clustering the reduced-dimensional data.

[0109] In step S51, the preset monitoring algorithm may be a t-SNE algorithm or another dimensionality reduction algorithm, such as the various monitoring algorithms included in the second configuration information, without limitation. The specific calculation process of the t-SNE algorithm will be described in detail later and is not described here. The monitoring entity, in accordance with the preset monitoring algorithm, reduces the dimension of the acquired first measurement data and training data into two-dimensional plane points, thereby obtaining reduced-dimensionality data.

[0110] In the above step S52, the monitoring algorithm clusters the dimensionality-reduced data, that is, clusters the points on the two-dimensional plane of the training data from the training set and the first measurement data from the environmental measurement, respectively, to obtain the first cluster to which the first measurement data belongs and the second cluster to which the training data belongs.

[0111] In some embodiments, the monitoring entity may pre-process the acquired first measurement data and training data. Step S51 may be implemented by converting the first measurement data and training data into intermediate data that matches the data processing model; and performing dimensionality reduction processing on the intermediate data according to a preset monitoring algorithm to obtain reduced-dimensionality data.

[0112] In an embodiment of the present application, the monitoring entity may pre-process the collected first measurement data and training data to obtain intermediate data that matches the input of a subsequent preset monitoring algorithm. For example, if the preset monitoring algorithm (such as the t-SNE algorithm) requires PDP as the input data of the positioning model, and the first measurement data and training data are CIR, the data needs to be processed to convert CIR into PDP to achieve preliminary data compression, while ensuring that the data monitored by the model is consistent with the input data of the positioning model, thereby ensuring the accuracy of the subsequent model monitoring results.

[0113] In some embodiments, referring to FIG. 6 , which is a detailed schematic diagram of step S34 provided in an embodiment of the present application, step S34 may include the following steps.

[0114] In step S61, the ratio of the first distance to the second distance is calculated to obtain a monitoring value. The first distance is the Euclidean distance between the center point of the first cluster and the center point of the second cluster, and the second distance is the sum of the radius of the first cluster and the radius of the second cluster. If the monitoring value is greater than a preset monitoring threshold, step S62 is executed; if the monitoring value is less than or equal to the preset monitoring threshold, step S63 is executed.

[0115] Step S62: determining a first monitoring result of the data processing model, where the first monitoring result indicates that the data processing model is unavailable.

[0116] Step S63: Determine a second monitoring result of the data processing model, where the second monitoring result indicates that the data processing model is available.

[0117] In the technical solution provided in the embodiments of this application, a conclusion is drawn and judged based on the obtained first and second clusters, combined with a preset monitoring threshold, through calculation and comparison, and the model monitoring result is output. For example, the ratio of the Euclidean distance between the center points of two clusters to the sum of the cluster radii is calculated and compared with a preset monitoring threshold generated, stored, or received locally to determine the monitoring result of the data processing model, thus achieving quantitative processing of model monitoring.

[0118] In step S61, after obtaining the first and second clusters, the monitoring entity calculates the Euclidean distance between the center of the first cluster and the center of the second cluster as the first distance, calculates the sum of the radius of the first cluster and the radius of the second cluster as the second distance, and then calculates the ratio of the first distance to the second distance, using this ratio as the monitoring value. For example, if the first distance is R, the radius of the first cluster is R1, and the radius of the second cluster is R2, the monitoring value is R / (R1+R2).

[0119] The monitoring entity determines the monitoring result of the data processing model based on the magnitude relationship between the monitoring value and a preset monitoring threshold. When the monitoring value is greater than the preset monitoring threshold, the monitoring entity executes step S62 above and determines that the monitoring result of the data processing model is the first monitoring result, i.e., the environmental data has deviated and the data processing model is unusable. When the monitoring value is less than or equal to the preset monitoring threshold, the monitoring entity executes step S63 above and determines that the monitoring result of the data processing model is the second monitoring result, i.e., the environmental data has not deviated and the data processing model is still applicable.

[0120] In some embodiments, the monitoring entity may further determine a monitoring result of the data processing model based on the first distance. The monitoring entity may implement step S34 as follows: if the Euclidean distance between the center point of the first cluster and the center point of the second cluster is greater than a preset monitoring threshold, then determining a first monitoring result of the data processing model, the first monitoring result indicating that the data processing model is unusable; if the Euclidean distance between the center point of the first cluster and the center point of the second cluster is less than or equal to the preset monitoring threshold, then determining a second monitoring result of the data processing model, the second monitoring result indicating that the data processing model is usable.

[0121] In an embodiment of the present application, the monitoring entity compares the Euclidean distance (i.e., the first distance) between the center point of the first cluster and the center point of the second cluster with a preset monitoring threshold. When the first distance is greater than the preset monitoring threshold, the monitoring entity determines that the monitoring result of the data processing model is the first monitoring result, and the data processing model is unusable. When the monitoring value is less than or equal to the preset monitoring threshold, the monitoring entity determines that the monitoring result of the data processing model is the second monitoring result, and the data processing model is still applicable. Applying the technical solution provided in the embodiment of the present application, comparing the first distance with the preset monitoring threshold to obtain the model monitoring result can reduce the amount of calculation and improve the efficiency of model monitoring.

[0122] In some embodiments, when a data processing model is deployed on a monitoring entity, after obtaining a monitoring result, the monitoring entity may use the monitoring result to determine whether to update the data processing model.

[0123] When the data processing model is deployed on another entity (such as a fifth entity), after obtaining monitoring results for the data processing model, the monitoring entity can also send the monitoring results to the fifth entity. The fifth entity can then use the monitoring results to determine whether to update the data processing model. The fifth entity is the entity that deploys the data processing model and can be a gNB, UE, or other entity. The monitoring entity broadcasts the monitoring results, transmits the model monitoring conclusions, and completes the data processing model update, thereby subsequently improving the accuracy of the data processing model.

[0124] Based on the above model monitoring method, when the data processing model is deployed on the monitoring entity, the embodiment of the present application also provides a model updating method. See Figure 7, which is a flow chart of the model updating method provided by the embodiment of the present application, applied to the monitoring entity. The above model updating method includes the following steps.

[0125] Step S71, when the monitoring result of the data processing model indicates that the data processing model is unavailable, obtain second measurement data of the second reference signal between the second target entity and the fourth entity in the current monitoring period, the fourth entity is located in the service area, and the processing result of the data processing model corresponding to the fourth entity is known, and the second target entity is the entity that sends the second reference signal between the first entity and the second entity.

[0126] Step S72: Update the data processing model according to the second measurement data.

[0127] In the technical solution provided in the embodiment of the present application, when the monitoring result of the data processing model is the first monitoring result, that is, when the monitoring entity determines that the data processing model is unavailable, the monitoring entity needs to collect data with true value labels and update the current data processing model to ensure the availability of the data processing model.

[0128] In step S71 above, the fourth entity may also be a physical device such as a UE, gNB, or PRU within the service area. The second target entity and the fourth entity are entities that transmit reference signals to each other. The second reference signal is a reference signal transmitted between the second target entity and the fourth entity. The fourth entity may also serve as an entity that measures the second reference signal.

[0129] The monitoring entity obtains second measurement data obtained by measuring the second reference signal. The plurality of second measurement data constitute an auxiliary data set, and each second measurement data carries a true value label. For example, when the data processing model is a positioning model, the true location of the fourth entity is known. To ensure the accuracy of model training, the accuracy of the measurement data obtained by the fourth entity is higher than a preset accuracy threshold.

[0130] In the above step S72, the monitoring entity obtains the second measurement data carrying the true value label, and retrains the current data processing model according to the second measurement data to update the network parameters of the data processing model.

[0131] In some embodiments, the monitoring entity may obtain a fourth measurement result from a third entity, where the fourth measurement result includes second measurement data of a second reference signal between the second target entity and the fourth entity. That is, the monitoring entity obtains the second measurement data from the third entity to update the model.

[0132] In the embodiment of the present application, when the monitoring entity is the first entity, the monitoring entity may obtain the fourth measurement result from the third entity in an active or passive manner.

[0133] 1) Passively obtaining the fourth measurement result: the third entity periodically sends the fourth measurement result to the monitoring entity.

[0134] 2) Actively obtaining a fourth measurement result: the monitoring entity sends a third request to the third entity, where the third request instructs the third entity to send the second measurement data to the monitoring entity; and receives a fourth measurement result corresponding to the third request sent by the third entity.

[0135] The monitoring entity actively sends a third request to the third entity. After receiving the third request, the third entity sends a fourth measurement result to the monitoring entity.

[0136] To ensure the accuracy of the updated data processing model, the third request includes a minimum number of samples. In this case, the fourth measurement result includes a quantity of second measurement data that is greater than or equal to the minimum number of samples. The minimum number of samples is expressed in units of units and bits, but may also be expressed in other forms, without limitation.

[0137] The third request is sent by the monitoring entity. When the monitoring entity detects data drift through a model monitoring algorithm or determines that the data processing model needs to be updated, the third request is used to request the third entity for an auxiliary dataset (i.e., the second measurement data) used for model update. The auxiliary information requested by the monitoring entity may include, but is not limited to, the minimum number of samples in the auxiliary dataset.

[0138] Considering that migrating and training existing data processing models or retraining new ones requires certain dataset size, an excessively small dataset may result in performance degradation. Therefore, when requesting an auxiliary dataset, a minimum sample size requirement should be specified. The number of bits used can be determined based on the number of samples or the actual size of the dataset. For example, if 1000 is used as a unit, 8 bits can be used to represent 1000 to 256000 samples. For example, when the minimum sample size is 00000001, it indicates that the minimum number of samples supported by the monitoring entity is 1000.

[0139] In some embodiments, after sending the third request to the third entity, the monitoring entity may also receive a first response or a second response corresponding to the third request sent by the third entity, where the first response indicates that the third entity is capable of sending measurement data greater than or equal to a minimum number of samples to the first entity, and the second response indicates that the third entity is not capable of sending measurement data greater than or equal to the minimum number of samples to the first entity. After receiving the first response, the monitoring entity performs the step of receiving a fourth measurement result corresponding to the third request sent by the third entity.

[0140] The response signaling is sent by the third entity. After receiving the third request from the monitoring entity, the third entity preliminarily assesses whether it is currently capable of generating and sending sufficient measurement data to the monitoring entity. If so, it responds with a first response; otherwise, it responds with a second response. To conserve bandwidth, the response signaling bit can occupy one bit. For example, a 1 in the response signaling bit indicates a first response, and a 0 in the response signaling bit indicates a second response.

[0141] In an embodiment of the present application, the third entity sends a response corresponding to the third request to the monitoring entity, so that the monitoring entity can decide whether to wait for receiving the fourth measurement result or to perform the next round of model monitoring, thereby avoiding the monitoring entity from continuing to wait for receiving the fourth measurement result when it cannot receive the fourth measurement result.

[0142] In the embodiment of the present application, when the monitoring entity is a third entity, the monitoring entity may directly obtain the fourth measurement result locally.

[0143] In some embodiments, the fourth measurement result may be a measurement result obtained by the third entity from the fourth entity.

[0144] When the third entity's assessment result indicates that it is capable of generating sufficient measurement data, the second measurement data is measurement data sent by the fourth entity to the third entity based on the third configuration information of the second reference signal issued by the third entity. The third entity sends the third configuration information to the fourth entity, such as a PRU or UE, whose location information is known. The fourth entity obtains the second measurement data based on the third configuration information of the second reference signal issued by the third entity, and feeds back a fifth measurement result including the second measurement data to the fourth entity.

[0145] The third configuration information may include at least one of the following: measurement-related information of the second reference signal and an identifier of the monitoring entity.

[0146] 1) Secondary reference signal measurement related information, which reuses some IEs and procedures in LPP, such as NR-On-Demand-DL-PRS-Configurations. This measurement related information is used to configure PRS measurement related information to the UE or PRU that generates the auxiliary data set.

[0147] 2) The identification of the monitoring entity, that is, the entity ID carrying the data processing model. Considering that the UE and PRU that generate the auxiliary data set can directly send the auxiliary data set to the monitoring entity after the collection is completed, it is necessary to identify the monitoring entity. Existing identification can be reused here, for example, the TMSI (Temporary Mobile Subscriber Identity) used by the UE.

[0148] In some embodiments, the second measurement data may include at least one of the following: a measurement value, a true value label corresponding to the measurement value, and a data quality corresponding to the true value label. The measurement value, the true value label, and the data quality are represented by a number of bits.

[0149] The second measurement data is sent by a fourth entity such as a UE or a PRU that assists in generating a data set, and the fourth entity transmits the auxiliary data set or the auxiliary measurement data to the monitoring entity.

[0150] 1) Measurement values, such as CIR and PDP, can be encoded to represent high-dimensional data according to a specific data format, and the number of bits used is determined by the amount of transmitted data.

[0151] 2) The true value label corresponding to the measurement value. For example, the terminal location or TOA prediction value corresponding to a set of CIR data (including CIR measurements from different base stations and at different times) can be determined by the specific data format. If decimeter-level (0.1m) accuracy is used, 17 bits can be used to represent distance or coordinate variables in the range of 0 to 13 km.

[0152] 3) The data quality corresponding to the true value label. For example, the terminal position error range, if using decimeter-level accuracy, can use 6 bits to represent the absolute value of the deviation from 0 to 5 meters. The data quality corresponding to the true value label here can be used to evaluate the accuracy of the measurement data obtained by the fourth entity.

[0153] In this embodiment of the present application, if the fourth entity only transmits auxiliary measurement data, the second measurement data may only include measurement values. In this case, the fourth entity may also serve as the first entity or the second entity for model monitoring.

[0154] In some embodiments, in scenarios where the monitoring entity is not a management entity, the monitoring entity may further send monitoring capability information to a third entity. The monitoring capability information includes at least one of the following: the maximum number of samples supported by the monitoring entity. After determining the sample unit, the maximum number of samples supported by the monitoring entity may be represented in bits. For example, if 1000 is used as a unit, 1000 to 256000 samples may be represented in 8 bits. For example, when the maximum number of samples is 00000001, it indicates that the maximum number of samples supported by the monitoring entity is 1000 samples.

[0155] Figure 8 illustrates the model monitoring capability request / provision process, where the monitoring entity is the UE / gNB and the third entity is the LMF. Before sending monitoring capability information to the third entity, the monitoring entity may also receive a fourth request from the third entity, indicating the acquisition of monitoring capability information (i.e., request monitoring capability signaling). Based on the fourth request, the monitoring entity sends the monitoring capability information to the third entity (i.e., provides monitoring capability signaling carrying the monitoring capability information). By sending the monitoring capability information to the third entity, the monitoring entity provides the third entity with its supported processing or storage capabilities to facilitate subsequent auxiliary data transmission.

[0156] The following is a detailed introduction to the t-SNE algorithm. The basic idea of ​​the SNE algorithm is to map data points to probability distributions. The main steps include:

[0157] 1) SNE is based on the similarity between high-dimensional data. The similarity can be measured using the Euclidean distance between a sample point and other sample points. The Euclidean distance is used to construct a Gaussian conditional probability distribution. The characteristic of this probability distribution is that for a sample point, sample points that are similar to it have a higher probability of being selected, while sample points that are dissimilar to it have a lower probability of being selected. The high-dimensional sample point x constructed by SNE i and x j The Gaussian conditional probability distribution between is as follows:

[0158] Among them, x i 、x j 、x k Represents a high-dimensional sample point, p j|i Represents a high-dimensional sample point x i and x j Gaussian conditional probability distribution between, that is, high-dimensional sample point x i Will choose x j As the probability of its neighbors, exp(·) represents the exponential function, ||·|| represents the modulus function, σ i Indicates x i The standard deviation of the Gaussian distribution centered at , and Σ(·) represents the summation function.

[0159] For the same sample point, the conditional probability is 0, that is, p i|i = 0. For each sample point x i , other sample points relative to this sample point x i The constructed Gaussian distribution has a standard deviation σ corresponding to the sample point i Under different data distributions, σ i The initialization step first defines the perplexity, and then determines the σ corresponding to the perplexity through binary search. i Value, perplexity is defined using the entropy of the constructed distribution:

[0160] in,

[0161] Perp(P i ) represents the perplexity, P i Represents the Gaussian distribution constructed by the relative distance between the i-th sample point and other sample points (Euclidean distance is used in the formula), H(P i ) represents the center point P i Cross entropy in binary terms, p j|i Represents a high-dimensional sample point x i and x j is the Gaussian conditional probability distribution between , and log(·) represents the logarithmic function.

[0162] In practical applications, the value of perplexity is determined by the user. SNE is robust to the value of perplexity and generally chooses a value between 5 and 50.

[0163] 2) SNE constructs a probability distribution of mapping points in low-dimensional space, under which each low-dimensional space data point corresponds to a data point in the original high-dimensional space. i with y j The conditional probability distribution between them is as follows:

[0164] Among them, q j|i is the low-dimensional sample point y i and y j The conditional probability distribution between y i 、y j and y k Represents low-dimensional sample points.

[0165] Similarly, for the same sample point, q i|i =0.

[0166] 3) In theory, when the dimensionality reduction effect is good enough, the distributions of the two structures should be the same, that is, p j|i =q j|i Therefore, the objective function is constructed using KL-divergence, and the optimization goal is to minimize the sum of the KL-divergence between all sample points in the high-dimensional space and all sample points in the low-dimensional space. The constructed objective function is as follows:

[0167] Among them, Cost represents the objective function, that is, the loss function, P i Represents the Gaussian distribution constructed by the relative distance between the i-th sample point and other samples (Euclidean distance is used in the formula), P i Expressed as p j|i The probability distribution {p 1|i ,p 2|i ...};Q i Represents the Gaussian distribution constructed from the relative distance between the i-th sample point and other sample points on the 2D mapping plane, Q i Represented as q j|i The probability distribution of {q 1|i ,q 2|i …}, i = 1, …, n.

[0168] The basic steps of the t-SNE algorithm are similar to those of SNE. To improve the shortcomings of the SNE algorithm (such as the crowding problem), t-SNE uses the joint distribution to construct the relationship between sample points and uses the heavier-tailed t-distribution to construct the distribution of mapping points in the low-dimensional space:

[0169] Among them, q ij represents the joint probability distribution, y i 、y j 、y k and y l Represents low-dimensional sample points.

[0170] For the joint distribution of high-dimensional data, a symmetric conditional distribution is used:

[0171] Among them, p ij represents the joint probability distribution, and n represents the number of sample points.

[0172] The corresponding objective function is:

[0173] Among them, P represents the joint probability distribution of the relative distances between all sample points; Q represents the joint probability distribution of the relative distances of all sample points on the two-dimensional mapping plane.

[0174] This application uses the t-SNE algorithm principle described above and applies this principle to the detection of input data distribution drift in model monitoring. In the 5G positioning scenario, the input of the ML model is generally high-dimensional data such as CIR and PDP obtained by measurement. In order to detect whether the distribution of the measurement data received during the model deployment process and the distribution of the measurement data used during model training have drifted, the training set data can be spliced ​​with the new measurement data and subjected to t-SNE dimensionality reduction processing. The low-dimensional space mapping points from the training set data and the new data after dimensionality reduction are then clustered separately. By judging the relative relationship between the Euclidean distance between the center points of the two clusters and the cluster radius, it is measured whether there is a significant offset between the training set and the test set.

[0175] The model monitoring method provided in the embodiment of the present application is described in detail below with reference to Figures 9 to 16.

[0176] The process of the model monitoring method shown in Figure 9 includes: 1) data collection step; 2) data processing step; 3) t-SNE-based model monitoring algorithm; 4) model monitoring result judgment; 5) model monitoring result broadcasting.

[0177] The monitoring entity collects measurement data (i.e., measured values) and training data and preprocesses the data to obtain intermediate quantities (i.e., intermediate data). The monitoring entity uses a t-SNE-based model monitoring algorithm to reduce the data dimension, obtaining a two-dimensional point set. Clustering is performed on this two-dimensional point set data to obtain clusters corresponding to the measurement data and training data, namely the first cluster and the second cluster, respectively. The monitoring entity determines the monitoring results based on the cluster center point and cluster radius and broadcasts the model monitoring results to other entities that deploy the data processing model, completing model monitoring.

[0178] Taking the use of AI / ML models for positioning enhancement in 5G NR positioning as an example, the positioning framework in 5G NR positioning includes the UE, gNB, LMF, PRU, etc. The positioning model is referred to as the ML model.

[0179] Depending on the location of the monitoring entity, the model monitoring scenarios are divided into: 1) the scenario where the monitoring entity is deployed on the UE as shown in Figures 10-12; 2) the scenario where the monitoring entity is deployed on the gNB as shown in Figures 13-15; 3) the scenario where the monitoring entity is deployed on the LMF end as shown in Figure 16.

[0180] Each scenario is described below.

[0181] Scenario 1) is suitable for UE-based AI / ML positioning using the UE-side ML model, using the AI / ML model direct or assisted positioning framework, and using the UE-side ML model based on the LMF end or other UE-assisted AI / ML positioning, using the AI / ML model assisted positioning framework. In the embodiment of the present application, scenario 1) can be subdivided into 3 sub-scenarios 11)-13).

[0182] In sub-scenario 11, the UE measures the RS (Reference Signal) from the gNB, i.e., the first reference signal, which is a downlink reference signal, and obtains the measurement value required for model monitoring (i.e., the first measurement data) through the measurement.

[0183] As shown in Figure 10, the LMF sends monitoring configuration signaling (e.g., monitoring configuration) to the UE. This monitoring configuration signaling carries the second configuration information. During each monitoring period, the LMF sends a reference signal configuration request signaling (e.g., RS configuration request), i.e., a first request, to the gNB. Based on the reference signal configuration request signaling, the gNB sends the reference signal configuration (RS configuration) to the UE and the RS (e.g., the first configuration information). The UE measures the RS (e.g., RS measurement), obtains measurement results, and performs model monitoring based on the measurement data included in the measurement results.

[0184] If the model monitoring result indicates that the ML model is unavailable, that is, the model monitoring result is the first monitoring result, the UE sends a request assistance signaling (the third request) to the LMF end. The LMF end, based on its own capabilities, feedbacks the UE with a request acceptance signaling or a request rejection signaling (request acceptance / rejection). The request acceptance signaling is the first response, and the request rejection signaling is the second response.

[0185] After feedback of the acceptance request signaling (i.e., acceptance) to the UE, the LMF sends assistance configuration signaling carrying third configuration information to the UEs / PRUs (i.e., the fourth entity) at known locations. The UEs / PRUs then provide feedback of assistance measurement result signaling carrying second measurement data of the second reference signal to the LMF. The LMF receives the feedback assistance measurement result signaling sent by the UEs / PRUs, assembles an assistance dataset, and sends it to the UE for model updating.

[0186] In sub-scenario 12, the gNB measures the RS from the UE, i.e., the first reference signal (RS). In this case, the first reference signal is an uplink reference signal, such as an SRS. The gNB obtains the measurement value data (i.e., the first measurement data) required for model monitoring through the measurement and transmits it back to the UE.

[0187] As shown in Figure 11, the LMF sends monitoring configuration signaling to the UE, which carries the second configuration information. During each monitoring period, the LMF sends a reference signal configuration request signaling (e.g., an SRS configuration request) to the gNB, i.e., the first request. The gNB sends the reference signal configuration (SRS configuration) to the UE based on the reference signal configuration request signaling and receives the SRS sent by the UE. The gNB then measures the SRS, obtains measurement results, and sends them to the UE. The UE then performs model monitoring based on the measurement data included in the measurement results.

[0188] If the model monitoring result indicates that the ML model is unavailable, that is, the model monitoring result is the first monitoring result, the UE sends a request for assistance signaling to the LMF end, that is, the third request. The LMF end feeds back a request acceptance signaling or a request rejection signaling to the UE based on its own capabilities.

[0189] After feedback request acceptance signaling is sent to the UE, the LMF sends auxiliary configuration signaling carrying third configuration information to the UEs / PRUs (i.e., the fourth entity) at known locations. The UEs / PRUs then feed back auxiliary measurement result signaling carrying second measurement data of the second reference signal to the LMF. The LMF receives the feedback auxiliary measurement result signaling sent by the UEs / PRUs, assembles an auxiliary data set, and sends it to the UE, which then performs a model update.

[0190] In sub-scenario 13), considering that the LMF end may have existing measurement values, the UE directly requests measurement data (ie, first measurement data) from the LMF end.

[0191] As shown in Figure 12, the LMF sends monitoring configuration signaling to the UE, which carries the second configuration information. During each monitoring period, the UE sends a request measurement result signaling (i.e., the second request) to the LMF. The LMF sends measurement result signaling to the UE, which carries the measurement result. The UE monitors the model based on the measurement data included in the measurement result.

[0192] If the model monitoring result indicates that the ML model is unavailable, that is, the model monitoring result is the first monitoring result, the UE sends a request for assistance signaling to the LMF end, that is, the third request. The LMF end feeds back a request acceptance signaling or a request rejection signaling to the UE based on its own capabilities.

[0193] After feedback request acceptance signaling is sent to the UE, the LMF sends auxiliary configuration signaling carrying third configuration information to the UEs / PRUs (i.e., the fourth entity) at known locations. The UEs / PRUs then feed back auxiliary measurement result signaling carrying second measurement data of the second reference signal to the LMF. The LMF receives the feedback auxiliary measurement result signaling sent by the UEs / PRUs, assembles an auxiliary data set, and sends it to the UE, which then performs a model update.

[0194] Scenario 2) is suitable for base station-assisted AI / ML positioning using the gNB-side ML model, using an AI / ML model-assisted positioning framework. In this embodiment of the present application, scenario 2) can be further divided into three sub-scenarios 21)-23).

[0195] In sub-scenario 21, the gNB measures the RS from the UE, i.e., the first reference signal. In this case, the first reference signal is an uplink reference signal, such as an SRS. The gNB obtains the measurement value data (i.e., the first measurement data) required for model monitoring through measurement.

[0196] As shown in Figure 13, the LMF sends monitoring configuration signaling to the gNB, which carries the second configuration information. During each monitoring period, the LMF sends reference signal configuration request signaling (i.e., the first request) to the gNB. Based on the reference signal configuration request signaling, the gNB sends the reference signal configuration to the UE and receives the SRS sent by the UE. The gNB measures the SRS, obtains measurement results, and performs model monitoring based on the measurement data included in the measurement results.

[0197] If the model monitoring result indicates that the ML model is unavailable, that is, the model monitoring result is the first monitoring result, the gNB sends a request for assistance signaling to the LMF, that is, the third request. The LMF responds to the gNB with a request acceptance signaling or a request rejection signaling based on its capabilities.

[0198] After receiving the feedback request acceptance signaling from the gNB, the LMF sends auxiliary configuration signaling carrying the third configuration information to the UEs / PRUs (i.e., the fourth entity) at known locations. The UEs / PRUs then send auxiliary measurement result signaling carrying the second measurement data of the second reference signal to the LMF. The LMF receives the feedback auxiliary measurement result signaling from the UEs / PRUs, assembles an auxiliary data set, and sends it to the gNB, which then performs a model update.

[0199] In sub-scenario 22, the UE measures the RS from the gNB, i.e., the first reference signal (RS). In this case, the first reference signal is a downlink reference signal. The UE obtains the measurement value data (i.e., the first measurement data) required for model monitoring through measurement and transmits it back to the gNB.

[0200] As shown in Figure 14, the LMF sends monitoring configuration signaling to the gNB, which carries the second configuration information. During each monitoring period, the LMF sends a reference signal configuration request signaling (i.e., the first request) to the gNB. Based on the reference signal configuration request signaling, the gNB sends the reference signal configuration to the UE and the RS to the UE. The UE measures the RS, obtains measurement results, and sends them to the gNB. The gNB then performs model monitoring based on the measurement data included in the measurement results.

[0201] If the model monitoring result indicates that the ML model is unavailable, that is, the model monitoring result is the first monitoring result, the gNB sends a request for assistance signaling to the LMF, that is, the third request. The LMF responds to the gNB with a request acceptance signaling or a request rejection signaling based on its capabilities.

[0202] After receiving the feedback request acceptance signaling from the gNB, the LMF sends auxiliary configuration signaling carrying the third configuration information to the UEs / PRUs (i.e., the fourth entity) at known locations. The UEs / PRUs then send auxiliary measurement result signaling carrying the second measurement data of the second reference signal to the LMF. The LMF receives the feedback auxiliary measurement result signaling from the UEs / PRUs, assembles an auxiliary data set, and sends it to the gNB, which then performs a model update.

[0203] In sub-scenario 23), considering that the LMF may have existing measurement values, the gNB can directly request the measurement data (i.e., the first measurement data) from the LMF.

[0204] As shown in Figure 15, the LMF sends monitoring configuration signaling to the gNB, which carries the second configuration information. During each monitoring period, the gNB sends a measurement result request signaling (i.e., the second request) to the LMF. The LMF sends the measurement result signaling to the gNB, and the gNB monitors the model based on the measurement data included in the measurement result.

[0205] If the model monitoring result indicates that the ML model is unavailable, that is, the model monitoring result is the first monitoring result, the gNB sends a request for assistance signaling to the LMF, that is, the third request. The LMF responds to the gNB with a request acceptance signaling or a request rejection signaling based on its capabilities.

[0206] After receiving the feedback request acceptance signaling from the gNB, the LMF sends auxiliary configuration signaling carrying the third configuration information to the UEs / PRUs (i.e., the fourth entity) at known locations. The UEs / PRUs then send auxiliary measurement result signaling carrying the second measurement data of the second reference signal to the LMF. The LMF receives the feedback auxiliary measurement result signaling from the UEs / PRUs, assembles an auxiliary data set, and sends it to the gNB, which then performs a model update.

[0207] Scenario 3) is suitable for LMF-based or UE-assisted AI / ML positioning using the LMF side ML model, direct positioning framework using the AI / ML model, and base station-assisted AI / ML positioning using the LMF side ML model, direct positioning framework using the AI / ML model.

[0208] As shown in Figure 16 , the UE / gNB sends the reference signal measurement result (RS / SRS measurement result) to the LMF. The LMF performs model monitoring based on its own second configuration information and the measurement data contained in the received measurement result. The process by which the UE / gNB obtains reference signal measurement data can be found in the description of Figures 10 to 15 above and is not repeated here.

[0209] If the model monitoring results indicate that the ML model is unavailable, the LMF sends auxiliary configuration signaling to UEs / PRUs with known locations. UEs / PRUs then feed back auxiliary measurement result signaling to the LMF. The LMF receives the feedback auxiliary measurement result signaling from the UEs / PRUs, assembles an auxiliary data set, and updates the model based on the measurement data.

[0210] The model monitoring method based on t-SNE data dimensionality reduction proposed in the embodiment of the present application can be used in monitoring scenarios without true value labels, and can effectively monitor whether there is a large difference between the distribution of the model training set and the real measurement value. The t-SNE algorithm is a new algorithm that is further optimized and designed based on the traditional SNE algorithm to solve the congestion problem. t-SNE is mostly used for the visualization of high-dimensional data. The t-SNE algorithm can be used to map high-dimensional data to low-dimensional space while retaining the relative relationship between high-dimensional data as much as possible. Finally, the relative relationship between the mapping points in two-dimensional or one-dimensional space is used to characterize the relative relationship between high-dimensional data.

[0211] The distribution of the measurement data after dimensionality reduction using the t-SNE algorithm can be seen in Figures 17-19. In these figures, the right sub-figures show the geographic distribution of the CIR samples, with the horizontal and vertical coordinates representing the positions in the X and Y directions, respectively, in meters. The left sub-figures show the distribution of the CIR samples on the two-dimensional XY mapping plane after dimensionality reduction using t-SNE, with the horizontal and vertical coordinates representing the coordinate values ​​in the X and Y directions, respectively. The two colored sample points represent the measurement data and training data, respectively.

[0212] Figure 17 shows the t-SNE dimensionality reduction and k-means clustering results for sample points in the same distribution area. Figure 18 shows the t-SNE dimensionality reduction and k-means clustering results for sample points in adjacent distribution areas. Figure 19 shows the t-SNE dimensionality reduction and k-means clustering results for sample points in distribution areas that are a certain distance apart.

[0213] The t-SNE model monitoring algorithm is used to monitor model input data. As the actual distributions of the two data sets continue to shift, the discriminability of the clustering results after t-SNE dimensionality reduction also increases. Therefore, the t-SNE algorithm can be used to process model inputs, thereby revealing differences in the geographic distribution characteristics of the data. This can further determine whether the distribution of the model input data has drifted relative to the training set, enabling effective model monitoring that is independent of ground truth labels. Furthermore, in actual model use, network training is not required; only threshold values ​​need to be passed to complete the model monitoring function. This significantly reduces computational complexity.

[0214] In the technical solution provided by the embodiments of the present application, for the problem of model monitoring without true value labels in 5G NR AI positioning scenarios, the embodiments of the present application provide a method and workflow for evaluating model performance using ML model input data when there are no true value labels by designing a model monitoring algorithm and model monitoring implementation process without true value labels. This also alleviates to a certain extent the waste of computing resources and time caused by the existing semi-supervised learning method of re-labeling data and repeatedly training the network parameters of the ML model.

[0215] Corresponding to the above-mentioned model monitoring method, an embodiment of the present application further provides a model monitoring device, as shown in FIG20 , which is applied to a monitoring entity and includes:

[0216] A first acquisition module 201 is configured to acquire first measurement data of a first reference signal between a first entity and a second entity in a current monitoring period, wherein the first entity and the second entity are located in a service area of ​​a data processing model, and the first measurement data is input data of the data processing model;

[0217] A second acquisition module 202 is used to acquire training data for the data processing model;

[0218] A first determining module 203 is configured to determine a first cluster to which the first measurement data belongs and a second cluster to which the training data belongs;

[0219] The second determining module 204 is configured to determine a monitoring result of the data processing model according to the Euclidean distance between the center point of the first cluster and the center point of the second cluster and a preset monitoring threshold.

[0220] In some embodiments, when the monitoring entity is a first entity, the first obtaining module 201 is specifically configured to:

[0221] receiving a first reference signal sent by a second entity within a current monitoring period;

[0222] The first reference signal is measured to obtain a first measurement result, where the first measurement result includes first measurement data of the first reference signal.

[0223] In some embodiments, when the first entity is a terminal, the second entity is a base station, and the first reference signal is a PRS, a CSI-RS, an SRS, an SSB, a DMRS, or a PTRS;

[0224] When the first entity is a base station, the second entity is a terminal, and the first reference signal is an SRS;

[0225] The first measurement data includes CIR and PDP.

[0226] In some embodiments, when the monitoring entity is a first entity, the first obtaining module 201 is specifically configured to:

[0227] sending a first reference signal to the second entity within a current monitoring period;

[0228] A second measurement result sent by the second entity is received, where the second measurement result includes first measurement data of the first reference signal.

[0229] In some embodiments, when the first entity is a terminal, the second entity is a base station, and the first reference signal is an SRS;

[0230] When the first entity is a base station, the second entity is a terminal, and the first reference signal is a PRS, a CSI-RS, an SRS, an SSB, a DMRS, or a PTRS;

[0231] The first measurement data includes CIR and PDP.

[0232] In some embodiments, the first reference signal is a reference signal sent or received by the terminal according to first configuration information sent by the base station, and the first configuration information indicates the time-frequency resources occupied by the first reference signal.

[0233] In some embodiments, the first configuration information is configuration information sent by the base station to the terminal according to a first request sent by a third entity, and the first request instructs the base station to send the first configuration information to the terminal.

[0234] In some embodiments, when the monitoring entity is a first entity, the first obtaining module 201 is specifically configured to:

[0235] Sending a second request to a third entity, where the third entity stores first measurement data of a first reference signal between the first entity and the second entity, the second request instructing the third entity to send the first measurement data to the monitoring entity;

[0236] A third measurement result corresponding to the second request sent by the third entity is received, where the third measurement result includes the first measurement data.

[0237] In some embodiments, the first entity is a terminal or a base station, and the third entity is a management entity;

[0238] The first measurement data includes CIR and PDP.

[0239] In some embodiments, when the monitoring entity is a third entity, the first obtaining module 201 is specifically configured to:

[0240] First measurement data of a first reference signal between the first entity and the second entity in a current monitoring period is obtained from a first target entity, where the first target entity is an entity, between the first entity and the second entity, that measures the first reference signal.

[0241] In some embodiments, the third entity is a management entity;

[0242] When the first entity is a terminal, the second entity is a base station, and the first target entity is the terminal, the first reference signal is a PRS, a CSI-RS, an SRS, an SSB, a DMRS, or a PTRS;

[0243] When the first entity is a base station, the second entity is a terminal, and the first target entity is the base station, the first reference signal is an SRS;

[0244] The first measurement data includes channel CIR and PDP.

[0245] In some embodiments, the first measurement data is first measurement data of the first reference signal obtained by the monitoring entity according to second configuration information sent by a third entity.

[0246] In some embodiments, the second configuration information includes at least one of the following: monitoring period information, measurement configuration information, a monitoring algorithm, and the preset monitoring threshold.

[0247] In some embodiments, the monitoring cycle information includes a cycle unit and a bit number.

[0248] In some embodiments, the measurement configuration information includes a measurement cycle length, a measurement time slice length, and a measurement frequency.

[0249] In some embodiments, the measurement configuration information, the monitoring algorithm, and the preset monitoring threshold are represented by a number of bits.

[0250] In some embodiments, the first determining module 203 is specifically configured to:

[0251] Performing dimensionality reduction processing on the first measurement data and the training data according to a preset monitoring algorithm to obtain dimensionality-reduced data;

[0252] The dimension-reduced data is clustered to obtain a first cluster to which the first measurement data belongs and a second cluster to which the training data belongs.

[0253] In some embodiments, the first determining module 203 is specifically configured to:

[0254] converting the first measurement data and the training data into intermediate data that matches the data processing model;

[0255] According to a preset monitoring algorithm, the intermediate data is subjected to dimensionality reduction processing to obtain dimensionality-reduced data.

[0256] In some embodiments, the preset monitoring algorithm is a t-SNE algorithm.

[0257] In some embodiments, the second determining module 204 is specifically configured to:

[0258] Calculating a ratio of a first distance to a second distance to obtain a monitoring value, where the first distance is a Euclidean distance between a center point of the first cluster and a center point of the second cluster, and the second distance is a sum of a radius of the first cluster and a radius of the second cluster;

[0259] If the monitoring value is greater than the preset monitoring threshold, determining a first monitoring result of the data processing model, the first monitoring result indicating that the data processing model is unusable;

[0260] If the monitoring value is less than or equal to the preset monitoring threshold, a second monitoring result of the data processing model is determined, and the second monitoring result indicates that the data processing model is available.

[0261] In some embodiments, the second determining module 204 is specifically configured to:

[0262] If the Euclidean distance between the center point of the first cluster and the center point of the second cluster is greater than a preset monitoring threshold, determining a first monitoring result of the data processing model, the first monitoring result indicating that the data processing model is unusable;

[0263] If the Euclidean distance between the center point of the first cluster and the center point of the second cluster is less than or equal to the preset monitoring threshold, a second monitoring result of the data processing model is determined, and the second monitoring result indicates that the data processing model is available.

[0264] In some embodiments, when the data processing model is deployed on the monitoring entity, the apparatus further comprises:

[0265] a third acquisition module, configured to, when the monitoring result indicates that the data processing model is unavailable, acquire second measurement data of a second reference signal between a second target entity and a fourth entity in a current monitoring period, where the fourth entity is located within the service area and a processing result of the data processing model corresponding to the fourth entity is known, and the second target entity is the entity, between the first entity and the second entity, that sends the second reference signal;

[0266] An updating module is configured to update the data processing model according to the second measurement data.

[0267] In some embodiments, the third acquisition module is specifically configured to:

[0268] A fourth measurement result is obtained from the third entity, where the fourth measurement result includes second measurement data of a second reference signal between the second target entity and the fourth entity.

[0269] In some embodiments, when the monitoring entity is the first entity, the third acquisition module is specifically configured to:

[0270] sending a third request to a third entity, where the third request instructs the third entity to send the second measurement data to the monitoring entity;

[0271] receiving a fourth measurement result corresponding to the third request sent by the third entity.

[0272] In some embodiments, the third request includes a minimum number of samples, and the fourth measurement result includes a number of second measurement data that is greater than or equal to the minimum number of samples.

[0273] In some embodiments, the third acquisition module is further configured to:

[0274] receiving a first response or a second response corresponding to the third request sent by the third entity, the first response indicating that the third entity is capable of sending measurement data greater than or equal to the minimum number of samples to the first entity, and the second response indicating that the third entity is incapable of sending measurement data greater than or equal to the minimum number of samples to the first entity;

[0275] After receiving the first response, the step of receiving a fourth measurement result corresponding to the third request sent by the third entity is performed.

[0276] In some embodiments, the minimum number of samples is expressed in units of quantity and bits.

[0277] In some embodiments, the second measurement data is measurement data sent by the fourth entity to the third entity according to third configuration information of the second reference signal sent by the third entity.

[0278] In some embodiments, the third configuration information includes at least one of the following: measurement-related information of the second reference signal and an identifier of a monitoring entity.

[0279] In some embodiments, the second measurement data includes at least one of the following: a measurement value, a true value label corresponding to the measurement value, and a data quality corresponding to the true value label.

[0280] In some embodiments, the measurement value, the true value label, and the data quality are represented by the number of bits.

[0281] In some embodiments, the fourth entity is a PRU or a terminal, and the accuracy of the measurement data obtained by the fourth entity is higher than a preset accuracy threshold.

[0282] In some embodiments, the apparatus further comprises:

[0283] The first sending module is configured to send monitoring capability information to a third entity.

[0284] In some embodiments, when the data processing model is deployed on the monitoring entity, the apparatus further comprises:

[0285] a receiving module, configured to receive a fourth request sent by a third entity, wherein the fourth request indicates obtaining monitoring capability information;

[0286] The first sending module is specifically configured to send monitoring capability information to the third entity according to the fourth request.

[0287] In some embodiments, the monitoring capability information includes at least one of the following: a maximum number of samples supported by the monitoring entity.

[0288] In some embodiments, when the data processing model is deployed on a fifth entity, the apparatus further comprises:

[0289] The second sending module is used to send the monitoring result to the fifth entity after obtaining the monitoring result.

[0290] In the technical solution provided by the embodiment of the present application, the monitoring entity clusters the input data (i.e., measurement data) of the data processing model collected in actual production and the training data of the data processing model into two clusters, namely, the first cluster to which the measurement data belongs and the second cluster to which the training data belongs, and compares the Euclidean distance between the center point of the first cluster and the center point of the second cluster with the preset monitoring threshold value, so as to determine whether there is a difference between the input data of the data processing model obtained in actual production and the training data of the data processing model, and obtain the corresponding monitoring result. If there is a difference between the input data of the data processing model obtained in actual production and the training data of the data processing model, it means that the environmental data has been offset and the data processing model cannot accurately process the input data in actual production. By using the above monitoring results, when there is a difference between the input data of the data processing model obtained in actual production and the training data of the data processing model, the data processing model can be retrained in time, thereby improving the data processing accuracy of the data processing model in actual production.

[0291] Corresponding to the above-mentioned model monitoring method, an embodiment of the present application further provides a monitoring entity, as shown in FIG21 , including a processor 211, a communication interface 212, a memory 213, and a communication bus 214, wherein the processor 211, the communication interface 212, and the memory 213 communicate with each other via the communication bus 214;

[0292] The memory 213 is used to store computer programs;

[0293] The processor 211 is configured to implement any of the above-mentioned steps of the model monitoring method when executing the program stored in the memory 213 .

[0294] The communication bus mentioned by the monitoring entity above may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, the figure shows only one thick line, but this does not mean that there is only one bus or only one type of bus.

[0295] The communication interface is used for communication between the above monitoring entity and other devices.

[0296] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage. Alternatively, the memory may be at least one storage device located away from the processor.

[0297] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0298] Corresponding to the above-mentioned model monitoring method, an embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, any of the above-mentioned steps of the model monitoring method is implemented.

[0299] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the steps of the model monitoring method described in any one of the above embodiments.

[0300] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0301] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0302] Each embodiment in this specification is described in a related manner. Similar portions between embodiments can be referenced to each other. Each embodiment focuses on the differences from other embodiments. In particular, the device, monitoring entity, storage medium, and program product embodiments are generally similar to the method embodiments, so their descriptions are simplified. For relevant portions, reference can be made to the descriptions of the method embodiments.

[0303] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A model monitoring method, characterized in that: Applied to a monitoring entity, the method comprises: Acquire first measurement data of a first reference signal between a first entity and a second entity in a current monitoring period, wherein the first entity and the second entity are located in a service area of ​​a data processing model, and the first measurement data is input data of the data processing model; Obtaining training data for the data processing model; Determine a first cluster to which the first measurement data belongs and a second cluster to which the training data belongs; The monitoring result of the data processing model is determined according to the Euclidean distance between the center point of the first cluster and the center point of the second cluster and a preset monitoring threshold.

2. The method according to claim 1, characterized in that When the monitoring entity is a first entity, the step of acquiring first measurement data of a first reference signal between the first entity and the second entity in a current monitoring period includes: receiving a first reference signal sent by a second entity within a current monitoring period; The first reference signal is measured to obtain a first measurement result, where the first measurement result includes first measurement data of the first reference signal.

3. The method according to claim 2, characterized in that When the first entity is a terminal, the second entity is a base station, and the first reference signal is a positioning reference signal PRS, a channel state information reference signal CSI-RS, a sounding reference signal SRS, a synchronization signal block SSB, a demodulation reference signal DMRS, or a phase tracking reference signal PTRS; When the first entity is a base station, the second entity is a terminal, and the first reference signal is an SRS; The first measurement data includes a channel impulse response CIR and a power delay profile PDP.

4. The method according to claim 1, characterized in that: When the monitoring entity is a first entity, the step of acquiring first measurement data of a first reference signal between the first entity and the second entity in a current monitoring period includes: Sending a first reference signal to the second entity within a current monitoring period; A second measurement result sent by the second entity is received, where the second measurement result includes first measurement data of the first reference signal.

5. The method according to claim 4, characterized in that When the first entity is a terminal, the second entity is a base station, and the first reference signal is an SRS; When the first entity is a base station, the second entity is a terminal, and the first reference signal is a PRS, a CSI-RS, an SRS, a SSB, a DMRS, or a PTRS; The first measurement data includes CIR and PDP.

6. The method according to claim 3 or 5, characterized in that: The first reference signal is a reference signal sent or received by the terminal according to first configuration information sent by the base station, and the first configuration information indicates the time-frequency resources occupied by the first reference signal.

7. The method according to claim 6, characterized in that The first configuration information is configuration information sent by the base station to the terminal according to a first request sent by a third entity, and the first request instructs the base station to send the first configuration information to the terminal.

8. The method according to claim 1, characterized in that When the monitoring entity is a first entity, the step of acquiring first measurement data of a first reference signal between the first entity and the second entity in a current monitoring period includes: Sending a second request to a third entity, wherein the third entity stores first measurement data of a first reference signal between the first entity and the second entity, and the second request instructs the third entity to send the first measurement data to the monitoring entity; A third measurement result corresponding to the second request sent by the third entity is received, where the third measurement result includes the first measurement data.

9. The method according to claim 8, characterized in that The first entity is a terminal or a base station, and the third entity is a management entity; The first measurement data includes CIR and PDP.

10. The method according to claim 1, characterized in that When the monitoring entity is a third entity, the step of acquiring first measurement data of a first reference signal between the first entity and the second entity in a current monitoring period includes: First measurement data of a first reference signal between the first entity and the second entity in a current monitoring period is acquired from a first target entity, where the first target entity is an entity between the first entity and the second entity that measures the first reference signal.

11. The method according to claim 10, characterized in that The third entity is a management entity; When the first entity is a terminal, the second entity is a base station, and the first target entity is the terminal, the first reference signal is a PRS, a CSI-RS, an SRS, an SSB, a DMRS, or a PTRS; When the first entity is a base station, the second entity is a terminal, and the first target entity is the base station, the first reference signal is an SRS; The first measurement data includes CIR and PDP.

12. The method according to any one of claims 1-5, 8-11, characterized in that: The first measurement data is first measurement data of the first reference signal obtained by the monitoring entity according to second configuration information sent by a third entity.

13. The method according to claim 12, characterized in that The second configuration information includes at least one of the following: monitoring cycle information, measurement configuration information, a monitoring algorithm, and the preset monitoring threshold.

14. The method according to claim 13, characterized in that The monitoring cycle information includes a cycle unit and a bit number.

15. The method according to claim 13, characterized in that The measurement configuration information includes a measurement cycle length, a measurement time slice length, and a measurement frequency.

16. The method according to claim 13, characterized in that The measurement configuration information, the monitoring algorithm and the preset monitoring threshold are represented by the number of bits.

17. The method according to claim 1, characterized in that The step of determining the first cluster to which the first measurement data belongs and the second cluster to which the training data belongs comprises: According to a preset monitoring algorithm, performing dimensionality reduction processing on the first measurement data and the training data to obtain dimensionality reduced data; The dimension-reduced data is clustered to obtain a first cluster to which the first measurement data belongs and a second cluster to which the training data belongs.

18. The method according to claim 17, characterized in that The step of performing dimensionality reduction processing on the first measurement data and the training data according to a preset monitoring algorithm to obtain dimensionality reduced data includes: converting the first measurement data and the training data into intermediate data matching the data processing model; According to a preset monitoring algorithm, the intermediate data is subjected to dimensionality reduction processing to obtain dimensionality reduced data.

19. The method according to claim 17 or 18, characterized in that The preset monitoring algorithm is a t-distributed random neighborhood embedding algorithm.

20. The method according to claim 1, characterized in that The step of determining the monitoring result of the data processing model according to the Euclidean distance between the center point of the first cluster and the center point of the second cluster and a preset monitoring threshold comprises: Calculating a ratio of a first distance to a second distance to obtain a monitoring value, wherein the first distance is a Euclidean distance between a center point of the first cluster and a center point of the second cluster, and the second distance is a sum of a radius of the first cluster and a radius of the second cluster; If the monitoring value is greater than the preset monitoring threshold, determining a first monitoring result of the data processing model, the first monitoring result indicating that the data processing model is unavailable; If the monitoring value is less than or equal to the preset monitoring threshold, a second monitoring result of the data processing model is determined, and the second monitoring result indicates that the data processing model is available.

21. The method according to claim 1, characterized in that The step of determining the monitoring result of the data processing model according to the Euclidean distance between the center point of the first cluster and the center point of the second cluster and a preset monitoring threshold comprises: If the Euclidean distance between the center point of the first cluster and the center point of the second cluster is greater than a preset monitoring threshold, determining a first monitoring result of the data processing model, the first monitoring result indicating that the data processing model is unavailable; If the Euclidean distance between the center point of the first cluster and the center point of the second cluster is less than or equal to the preset monitoring threshold, a second monitoring result of the data processing model is determined, and the second monitoring result indicates that the data processing model is available.

22. The method according to claim 1, characterized in that When the data processing model is deployed on the monitoring entity, the method further includes: When the monitoring result indicates that the data processing model is unavailable, obtaining second measurement data of a second reference signal between a second target entity and a fourth entity in a current monitoring period, the fourth entity being located in the service area, and a processing result of the data processing model corresponding to the fourth entity being known, and the second target entity being an entity of the first entity and the second entity that sends the second reference signal; The data processing model is updated according to the second measurement data.

23. The method according to claim 22, characterized in that The step of acquiring second measurement data of a second reference signal between the second target entity and the fourth entity in the current monitoring period includes: A fourth measurement result is obtained from the third entity, where the fourth measurement result includes second measurement data of a second reference signal between the second target entity and the fourth entity.

24. The method according to claim 23, characterized in that When the monitoring entity is the first entity, the step of obtaining the fourth measurement result from the third entity includes: Sending a third request to a third entity, wherein the third request instructs the third entity to send the second measurement data to the monitoring entity; A fourth measurement result corresponding to the third request sent by the third entity is received.

25. The method according to claim 24, characterized in that The third request includes a minimum number of samples, and the fourth measurement result includes a number of second measurement data that is greater than or equal to the minimum number of samples.

26. The method according to claim 25, characterized in that The method further comprises: receiving a first response or a second response corresponding to the third request sent by the third entity, the first response indicating that the third entity is capable of sending measurement data greater than or equal to the minimum number of samples to the first entity, and the second response indicating that the third entity is not capable of sending measurement data greater than or equal to the minimum number of samples to the first entity; After receiving the first response, the step of receiving a fourth measurement result corresponding to the third request sent by the third entity is performed.

27. The method according to claim 25 or 26, characterized in that The minimum number of samples is expressed in a quantity unit and a number of bits.

28. The method according to claim 23, characterized in that The second measurement data is measurement data sent by the fourth entity to the third entity according to third configuration information of the second reference signal sent by the third entity.

29. The method according to claim 28, characterized in that The third configuration information includes at least one of the following: measurement-related information of the second reference signal and an identifier of a monitoring entity.

30. The method according to any one of claims 22-26, 28-29, characterized in that: The second measurement data includes at least one of the following: a measurement value, a true value label corresponding to the measurement value, and data quality corresponding to the true value label.

31. The method according to claim 30, characterized in that The measured value, the true value label, and the data quality are represented by the number of bits.

32. The method according to any one of claims 22-26, 28-29, characterized in that: The fourth entity is a positioning reference unit PRU or a terminal, and the accuracy of the measurement data obtained by the fourth entity is higher than a preset accuracy threshold.

33. The method according to claim 1, characterized in that The method further comprises: The monitoring capability information is sent to the third entity.

34. The method according to claim 33, characterized in that When the data processing model is deployed on the monitoring entity, the method further includes: receiving a fourth request sent by the third entity, where the fourth request indicates obtaining monitoring capability information; According to the fourth request, the step of sending monitoring capability information to the third entity is performed.

35. The method according to claim 33 or 34, characterized in that The monitoring capability information includes at least one of the following: a maximum number of samples supported by the monitoring entity.

36. The method according to claim 1, characterized in that When the data processing model is deployed on the fifth entity, after obtaining the monitoring result, the method further includes: The monitoring result is sent to the fifth entity.

37. A model monitoring device, characterized in that: Applied to a monitoring entity, the device comprises: A first acquisition module, configured to acquire first measurement data of a first reference signal between a first entity and a second entity in a current monitoring period, wherein the first entity and the second entity are located in a service area of ​​a data processing model, and the first measurement data is input data of the data processing model; A second acquisition module, used to acquire training data of the data processing model; A first determination module, configured to determine a first cluster to which the first measurement data belongs and a second cluster to which the training data belongs; The second determination module is used to determine the monitoring result of the data processing model according to the Euclidean distance between the center point of the first cluster and the center point of the second cluster and a preset monitoring threshold.

38. The device according to claim 37, characterized in that When the monitoring entity is a first entity, the first acquisition module is specifically configured to: receiving a first reference signal sent by a second entity within a current monitoring period; The first reference signal is measured to obtain a first measurement result, where the first measurement result includes first measurement data of the first reference signal.

39. The device according to claim 38, characterized in that When the first entity is a terminal, the second entity is a base station, and the first reference signal is a positioning reference signal PRS, a channel state information reference signal CSI-RS, a sounding reference signal SRS, a synchronization signal block SSB, a demodulation reference signal DMRS, or a phase tracking reference signal PTRS; When the first entity is a base station, the second entity is a terminal, and the first reference signal is an SRS; The first measurement data includes a channel impulse response CIR and a power delay profile PDP.

40. The device according to claim 37, characterized in that When the monitoring entity is a first entity, the first acquisition module is specifically configured to: Sending a first reference signal to the second entity within a current monitoring period; A second measurement result sent by the second entity is received, where the second measurement result includes first measurement data of the first reference signal.

41. The device according to claim 40, characterized in that When the first entity is a terminal, the second entity is a base station, and the first reference signal is an SRS; When the first entity is a base station, the second entity is a terminal, and the first reference signal is a PRS, a CSI-RS, an SRS, a SSB, a DMRS, or a PTRS; The first measurement data includes CIR and PDP.

42. The device according to claim 39 or 41, characterized in that The first reference signal is a reference signal sent or received by the terminal according to first configuration information sent by the base station, and the first configuration information indicates the time-frequency resources occupied by the first reference signal.

43. The device according to claim 42, characterized in that The first configuration information is configuration information sent by the base station to the terminal according to a first request sent by a third entity, and the first request instructs the base station to send the first configuration information to the terminal.

44. The device according to claim 37, characterized in that When the monitoring entity is a first entity, the first acquisition module is specifically configured to: Sending a second request to a third entity, wherein the third entity stores first measurement data of a first reference signal between the first entity and the second entity, and the second request instructs the third entity to send the first measurement data to the monitoring entity; A third measurement result corresponding to the second request sent by the third entity is received, where the third measurement result includes the first measurement data.

45. The device according to claim 44, characterized in that The first entity is a terminal or a base station, and the third entity is a management entity; The first measurement data includes CIR and PDP.

46. ​​The device according to claim 37, characterized in that When the monitoring entity is a third entity, the first acquisition module is specifically configured to: First measurement data of a first reference signal between the first entity and the second entity in a current monitoring period is acquired from a first target entity, where the first target entity is an entity between the first entity and the second entity that measures the first reference signal.

47. The device according to claim 46, characterized in that The third entity is a management entity; When the first entity is a terminal, the second entity is a base station, and the first target entity is the terminal, the first reference signal is a PRS, a CSI-RS, an SRS, an SSB, a DMRS, or a PTRS; When the first entity is a base station, the second entity is a terminal, and the first target entity is the base station, the first reference signal is an SRS; The first measurement data includes CIR and PDP.

48. The device according to any one of claims 37-41, 44-47, characterized in that The first measurement data is first measurement data of the first reference signal obtained by the monitoring entity according to second configuration information sent by a third entity.

49. The device according to claim 48, characterized in that The second configuration information includes at least one of the following: monitoring cycle information, measurement configuration information, a monitoring algorithm, and the preset monitoring threshold.

50. The device according to claim 49, characterized in that The monitoring cycle information includes a cycle unit and a bit number.

51. The device according to claim 49, characterized in that The measurement configuration information includes a measurement cycle length, a measurement time slice length, and a measurement frequency.

52. The device according to claim 49, characterized in that The measurement configuration information, the monitoring algorithm and the preset monitoring threshold are represented by the number of bits.

53. The device according to claim 37, characterized in that The first determining module is specifically configured to: According to a preset monitoring algorithm, performing dimensionality reduction processing on the first measurement data and the training data to obtain dimensionality reduced data; The dimension-reduced data is clustered to obtain a first cluster to which the first measurement data belongs and a second cluster to which the training data belongs.

54. The device according to claim 53, characterized in that The first determining module is specifically configured to: converting the first measurement data and the training data into intermediate data matching the data processing model; According to a preset monitoring algorithm, the intermediate data is subjected to dimensionality reduction processing to obtain dimensionality reduced data.

55. The device according to claim 53 or 54, characterized in that The preset monitoring algorithm is a t-distributed random neighborhood embedding algorithm.

56. The device according to claim 37, characterized in that The second determining module is specifically used to: Calculating a ratio of a first distance to a second distance to obtain a monitoring value, wherein the first distance is a Euclidean distance between a center point of the first cluster and a center point of the second cluster, and the second distance is a sum of a radius of the first cluster and a radius of the second cluster; If the monitoring value is greater than the preset monitoring threshold, determining a first monitoring result of the data processing model, the first monitoring result indicating that the data processing model is unavailable; If the monitoring value is less than or equal to the preset monitoring threshold, a second monitoring result of the data processing model is determined, and the second monitoring result indicates that the data processing model is available.

57. The device according to claim 37, characterized in that The second determining module is specifically used to: If the Euclidean distance between the center point of the first cluster and the center point of the second cluster is greater than a preset monitoring threshold, determining a first monitoring result of the data processing model, the first monitoring result indicating that the data processing model is unavailable; If the Euclidean distance between the center point of the first cluster and the center point of the second cluster is less than or equal to the preset monitoring threshold, a second monitoring result of the data processing model is determined, and the second monitoring result indicates that the data processing model is available.

58. The device according to claim 37, characterized in that When the data processing model is deployed on the monitoring entity, the device further includes: a third acquisition module, configured to, when the monitoring result indicates that the data processing model is unavailable, acquire second measurement data of a second reference signal between a second target entity and a fourth entity in a current monitoring period, wherein the fourth entity is located in the service area, and a processing result of the data processing model corresponding to the fourth entity is known, and the second target entity is an entity of the first entity and the second entity that sends the second reference signal; An updating module is used to update the data processing model according to the second measurement data.

59. The device according to claim 58, characterized in that The third acquisition module is specifically used for: A fourth measurement result is obtained from the third entity, where the fourth measurement result includes second measurement data of a second reference signal between the second target entity and the fourth entity.

60. The device according to claim 59, characterized in that When the monitoring entity is the first entity, the third acquisition module is specifically configured to: Sending a third request to a third entity, wherein the third request instructs the third entity to send the second measurement data to the monitoring entity; A fourth measurement result corresponding to the third request sent by the third entity is received.

61. The device according to claim 60, characterized in that The third request includes a minimum number of samples, and the fourth measurement result includes a number of second measurement data that is greater than or equal to the minimum number of samples.

62. The device according to claim 61, characterized in that The third acquisition module is further used for: receiving a first response or a second response corresponding to the third request sent by the third entity, the first response indicating that the third entity is capable of sending measurement data greater than or equal to the minimum number of samples to the first entity, and the second response indicating that the third entity is not capable of sending measurement data greater than or equal to the minimum number of samples to the first entity; After receiving the first response, the step of receiving a fourth measurement result corresponding to the third request sent by the third entity is performed.

63. The device according to claim 61 or 62, characterized in that The minimum number of samples is expressed in a quantity unit and a number of bits.

64. The device according to claim 59, characterized in that The second measurement data is measurement data sent by the fourth entity to the third entity according to third configuration information of the second reference signal sent by the third entity.

65. The device according to claim 64, characterized in that The third configuration information includes at least one of the following: measurement-related information of the second reference signal and an identifier of a monitoring entity.

66. The device according to any one of claims 58-62, 64-65, characterized in that The second measurement data includes at least one of the following: a measurement value, a true value label corresponding to the measurement value, and data quality corresponding to the true value label.

67. The device according to claim 66, characterized in that The measured value, the true value label, and the data quality are represented by the number of bits.

68. The device according to any one of claims 58-62, 64-65, characterized in that The fourth entity is a positioning reference unit PRU or a terminal, and the accuracy of the measurement data obtained by the fourth entity is higher than a preset accuracy threshold.

69. The device according to claim 37, characterized in that The device also includes: The first sending module is used to send monitoring capability information to the third entity.

70. The device according to claim 69, characterized in that When the data processing model is deployed on the monitoring entity, the device further includes: A receiving module, configured to receive a fourth request sent by a third entity, wherein the fourth request indicates obtaining monitoring capability information; The first sending module is specifically configured to send monitoring capability information to the third entity according to the fourth request.

71. The device according to claim 69 or 70, characterized in that The monitoring capability information includes at least one of the following: a maximum number of samples supported by the monitoring entity.

72. The device according to claim 37, characterized in that When the data processing model is deployed on the fifth entity, the apparatus further includes: The second sending module is used to send the monitoring result to the fifth entity after obtaining the monitoring result.

73. A monitoring entity, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor, when used to execute the program stored in the memory, implements the method steps described in any one of claims 1-36.

74. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1-36 are implemented.

Images