Data analysis method and device, computer device and storage medium

Abstract

The application relates to a data analysis method and device, computer equipment, a storage medium and a computer program product. The method is applied to a network data analysis network element, and comprises the following steps: obtaining a subscription event and original data; obtaining a machine learning model and a data preprocessing scheme of the machine learning model according to the subscription event; converting the original data according to the data preprocessing scheme to obtain feature data; and inputting the feature data into the machine learning model for prediction to obtain an analysis result of the subscription event. The method can convert the original data into multiple model-adapted input data through the preprocessing scheme indicated by the subscription event, and can perform inference analysis, thereby realizing the reuse of the original data. Therefore, the frequency of obtaining the original data in the inference process for each subscription event is relatively low, and the pressure of a data source network element and other equipment is small.

Description

Technical Field

[0001] This application relates to the field of communications, and in particular to a data analysis method, apparatus, computer equipment, storage medium, and computer program product. Background Technology

[0002] With the continuous development of mobile communication networks, the process of core network elements handling data throughput needs to be implemented based on the Network Data Analytics Function (NWDAF).

[0003] The network data analysis element collects data from the data source element based on subscribed events, and then uses a machine learning model to perform inference and analysis on the collected data, pushing the inference results to the core network element, which acts as the consumer. If the network data analysis element collects data from the data source element and other devices for every subscribed event, it will put excessive pressure on those devices. Summary of the Invention

[0004] Therefore, it is necessary to provide a data analysis method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can alleviate the pressure on data source network elements and other equipment to address the aforementioned technical problems.

[0005] Firstly, this application provides a data analysis method applied to network data analysis network elements, the method comprising:

[0006] Retrieve subscribed events and raw data;

[0007] Based on the subscribed events, obtain the machine learning model and the data preprocessing scheme for the machine learning model;

[0008] The original data is transformed according to the data preprocessing scheme to obtain feature data;

[0009] The feature data is input into the machine learning model for prediction to obtain the analysis results of the subscribed event.

[0010] In one embodiment, obtaining the machine learning model and its data preprocessing scheme based on the subscribed event includes:

[0011] Based on the model information indicated by the subscription event, obtain the machine learning model indicated by the model information from the model training module;

[0012] According to the model information, the data preprocessing scheme of the machine learning model is obtained from the model training module.

[0013] In one embodiment, the model training module performs the training process of the machine learning model, and the model training module caches the data preprocessing steps of the training process as the data preprocessing scheme.

[0014] In one embodiment, the data preprocessing scheme includes at least one of a data filtering scheme, a format conversion scheme, and a data normalization scheme; the step of converting the original data according to the data preprocessing scheme to obtain feature data includes:

[0015] The raw data is filtered according to the data filtering scheme to obtain feature data that matches the content of the machine learning model; and / or,

[0016] The original data is converted according to the format conversion scheme to obtain feature data that matches the format of the machine learning model; and / or,

[0017] The original data is processed according to the data normalization scheme to obtain feature data that matches the scale of the machine learning model.

[0018] In one embodiment, obtaining the subscription event and raw data includes:

[0019] Retrieve different subscribed events and obtain raw data from data source network elements;

[0020] The original data corresponds to the different subscription events.

[0021] In one embodiment, the raw data and the training data of the machine learning model belong to the same data category, which is divided based on core network elements and is determined according to the subscription event.

[0022] In one embodiment, the data preprocessing scheme is determined based on the processing efficiency and metrics of the machine learning model.

[0023] Secondly, this application also provides a data analysis device applied to a network data analysis element, the device comprising:

[0024] The data acquisition module is used to acquire subscribed events and raw data;

[0025] The scheme acquisition module is used to acquire the machine learning model and the data preprocessing scheme of the machine learning model based on the subscribed event;

[0026] The feature transformation module is used to transform the original data according to the data preprocessing scheme to obtain feature data;

[0027] The analysis module is used to input the feature data into the machine learning model for prediction and obtain the analysis results of the subscribed event.

[0028] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the data analysis steps in any of the above embodiments.

[0029] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the data analysis steps in any of the above embodiments.

[0030] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the data analysis steps in any of the above embodiments.

[0031] The aforementioned data analysis methods, devices, computer equipment, storage media, and computer program products do not have a one-to-one correspondence between subscription events and raw data. To ensure that each acquired raw data is usable by one or more subscription events, it is transformed using a preprocessing scheme indicated by the subscription event. This allows a single set of raw data to be converted into input data suitable for multiple models, enabling inference analysis and thus reusing the raw data. Consequently, the frequency of acquiring raw data during the inference process for each subscription event is relatively low, reducing the pressure on data source network elements and other equipment. Attached Figure Description

[0032] Figure 1 This is a diagram illustrating the application environment of a data analysis method in one embodiment.

[0033] Figure 2 This is a flowchart illustrating a data analysis method in one embodiment;

[0034] Figure 3 This is a flowchart illustrating the data analysis method in another embodiment;

[0035] Figure 4 This is a flowchart illustrating the data analysis method in yet another embodiment;

[0036] Figure 5 This is a structural block diagram of a data analysis device in one embodiment;

[0037] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0039] The data analysis method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, network data analysis element 104 collects raw data from the data stored in data source element 102 based on the subscription events received from the subscription, performs data analysis, and generates corresponding analysis results for the subscription events. These analysis results can be fed back to consumer 106 or to devices other than consumer 106. Both network data analysis element 104 and data source element 102 are core network elements serving consumer 106.

[0040] Consumer 106 includes core network elements. The data categories of the data processed by the core network elements in Consumer 106 can be broadly categorized according to device network management, such as performance, alarms, configuration, and logs, or further refined according to network application data, such as services, traffic, and signaling. The raw data matches the data categories processed by Consumer 106. Network data analysis element 104 obtains raw data from data source element 102, and performs preprocessing and inference analysis on the raw data to obtain the analysis results of the corresponding subscribed events. Each core network element can be a specific hardware device or a software module used to implement a certain function, and each core network element can belong to the 5G domain or even the 6G domain.

[0041] In one embodiment, such as Figure 2 As shown, a data analysis method is provided, which is applied to Figure 1 The network data analysis element 104 in the middle includes the following steps:

[0042] Step 202: Obtain the subscribed events and raw data.

[0043] Subscription events are events subscribed to and received by the network data analysis element 104. Subscription events are sent for data categories related to machine learning models, used to select a specific machine learning model to perform predictive analysis on data of that data category, in order to obtain the analysis results of that machine learning model; the analysis results of that machine learning model are the analysis results of the subscription events.

[0044] Optionally, the subscribed events can be related to the machine learning model and fall under the categories of performance issues, network error alerts, configuration security threats, log anomalies, or other types of notifications. For example, a subscribed event could be a request failure event, a database timeout event, or a connection rejection event from a network element.

[0045] The raw data is the input data to be transformed by the machine learning model indicated by the subscribed event. There is no one-to-one correspondence between the raw data and the subscribed event, nor is there a one-to-one correspondence between the raw data and the machine learning model. Optionally, the raw data is the data that consumer 106 can process; it can be either input or output data of consumer 106.

[0046] Optionally, the input data to be transformed is data that can be input into a machine learning model. Specifically, when a piece of data can be directly input into a machine learning model, that data is raw data; when a piece of data has undergone preprocessing, and the preprocessing result can be directly input into a machine learning model, that data is considered input data to be transformed.

[0047] Optionally, the network data analysis element 104 can obtain subscribed events and raw data according to the event categories it subscribes to. The event category can be the data category that the consumer is processing, or it can be a further refinement of the data category using different processing methods.

[0048] In one embodiment, acquiring subscription events and raw data includes: subscribing to at least one device among a data source network element and a consumer; if a subscription event is acquired, raw data is collected from the data source network element; wherein, the consumer is a core network element that processes different data. Thus, the network data analysis network element 104 can acquire subscription events from different devices, and the acquisition process of subscription events is a condition for acquiring raw data, thereby adaptively adjusting the load on devices such as the data source network element.

[0049] Step 204: Obtain the machine learning model and its data preprocessing scheme based on the subscribed events.

[0050] A machine learning model (ML) is a pre-trained model. There is a correspondence between the ML model and the subscribed events; this correspondence indicates the ML model and the execution process it undergoes. Optionally, the ML model is obtained by invoking the network data analysis element 104.

[0051] The data preprocessing scheme is based on the input data settings of the machine learning model and is used to transform the raw data for the machine learning model so that the transformed feature data matches the input data of the machine learning model. Optionally, the data preprocessing scheme involves data preprocessing processes such as content, format, and scaling.

[0052] In one embodiment, obtaining a machine learning model and a data preprocessing scheme for the machine learning model based on a subscription event includes: obtaining the trained machine learning model indicated by the subscription event; and obtaining the data preprocessing scheme indicated by the subscription event; wherein the trained machine learning model and the data preprocessing scheme are in one-to-one correspondence.

[0053] Step 206: Transform the raw data according to the data preprocessing scheme to obtain feature data.

[0054] Feature data is matched with the input data of the machine learning model, and the content of the feature data is part of the content of the original data. Feature data is candidate input data for the machine learning model indicated by the subscribed event. There is not a one-to-one correspondence between feature data and subscribed events, and one set of feature data corresponds to one machine learning model, while one machine learning model can correspond to different sets of feature data. Optionally, all the content of the feature data is the same as at least part of the content of the original data.

[0055] In one embodiment, the raw data is transformed according to a data preprocessing scheme, including: determining a data preprocessing sub-scheme for the raw data based on the input data requirements of the raw data and the machine learning model; and transforming the raw data using the data preprocessing sub-scheme. The input data requirements include content, format, and / or scaling requirements. Thus, by determining a data preprocessing sub-scheme from the data preprocessing scheme based on the raw data, a data preprocessing sub-scheme can be adaptively selected for different types of raw data, ensuring processing efficiency.

[0056] Optionally, the data preprocessing sub-scheme may further include: a preprocessing sub-scheme for completeness and quality, used to find missing values, erroneous values, or outliers in the original data, and to clean the found missing values, erroneous values, or outliers to improve processing efficiency. Optionally, the data preprocessing sub-scheme may further include: a preprocessing sub-scheme for feature sparsity and correlation, to adjust the data distribution to ensure a balance between the accuracy of feature data category identification, thereby keeping the sparsity and correlation within the feature data within a suitable range, avoiding overfitting, and improving the accuracy of the analysis results.

[0057] Step 208: Input the feature data into the machine learning model for prediction and obtain the analysis results of the subscribed events.

[0058] The analysis results include predicted values ​​from machine learning models based on feature data. These results can also be analysis reports for subscribed events, accessible to consumers who require them. For example, when consumers belong to different core network elements, the analysis results can be used for in-depth analysis of network performance and efficiency to improve the network performance and bandwidth utilization of core network elements. Furthermore, the analysis results can be used for network fault detection, security checks, and other applications.

[0059] Optionally, the feature data is input into a machine learning model to perform inference analysis on the feature data according to the inference analysis method obtained from the training of the machine learning model, so as to obtain the predicted value of the feature data; the predicted value can be used as the analysis result of the subscription event. Optionally, the analysis result of the subscription event may also include information such as a graph or table drawn from the predicted value and the confidence level of the predicted value, so as to reduce the probability of the subscription event being triggered multiple times and reduce the pressure on the data source network element and other devices.

[0060] In the aforementioned data analysis method, there is not a one-to-one correspondence between subscribed events and raw data. To ensure that each acquired raw data can be used by one or more subscribed events, it is transformed using a preprocessing scheme indicated by the subscribed event. This allows a single set of raw data to be transformed into input data suitable for multiple models for inference analysis, thus achieving data reuse. Consequently, the frequency of acquiring raw data during the inference process for each subscribed event is relatively low, reducing the pressure on data source network elements and other equipment. Correspondingly, the infrequent acquisition of raw data avoids redundant occupation of storage space in the network data analysis network elements, preventing storage space waste. Furthermore, the network data analysis network elements can acquire raw data from data source network elements from different vendors. Through data preprocessing schemes, the network data analysis network elements standardize the input data for machine learning models, expanding the selection range of raw data acquisition.

[0061] In one embodiment, obtaining subscription events and raw data includes: obtaining different subscription events and obtaining raw data from a data source network element; wherein the raw data corresponds to different subscription events.

[0062] Different subscription events are used to indicate different machine learning models and their respective data preprocessing schemes. Different subscription events can be obtained by the network data analysis network element from subscribing to different core network elements, or they can be different events subscribed to by the network data analysis network from the same core network element.

[0063] In an optional embodiment, obtaining different subscription events and obtaining raw data from the data source network element includes: obtaining raw data from the data source network element according to the data category indicated by the different subscription events, the data category to which the model's training data belongs, or the data category of the core network element.

[0064] Therefore, for different subscription events obtained by the network data analysis network element, a separate original data can be obtained; by using this original data and different subscription events, steps 204-208 are executed respectively, so that the frequency of the network data analysis network element obtaining data from the data source network element is reduced, thereby alleviating the pressure on the data source network element.

[0065] In one embodiment, the relevant elements of the raw data are described more accurately; the raw data and the training data of the machine learning model belong to the same data category, which is divided based on core network elements and determined according to subscription events.

[0066] Training data refers to the data used during the training process of a machine learning model. Training data includes training samples and training parameters. The data category to which the training data belongs is the data category that the machine learning model can predict. This embodiment does not involve the application process of the training data, but rather directly compares the data category of the training data with the data category of the original data to obtain the corresponding original data.

[0067] The original data and the training data are not necessarily the same, but they belong to the same data category so that the feature data transformed by the data preprocessing scheme can be directly input into the machine learning model for prediction in order to obtain the corresponding analysis results.

[0068] Optionally, the data categories are based on core network elements. They are divided into major categories such as performance, alarms, configuration, and logs according to device network management, or further subdivided into categories such as services, traffic, and signaling according to network application data.

[0069] Optionally, the feature data obtained by the data preprocessing scheme belongs to the data category. Therefore, the data preprocessing scheme will not analyze across data categories, ensuring that this embodiment can be performed accurately.

[0070] In one embodiment, if the original data d belongs to type A call data and the A machine learning model is trained using type A call data, then the original data d is transformed using the data preprocessing scheme of the A machine learning model and then input into the A machine learning model for inference analysis.

[0071] Based on this, the raw data and the training data of the machine learning model belong to the same data category, which allows for flexible selection of raw data within the data category, thereby increasing the probability of performing inference analysis. The data category is based on the core network elements to make adaptive adjustments to the data throughput of different core network element devices, ensuring that this embodiment can be performed stably. The data category is determined based on the subscription event, so that the raw data is obtained based on the data category.

[0072] In one embodiment, obtaining a machine learning model and a data preprocessing scheme for the machine learning model based on a subscription event includes: obtaining the machine learning model indicated by the model information from the model training module based on the model information indicated by the subscription event; and obtaining the data preprocessing scheme for the machine learning model from the model training module according to the model information.

[0073] Model information is the information indicated by the subscribed event, used to determine a specific machine learning model and its data preprocessing scheme. Optionally, model information is set according to a specific data category, used to determine a specific machine learning model for that data category, and its data preprocessing scheme.

[0074] The model training module is used to train machine learning models and to cache the trained machine learning models and their data preprocessing schemes. By storing the machine learning models and data preprocessing schemes through the model training module, and using the model training module as the data source for these models and schemes, not only is the pressure on the data source network element alleviated, but the model training and storage processes are also nearly integrated, allowing for more efficient storage of the machine learning models and data preprocessing schemes. Furthermore, the model training module can be the Model Training Logical Function (MTLN) within the network data analysis network element. In this case, the network data analysis network element only needs to obtain the subscribed events and a single set of raw data to perform the data analysis process independently, resulting in lower dependence on external network elements and avoiding pressure on other core network elements.

[0075] In one implementation, obtaining the machine learning model indicated by the model information from the model training module based on the model information indicated by the subscription event includes: the logical analysis module of the network data analysis element obtaining the machine learning model indicated by the model identifier from the model training module of the network data analysis element based on the model identifier indicated by the subscription event.

[0076] Correspondingly, based on the model information, the data preprocessing scheme for the machine learning model is obtained from the model training module. This includes: the logical analysis module of the network data analysis element obtaining the data preprocessing scheme indicated by the model identifier from the model training module of the network data analysis element. Thus, by obtaining the machine learning model and data preprocessing scheme in parallel based on a single model identifier, the efficiency of data processing is improved.

[0077] Both the machine learning model and its data preprocessing scheme are pre-stored in the model training module and are invoked by the network data analysis network element according to the model information indicated by the subscribed event. This not only standardizes the data input to the machine learning model based on the data preprocessing scheme, but also expands the range of data sources. Furthermore, from an engineering perspective, there are fewer changes to the network element data transmission.

[0078] The model training module executes the training process of the machine learning model, and caches the data preprocessing steps of the training process as a data preprocessing scheme.

[0079] The training process is a set of steps by which the model training module trains a machine learning model based on training data. This set of steps includes at least two subsets: data preprocessing steps and training execution steps. Each subset includes one or more specific scheme steps. The data preprocessing steps precede the training execution steps. The data preprocessing steps are used to filter, transform, normalize, and / or perform other transformation operations on the training data so that the training data is suitable for training the machine learning model. The training execution steps are the steps by which the model training module trains the machine learning model based on the transformed training data.

[0080] Optionally, the data preprocessing step includes filtering, transforming, normalizing, and / or other operations performed on the training data by algorithm engineers. These operations are cached as a data preprocessing scheme by the model training module. Thus, caching the manual preprocessing steps allows for the efficient construction of a data preprocessing scheme, facilitating the reuse of these steps and enabling the feature data obtained from the transformation of the original data to better serve as input data for the machine learning model.

[0081] During the training process of machine learning models, the data preprocessing steps of the training process are cached as a data preprocessing scheme. The data preprocessing scheme can be configured, improved or adjusted through the model training process, reducing the number of times the data preprocessing scheme is created. This makes the data reuse efficiency higher, and allows the same raw data to be differentiated into more feature data for more subscribed events, thereby better reducing the pressure on data source network elements and saving resource storage space of network data analysis network elements.

[0082] In an exemplary embodiment, the network data analysis element includes an Analytics Logical Function (AnLF) module, through which steps 202-208 are executed; and the network data analysis element includes a Model Training Logical Function (MTLF) module, through which the training process of the machine learning model (ML model), the generation process of the data preprocessing scheme, and the caching process of the model and the data preprocessing scheme are performed.

[0083] In an exemplary embodiment, prior to step 202, the model training module performs model training and, in addition to storing the machine learning model file, also caches the data preprocessing steps performed by the algorithm engineer during the training process; step 202 includes: whenever an analysis report of a subscription event needs to be provided, the network data analysis element collects raw data from the data source element.

[0084] In step 204, the machine learning model is obtained based on the subscribed events, including: the AnLF module requests the ML model from the MTLF module based on the user's subscribed events.

[0085] In step 204, the process of obtaining the data preprocessing scheme includes: the AnLF module requesting the data processing scheme of the ML model from the MTLF module.

[0086] Steps 206 and 208 correspond to: AnLF processes the collected raw data according to the data processing scheme to obtain feature data, and inputs the feature data into the ML model for inference and prediction.

[0087] After step 208, the method further includes: NWDAF sending the corresponding subscribed inference reports to the users respectively.

[0088] In another exemplary embodiment, such as Figure 3 As shown, when the logical analysis module of the network data analysis element obtains two subscription events a and b, and provides analysis reports for these two subscription events a and b, it includes:

[0089] Before step 202, there is also step 0, in which the model training module trains the model and stores the machine learning model file, caching the data preprocessing steps performed by the algorithm engineer during the training process;

[0090] Step 202, including step 1, specifically: whenever an event report needs to be provided, the network data analysis element collects data D1 from the data source element so that the logical analysis module in the network data analysis element can obtain the subscribed event and the raw data;

[0091] In step 204, the machine learning model is obtained based on the subscription event, including: 2a. The logic analysis module requests machine learning model a from the model training module based on the subscription event a subscribed by the user; and 2b. The model training module requests machine learning model b from the model training module.

[0092] In step 204, the process of obtaining the data preprocessing scheme includes: 3a. The AnLF module requests the data processing scheme a of ML model a from the MTLF module; and 3b. The AnLF module requests the data processing scheme b of ML model b.

[0093] Steps 206 and 208 correspond to: 4a. AnLF processes the collected raw data D1 according to data processing scheme a to obtain feature data a1, inputs feature data a1 into ML model a for inference and prediction, and obtains the analysis result of subscription event a; and 4b. AnLF processes data D1 with scheme b to obtain feature data b1, uses feature data b1 as data of ML model b for inference and prediction, and obtains the analysis result of subscription event b.

[0094] In one embodiment, the data preprocessing scheme includes at least one of a data filtering scheme, a format conversion scheme, and a data warping scheme. The data filtering scheme, format conversion scheme, and data warping scheme are different data preprocessing sub-schemes, each used to perform different transformations of the original data; the data filtering scheme is used to filter out original data that conforms to the content of the machine learning model; the format conversion scheme is used to convert the original data into the format of the input data for the machine learning model; and the data warping scheme is used to adjust the scale of the original data to a suitable specification.

[0095] Optionally, the input data requirements for the machine learning model include requirements for content, format, and scale. If the original data does not conform to any of these requirements, it is transformed into feature data according to the scheme for addressing the inconsistency. For example, if the content of the original data covers the content required by the machine learning model, and the amount of content data in the original data exceeds the threshold required by the machine learning model, then the original data needs to be filtered according to a data filtering scheme to obtain feature data that matches the content of the machine learning model. It is understood that content limitations can be not only limited by the amount of data, but also by certain fields or by certain identifiers.

[0096] The above process transforms the raw data according to the data preprocessing scheme to obtain feature data, including:

[0097] The raw data is filtered according to a data filtering scheme to obtain feature data that matches the content of the machine learning model; and / or,

[0098] The original data is converted according to a format conversion scheme to obtain feature data that matches the format of the machine learning model; and / or,

[0099] The original data is processed according to the data normalization scheme to obtain feature data that matches the scale of the machine learning model.

[0100] In one implementation, the raw data is filtered according to a data filtering scheme to obtain feature data that matches the content of the machine learning model. This includes filtering the raw data according to the fields indicated by the data filtering scheme to obtain feature data that meets the content conditions of the machine learning model fields. Optionally, for different machine learning models, adjustments are made based on content factors such as business data, variable correlation, feature importance, feature fitting, and encoding methods to improve the quality of the machine learning models. This quality can include aspects such as processing efficiency, accuracy, and robustness. After model adjustments, the input fields required by different machine learning models will also be adjusted according to content factors, and the process of adjusting these input fields will efficiently and synchronously adjust the data filtering scheme.

[0101] In one implementation, the original data is format-converted according to a format conversion scheme to obtain feature data that matches the format of the machine learning model. This includes: converting the original data according to the input data field format of the machine learning model to obtain feature data that conforms to the format conditions of the machine learning model. Optionally, the process of format conversion of the original data may include dimensions such as one-hot encoding, label encoding, and EMBEDDING, used to convert the data types in the original data into numeric types. The numeric types include integer, long integer, or floating-point formats, so that the machine learning model can efficiently use feature data that conforms to the format conditions.

[0102] In one implementation, the original data is processed according to a data normalization scheme to obtain feature data that matches the scale of the machine learning model. This includes scaling the original data according to the data normalization scheme to obtain feature data that meets the scale conditions of the machine learning model. Optionally, the scaling scale involves normalization, centering, or standardization of the original data to enable the machine learning model to generate analysis reports of subscribed events more efficiently.

[0103] Optionally, the data filtering scheme, format conversion scheme, and data warping scheme are performed sequentially. The result obtained from the current preprocessing sub-scheme is input into the next preprocessing sub-scheme to continue the conversion process, until all preprocessing sub-schemes containing the original data have been executed. Optionally, any number of preprocessing sub-schemes can be selected from the data filtering scheme, format conversion scheme, and data warping scheme for execution.

[0104] In one feasible implementation, the raw data is processed sequentially according to a data filtering scheme, a format conversion scheme, and a data regularization scheme, including: filtering the raw data according to the data filtering scheme to obtain first feature data that matches the content of the machine learning model; converting the first feature data according to the format conversion scheme to obtain second feature data that matches the format of the machine learning model; and regularizing the second feature data according to the data regularization scheme to obtain third feature data that matches the scale of the machine learning model; the third feature data is the feature data used to generate the analysis report.

[0105] In one exemplary embodiment, such as Figure 4 As shown, the data processing pre-planning scheme consists of three sub-schemes: a data filtering scheme, a data transformation scheme, and a data normalization scheme, corresponding to the data filtering, transformation, and normalization modules in the processing module. The filtering module is used to select fields that meet the requirements of the machine learning model; the transformation module is used to transform the format of the fields, such as converting strings to integers and floating-point numbers; normalization is used to perform scaling operations such as data normalization, standardization, and centering. In this embodiment, the network data analysis element only needs to collect data from the data source element once. After the data is transformed by the data processing module, feature data a, b, and c are formed. Then, feature data a, b, and c are input into their respective machine learning models to obtain analysis reports for subscription events A, B, and C, thus completing the sending of analysis reports obtained from inference based on the subscription time to the user's terminal for each subscription event.

[0106] Optionally, the data preprocessing scheme is determined based on the processing efficiency and metrics of the machine learning model.

[0107] The configuration of the data preprocessing scheme is matched with the input data of the machine learning model to which the data preprocessing scheme belongs. Because the machine learning model can be changed based on requirements such as algorithm efficiency and metrics, the data processing scheme can also be changed accordingly, adaptively performing feature engineering such as data filtering to improve the accuracy and recall of the analysis results.

[0108] Optionally, if the data prediction processing step in the training process of the machine learning model is cached as a data preprocessing scheme, the efficiency of adjusting the data preprocessing scheme can be further improved.

[0109] In one embodiment, the data analysis method of this application belongs to a data processing method for network data analysis network elements. It adds a data preprocessing scheme caching module to the model training module, adds a data processing module to the logic analysis module, and implements caching and scheduling methods for the data processing scheme.

[0110] The standard specifications for network data analysis network elements define the input data for each type of subscription event. These elements collect data in real time for inference and prediction. Through data preprocessing, simulation algorithm engineers perform operations such as filtering, transforming, and normalizing the raw data, thereby converting the input data into feature data required by machine learning models for inference and analysis.

[0111] Furthermore, in application scenarios where data preprocessing schemes collect raw data from data source network elements based on different subscription events, the issue of the increased raw data collection required for more subscription events is analyzed: repetitive data collection may put significant pressure on the data source network element; moreover, large-scale data collection wastes resources for data storage. The method provided in this patent can provide a single set of raw data collected by a network element to multiple subscription events that use this data, thereby achieving data reuse.

[0112] In another specific embodiment, the probability of the network data analysis element collecting the same raw data multiple times is relatively small, thus reducing resource waste. This solves the problem of the network data analysis element supplying the same raw data to different machine learning models.

[0113] Specifically, network data analysis network elements can reuse collected data. On the one hand, network data analysis network elements do not repeatedly collect the same data, reducing the pressure on network elements to provide the same data for different subscription events; on the other hand, by using different data preprocessing schemes for the same raw data, it can be converted into multiple segments of data with different features, which can be used by machine learning models for different subscription events, saving storage resources.

[0114] Furthermore, network data analysis network elements standardize input data themselves, expanding the range of data sources and reducing modifications to network data output in engineering. In addition, network data analysis network elements can configure data processing schemes according to the efficiency and metrics requirements of machine learning models, and can perform feature engineering such as data filtering to improve the accuracy and recall of analysis results.

[0115] It is understandable that as communication networks continue to develop, network data analysis elements can accept more and more subscription events. Differentiating the same data based on different subscription events can reduce the pressure on data source elements and save storage resources. Furthermore, network data analysis elements can collect data from network elements from different manufacturers, and by standardizing the input data themselves, they can expand the range of data collection options.

[0116] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0117] Based on the same inventive concept, this application also provides a data analysis apparatus for implementing the data analysis method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data analysis apparatus embodiments provided below can be found in the limitations of the data analysis method described above, and will not be repeated here.

[0118] In one embodiment, such as Figure 5 As shown, a data analysis device is provided, applied to a network data analysis network element, the device comprising:

[0119] Data acquisition module 502 is used to acquire subscribed events and raw data;

[0120] The scheme acquisition module 504 is used to acquire the machine learning model and the data preprocessing scheme of the machine learning model according to the subscribed event;

[0121] The feature conversion module 506 is used to convert the original data according to the data preprocessing scheme to obtain feature data;

[0122] The analysis module 508 is used to input the feature data into the machine learning model for prediction and obtain the analysis results of the subscribed event.

[0123] In one embodiment, the scheme acquisition module 504 is used for:

[0124] Based on the model information indicated by the subscription event, obtain the machine learning model indicated by the model information from the model training module;

[0125] According to the model information, the data preprocessing scheme of the machine learning model is obtained from the model training module.

[0126] In one embodiment, the model training module performs the training process of the machine learning model, and the model training module caches the data preprocessing steps of the training process as the data preprocessing scheme.

[0127] In one embodiment, the data preprocessing scheme includes at least one of a data filtering scheme, a format conversion scheme, and a data normalization scheme; the feature conversion module 506 is used for:

[0128] The raw data is filtered according to the data filtering scheme to obtain feature data that matches the content of the machine learning model; and / or,

[0129] The original data is converted according to the format conversion scheme to obtain feature data that matches the format of the machine learning model; and / or,

[0130] The original data is processed according to the data normalization scheme to obtain feature data that matches the scale of the machine learning model.

[0131] In one embodiment, the data acquisition module 502 is configured to:

[0132] Retrieve different subscribed events and obtain raw data from data source network elements;

[0133] The original data corresponds to the different subscription events.

[0134] In one embodiment, the raw data and the training data of the machine learning model belong to the same data category, which is divided based on core network elements and is determined according to the subscription event.

[0135] In one embodiment, the data preprocessing scheme is determined based on the processing efficiency and metrics of the machine learning model.

[0136] Each module in the aforementioned data analysis device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0137] In one embodiment, a computer device is provided, which may be a network data analysis element, and its internal structure diagram may be as follows: Figure 6As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores machine learning models and their data preprocessing schemes. The I / O interfaces facilitate information exchange between the processor and external devices. The communication interface allows communication with external terminals via a network connection. When executed by the processor, the computer program implements a data analysis method.

[0138] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0139] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0140] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0141] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0142] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0143] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0144] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0145] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method of data analysis, characterized by, The method is applied to a network data analysis network element, and comprises the following steps: Obtaining a subscription event and original data; According to model information indicated by the subscription event, a machine learning model indicated by the model information is obtained from a model training module; according to the model information, a data preprocessing scheme of the machine learning model is obtained from the model training module; a training process of the machine learning model is executed by the model training module, and the model training module caches a data preprocessing step of the training process as the data preprocessing scheme; The original data is converted according to the data preprocessing scheme to obtain feature data; The feature data is input into the machine learning model for prediction to obtain an analysis result of the subscription event.

2. The method of claim 1, wherein, The data preprocessing scheme comprises at least one of a data screening scheme, a format conversion scheme and a data normalization scheme; The original data is converted according to the data preprocessing scheme to obtain feature data, which comprises the following steps: The original data is content-screened according to the data screening scheme to obtain feature data matched in content with the machine learning model; And / or, The original data is format-converted according to the format conversion scheme to obtain feature data matched in format with the machine learning model; and / or, The original data is data-normalized according to the data normalization scheme to obtain feature data matched in scale with the machine learning model.

3. The method of claim 1, wherein, The original data is converted according to the data preprocessing scheme to obtain feature data, which comprises the following steps: Different subscription events are obtained, and original data is obtained from a data source network element; The original data corresponds to the different subscription events.

4. The method of claim 1, wherein, The original data and training data of the machine learning model belong to the same data category, the data category is divided based on a core network element, and the data category is determined according to the subscription event.

5. The method of claim 1, wherein, The data preprocessing scheme is determined based on processing efficiency and indicators of the machine learning model.

6. The method of claim 1, wherein, The original data is to-be-converted input data of a machine learning model indicated by a subscription event.

7. The method of claim 1, wherein, The data preprocessing step comprises at least one of a data screening step, a format conversion step and a data normalization step; Through the data screening step, original data conforming to the content of the machine learning model is screened out; Through the format conversion step, the original data is converted into a format possessed by input data of the machine learning model; Through the data normalization step, the scale specification of the original data is adjusted.

8. A data analysis device, characterized by, The device is applied to a network data analysis network element, and comprises the following steps: A data acquisition module is configured to obtain a subscription event and original data; A scheme acquisition module is configured to obtain a machine learning model indicated by model information from a model training module according to the model information indicated by the subscription event; obtain a data preprocessing scheme of the machine learning model from the model training module according to the model information; and execute a training process of the machine learning model by the model training module, wherein the model training module caches a data preprocessing step of the training process as the data preprocessing scheme; a feature conversion module, configured to convert the original data according to the data preprocessing scheme to obtain feature data; an analysis module, configured to input the feature data into the machine learning model to perform prediction and obtain an analysis result of the subscription event. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-8 when the computer program is executed by the processor. The processor executes the computer program to implement the steps of the method in any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 7. The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 7.

Images