Data analysis method and apparatus, computer device, and storage medium

Abstract

The present application relates to a data analysis method and apparatus, a computer device, a storage medium, and a computer program product. The method comprises: acquiring a subscription event and raw data; on the basis of the subscription event, acquiring a machine learning model and a data preprocessing scheme of the machine learning model; according to the data preprocessing scheme, performing conversion on the raw data, and obtaining feature data; and inputting the feature data into the machine learning model for prediction, and obtaining an analysis result of the subscription event. By using the method, conversion can be performed by means of the preprocessing scheme indicated by the subscription event, such that a copy of the raw data can be converted into input data adapted to a plurality of models to undergo inference analysis, thereby achieving multiplexing of the raw data. In this way, in the process of performing inference for each subscription event, the frequency of acquiring raw data is relatively low, and thus less interaction stress is imposed on devices such as a data source network element.

Description

Data analysis method, device, computer equipment and storage medium

[0001] Related applications

[0002] This application claims priority to Chinese patent application number 202310875672.6, filed on July 17, 2023, entitled “Data Analysis Method, Device, Computer Equipment and Storage Medium,” the entire text of which is hereby incorporated by reference. Technical Field

[0003] The present application relates to the field of communications, and in particular to a data analysis method, apparatus, computer equipment, storage medium, and computer program product. Background Art

[0004] With the continuous development of mobile communication networks, the data throughput process of core network elements needs to be implemented based on the Network Data Analytics Function (NWDAF).

[0005] The network data analysis NE collects data from the data source NE based on subscription events, uses machine learning models to perform inference analysis on the collected data, and pushes the inference and analysis results to the core network NE, which serves as the consumer. If the network data analysis NE collects data from the data source NE and other devices for every subscription event, it will place excessive interaction pressure on the data source NE and other devices.

[0006] Summary of the Invention

[0007] Based on this, it is necessary to provide a data analysis method, device, computer equipment, computer-readable storage medium and computer program product that can alleviate the interaction pressure of data source network elements and other devices in response to the above technical problems.

[0008] In a first aspect, the present application provides a data analysis method, comprising:

[0009] Get subscription events and raw data;

[0010] Acquire a machine learning model and a data preprocessing solution for the machine learning model according to the subscription event;

[0011] Converting the raw data according to the data preprocessing scheme to obtain feature data;

[0012] The feature data is input into the machine learning model for prediction to obtain an analysis result of the subscription event.

[0013] In one embodiment, obtaining a machine learning model and a data preprocessing solution for the machine learning model according to the subscription event includes:

[0014] According to the model information indicated by the subscription event, obtaining the machine learning model indicated by the model information from the model training module;

[0015] According to the model information, a data preprocessing solution for the machine learning model is obtained from the model training module.

[0016] In one embodiment, the training process of the machine learning model is performed by the model training module, and the model training module caches the data preprocessing steps of the training process as the data preprocessing scheme.

[0017] In one embodiment, the data preprocessing solution includes at least one of a data screening solution, a format conversion solution, and a data regularization solution.

[0018] In one embodiment, converting the raw data according to the data preprocessing scheme to obtain feature data includes:

[0019] The original data is content-filtered according to the data filtering scheme to obtain feature data that matches the content of the machine learning model.

[0020] In one embodiment, converting the raw data according to the data preprocessing scheme to obtain feature data includes:

[0021] The original data is format converted according to the format conversion scheme to obtain feature data that matches the format of the machine learning model.

[0022] In one embodiment, converting the raw data according to the data preprocessing scheme to obtain feature data includes:

[0023] The raw data is processed according to the data regularization scheme to obtain feature data that matches the scale of the machine learning model.

[0024] In one embodiment, content screening of the original data according to the data screening scheme to obtain feature data that matches the content of the machine learning model includes:

[0025] The original data is filtered according to the fields indicated by the data filtering scheme to obtain feature data that meets the field content conditions of the machine learning model.

[0026] In one embodiment, converting the format of the original data according to the format conversion scheme to obtain feature data that matches the format of the machine learning model includes:

[0027] The original data is formatted according to the input data field format of the machine learning model to obtain feature data that meets the field format conditions of the machine learning model.

[0028] In one embodiment, performing data regularization on the raw data according to the data regularization scheme to obtain feature data that matches the scale of the machine learning model includes:

[0029] The original data is scaled according to the data regularization scheme to obtain feature data that meets the scale conditions of the machine learning model.

[0030] In one embodiment, obtaining subscription events and raw data includes:

[0031] Get different subscription events and obtain raw data from data source network elements;

[0032] The original data corresponds to the different subscription events.

[0033] In one embodiment, the original data and the training data of the machine learning model belong to the same data category, the data category is divided based on core network elements, and the data category is determined according to the subscription event.

[0034] In one embodiment, the data preprocessing scheme is determined based on the processing efficiency and indicators of the machine learning model.

[0035] In a second aspect, the present application further provides a data analysis device, comprising:

[0036] Data acquisition module, used to obtain subscription events and raw data;

[0037] A solution acquisition module, configured to acquire a machine learning model and a data preprocessing solution for the machine learning model based on the subscription event;

[0038] A feature conversion module, configured to convert the raw data according to the data preprocessing scheme to obtain feature data;

[0039] An analysis module is used to input the feature data into the machine learning model for prediction to obtain an analysis result of the subscription event.

[0040] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the data analysis method in any of the above embodiments.

[0041] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the data analysis method in any of the above embodiments.

[0042] In a fifth aspect, the present application further provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of data analysis in any of the above embodiments.

[0043] In the aforementioned data analysis methods, apparatuses, computer devices, storage media, and computer program products, subscription events and raw data do not correspond one-to-one. To ensure that the raw data obtained each time can be used by one or more subscription events, the data is converted using a preprocessing scheme indicated by the subscription event. This allows a single piece of raw data to be converted into input data suitable for multiple models, allowing for inference analysis and reuse of the raw data. Consequently, the frequency of raw data acquisition during inference for each subscription event is relatively low, placing less pressure on data source network elements and other devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technology descriptions. Obviously, the drawings described below are only some embodiments of this application. For those of ordinary skill in the art, other drawings can be derived from these drawings without inventive effort.

[0045] FIG1 is a diagram showing an application environment of a data analysis method in an embodiment of the present application.

[0046] FIG2 is a flow chart of a data analysis method in one embodiment of the present application.

[0047] FIG3 is a flow chart of a data analysis method in another embodiment of the present application.

[0048] FIG4 is a flow chart of a data analysis method in another embodiment of the present application.

[0049] FIG5 is a structural block diagram of a data analysis device in an embodiment of the present application.

[0050] FIG6 is a diagram showing the internal structure of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0052] The data analysis method provided in the embodiments of the present application can be applied, for example, to the application environment shown in FIG1 . In this example application environment, the network data analysis network element 104 collects raw data from the stored data of the data source network element 102 based on the subscription events obtained by subscription, and performs data analysis to generate corresponding analysis results for the subscription events. The analysis results can be fed back to the consumer 106 of the network data analysis network element, or fed back to other devices other than the consumer 106. The network data analysis network element 104 and the data source network element 102 are both core network elements used to serve the consumer 106.

[0053] In some embodiments, the consumer 106 includes a core network element. The data categories of the data handled by the core network element of the consumer 106 can be divided into major categories such as performance, alarm, configuration, and log according to the type of device network management, or can be further divided into categories such as business, traffic, and signaling according to the type of network application data; and the raw data matches the data category of the data handled by the consumer 106. The network data analysis element 104 obtains the raw data from the data source element 102, and obtains analysis results of the corresponding subscription events by preprocessing and reasoning analysis of the raw data. Each core network element can be a specific hardware device or a software module used to implement a certain function, and each core network element can belong to the 5G field or even the 6G field.

[0054] In one embodiment of the present application, as shown in FIG2 , a data analysis method is provided. The method can be applied to the network data analysis network element 104 in FIG1 , including the following steps:

[0055] Step S202: Obtain subscription events and original data.

[0056] Subscription events are events subscribed to and received by the network data analysis network element 104. Subscription events are sent for data categories related to machine learning models and are used to select a machine learning model to perform predictive analysis on the data in that data category to obtain the analysis results of the machine learning model; the analysis results of the machine learning model are the analysis results of the subscription events.

[0057] In one embodiment, the subscribed events may be related to the machine learning model and may be notifications of performance issues, network error alerts, configuration security threats, log anomalies, or other types of notifications. For example, the subscribed events may be request failure events, database timeout events, or connection rejection events from a network element.

[0058] Raw data is the input data to be transformed by the machine learning model indicated by the subscription event. There is no one-to-one correspondence between raw data and subscription events, nor is there a one-to-one correspondence between raw data and machine learning models. In one embodiment, raw data is data that can be processed by consumer 106, such as input data or output data of consumer 106.

[0059] In one embodiment, the input data to be converted is data that can be input into a machine learning model. Specifically, when the data can be directly input into the machine learning model, the data is raw data; when the data is preprocessed and the preprocessed data can be directly input into the machine learning model, the data is considered input data to be converted.

[0060] In one embodiment, the event categories to which the network element 104 subscribes can be analyzed according to network data to obtain subscription events and original data. The event category can be the data category consumed by the consumer, or it can be a category obtained by further refining the data category with different processing methods.

[0061] In one embodiment, the steps of obtaining subscription events and raw data may include: subscribing to at least one of the data source network element and the consumer device; if a subscription event is obtained, collecting raw data from the data source network element; wherein the consumer is a core network element that processes different data. Thus, the network data analysis network element 104 can obtain subscription events from different devices, and the process of obtaining subscription events is a condition for obtaining raw data, thereby adaptively adjusting the interaction pressure of devices such as the data source network element.

[0062] Step S204: Obtain a machine learning model and a data preprocessing solution for the machine learning model based on the subscription event.

[0063] A machine learning model (ML) is a pre-trained model. A corresponding relationship exists between the ML model and the subscription event. This relationship is used to indicate the ML model and the execution process of acquiring the ML model. In one embodiment, the ML model is acquired by calling the network data analysis network element 104.

[0064] A data preprocessing solution is defined based on the input data of a machine learning model. It is used to transform the raw data for the machine learning model so that the resulting feature data matches the input data of the machine learning model. In one embodiment, the data preprocessing solution involves data preprocessing in terms of content, format, and scaling.

[0065] In one embodiment, based on a subscription event, the steps of obtaining a machine learning model and a data preprocessing scheme for the machine learning model include: obtaining a trained machine learning model indicated by the subscription event, and obtaining the data preprocessing scheme indicated by the subscription event; wherein the trained machine learning model and the data preprocessing scheme are one-to-one corresponding.

[0066] Step S206: convert the original data according to the data preprocessing solution to obtain feature data.

[0067] The feature data matches the input data of the machine learning model, and the content of the feature data is consistent with the content of the original data. The feature data is candidate input data for the machine learning model indicated by the subscription event. There is no one-to-one correspondence between feature data and subscription events; a set of feature data corresponds to a machine learning model, and a machine learning model can correspond to different sets of feature data. In one embodiment, the entire content of the feature data is identical to at least part of the content of the original data.

[0068] In one embodiment, the step of converting the raw data according to the data preprocessing scheme includes: determining a data preprocessing sub-scheme for the raw data based on the raw data and the input data requirements of the machine learning model; and converting the raw data using the data preprocessing sub-scheme, wherein the input data requirements include content, format, and / or scaling requirements. Thus, based on the raw data, the data preprocessing sub-scheme is determined from the data preprocessing scheme to select a data preprocessing sub-scheme that is adaptable to different raw data, thereby ensuring processing efficiency.

[0069] In one embodiment, the data preprocessing sub-scheme may further include: a preprocessing sub-scheme for integrity and quality, which is used to find missing values, erroneous values, or outliers in the original data, and to perform data cleaning on the found missing values, erroneous values, or outliers to improve processing efficiency. In one embodiment, the data preprocessing sub-scheme may further include: a preprocessing sub-scheme for feature sparsity and correlation, which adjusts the data distribution to ensure a balance between the accuracy of feature data classification, thereby ensuring that the sparsity of the feature data and the correlation within the feature data are within a moderate range, avoiding overfitting, and improving the accuracy of the analysis results.

[0070] Step S208: Input the feature data into the machine learning model for prediction to obtain the analysis result of the subscription event.

[0071] Specifically, the analysis results include the predicted values ​​derived from the machine learning model's predictions of the feature data. The analysis results can also be provided as analysis reports for subscription events, which can be accessed by consumers who request them. For example, if consumers belong to different core network elements, the analysis results can be used to conduct in-depth analysis of network performance and efficiency, thereby improving the core network element's network performance and bandwidth utilization. Furthermore, the analysis results can be used for network fault detection, security monitoring, and other aspects.

[0072] In one embodiment, feature data is input into a machine learning model, which then performs inference analysis on the feature data using the inference analysis method developed through machine learning model training to obtain a predicted value for the feature data; this predicted value can serve as the analysis result for the subscription event. In one embodiment, the analysis result for the subscription event can also include a graph or table plotted using the predicted value, the credibility of the predicted value, and other information to reduce the probability of the subscription event being triggered multiple times, thereby reducing the interaction pressure on devices such as data source network elements.

[0073] In the above-mentioned data analysis method, there is not a one-to-one correspondence between subscription events and raw data. In order to make the raw data obtained each time available for one or more subscription events, it is converted through the preprocessing scheme indicated by the subscription event, so that a set of raw data can be converted into input data adapted to multiple models, and reasoning analysis is performed to achieve the reuse of raw data. Therefore, in the process of reasoning for each subscription event, the frequency of obtaining raw data is relatively low, and the interaction pressure on devices such as data source network elements is relatively low. Therefore, the frequency of raw data collection is low, and the storage space of the network data analysis network element will not be repeatedly occupied, which can avoid the problem of storage space waste. In addition, the network data analysis network element can obtain raw data from data source network elements of different manufacturers. The network data analysis network element can achieve the standardization of the input data of the machine learning model through the data preprocessing scheme, thereby expanding the selection range of raw data collection.

[0074] In one embodiment, the step of obtaining subscription events and raw data may include: obtaining different subscription events, and obtaining raw data from a data source network element; wherein the raw data corresponds to different subscription events.

[0075] Different subscription events are used to indicate different machine learning models and their respective data preprocessing solutions. Different subscription events can be obtained by the network data analysis network element for different core network elements, and can also be different events subscribed by the network data analysis network for the same core network element.

[0076] In an optional embodiment, the steps of obtaining different subscription events and obtaining original data from the data source network element include: obtaining original data from the data source network element according to the data category indicated by the different subscription events, the data category to which the training data of the model belongs, or the data category of the core network network element.

[0077] Specifically, for different subscription events obtained by the network data analysis network element, a set of original data can be obtained separately; steps S204-S208 are executed respectively through this set of original data and different subscription events, so that the frequency of the network data analysis network element obtaining data from the data source network element is reduced, thereby alleviating the interaction pressure of the data source network element.

[0078] In one embodiment, in order to more accurately illustrate the relevant elements of the original data, the original data and the training data of the machine learning model belong to the same data category, the data category is divided based on the core network element, and the data category is determined based on the subscription event.

[0079] Training data is the data used during the training of a machine learning model; it includes data such as training samples and training parameters. The data categories to which the training data of a machine learning model belongs are the data categories that the machine learning model can predict. This embodiment does not involve the application of training data; instead, it directly compares the data categories of the training data with those of the original data to obtain the corresponding original data.

[0080] The original data and the training data are not necessarily the same, but they belong to the same data category, so that the feature data converted by the data preprocessing scheme can be directly input into the machine learning model for prediction to obtain the corresponding analysis results.

[0081] In one embodiment, data categories are divided based on core network elements, and are divided into major categories such as performance, alarm, configuration, and log according to the device network management type. They can also be further divided into categories such as business, traffic, and signaling according to the network application data type.

[0082] In one embodiment, the feature data converted by the data preprocessing scheme belong to the same data category, so the data preprocessing scheme does not perform analysis across different data categories, ensuring that this embodiment can be performed accurately.

[0083] In one embodiment, if the original data d belongs to Class A traffic data, and the A machine learning model is trained using Class A traffic data, then the original data d is converted using the data preprocessing solution of the A machine learning model and then input into the A machine learning model for inference analysis.

[0084] Based on this, the raw data and the training data for the machine learning model belong to the same data category, allowing for flexible selection of raw data within the data category, increasing the probability of successful inference analysis. Data categories are divided based on core network elements, enabling adaptive adjustments to the data throughput of different core network element devices to ensure the stable operation of this embodiment. Data categories are determined based on subscription events, allowing for the acquisition of raw data based on the data category.

[0085] In one embodiment, the steps of obtaining a machine learning model and a data preprocessing solution for the machine learning model based on a subscription event may include: obtaining the machine learning model indicated by the model information from a model training module based on the model information indicated by the subscription event; and obtaining the data preprocessing solution for the machine learning model from the model training module based on the model information.

[0086] Model information is information indicated by a subscription event and is used to identify a specific machine learning model and its data preprocessing solution. In one embodiment, model information is defined based on a data category and is used to identify a specific machine learning model for that data category and its data preprocessing solution.

[0087] The model training module is used to train the machine learning model, and to cache the trained machine learning model and the data preprocessing solution of the machine learning model. The machine learning model and the data preprocessing solution are stored through the model training module. Using the model training module as the data source for the machine learning model and the data preprocessing solution not only alleviates the interaction pressure of the data source network element, but also makes the model training and storage processes almost integrated, so that the machine learning model and the data preprocessing solution can be stored more efficiently. In addition, the model training module can be a model training module (Model Training Logical Function, MTLN) in the network data analysis network element. At this time, the network data analysis network element only needs to obtain subscription events and a set of raw data to perform the data analysis process by itself. It has low dependence on external network elements and will not bring interaction pressure to other core network elements.

[0088] In one embodiment, the step of obtaining the machine learning model indicated by the model information from the model training module according to the model information indicated by the subscription event may include: the logic analysis module of the network data analysis network element obtains the machine learning model indicated by the model identifier from the model training module of the network data analysis network element according to the model identifier indicated by the subscription event.

[0089] Accordingly, the step of obtaining a data preprocessing solution for the machine learning model from the model training module based on the model information includes: the logic analysis module of the network data analysis network element obtains the data preprocessing solution indicated by the model identifier from the model training module of the network data analysis network element based on the model identifier. Thus, the machine learning model and the data preprocessing solution are obtained in parallel based on a single model identifier, thereby improving data processing efficiency.

[0090] The machine learning model and the data preprocessing scheme for the machine learning model are both pre-stored in the model training module and are called by the network data analysis network element according to the model information indicated by the subscription event. In this way, not only the data input into the machine learning model based on the data preprocessing scheme is standardized, but also the range of data source selection is expanded; and from an engineering perspective, the changes to the network element data transmission are relatively small.

[0091] Among them, the training process of the machine learning model is executed by the model training module, and the model training module caches the data preprocessing steps of the training process as a data preprocessing solution.

[0092] The training process is a set of steps used by the model training module to train the machine learning model based on the training data. This set of steps includes at least two subsets: data preprocessing and training execution. Each subset includes one or more specific solution steps. The data preprocessing step precedes the training execution step and is used to filter, convert, regularize, and / or perform other transformations on the training data to make it suitable for training the machine learning model. The training execution step is the step in which the model training module trains the machine learning model based on the data obtained from the conversion of the training data.

[0093] In one embodiment, the data preprocessing step includes the algorithm engineer performing screening, conversion, regularization, and / or other operations on the training data. The operations involved in the data preprocessing step are cached by the model training module as a data preprocessing solution. Thus, the manual preprocessing steps are cached to efficiently construct a data preprocessing solution, which facilitates the reuse of manual preprocessing steps and enables the feature data obtained from the raw data conversion to better serve as input data for the machine learning model.

[0094] During the training process of the machine learning model, the data preprocessing steps of the training process are cached as a data preprocessing scheme. The data preprocessing scheme can be configured, improved or adjusted through the model training process, reducing the number of times the data preprocessing scheme is created, making the data reuse efficiency higher, and the same set of original data can be differentiated into feature data of more machine learning models for more subscription events, thereby further reducing the interaction pressure of the data source network element and saving the resource storage space of the network data analysis network element.

[0095] In an exemplary embodiment, the network data analysis network element is provided with an Analytics Logical Function (AnLF), which performs steps S202-S208. The network data analysis network element is provided with a Model Training Logical Function (MTLF), which performs the training process of the machine learning model (ML model), the generation process of the data preprocessing solution, and the caching process of the model and the data preprocessing solution.

[0096] In an exemplary embodiment, prior to step S202, the data analysis method further includes: a model training module performing model training, storing the machine learning model file, and caching the data preprocessing steps performed by the algorithm engineer during the training process. In one embodiment, step S202 may further include: whenever an analysis report for a subscription event is required, the network data analysis network element collects raw data from the data source network element.

[0097] In step S204, the step of obtaining the machine learning model according to the subscription event may include: the AnLF module requests the ML model from the MTLF module according to the subscription event subscribed by the user.

[0098] In step S204 , the process of obtaining the data preprocessing solution may include: the AnLF module requests the MTLF module for the data preprocessing solution of the ML model.

[0099] Step S206 corresponds to step S208: AnLF processes the collected raw data according to the data preprocessing scheme to obtain feature data, and inputs the feature data into the ML model for inference and prediction.

[0100] After step S208 , the data analysis method may further include: NWDAF sending corresponding subscribed reasoning reports to users respectively.

[0101] In another exemplary embodiment, as shown in FIG3 , when the logic analysis module of the network data analysis network element obtains two subscription events a and b and provides analysis reports for the two subscription events a and b, the data analysis method may include:

[0102] Before step S202, step 0 is also included, in which the model training module performs model training, stores the machine learning model file, and caches the data preprocessing steps of the algorithm engineer during the training process.

[0103] In this embodiment, step S202 may include step 1, specifically: whenever an event report needs to be provided, the network data analysis network element collects data D1 from the data source network element, so that the logic analysis module in the network data analysis network element obtains the subscription event and original data.

[0104] In this embodiment, the step of obtaining the machine learning model according to the subscription event in step S204 may include: step 2a. The logic analysis module requests the machine learning model a from the model training module according to the subscription event a subscribed by the user; and step 2b. The model training module requests the machine learning model b from the model training module.

[0105] In step S204 , the step of obtaining the data preprocessing solution may include: step 3a. The AnLF module requests the data preprocessing solution a of the ML model a from the MTLF module; and step 3b. The AnLF module requests the data preprocessing solution b of the ML model b.

[0106] Step S206 corresponds to step S208: Step 4a. AnLF processes the collected raw data D1 according to data preprocessing scheme a to obtain feature data a1, inputs the feature data a1 into ML model a for inference and prediction, and obtains the analysis result of subscription event a; and Step 4b. AnLF processes the data D1 using scheme b to obtain feature data b1, inputs the feature data b1 into ML model b for inference and prediction, and obtains the analysis result of subscription event b.

[0107] In one embodiment, the data preprocessing solution includes at least one of a data screening solution, a format conversion solution, and a data regularization solution. The data screening solution, the format conversion solution, and the data regularization solution are different data preprocessing sub-solutions, each used to perform different conversions on the raw data. The data screening solution is used to filter out raw data that meets the requirements of the machine learning model; the format conversion solution is used to convert the raw data into the format required by the machine learning model input data; and the data regularization solution is used to adjust the scale of the raw data to an appropriate specification.

[0108] In one embodiment, the input data requirements of the machine learning model include requirements for content, format, and scale. If any of the content, format, and scale of the original data do not meet the requirements, the original data is converted according to a scheme that does not conform to the requirements, so that the original data is converted into feature data. For example: if the content of the original data covers the content required by the machine learning model, and the content data volume of the original data is greater than the content data volume threshold required by the machine learning model, the original data needs to be content screened according to the data screening scheme to obtain feature data that matches the content data volume of the machine learning model. It can be understood that the content aspect can not only be a limitation on the amount of data, but also a limitation on certain fields or a limitation with certain identifiers.

[0109] In one embodiment, the step of converting the raw data according to the data preprocessing solution to obtain feature data may include:

[0110] Perform content screening on the original data according to the data screening scheme to obtain feature data that matches the content of the machine learning model; and / or,

[0111] Convert the original data into a format according to the format conversion scheme to obtain feature data that matches the format of the machine learning model; and / or,

[0112] The original data is processed according to the data regularization scheme to obtain feature data that matches the scale of the machine learning model.

[0113] In one embodiment, the step of content screening the original data according to the data screening scheme to obtain feature data that matches the content of the machine learning model may include: screening the original data according to the fields indicated by the data screening scheme to obtain feature data that meets the content conditions of the machine learning model fields. Optionally, for different machine learning models, adjustments are made based on content factors such as business data, variable correlation, feature importance, feature fitting, encoding method, etc. to improve the quality of the machine learning model, which can be in terms of processing efficiency, accuracy, and robustness. After the model is adjusted, the input fields required by different machine learning models will also be adjusted according to the content factors, and the adjustment process of these input fields will efficiently and synchronously adjust the data screening scheme.

[0114] In one embodiment, the step of converting the format of the raw data according to the format conversion scheme to obtain feature data that matches the format of the machine learning model may include: converting the format of the raw data according to the input data field format of the machine learning model to obtain feature data that meets the field format conditions of the machine learning model. Optionally, the step of converting the format of the raw data may include dimensions such as one-hot encoding, label encoding, and EMBEDDING, which are used to convert the data type in the raw data into a numeric type, wherein the format of the numeric type includes integer, long integer, or floating point format, so that the machine learning model can efficiently use the feature data that meets the field format conditions.

[0115] In one embodiment, performing data regularization on the raw data according to the data regularization scheme to obtain feature data that matches the scale of the machine learning model may include: scaling the raw data according to the data regularization scheme to obtain feature data that meets the scale conditions of the machine learning model. Optionally, scaling involves normalizing, centering, or standardizing the raw data to enable the machine learning model to more efficiently generate analysis reports for subscription events.

[0116] In one embodiment, the data screening scheme, format conversion scheme, and data regularization scheme are executed sequentially. The result of the conversion of the current preprocessing sub-scheme is input into the next preprocessing sub-scheme of the current preprocessing sub-scheme to continue the conversion process until all preprocessing sub-schemes existing in the original data have been executed. Optionally, any number of preprocessing sub-schemes can be selected for execution from the data screening scheme, format conversion scheme, and data regularization scheme.

[0117] In one embodiment, the original data is processed in sequence according to a data screening scheme, a format conversion scheme, and a data regularization scheme, including: performing content screening on the original data according to the data screening scheme to obtain first feature data that matches the content of the machine learning model; performing format conversion on the first feature data according to the format conversion scheme to obtain second feature data that matches the format of the machine learning model; performing data regularization on the second feature data according to the data regularization scheme to obtain third feature data that matches the scale of the machine learning model; and determining the third feature data as feature data for generating an analysis report.

[0118] In an exemplary embodiment, as shown in FIG4 , the data preprocessing scheme has three data preprocessing sub-schemes, namely, a data screening scheme, a data conversion scheme, and a data regularization scheme, corresponding to the data screening module, the data conversion module, and the data regularization module in the processing module. The data screening module is used to select data that meets the content requirements of the machine learning model field; the data conversion module is used to convert the format of the field, such as converting a string into an integer and a floating point number; and the data regularization module is used to implement scaling operations such as normalization, standardization, and centralization of the data. In this embodiment, the network data analysis network element only needs to collect data from the data source network element once. After the data is converted by the data processing module, feature data a, b, and c are formed. The feature data a, b, and c are then input into the machine learning models corresponding to the feature data a, b, and c, respectively, to obtain analysis reports of the subscription events A, B, and C, thereby completing the analysis of the subscription events, and sending the analysis report obtained by reasoning at the subscription time to the user's terminal.

[0119] Optionally, the data preprocessing scheme is determined based on the processing efficiency and indicators of the machine learning model.

[0120] The configuration of a data preprocessing solution is designed to match the input data of the machine learning model to which it belongs. Because machine learning models can be modified based on algorithmic efficiency and performance requirements, the data preprocessing solution can also be modified accordingly, allowing for adaptive feature engineering such as data screening to improve the accuracy and recall of analysis results.

[0121] Optionally, if during the training process of the machine learning model, the data prediction processing steps of the training process are cached as a data preprocessing scheme, the adjustment efficiency of the data preprocessing scheme can be further improved.

[0122] In one embodiment, the data analysis method of the present application belongs to a data processing method of a network data analysis network element, a cache module of a data preprocessing scheme is added to the model training module, a data processing module is added to the logic analysis module, and the data preprocessing scheme is cached and scheduled.

[0123] The standard specifications for network data analysis elements specify the input data for each subscription event. These elements collect data in real time for inference and prediction. Using data preprocessing solutions, simulation algorithm engineers filter, transform, and normalize the raw data, converting it into the feature data required by machine learning models for inference and analysis.

[0124] In addition, in the application scenario where the data preprocessing solution collects raw data from the data source network element based on different subscription events, the problem that the more subscription events, the more raw data needs to be collected is analyzed: repeated data collection may cause greater interaction pressure on the data source network element; and large amounts of data collection will waste resources for data storage. The data analysis method provided in this application can provide a set of raw data collected by the network element to multiple subscription events that use this data, thereby achieving data reuse.

[0125] In another specific embodiment, the probability of the network data analysis element collecting the same raw data multiple times is low, which is less likely to waste resources, thereby solving the problem of the network data analysis element providing the same raw data to different machine learning models.

[0126] Specifically, the network data analysis element can reuse collected data. This avoids repeated data collection, alleviating the problem of providing the same data for different subscription events. Furthermore, by applying different data preprocessing solutions to the same set of raw data, it can be converted into multiple sets of distinct feature data to feed into machine learning models for different subscription events, saving storage resources.

[0127] Furthermore, the network data analysis element standardizes its own input data, expanding the range of data sources and reducing engineering modifications to the output data. Furthermore, the network data analysis element can configure data processing solutions based on the efficiency and metrics of the machine learning model, performing feature engineering such as data screening to improve the accuracy and recall of analysis results.

[0128] It's understandable that with the continuous development of communication networks, network data analysis elements are accepting an increasing number of subscription events. Differentiating the same piece of data based on different subscription events can reduce the interaction pressure on data source elements and save storage resources. Network data analysis elements can collect data from network elements from different manufacturers. This standardizes input data while also expanding the scope of data collection.

[0129] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0130] Based on the same inventive concept, the present application also provides a data analysis device for implementing the aforementioned data analysis method. The solution to the problem provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more data analysis device embodiments provided below can be found in the above-mentioned limitations of the data analysis method and will not be repeated here.

[0131] In one embodiment, as shown in FIG5 , the present application further provides a data analysis device that can be applied to a network data analysis network element, the device comprising:

[0132] Data acquisition module 502, used to obtain subscription events and original data;

[0133] A solution acquisition module 504 is configured to acquire a machine learning model and a data preprocessing solution for the machine learning model based on the subscription event;

[0134] A feature conversion module 506 is used to convert the raw data according to the data preprocessing scheme to obtain feature data;

[0135] The analysis module 508 is used to input the feature data into the machine learning model for prediction to obtain the analysis result of the subscription event.

[0136] In one embodiment, the solution acquisition module 504 is used to:

[0137] According to the model information indicated by the subscription event, obtaining the machine learning model indicated by the model information from the model training module;

[0138] According to the model information, a data preprocessing solution for the machine learning model is obtained from the model training module.

[0139] In one embodiment, the training process of the machine learning model is performed by the model training module, and the model training module caches the data preprocessing steps of the training process as the data preprocessing scheme.

[0140] In one embodiment, the data preprocessing solution includes at least one of a data screening solution, a format conversion solution, and a data regularization solution; the feature conversion module 506 is used to:

[0141] Performing content screening on the original data according to the data screening scheme to obtain feature data that matches the content of the machine learning model; and / or,

[0142] Performing format conversion on the original data according to the format conversion scheme to obtain feature data that matches the format of the machine learning model; and / or,

[0143] The raw data is processed according to the data regularization scheme to obtain feature data that matches the scale of the machine learning model.

[0144] In one embodiment, the data acquisition module 502 is used to:

[0145] Get different subscription events and obtain raw data from data source network elements;

[0146] The original data corresponds to the different subscription events.

[0147] In one embodiment, the original data and the training data of the machine learning model belong to the same data category, the data category is divided based on the core network element, and the data category is determined according to the subscription event.

[0148] In one embodiment, the data preprocessing scheme is determined based on the processing efficiency and indicators of the machine learning model.

[0149] Each module in the above-mentioned data analysis device can be implemented in whole or in part through software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor of the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each of the above modules.

[0150] In one embodiment of the present application, a computer device is also provided. The computer device can be a network data analysis network element, and its internal structure diagram can be shown in Figure 6. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store a machine learning model and a data preprocessing scheme for the machine learning model. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data analysis method is implemented.

[0151] Those skilled in the art will understand that the structure shown in FIG6 is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0152] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0153] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0154] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0155] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0156] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0157] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0158] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A data analysis method, comprising: Get subscription events and raw data; According to the subscription event, obtain a machine learning model and a data preprocessing solution for the machine learning model; Converting the raw data according to the data preprocessing scheme to obtain feature data; The feature data is input into the machine learning model for prediction to obtain the analysis result of the subscription event.

2. The method according to claim 1, wherein obtaining the machine learning model and the data preprocessing scheme of the machine learning model according to the subscription event comprises: According to the model information indicated by the subscription event, obtaining the machine learning model indicated by the model information from the model training module; According to the model information, a data preprocessing solution for the machine learning model is obtained from the model training module.

3. The method according to claim 2, wherein the training process of the machine learning model is performed by the model training module, and the model training module caches the data preprocessing steps of the training process as the data preprocessing scheme.

4. The method according to claim 1, wherein the data preprocessing scheme comprises: At least one of a data screening solution, a format conversion solution, and a data regularization solution.

5. The method according to claim 4, wherein converting the raw data according to the data preprocessing scheme to obtain the characteristic data comprises: The original data is content-filtered according to the data filtering scheme to obtain feature data that matches the content of the machine learning model.

6. The method according to claim 4, wherein converting the raw data according to the data preprocessing scheme to obtain the characteristic data comprises: The original data is format converted according to the format conversion scheme to obtain feature data that matches the format of the machine learning model.

7. The method according to claim 4, wherein converting the raw data according to the data preprocessing scheme to obtain feature data comprises: The raw data is processed according to the data regularization scheme to obtain feature data that matches the scale of the machine learning model.

8. The method according to claim 5, wherein the original data is subjected to content screening according to the data screening scheme to obtain feature data matching the content of the machine learning model, comprising: The original data is filtered according to the fields indicated by the data filtering scheme to obtain feature data that meets the field content conditions of the machine learning model.

9. The method according to claim 6, wherein the format conversion of the original data is performed according to the format conversion scheme to obtain feature data matching the format of the machine learning model comprises: The original data is formatted according to the input data field format of the machine learning model to obtain feature data that meets the field format conditions of the machine learning model.

10. The method according to claim 7, wherein performing data regularization processing on the original data according to the data regularization scheme to obtain feature data matching the scale of the machine learning model comprises: The original data is scaled according to the data regularization scheme to obtain feature data that meets the scale conditions of the machine learning model.

11. The method according to claim 1, wherein obtaining subscription events and raw data comprises: Get different subscription events and get raw data from data source network elements; The original data corresponds to the different subscription events.

12. The method according to claim 1, wherein the original data and the training data of the machine learning model belong to the same data category, the data category is divided based on core network elements, and the data category is determined according to the subscription event.

13. The method according to claim 1, wherein the data preprocessing scheme is determined based on the processing efficiency and indicators of the machine learning model.

14. A data analysis device, comprising: Data acquisition module, used to obtain subscription events and raw data; A solution acquisition module, used to acquire a machine learning model and a data preprocessing solution for the machine learning model according to the subscription event; A feature conversion module, used to convert the original data according to the data preprocessing scheme to obtain feature data; The analysis module is used to input the feature data into the machine learning model for prediction to obtain the analysis result of the subscription event.

15. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein the processor implements the steps of the method according to any one of claims 1 to 13 when executing the computer program.

16. A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the method according to any one of claims 1 to 13 when executed by a processor.

Images