A data processing method for industrial data distillation in elastic interaction

Abstract

The present invention discloses a data processing method for industrial data distillation in a flexible interactive environment, comprising: obtaining a monitoring data stream through real-time monitoring in a dynamic data environment of flexible interaction; preprocessing the monitoring data stream to obtain preprocessed data; performing data identification on the preprocessed data to determine key data; processing the key data through knowledge distillation to determine target data; and visualizing and displaying the target data. In a dynamic data environment of flexible interaction, data is accurately and rapidly collected and processed, shortening processing time and improving the accuracy and effectiveness of data processing.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing method for performing industrial data distillation in elastic interaction. Background Art

[0002] Currently, the following technical issues exist when extracting and visualizing important data in a dynamic, interactive environment: 1. It is difficult to comprehensively and accurately collect data in a dynamic environment, which easily leads to low data collection rates and difficulty in data collection. 2. It is difficult to effectively process the collected data, making it difficult to accurately identify important data and visualize it. This results in a long processing time and poor data processing results. Summary of the Invention

[0003] The present invention aims to at least partially address one of the technical problems encountered in the aforementioned technologies. To this end, the present invention proposes a data processing method for industrial data distillation in a flexible interactive environment, accurately and rapidly collecting data in a dynamic data environment of flexible interaction, accurately processing the collected data, shortening processing time, and improving the accuracy and effectiveness of data processing.

[0004] To achieve the above objectives, an embodiment of the present invention proposes a data processing method for performing industrial data distillation in elastic interaction, comprising:

[0005] In a dynamic data environment with elastic interaction, monitoring data streams can be obtained through real-time monitoring;

[0006] Preprocessing the monitoring data stream to obtain preprocessed data;

[0007] Perform data identification on pre-processed data to determine key data;

[0008] Process key data through knowledge distillation to determine target data;

[0009] Visualize the target data and present it visually.

[0010] According to some embodiments of the present invention, in a dynamic data environment with elastic interaction, obtaining a monitoring data stream through real-time monitoring includes:

[0011] Determine the objects and indicators to be monitored in a dynamic data environment with flexible interactions;

[0012] Based on the objects and indicators to be monitored, determine the monitoring tools to be used and their key locations; wherein the monitoring tools include log analysis tools, performance monitoring software and network traffic monitors;

[0013] Real-time monitoring is carried out based on monitoring tools at corresponding key locations to obtain monitoring data streams.

[0014] According to some embodiments of the present invention, preprocessing the monitoring data stream to obtain preprocessed data includes:

[0015] The monitoring data stream is cleaned, denoised, converted, integrated and reduced to obtain preprocessed data.

[0016] According to some embodiments of the present invention, performing data denoising on a monitoring data stream includes: inputting the monitoring data stream into a pre-trained denoising model to perform data denoising.

[0017] According to some embodiments of the present invention, performing data identification on pre-processed data to determine key data includes:

[0018] Based on streaming processing technology, feature extraction is performed on pre-processed data to determine key features;

[0019] Select the corresponding data recognition algorithm based on key features;

[0020] Perform data identification on pre-processed data based on data identification algorithm to determine key data.

[0021] According to some embodiments of the present invention, performing data identification on pre-processed data to determine key data includes:

[0022] Input the pre-processed data into the pre-trained entity recognition model and output several entities;

[0023] Perform cluster analysis on several entities to obtain several entity sets;

[0024] Calculate the correlation coefficient between several entity sets;

[0025]

[0026] Where r is the correlation coefficient between entity set X and entity set Y; cov(X, Y) is the covariance between entity set X and entity set Y; σ X , σ y is the standard deviation of entity set X and entity set Y respectively; is the average value of entity set X and entity set Y; n is the smaller value of the number of entities included in entity set X and entity set Y respectively; X i is the i-th entity in the entity set X; Y i is the i-th entity in the entity set Y;

[0027] Construct a correlation diagram between several entity sets based on the correlation coefficient;

[0028] In the association diagram, the size of the correlation coefficient between each entity set and other entity sets and the preset correlation coefficient is determined, and the number of entity sets with a correlation coefficient greater than the preset correlation coefficient is determined; and the entity sets with a number greater than the preset number are screened out as key data.

[0029] According to some embodiments of the present invention, a method for obtaining an entity recognition model includes:

[0030] Acquire training data; the training data includes entity annotation results;

[0031] Input the training data into the semantic determination layer of the neural network model to determine the initial semantic vector of each semantic unit in the training data; match each semantic unit with the corresponding knowledge graph to determine the loaded semantic vector of each semantic vector; splice the initial semantic vector of each semantic vector with the loaded semantic vector to determine the spliced semantic vector, and construct a spliced semantic vector sequence; input the spliced semantic vector sequence into the semantic recognition layer of the neural network model, split the semantic structure of the spliced semantic vector sequence to obtain a number of units to be recognized; match the several units to be recognized with a preset entity vector library, and determine the entity recognition result based on the matching result;

[0032] The difference between the entity labeling result and the entity recognition result is calculated, and iterative training is performed according to the calculation result until the entity labeling result is determined to be consistent with the entity recognition result, thereby obtaining the initial entity recognition model;

[0033] Obtain test data and input it into the initial entity recognition model;

[0034] Determine the cross entropy loss function between the predicted entities and the real entities of the test data based on the initial entity recognition model;

[0035] Determine the degree of match between the predicted entities of the test data and the real entities based on the initial entity recognition model;

[0036] Determine the cosine distance between the vectors of the predicted entity and the true entity of the test data based on the initial entity recognition model;

[0037] When it is determined that the cross entropy loss function, the matching degree and the cosine distance are all less than the corresponding preset thresholds, it means that the initial entity recognition model is qualified and is used as the final trained entity recognition model.

[0038] According to some embodiments of the present invention, key data is processed by knowledge distillation to determine target data, including:

[0039] The key data is input into the encoder for compression encoding. The encoder projects the key data into the latent space, obtains the hidden vector in the latent space and performs numerical processing;

[0040] Sort the numerically processed hidden vectors from large to small to obtain a sorted queue;

[0041] Set the distillation ratio and select the data with the distillation ratio from the front of the sorting queue as the target data.

[0042] According to some embodiments of the present invention, visual processing and visual display of target data include: using data visualization technology to visually display the target data in the form of charts or reports.

[0043] According to some embodiments of the present invention, data visualization technology is used to visualize target data in the form of charts, including:

[0044] Classify the target data to obtain several classified data;

[0045] Add several classified data to the preset graphic matrix to obtain a display graph for visual display.

[0046] The present invention proposes a data processing method for industrial data distillation in elastic interaction, which accurately and quickly collects data in the dynamic data environment of elastic interaction, accurately processes the collected data, shortens the processing time, and improves the accuracy and effect of data processing.

[0047] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0048] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0050] Figure 1 is a flow chart of a data processing method for performing industrial data distillation in elastic interaction according to one embodiment of the present invention;

[0051] Figure 2 is a flowchart of a method for obtaining a monitoring data stream according to an embodiment of the present invention;

[0052] Figure 3 is a flowchart of a method for determining key data according to an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0054] like Figure 1 As shown, an embodiment of the present invention proposes a data processing method for industrial data distillation in elastic interaction, including steps S1-S5:

[0055] S1. Obtain monitoring data streams through real-time monitoring in a dynamic data environment with elastic interaction;

[0056] S2. Preprocess the monitoring data stream to obtain preprocessed data;

[0057] S3. Perform data identification on the pre-processed data to determine key data;

[0058] S4. Process the key data through knowledge distillation to determine the target data;

[0059] S5. Visualize the target data and display it visually.

[0060] The working principle of the above technical solution is as follows: In a dynamic data environment with flexible interaction, real-time monitoring is used to obtain monitoring data streams; preprocessing of the monitoring data streams is performed to obtain preprocessed data; data identification is performed on the preprocessed data to determine key data; key data is the selected and relatively important data. This key data is processed through knowledge distillation to determine target data; this facilitates further extraction of the most important data, which is then visualized and displayed.

[0061] The beneficial effects of the above technical solution are: accurately and quickly collecting data in a flexible and interactive dynamic data environment, accurately processing the collected data, shortening the processing time, and improving the accuracy and effect of data processing.

[0062] like Figure 2 As shown, according to some embodiments of the present invention, in a dynamic data environment with elastic interaction, obtaining a monitoring data stream through real-time monitoring includes steps S11-S13:

[0063] S11. Determine the objects and indicators to be monitored in a dynamic data environment with flexible interaction;

[0064] S12. Determine the monitoring tools to be used and their key locations based on the objects and indicators to be monitored; wherein the monitoring tools include log analysis tools, performance monitoring software, and network traffic monitors;

[0065] S13. Perform real-time monitoring based on monitoring tools at corresponding key locations to obtain monitoring data streams.

[0066] The working principle and beneficial effects of the above technical solution: In this embodiment, the objects to be monitored include data senders, data receivers, intermediaries, etc. Indicators include data transmission time, transmission duration, efficiency, data transmission trend, response time of each party, throughput, etc. Based on the objects and indicators to be monitored, determine the monitoring tools to be used and their key positions; wherein, the monitoring tools include log analysis tools, performance monitoring software and network traffic monitors; based on the monitoring tools in the corresponding key positions, real-time monitoring is performed to obtain monitoring data streams. Accurately determine the objects and indicators to be monitored and then select the corresponding monitoring tools, and adjust the layout of the monitoring tools to facilitate accurate acquisition of monitoring data.

[0067] According to some embodiments of the present invention, preprocessing the monitoring data stream to obtain preprocessed data includes:

[0068] The monitoring data stream is cleaned, denoised, converted, integrated and reduced to obtain preprocessed data.

[0069] The working principle and beneficial effects of the above technical solution are as follows: Data cleaning includes missing value processing: checking for missing values in the data and filling or deleting these missing values based on business logic or statistical methods. Outlier processing: identifying and processing outliers in the data, which usually involves setting thresholds or using statistical methods to identify and correct or delete abnormal data. Duplicate value processing: deleting or merging duplicate data records to ensure data uniqueness. Data noise reduction includes: Smoothing: reducing random fluctuations and noise in the data through methods such as moving average and exponential smoothing. Filtering: using filters (such as low-pass filters) to remove high-frequency noise and retain the main trends of the data. Signal decomposition: such as wavelet transform, decomposing the signal into components of different frequencies, thereby removing the noise component. Data transformation includes: Standardization / normalization: scaling the data so that it falls into a small specific interval, such as [0,1] or [-1,1], which helps the stability and convergence of subsequent algorithms. Discretization: converting continuous variables into discrete variables to meet the requirements of certain algorithms or models. Logarithmic transformation: For skewed data, logarithmic transformation can make it closer to a normal distribution. Data integration includes data merging: Data from different data sources are combined according to specific rules to form a complete dataset. Conflict resolution: When data from different data sources conflict, rules or algorithms are needed to resolve these conflicts. Data redundancy processing: During the integration process, redundant data may occur, requiring deduplication or merging. Data reduction includes: Dimensionality reduction: Using methods such as principal component analysis (PCA) and feature selection to reduce data dimensionality and computational complexity. Numerical reduction: Using methods such as clustering and parameterization to reduce the number of objects in a dataset or replace the original data with a simpler model. Data compression: Using lossless or lossy compression algorithms to reduce data storage space and transmission costs. After the above preprocessing steps, the raw monitoring data stream is transformed into a clearer, more accurate dataset suitable for subsequent analysis. This not only improves the efficiency and accuracy of data analysis but also provides a more reliable basis for decision-making.

[0070] According to some embodiments of the present invention, performing data denoising on a monitoring data stream includes: inputting the monitoring data stream into a pre-trained denoising model to perform data denoising.

[0071] like Figure 3 As shown, according to some embodiments of the present invention, data identification is performed on pre-processed data to determine key data, including steps S31-S33:

[0072] S31. Based on streaming processing technology, perform feature extraction on pre-processed data to determine key features;

[0073] S32. Selecting a corresponding data recognition algorithm based on the key features;

[0074] S33. Perform data recognition on the pre-processed data based on the data recognition algorithm to determine key data.

[0075] The above technical solution works as follows: Streaming allows data to enter the processing system in a stream, without first storing the entire dataset. This is particularly useful for real-time data analysis, as it allows for immediate processing and results. Feature extraction is performed on the preprocessed data to identify key features; these include numerical values, text, images, or other types of data that describe certain aspects or attributes of the data. Based on these key features, a corresponding data recognition algorithm, such as a text recognition algorithm, image recognition algorithm, or numerical recognition algorithm, is selected. Data recognition is performed on the preprocessed data using this data recognition algorithm to identify key data.

[0076] The beneficial effects of the above technical solution are: based on streaming processing technology, it is convenient to quickly extract features, avoid occupying large memory, improve data extraction efficiency, and perform data recognition on the corresponding pre-processed data based on different data recognition algorithms, thereby improving the accuracy of determining key data.

[0077] According to some embodiments of the present invention, performing data identification on pre-processed data to determine key data includes:

[0078] Input the pre-processed data into the pre-trained entity recognition model and output several entities;

[0079] Perform cluster analysis on several entities to obtain several entity sets;

[0080] Calculate the correlation coefficient between several entity sets;

[0081]

[0082] Where r is the correlation coefficient between entity set X and entity set Y; cov(X, Y) is the covariance between entity set X and entity set Y; σ X , σ y is the standard deviation of entity set X and entity set Y respectively; is the average value of entity set X and entity set Y; n is the smaller value of the number of entities included in entity set X and entity set Y respectively; X i is the i-th entity in the entity set X; Y i is the i-th entity in the entity set Y;

[0083] Construct a correlation diagram between several entity sets based on the correlation coefficient;

[0084] In the association diagram, the size of the correlation coefficient between each entity set and other entity sets and the preset correlation coefficient is determined, and the number of entity sets with a correlation coefficient greater than the preset correlation coefficient is determined; and the entity sets with a number greater than the preset number are screened out as key data.

[0085] The working principle of the above technical solution: In this embodiment, entity recognition, also known as named entity recognition (NER), is the process of identifying entities with specific meanings from data, such as names of people, places, and organizations. Cluster analysis is performed on multiple entities to obtain multiple entity sets; correlation coefficients between the multiple entity sets are calculated; a relationship diagram between the multiple entity sets is constructed based on the correlation coefficients; the correlation coefficient between each entity set and other entity sets in the relationship diagram is compared with a preset correlation coefficient, and the number of entity sets with correlation coefficients greater than the preset correlation coefficient is determined; and entity sets greater than the preset number are selected as key data.

[0086] The beneficial effects of the above technical solution are: based on identifying the entity information of the preprocessed data and performing cluster analysis, several entity sets are obtained; the correlation coefficients between the several entity sets are calculated; based on the correlation coefficients, the number of entity sets whose correlation coefficients are greater than the preset correlation coefficients is determined, and the entity sets whose number is greater than the preset number are screened out as key data, which facilitates the accurate determination of key data.

[0087] According to some embodiments of the present invention, a method for obtaining an entity recognition model includes:

[0088] Acquire training data; the training data includes entity annotation results;

[0089] Input the training data into the semantic determination layer of the neural network model to determine the initial semantic vector of each semantic unit in the training data; match each semantic unit with the corresponding knowledge graph to determine the loaded semantic vector of each semantic vector; splice the initial semantic vector of each semantic vector with the loaded semantic vector to determine the spliced semantic vector, and construct a spliced semantic vector sequence; input the spliced semantic vector sequence into the semantic recognition layer of the neural network model, split the semantic structure of the spliced semantic vector sequence to obtain a number of units to be recognized; match the several units to be recognized with a preset entity vector library, and determine the entity recognition result based on the matching result;

[0090] The difference between the entity labeling result and the entity recognition result is calculated, and iterative training is performed according to the calculation result until the entity labeling result is determined to be consistent with the entity recognition result, thereby obtaining the initial entity recognition model;

[0091] Obtain test data and input it into the initial entity recognition model;

[0092] Determine the cross entropy loss function between the predicted entities and the real entities of the test data based on the initial entity recognition model;

[0093] Determine the degree of match between the predicted entities of the test data and the real entities based on the initial entity recognition model;

[0094] Determine the cosine distance between the vectors of the predicted entity and the true entity of the test data based on the initial entity recognition model;

[0095] When it is determined that the cross entropy loss function, the matching degree and the cosine distance are all less than the corresponding preset thresholds, it means that the initial entity recognition model is qualified and is used as the final trained entity recognition model.

[0096] The working principle and beneficial effects of the above technical solution are as follows: obtaining training data; the training data contains entity labeling results; inputting the training data into the semantic determination layer of the neural network model to determine the initial semantic vector of each semantic unit in the training data; matching each semantic unit with the corresponding knowledge graph to determine the loaded semantic vector of each semantic vector; splicing the initial semantic vector of each semantic vector with the loaded semantic vector to determine the spliced semantic vector, and constructing a spliced semantic vector sequence; inputting the spliced semantic vector sequence into the semantic recognition layer of the neural network model, splitting the semantic structure of the spliced semantic vector sequence to obtain several units to be recognized; matching several units to be recognized with a preset entity vector library, and determining the entity recognition result according to the matching result; determining the entity recognition result is based on matching the units to be recognized obtained by splitting the semantic structure of the determined semantic vector sequence with the preset entity vector library, recognizing based on semantics, and connecting with the context, thereby improving the accuracy of the entity recognition result. The entity labeling results are compared with the entity recognition results, and iterative training is performed based on the calculation results until the entity labeling results are consistent with the entity recognition results. The initial entity recognition model is then obtained and input into the obtained initial entity recognition model. The initial entity recognition model is then tested for accuracy. The cross-entropy loss function between the predicted entities and the real entities in the test data is determined based on the initial entity recognition model. The degree of match between the predicted entities and the real entities in the test data is determined based on the initial entity recognition model. The cosine distance between the vectors of the predicted entities and the real entities in the test data is determined based on the initial entity recognition model. When the cross-entropy loss function, degree of match, and cosine distance are all less than corresponding preset thresholds, the initial entity recognition model is deemed qualified and is used as the final trained entity recognition model. This improves the accuracy of the obtained entity recognition model.

[0097] According to some embodiments of the present invention, key data is processed by knowledge distillation to determine target data, including:

[0098] The key data is input into the encoder for compression encoding. The encoder projects the key data into the latent space, obtains the hidden vector in the latent space and performs numerical processing;

[0099] Sort the numerically processed hidden vectors from large to small to obtain a sorted queue;

[0100] Set the distillation ratio and select the data with the distillation ratio from the front of the sorting queue as the target data.

[0101] The above technical solution works as follows: Key data is provided as input to an encoder, which maps (or projects) the input key data into a low-dimensional latent space. This reduces the data's dimensionality, simplifying subsequent processing while preserving the key information within the data. In the latent space, each key data point is represented as a latent vector. These vectors are typically numerical, making them easier for computers to process and analyze. Numerical processing may include converting the elements of the vector to a specific data type (such as floating-point numbers) or performing normalization. The converted latent vectors are sorted according to a metric (such as vector size, length, or a specific scoring function). The purpose of sorting is to identify those vectors that contribute most to the subsequent task. After sorting, the latent vectors are sorted from largest to smallest to form a sorted queue. The vectors in this queue are arranged according to their importance or contribution. The distillation ratio is a preset parameter that determines the proportion of data from the sorted queue that is selected as target data. This ratio can be adjusted based on specific task requirements and resource constraints. Based on the set distillation ratio, a corresponding proportion of data is selected from the front of the sorted queue. These selected data are considered to be the most important or representative and will serve as target data for subsequent tasks.

[0102] The beneficial effects of the above technical solution are: effectively compressing, encoding, sorting and distilling key data, thereby obtaining a set of streamlined and representative target data, which makes it easier to display the target data.

[0103] According to some embodiments of the present invention, visual processing and visual display of target data include: using data visualization technology to visually display the target data in the form of charts or reports.

[0104] According to some embodiments of the present invention, data visualization technology is used to visualize target data in the form of charts, including:

[0105] Classify the target data to obtain several classified data;

[0106] Add several classified data to the preset graphic matrix to obtain a display graph for visual display.

[0107] The working principle and beneficial effects of the above technical solution are as follows: Defining classification criteria: First, the classification objectives and criteria must be clearly defined. Applying a classification algorithm: Based on the defined classification criteria, an appropriate classification algorithm is used to classify the target data. Classification algorithms can be rule-based, statistical, or machine-learning based, such as decision trees, support vector machines, and neural networks. Obtaining classified data: After processing by the classification algorithm, the target data is divided into several distinct categories, forming categorical data. The data within each category has similar characteristics or attributes. Visualization and Presetting a Graphical Matrix: Before visualization, a graphical matrix must be pre-set. A graphical matrix is a template or layout for displaying data that defines how the different categorical data are presented graphically. The graphical matrix can be a grid layout, pie chart, bar chart, scatter plot, or other format, depending on the nature of the data and presentation requirements. Adding categorical data: The obtained categorical data is added according to the pre-set graphical matrix. This involves mapping the data to different components of the graph, such as different bars, colors, sizes, or positions, to clearly display the data for each category. Generating a presentation graph: Based on the mapping relationship between the graphical matrix and the categorical data, the final presentation graph is generated. The presentation should intuitively reflect the relationships, distribution, and proportions between different data categories, allowing users to quickly understand and analyze the data. Visualization: Display the generated presentation to users or stakeholders. Through visualization, users can more intuitively understand the characteristics and patterns of the data, enabling better decisions or analysis.

[0108] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A data processing method for industrial data distillation in elastic interaction, characterized in that: include: In a dynamic data environment with elastic interaction, monitoring data streams can be obtained through real-time monitoring; Preprocessing the monitoring data stream to obtain preprocessed data; Perform data identification on pre-processed data to determine key data; Process key data through knowledge distillation to determine target data; Visualize the target data and present it visually; Perform data identification on pre-processed data to determine key data, including: Input the pre-processed data into the pre-trained entity recognition model and output several entities; Perform cluster analysis on several entities to obtain several entity sets; Calculate the correlation coefficient between several entity sets; Where r is the correlation coefficient between entity set X and entity set Y; cov(X, Y) is the covariance between entity set X and entity set Y; σ X , σ y is the standard deviation of entity set X and entity set Y respectively; is the average value of entity set X and entity set Y; n is the smaller value of the number of entities included in entity set X and entity set Y respectively; X i is the i-th entity in the entity set X; Y i is the i-th entity in the entity set Y; Construct a correlation diagram between several entity sets based on the correlation coefficient; In the association diagram, the size of the correlation coefficient between each entity set and other entity sets and the preset correlation coefficient is determined, and the number of entity sets with a correlation coefficient greater than the preset correlation coefficient is determined; and the entity sets with a number greater than the preset number are screened out as key data.

2. The data processing method for industrial data distillation in elastic interaction according to claim 1, characterized in that: In a dynamic data environment with flexible interactions, real-time monitoring is used to obtain monitoring data streams, including: Determine the objects and indicators to be monitored in a dynamic data environment with flexible interactions; Based on the objects and indicators to be monitored, determine the monitoring tools to be used and their key locations; wherein the monitoring tools include log analysis tools, performance monitoring software and network traffic monitors; Real-time monitoring is carried out based on monitoring tools at corresponding key locations to obtain monitoring data streams.

3. The data processing method for industrial data distillation in elastic interaction according to claim 1, characterized in that: Preprocess the monitoring data stream to obtain preprocessed data, including: The monitoring data stream is cleaned, denoised, converted, integrated and reduced to obtain preprocessed data.

4. The data processing method for industrial data distillation in elastic interaction according to claim 3, characterized in that: Performing data denoising on the monitoring data stream includes: inputting the monitoring data stream into a pre-trained denoising model to perform data denoising.

5. The data processing method for industrial data distillation in elastic interaction according to claim 1, characterized in that: Perform data identification on pre-processed data to determine key data, including: Based on streaming processing technology, feature extraction is performed on pre-processed data to determine key features; Select the corresponding data recognition algorithm based on key features; Perform data identification on pre-processed data based on data identification algorithm to determine key data.

6. The data processing method for performing industrial data distillation in elastic interaction according to claim 1, characterized in that: The method of obtaining the entity recognition model includes: Acquire training data; the training data includes entity annotation results; Input the training data into the semantic determination layer of the neural network model to determine the initial semantic vector of each semantic unit in the training data; match each semantic unit with the corresponding knowledge graph to determine the loaded semantic vector of each semantic vector; splice the initial semantic vector of each semantic vector with the loaded semantic vector to determine the spliced semantic vector, and construct a spliced semantic vector sequence; input the spliced semantic vector sequence into the semantic recognition layer of the neural network model, split the semantic structure of the spliced semantic vector sequence to obtain a number of units to be recognized; match the several units to be recognized with a preset entity vector library, and determine the entity recognition result based on the matching result; The difference between the entity labeling result and the entity recognition result is calculated, and iterative training is performed according to the calculation result until the entity labeling result is determined to be consistent with the entity recognition result, thereby obtaining the initial entity recognition model; Obtain test data and input it into the initial entity recognition model; Determine the cross entropy loss function between the predicted entities and the real entities of the test data based on the initial entity recognition model; Determine the degree of match between the predicted entities of the test data and the real entities based on the initial entity recognition model; Determine the cosine distance between the vectors of the predicted entity and the true entity of the test data based on the initial entity recognition model; When it is determined that the cross entropy loss function, the matching degree and the cosine distance are all less than the corresponding preset thresholds, it means that the initial entity recognition model is qualified and is used as the final trained entity recognition model.

7. The data processing method for performing industrial data distillation in elastic interaction according to claim 1, characterized in that: Key data is processed through knowledge distillation to determine target data, including: The key data is input into the encoder for compression encoding. The encoder projects the key data into the latent space, obtains the hidden vector in the latent space and performs numerical processing; Sort the numerically processed hidden vectors from large to small to obtain a sorted queue; Set the distillation ratio and select the data with the distillation ratio from the front of the sorting queue as the target data.

8. The data processing method for performing industrial data distillation in elastic interaction according to claim 1, characterized in that: Visualize the target data and present it visually, including: using data visualization technology to visualize the target data in the form of charts or reports.

9. The data processing method for performing industrial data distillation in elastic interaction according to claim 8, characterized in that: Use data visualization technology to visualize the target data in the form of charts, including: Classify the target data to obtain several classified data; Add several classified data to the preset graphic matrix to obtain a display graph for visual display.

Images