Distributed data processing method, system, device and storage medium

Abstract

The present application provides a distributed data processing method, system, device and storage medium, which relates to the field of distributed data processing technology, obtains industrial data packets to be processed in a distributed system, performs data compression on the industrial data packets to be processed, and then obtains multiple data compression slices; extracts the industrial data covariates corresponding to each data compression slice, thereby determining the distribution density disorder of each data compression slice, and then determining the data covariance threshold value of each data compression slice; determines the potential abnormal data of each data compression slice based on the corresponding data covariance threshold value, and determines the de-differentiated data packets in the distributed processing of industrial data through the potential abnormal data of each data compression slice; stores the de-differentiated data packets to complete the distributed processing of industrial data. The present application can set a dynamic adaptive threshold that takes into account both global accuracy and local sensitivity to improve the accuracy of anomaly elimination in distributed data processing.

Description

Technical Field

[0001] The present application relates to the technical field of distributed data processing, and more specifically, to a distributed data processing method, system, device and storage medium. Background Art

[0002] Distributed data processing involves partitioning large amounts of data into subsets, processing them in parallel across multiple computing nodes, and then aggregating the results. With the advent of the big data era, data volumes are exploding. Traditional single-machine processing models are no longer able to meet the requirements of real-time performance, high availability, and scalability. Therefore, distributed data processing technology has become a crucial foundation for supporting a wide range of applications. From basic data storage (such as the distributed file system HDFS) to computing frameworks (such as MapReduce, Spark, and Flink) to data transmission systems (such as Kafka and Pulsar), the distributed data processing landscape continues to expand and improve.

[0003] In existing technologies, industrial data used in distributed data processing is often highly heterogeneous, with significant differences in data distribution across different devices, production lines, or time windows. Fixed thresholds struggle to balance global accuracy and local sensitivity, leading to false positives or missed detections. Furthermore, data streams often exhibit non-stationary characteristics (such as distribution drift caused by equipment aging and operating mode switching). Static thresholds cannot be adaptively adjusted, requiring frequent manual intervention. In large-scale distributed environments, statistical characteristics between data shards may differ significantly, and a unified threshold can easily waste resources or miss key anomalies. Many methods rely on manually set static thresholds, which struggle to cope with the dynamic changes in statistical characteristics of different data segments, making anomalies prone to misjudgment or missed detection. Therefore, setting dynamic, adaptive thresholds that balance global accuracy and local sensitivity to improve the accuracy of anomaly rejection in distributed data processing has become a challenging issue for the industry. Summary of the Invention

[0004] The present application provides a distributed data processing method, system, device and storage medium, which can set a dynamic adaptive threshold that takes into account both global accuracy and local sensitivity to improve the accuracy of anomaly elimination in distributed data processing.

[0005] In a first aspect, the present application provides a distributed data processing method, the processing method comprising the following steps:

[0006] Obtaining industrial data packets to be processed in a distributed system, performing data sharding and compression on the industrial data packets to be processed, and thereby obtaining multiple data compression shards;

[0007] Extracting industrial data covariates corresponding to each data compression slice, determining the distribution density disorder of each data compression slice based on the corresponding industrial data covariates, and determining the data covariance threshold value of each data compression slice based on the corresponding industrial data covariates and the distribution density disorder;

[0008] The industrial data in each data compression slice is divided according to the corresponding data covariance threshold value, and the potential abnormal data of each data compression slice is obtained. The de-differentiated data packets in the distributed processing of industrial data are determined by the potential abnormal data of each data compression slice;

[0009] The de-identified data packets are stored to complete the distributed processing of the industrial data.

[0010] In this embodiment, the industrial data packet to be processed is compressed into slices to obtain multiple industrial slices of data, specifically including:

[0011] Slice the data in the industrial data packet to be processed to obtain multiple sliced data;

[0012] Determine the data variability and data level of each shard data;

[0013] Each shard data is converted into a corresponding data compression shard according to the corresponding data variance and data level, thereby obtaining multiple data compression shards.

[0014] In this embodiment, extracting the industrial data covariates corresponding to each data compression slice specifically includes:

[0015] Obtain the covariate extraction parameters and spatial spectrum distribution function corresponding to each data compression slice;

[0016] The industrial data covariates corresponding to each data compression slice are determined according to the corresponding covariate extraction parameters and spatial spectrum distribution function.

[0017] In this embodiment, determining the distribution density disorder of each data compression slice according to the corresponding industrial data covariate specifically includes:

[0018] Determine the distribution density of each data compression slice based on the industrial data covariate corresponding to each data compression slice;

[0019] The distribution density disorder of each data compression slice is determined according to the corresponding distribution density.

[0020] In this embodiment, determining the data covariance threshold value of each data compression slice based on the corresponding industrial data covariate and distribution density disorder specifically includes:

[0021] For each data compression slice, determining the weighted disorder of the data compression slice by the industrial data covariate and distribution density disorder corresponding to the data compression slice;

[0022] The data covariance threshold value of the data compression slice is determined according to the weighted disorder degree, and then the data covariance threshold value of each data compression slice is obtained.

[0023] In this embodiment, the industrial data in each data compression slice is divided according to the corresponding data covariance threshold value, and the potential abnormal data of each data compression slice is obtained, which specifically includes:

[0024] For each data compression slice, obtaining a data covariance threshold value of the data compression slice;

[0025] Taking compressed data points in the data compression slice that are greater than the data covariance threshold value as potential abnormal data points, thereby obtaining multiple potential abnormal data points in the data compression slice;

[0026] The potential abnormal data of the data compression slices are constructed through all potential abnormal data points, and then the potential abnormal data of each data compression slice is obtained.

[0027] In this embodiment, determining the denormalized data packets in the distributed processing of industrial data based on the potential abnormal data of each data compression slice specifically includes:

[0028] For the potential abnormal data of each data compression slice, determining the data difference between each potential abnormal data point in the potential abnormal data of the data compression slice;

[0029] Determining anomaly detection limits corresponding to potential anomaly data in the data compression slice;

[0030] Determine the de-differentiated data slices corresponding to the data compression slices based on all data differences and the anomaly detection limit, and then obtain the de-differentiated data slices corresponding to each data compression slice;

[0031] Build de-identified data packets for distributed processing of industrial data based on all de-identified data shards.

[0032] In a second aspect, the present application provides a distributed data processing system for executing a distributed data processing method, the processing system comprising:

[0033] A fragment compression module is used to obtain the industrial data packets to be processed in the distributed system, perform fragment data compression on the industrial data packets to be processed, and thus obtain multiple data compression fragments;

[0034] A threshold module is used to extract the industrial data covariates corresponding to each data compression slice, determine the distribution density disorder of each data compression slice based on the corresponding industrial data covariates, and determine the data covariance threshold value of each data compression slice based on the corresponding industrial data covariates and the distribution density disorder;

[0035] The de-anomaly module is used to divide the industrial data in each data compression slice according to the corresponding data covariance threshold value, thereby obtaining the potential abnormal data of each data compression slice, and determining the de-anomaly data packet in the distributed processing of industrial data based on the potential abnormal data of each data compression slice;

[0036] The storage module is used to store the de-identified data packets and complete the distributed processing of industrial data.

[0037] In a third aspect, the present application provides a computer device, comprising a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the computer device executes the above-mentioned distributed data processing method.

[0038] In a fourth aspect, the present application provides a computer-readable storage medium, in which instructions or codes are stored. When the instructions or codes are run on a computer, the computer implements the above-mentioned distributed data processing method when executing.

[0039] The technical solutions provided by the embodiments disclosed in this application have the following beneficial effects:

[0040] By acquiring the industrial data packets to be processed in the distributed system, the industrial data packets to be processed are sliced and compressed to obtain a plurality of data compression slices; the industrial data covariates corresponding to each data compression slice are extracted, and the distribution density disorder of each data compression slice is determined according to the corresponding industrial data covariates, and the data covariance threshold value of each data compression slice is determined according to the corresponding industrial data covariates and the distribution density disorder; the industrial data in each data compression slice is divided according to the corresponding data covariance threshold value, and the potential abnormal data of each data compression slice is obtained, and the de-noised data packets in the distributed processing of the industrial data are determined according to the potential abnormal data of each data compression slice; the de-noised data packets are stored to complete the distributed processing of the industrial data.

[0041] It can be seen that in this application, first, the industrial data packet to be processed is sliced and compressed to obtain multiple data compression slices, which can effectively reduce the storage and transmission overhead of data while retaining important information and improving the processing efficiency of the system; then, by extracting the industrial data covariates corresponding to each data compression slice and combining the distribution density disorder to determine the data covariance threshold, the distribution characteristics and potential anomalies of the data can be captured more accurately. Setting an adaptable data covariance threshold for each data compression slice can improve the accuracy of abnormal data detection and can flexibly respond to changes in different data compression slices; finally, the industrial data in the data compression slices is divided according to the data covariance threshold, and potential abnormal data is identified, which can effectively and accurately remove outliers from large-scale data and improve the quality and accuracy of the data. By analyzing each slice and combining the data characteristics, potential abnormal data can be targeted and screened out, avoiding the roughness of the traditional global threshold setting. In addition, the setting of the dynamic adaptive threshold (i.e., the anomaly detection limit) takes into account both global accuracy and local sensitivity, and can flexibly adjust the threshold according to the actual situation of different data slices, thereby improving the accuracy of anomaly removal in distributed data processing.

[0042] In summary, the technical solution adopted in this application can set a dynamic adaptive threshold that takes into account both global accuracy and local sensitivity, so as to improve the accuracy of anomaly elimination in distributed data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0044] Figure 1 is an exemplary flow chart of the distributed data processing method provided by this application;

[0045] Figure 2 is an exemplary flow chart for determining the distribution density disorder of each data compression slice provided by the present application;

[0046] Figure 3 is an exemplary flow chart for determining potential abnormal data for each data compression slice provided by the present application;

[0047] Figure 4 It is a module structure diagram of the distributed data processing system provided by this application;

[0048] Figure 5It is a structural diagram of a computer device for implementing a distributed data processing method provided by the present application. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0050] The embodiment of the present application provides a distributed data processing method, system, device and storage medium, the core of which is to obtain the industrial data packets to be processed in the distributed system, perform data compression on the industrial data packets to be processed, and thus obtain multiple data compression slices; extract the industrial data covariates corresponding to each data compression slice, determine the distribution density disorder of each data compression slice according to the corresponding industrial data covariates, and determine the data covariance threshold value of each data compression slice through the corresponding industrial data covariates and distribution density disorder; divide the industrial data in each data compression slice according to the corresponding data covariance threshold value, and thus obtain the potential abnormal data of each data compression slice, and determine the de-differentiation data packet in the distributed processing of industrial data through the potential abnormal data of each data compression slice; store the de-differentiation data packet to complete the distributed processing of industrial data. The above scheme can be used to set a dynamic adaptive threshold that takes into account both global accuracy and local sensitivity to improve the accuracy of anomaly elimination in distributed data processing.

[0051] Example 1: In order to better understand the above technical solution, the above technical solution will be described in detail below with reference to the accompanying drawings and specific implementation methods. Figure 1 As shown in FIG, this figure is an exemplary flow chart of a distributed data processing method according to this embodiment of the present application, and the processing method includes the following steps:

[0052] In step S1, an industrial data packet to be processed in a distributed system is obtained, and data fragmentation compression is performed on the industrial data packet to be processed to obtain multiple data compression fragments.

[0053] In a specific implementation, the industrial data packet to be processed may be obtained from the memory of the distributed system, wherein the industrial data packet to be processed includes a plurality of industrial data.

[0054] In this embodiment, the industrial data packet to be processed is subjected to data fragment compression, thereby obtaining multiple data compression fragments, which can be specifically implemented by the following steps:

[0055] Slice the data in the industrial data packet to be processed to obtain multiple data slices;

[0056] Determine the data variability and data level of each data shard;

[0057] Each data slice is converted into a corresponding data compression slice according to the corresponding data variation and data level, thereby obtaining multiple data compression slices.

[0058] It should be noted that, in the present application, data compression shards are data shards obtained after sharding and compressing the industrial data in the to-be-processed industrial data packet; in specific implementation, first, the to-be-processed data packet is obtained from the memory of the distributed system, and according to the preset sharding strategy, the industrial data in the to-be-processed industrial data packet is sharded to obtain multiple data shards, wherein the preset sharding strategy can be set according to historical experience; then, the data variability and data level of each data shard can be determined, wherein the data variability represents the degree of abnormal fluctuation of the overall value of all industrial data points in the data shard, and the data level is the overall value level of all industrial data points in the corresponding data shard. For each data shard, the data variability can be calculated. The average value of all industrial data points in the data slice is used as the data level of the data slice, and the standard deviation of all industrial data points in the data slice is used as the data variability of the data slice. The data variability and data level of each data slice can be obtained through the above method; finally, each data slice can be converted into a corresponding data compression slice according to the corresponding data variability and data level. That is, for each industrial data point of the data slice, the industrial data point can be converted according to the data variability and data level of the data slice to obtain the compressed data point corresponding to the industrial data point. In actual implementation, the compressed data point can be determined according to the following formula:

[0059]

[0060] in, Represents industrial data points The corresponding compressed data points, represents the t-th industrial data point in the data shard, Indicates the data level, Indicates the data variation, represents a low-dimensional space, It means taking a specific value of the low-dimensional space. The specific value of the low-dimensional space can be calibrated as a constant based on historical experience. All compressed data points can be obtained through the above steps. The slice composed of all compressed data points is used as the data compression slice corresponding to the data slice, so that multiple data compression slices can be obtained.

[0061] It should be noted that slicing and compressing the industrial data packets to be processed and obtaining multiple compressed data slicing can effectively reduce data storage and transmission overhead while retaining important information and improving the processing efficiency of the system. The compression strategy of each data slicing can be adjusted according to its data variability and data level, thereby optimizing the data storage method and improving the speed of data transmission.

[0062] In step S2, the industrial data covariates corresponding to each data compression slice are extracted, the distribution density disorder of each data compression slice is determined according to the corresponding industrial data covariates, and the data covariance threshold value of each data compression slice is determined by the corresponding industrial data covariates and the distribution density disorder.

[0063] In this embodiment, the extraction of industrial data covariates corresponding to each data compression slice can be specifically implemented by the following steps:

[0064] Obtain the covariate extraction parameters and spatial spectrum distribution function corresponding to each data compression slice;

[0065] The industrial data covariates corresponding to each data compression slice are determined according to the corresponding covariate extraction parameters and spatial spectrum distribution function.

[0066] It should be noted that in this application, the covariate extraction parameter is a set of numerical values extracted from the original data that can quantitatively describe its behavioral characteristics; the spatial spectrum distribution function is a function that represents the distribution of the covariate on the spatial spectrum; and the industrial data covariate is a characteristic vector used to represent important information in the industrial data.

[0067] In specific implementation, first, the mean, variance and skewness of each data compression slice can be calculated, and the corresponding mean, variance and skewness can be used as the specific values in the covariate extraction parameters; then, the data compression slice can be subjected to fast Fourier transform to obtain its spectrum, so that the square of the absolute value of its spectrum can be calculated, and the calculated result can be used as the spatial spectrum distribution function. The covariate extraction parameters and spatial spectrum distribution function corresponding to each data compression slice can be obtained in the above manner; finally, for each data compression slice, the characteristic vector composed of the covariate extraction parameters and spatial spectrum distribution function corresponding to the data compression slice can be used as the industrial data covariate corresponding to the data compression slice. The industrial data covariate corresponding to each data compression slice can be obtained in the above manner.

[0068] Preferably, in this embodiment, reference Figure 2 As shown in FIG. 1 , this figure is an exemplary flow chart for determining the distribution density disorder of each data compression slice in an embodiment of the present application. In this embodiment, the distribution density disorder of each data compression slice is determined based on the corresponding industrial data covariate, which can be implemented by the following steps:

[0069] In step S21, the distribution density of each data compression slice is determined based on the industrial data covariate corresponding to each data compression slice;

[0070] In step S22, the distribution density disorder of each data compression slice is determined according to the corresponding distribution density.

[0071] In specific implementation, first, the distribution density of each data compression slice can be determined based on the industrial data covariates corresponding to each data compression slice. The distribution density refers to the degree of clustering of the compressed data points in each data compression slice in its feature space. In actual implementation, the distribution density can be determined by the following formula:

[0072]

[0073] in, represents the distribution density of data compression fragment x, represents the total number of features in the industrial data covariate corresponding to the data compression shard x, Represents the value of the mth feature in the industrial data covariate; then, the distribution density disorder of each data compression slice can be determined according to the corresponding distribution density, wherein the distribution density disorder indicates the degree of disorder of the information contained in the data compression slice relative to the historical information. In actual implementation, the historical distribution density data of the data compression slice can be obtained, so that the ratio of the distribution density of the data compression slice to the mean of all distribution densities in the historical distribution density data can be calculated, and the calculation result is used as the distribution density disorder of the data compression slice. The distribution density disorder of each data compression slice can be obtained through the above steps.

[0074] In this embodiment, the data covariance threshold value of each data compression slice is determined by the corresponding industrial data covariate and distribution density disorder, which can be specifically implemented by the following steps:

[0075] For each data compression slice, determining the weighted disorder of the data compression slice by the industrial data covariate and distribution density disorder corresponding to the data compression slice;

[0076] The data covariance threshold value of the data compression slice is determined according to the weighted disorder degree, and then the data covariance threshold value of each data compression slice is obtained.

[0077] In the specific implementation, first, each feature in the industrial data covariate corresponding to the data compression slice is multiplied by the distribution density disorder, and the obtained results are summed up, so that the summation result is used as the weighted disorder of the data compression slice. It should be noted that in this application, the weighted disorder is a comprehensive indicator used to reflect the sensitivity of the importance of actual data; then, the ratio of the weighted disorder to the number of data points in the data compression slice can be calculated, and the calculation result is used as the data covariance threshold value of the data compression slice. The above steps are performed on all data compression slices to obtain the data covariance threshold value of each data compression slice. It should be noted that in this application, the data covariance threshold value is a threshold parameter used to measure the degree of data anomaly in the data compression slice.

[0078] It should be noted that by extracting the industrial data covariates corresponding to each data compression slice and combining them with the distribution density disorder to determine the data covariance threshold, the distribution characteristics and potential anomalies of the data can be captured more accurately. By dynamically evaluating the characteristics of each data compression slice and setting a highly adaptable data covariance threshold for each data compression slice, the accuracy of abnormal data detection can be improved. It can flexibly respond to changes in different data compression slices, allowing the system to ensure overall data accuracy when processing large-scale distributed data while sensitively detecting abnormal data in local areas, thereby significantly improving the accuracy of anomaly rejection and reducing the occurrence of false detections and missed detections.

[0079] In step S3, the industrial data in each data compression slice is divided according to the corresponding data covariance threshold value, and then the potential abnormal data of each data compression slice is obtained. The de-normalized data packets in the distributed processing of industrial data are determined by the potential abnormal data of each data compression slice.

[0080] Preferably, in this embodiment, reference Figure 3 As shown in the figure, this is an exemplary flow chart for determining potential abnormal data of each data compression slice in an embodiment of the present application. In this embodiment, the industrial data in each data compression slice is divided according to the corresponding data covariance threshold value, and then the potential abnormal data of each data compression slice is obtained. Specifically, the following steps can be used to achieve this:

[0081] In step S31, for each data compression slice, a data covariance threshold value of the data compression slice is obtained;

[0082] In step S32, compressed data points in the data compression slice that are greater than the data covariance threshold value are taken as potential abnormal data points, thereby obtaining multiple potential abnormal data points in the data compression slice;

[0083] In step S33, potential abnormal data of the data compression slices are constructed using all potential abnormal data points, thereby obtaining potential abnormal data of each data compression slice.

[0084] In specific implementation, first, the data covariance threshold value of the data compression slice can be used; then, the compressed data points in the data compression slice are compared with the data covariance threshold value of the data compression slice, and the compressed data points in the data compression slice that are greater than the data covariance threshold value are taken as potential abnormal data points, wherein the data covariance threshold value is calculated based on the covariate of the slice and the distribution density disorder, and is the "fluctuation tolerance boundary" of the slice data in a statistical sense. The data within this boundary is considered to be a reasonable fluctuation, that is, the compressed data points in the data compression slice that are greater than the data covariance threshold value are taken as potential abnormal data points. It should be noted that in this application, potential abnormal data points refer to data points that significantly deviate from the overall data point distribution pattern in the data compression slice and have the possibility of being abnormal, but have not yet been fully determined to be abnormal; finally, the data set consisting of all potential abnormal data points is taken as the potential abnormal data of the data compression slice, and the above steps are repeated to obtain the potential abnormal data of each data compression slice.

[0085] In this embodiment, determining the outlier data packets in the distributed processing of industrial data by using the potential outlier data in each data compression slice can be specifically implemented by the following steps:

[0086] For the potential abnormal data of each data compression slice, determining the data difference between each potential abnormal data point in the potential abnormal data of the data compression slice;

[0087] Determining anomaly detection limits corresponding to potential anomaly data in the data compression slice;

[0088] Determine the de-differentiated data slices corresponding to the data compression slices based on all data differences and the anomaly detection limit, and then obtain the de-differentiated data slices corresponding to each data compression slice;

[0089] Build de-identified data packets for distributed processing of industrial data based on all de-identified data shards.

[0090] In specific implementation, first, for each potential abnormal data point in the potential abnormal data of the data compression slice, the Euclidean distance between each potential abnormal data point can be calculated, and the corresponding Euclidean distance can be used as the data difference between the potential abnormal data points, wherein the data difference is an indicator used to measure the degree of data difference between the potential abnormal data points; then, the data mean and data standard deviation of the potential abnormal data of the data compression slice can be calculated, and the sum of the data mean and 3 times the data standard deviation can be used as the anomaly detection limit corresponding to the potential abnormal data of the data compression slice, wherein the anomaly detection limit is a threshold used to determine whether the potential abnormal data point is abnormal.

[0091] In addition, in the specific implementation, first, each potential abnormal data point in the potential abnormal data of the data compression slice can be compared with the corresponding abnormal detection limit, so that the potential abnormal data point greater than the abnormal detection limit can be used as the abnormal data point, thereby obtaining the first abnormal data point set of the data compression slice; then, the data difference between each potential abnormal data point can be used as the input of the local outlier factor algorithm, so that all abnormal data points in the potential abnormal data of the data compression slice can be screened out by the local outlier factor algorithm, and all screened out abnormal data points are used as the second abnormal data point set; finally, all abnormal data points contained in the first abnormal data point set and the second abnormal data point set can be eliminated in the data compression slice, so that the data compression slice after eliminating the abnormal data points is used as the corresponding de-noised data slice, and the above steps are repeated to obtain the de-noised data slice corresponding to each data compression slice, so that the data packet composed of all the de-noised data slices can be used as the de-noised data packet in the distributed processing of industrial data. It should be noted that in this application, the de-noised data packet refers to the industrial data packet obtained after the abnormality is eliminated.

[0092] It's important to note that partitioning industrial data within data compression shards based on data covariance thresholds and identifying potential anomalies effectively and accurately removes outliers from large-scale data, improving data quality and accuracy. By analyzing each shard and incorporating data characteristics, this method specifically filters out potential anomalies, avoiding the harshness of traditional global threshold settings. Furthermore, the dynamic adaptive threshold (i.e., anomaly detection limit) balances global accuracy with local sensitivity, enabling flexible adjustment of the threshold based on the actual conditions of each data shard, thereby improving the accuracy of anomaly removal.

[0093] In step S4, the de-identified data packets are stored to complete the distributed processing of the industrial data.

[0094] In this embodiment, storing the de-identified data packets is to re-store the de-identified data packets in a distributed system, thereby completing the distributed processing of the industrial data.

[0095] It can be seen that in this application, first, the industrial data packet to be processed is sliced and compressed to obtain multiple data compression slices, which can effectively reduce the storage and transmission overhead of data while retaining important information and improving the processing efficiency of the system; then, by extracting the industrial data covariates corresponding to each data compression slice and combining the distribution density disorder to determine the data covariance threshold, the distribution characteristics and potential anomalies of the data can be captured more accurately. Setting an adaptable data covariance threshold for each data compression slice can improve the accuracy of abnormal data detection and can flexibly respond to changes in different data compression slices; finally, the industrial data in the data compression slices is divided according to the data covariance threshold, and potential abnormal data is identified, which can effectively and accurately remove outliers from large-scale data and improve the quality and accuracy of the data. By analyzing each slice and combining the data characteristics, potential abnormal data can be targeted and screened out, avoiding the roughness of the traditional global threshold setting. In addition, the setting of the dynamic adaptive threshold (i.e., the anomaly detection limit) takes into account both global accuracy and local sensitivity, and can flexibly adjust the threshold according to the actual situation of different data slices, thereby improving the accuracy of anomaly removal in distributed data processing.

[0096] In summary, the technical solution adopted in this application can set a dynamic adaptive threshold that takes into account both global accuracy and local sensitivity, so as to improve the accuracy of anomaly elimination in distributed data processing.

[0097] In the second embodiment, the present application provides a distributed data processing system, referring to Figure 4 As shown, this figure is a module structure diagram of the processing system shown in this embodiment of the present application, and the processing system includes:

[0098] The fragment compression module 100 is used to obtain the industrial data packets to be processed in the distributed system, perform fragment data compression on the industrial data packets to be processed, and thereby obtain multiple data compression fragments;

[0099] Threshold module 200, for extracting industrial data covariates corresponding to each data compression slice, determining the distribution density disorder of each data compression slice based on the corresponding industrial data covariates, and determining the data covariance threshold value of each data compression slice based on the corresponding industrial data covariates and the distribution density disorder;

[0100] The de-anomaly removal module 300 is used to divide the industrial data in each data compression slice according to the corresponding data covariance threshold value, thereby obtaining potential abnormal data of each data compression slice, and determining the de-anomaly data packet in the distributed processing of industrial data based on the potential abnormal data of each data compression slice;

[0101] The storage module 400 is used to store the de-identified data packets and complete the distributed processing of industrial data.

[0102] The examples of the distributed data processing methods and systems provided by the embodiments of the present application are described in detail above. It is understandable that the corresponding devices include hardware structures and / or software modules for performing the corresponding functions in order to implement the above functions. Those skilled in the art should easily appreciate that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0103] In a third embodiment, the present application further provides a computer device comprising a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the computer device executes the above-mentioned distributed data processing method.

[0104] In this embodiment, reference Figure 5 , the dotted line in the figure indicates that the unit or module is optional, and the figure is a structural diagram of a computer device according to a distributed data processing method provided in an embodiment of the present application. The above-mentioned distributed data processing method in the above embodiment can be Figure 5 The computer device shown in the figure is implemented, and the computer device includes at least one processor 501, a memory 502 and at least one communication unit 505. The computer device can be a terminal device, a server or a chip.

[0105] The processor 501 may be a general-purpose processor or a dedicated processor. For example, the processor 501 may be a central processing unit (CPU). The CPU may be used to control the computer device, execute software programs, and process data from the software programs. The computer device may also include a communication unit 505 to implement signal input (reception) and output (transmission).

[0106] For example, the computer device may be a chip, the communication unit 505 may be an input and / or output circuit of the chip, or the communication unit 505 may be a communication interface of the chip, and the chip may be a component of a terminal device, a network device, or other device.

[0107] For another example, the computer device may be a terminal device or a server, and the communication unit 505 may be a transceiver of the terminal device or the server, or the communication unit 505 may be a transceiver circuit of the terminal device or the server.

[0108] Computer device 500 may include one or more memories 502, on which a program 504 is stored. Program 504 can be executed by processor 501 to generate instructions 503, causing processor 501 to execute the method described in the above method embodiment according to instructions 503. Optionally, memory 502 may also store data (such as a target audit model). Optionally, processor 501 may also read data stored in memory 502. This data may be stored at the same memory address as program 504, or at a different memory address.

[0109] The processor 501 and the memory 502 may be provided separately or integrated together, for example, integrated on a system on chip (SOC) of a terminal device.

[0110] It should be understood that each step of the above method embodiment can be completed by a hardware-based logic circuit or software-based instructions in the processor 501. The processor 501 can be a central processing unit, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, such as discrete gates, transistor logic devices, or discrete hardware components.

[0111] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0112] In a fourth embodiment, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions or codes. When the instructions or codes are executed on a computer, the computer implements the above-mentioned distributed data processing method.

[0113] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0114] Obviously, those skilled in the art may make various changes and modifications to the present application without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present application fall within the scope of the claims and their equivalents, the present application is intended to include such modifications and variations.

Claims

1. A distributed data processing method, characterized in that: The processing method comprises the following steps: Obtaining industrial data packets to be processed in a distributed system, performing data sharding and compression on the industrial data packets to be processed, and thereby obtaining multiple data compression shards; Extract the industrial data covariates corresponding to each data compression slice, wherein the industrial data covariates are feature vectors used to represent important information in the industrial data, and the industrial data covariates are feature vectors composed of the covariate extraction parameters and the spatial spectrum distribution function corresponding to the data compression slice; determine the distribution density disorder of each data compression slice according to the corresponding industrial data covariates, wherein the distribution density disorder is the ratio of the distribution density of the data compression slice to the mean of all distribution densities in the historical distribution density data, wherein the distribution density refers to the degree of clustering of the compressed data points in each data compression slice in its feature space; determine the data covariance threshold value of each data compression slice by the corresponding industrial data covariates and the distribution density disorder, wherein determining the data covariance threshold value of each data compression slice by the corresponding industrial data covariates and the distribution density disorder specifically includes: For each data compression slice, a weighted disorder degree of the data compression slice is determined by the industrial data covariate corresponding to the data compression slice and the distribution density disorder degree, wherein each feature in the industrial data covariate corresponding to the data compression slice is multiplied by the distribution density disorder degree, the obtained results are summed, and the summed result is used as the weighted disorder degree of the data compression slice; Determining a data covariance threshold value of the data compression slice according to the weighted disorder degree, and then obtaining a data covariance threshold value of each data compression slice; The industrial data in each data compression slice is divided according to the corresponding data covariance threshold value, and the potential abnormal data of each data compression slice is obtained. The de-differentiated data packets in the distributed processing of industrial data are determined by the potential abnormal data of each data compression slice; The de-identified data packets are stored to complete the distributed processing of the industrial data.

2. A distributed data processing method according to claim 1, characterized in that: The industrial data packets to be processed are compressed into fragments, thereby obtaining multiple data compression fragments, specifically including: Slice the data in the industrial data packet to be processed to obtain multiple sliced data; Determine the data variability and data level of each shard data, where the data variability is the standard deviation of all industrial data points in the corresponding data shard, and the data level is the average value of all industrial data points in the corresponding data shard; Each shard data is converted into a corresponding data compression shard according to the corresponding data variance and data level, thereby obtaining multiple data compression shards.

3. A distributed data processing method according to claim 1, characterized in that: Extracting the industrial data covariates corresponding to each data compression slice specifically includes: Obtaining the covariate extraction parameters and spatial spectrum distribution function corresponding to each data compression slice, wherein the covariate extraction parameters include the mean, variance, and skewness of all compressed data points in the corresponding data compression slice; the spatial spectrum distribution function is the result of performing a fast Fourier transform on the data compression slice to obtain its spectrum and calculating the square of the absolute value of its spectrum; The industrial data covariates corresponding to each data compression slice are determined according to the corresponding covariate extraction parameters and spatial spectrum distribution function.

4. A distributed data processing method according to claim 1, characterized in that: The industrial data in each data compression slice is divided according to the corresponding data covariance threshold value, and the potential abnormal data of each data compression slice is obtained, including: For each data compression slice, obtaining a data covariance threshold value of the data compression slice; Taking compressed data points in the data compression slice that are greater than the data covariance threshold value as potential abnormal data points, thereby obtaining multiple potential abnormal data points in the data compression slice; The potential abnormal data of the data compression slices are constructed through all potential abnormal data points, and then the potential abnormal data of each data compression slice is obtained.

5. A distributed data processing method according to claim 1, characterized in that: Determining the denormalized data packets in distributed industrial data processing by using the potential abnormal data in each data compression shard specifically includes: For the potential abnormal data of each data compression slice, determining the data difference between each potential abnormal data point in the potential abnormal data of the data compression slice; Determining anomaly detection limits corresponding to potential anomaly data in the data compression slice; Determine the de-differentiated data slices corresponding to the data compression slices based on all data differences and the anomaly detection limit, and then obtain the de-differentiated data slices corresponding to each data compression slice; Build de-identified data packets for distributed processing of industrial data based on all de-identified data shards.

6. A distributed data processing system for executing a distributed data processing method according to any one of claims 1 to 5, characterized in that: The processing system comprises: A fragment compression module is used to obtain the industrial data packets to be processed in the distributed system, perform fragment data compression on the industrial data packets to be processed, and then obtain multiple data compression fragments; The threshold module is used to extract the industrial data covariates corresponding to each data compression slice, wherein the industrial data covariates are characteristic vectors used to represent important information in the industrial data, and the industrial data covariates are characteristic vectors composed of the covariate extraction parameters and the spatial spectrum distribution function corresponding to the data compression slice; the distribution density disorder of each data compression slice is determined according to the corresponding industrial data covariates, wherein the distribution density disorder is the ratio of the distribution density of the data compression slice to the mean of all distribution densities in the historical distribution density data, wherein the distribution density refers to the degree of clustering of the compressed data points in each data compression slice in its feature space; the data covariance threshold value of each data compression slice is determined by the corresponding industrial data covariates and the distribution density disorder, wherein the data covariance threshold value of each data compression slice is determined by the corresponding industrial data covariates and the distribution density disorder, wherein the data covariance threshold value of each data compression slice is determined by the corresponding industrial data covariates and the distribution density disorder specifically including: For each data compression slice, a weighted disorder degree of the data compression slice is determined by the industrial data covariate corresponding to the data compression slice and the distribution density disorder degree, wherein each feature in the industrial data covariate corresponding to the data compression slice is multiplied by the distribution density disorder degree, the obtained results are summed, and the summed result is used as the weighted disorder degree of the data compression slice; Determining a data covariance threshold value of the data compression slice according to the weighted disorder degree, and then obtaining a data covariance threshold value of each data compression slice; The de-anomaly module is used to divide the industrial data in each data compression slice according to the corresponding data covariance threshold value, thereby obtaining the potential abnormal data of each data compression slice, and determining the de-anomaly data packet in the distributed processing of industrial data based on the potential abnormal data of each data compression slice; The storage module is used to store the de-identified data packets and complete the distributed processing of industrial data.

7. A computer device, characterized in that: The computer device includes a memory and a processor, the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the computer device executes a distributed data processing method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores instructions or codes, which, when executed on a computer, enable the computer to implement a distributed data processing method according to any one of claims 1 to 5.

Images