A data processing method, device, apparatus and storage medium

Abstract

A data processing method, device, equipment and storage medium are disclosed. The method comprises: obtaining to-be-processed data, wherein the to-be-processed data is determined based on original data collected by a data collection edge device from a preset device; determining a data feature of the to-be-processed data, judging whether the to-be-processed data has a preset problem based on the data feature, and if so, determining a target algorithm according to the data feature, wherein the preset problem comprises at least one of data anomaly, data loss and data noise; and processing the to-be-processed data by using the target algorithm to obtain target data. The technical scheme of the embodiment of the present application determines the algorithm for processing the to-be-processed data according to different data features in the to-be-processed data, avoids the problem of poor data processing effect caused by a single processing mode, and effectively processes a large amount of and various types of data at the park level.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data processing method, apparatus, device, and storage medium. Background Technology

[0002] With the rapid development of zero-carbon solutions for integrated energy systems, the demand for data acquisition edge devices that can connect to various energy supply / consumption equipment (such as wind, solar, storage, charging, hydrogen, electricity loads, and heating and cooling loads) and multiple energy forms (such as cooling, heating, electricity, and hydrogen) is becoming increasingly urgent.

[0003] Typically, edge acquisition devices can read data from various power supply / consumption devices using communication protocols, and then transmit the read data to the cloud platform using communication technology. When the cloud platform needs to configure data points, the edge acquisition device can also use communication technology and communication protocols to configure the corresponding data of external devices, thereby realizing the transmission of data between the cloud platform and external devices, as well as the control of external devices by the cloud platform.

[0004] However, the characteristics of data from various energy-consuming and energy-supplying devices often differ significantly. Conventional data acquisition edge devices have relatively simple processing methods for the acquired data, resulting in many problems with the processed data, such as the presence of outliers, and the inability to effectively process large amounts of diverse data at the park level. Summary of the Invention

[0005] This invention provides a data processing method, apparatus, device, and storage medium to achieve effective processing of large amounts of diverse data.

[0006] In a first aspect, embodiments of the present invention provide a data processing method applied to a data acquisition edge device, the method comprising:

[0007] Acquire data to be processed, wherein the data to be processed is determined based on the raw data collected by the data acquisition edge device from a preset device;

[0008] The data characteristics of the data to be processed are determined. Based on the data characteristics, it is determined whether the data to be processed has a preset problem. If it does, a target algorithm is determined according to the data characteristics.

[0009] The preset problem includes at least one of data anomalies, data missing, and data noise;

[0010] The target algorithm is used to process the data to be processed to obtain the target data.

[0011] In a second aspect, embodiments of the present invention provide a data processing apparatus configured in a data acquisition edge device, the apparatus comprising:

[0012] A data acquisition module is used to acquire data to be processed, wherein the data to be processed is determined based on the raw data acquired by the data acquisition edge device from a preset device;

[0013] The target algorithm determination module is used to determine the data characteristics of the data to be processed, and to determine whether there is a preset problem in the data to be processed based on the data characteristics. If there is, the target algorithm is determined according to the data characteristics. The preset problem includes at least one of data anomaly, data missing, and data noise.

[0014] The data processing module is used to process the data to be processed using the target algorithm to obtain the target data.

[0015] Thirdly, embodiments of the present invention provide a data acquisition edge device, the data acquisition edge device comprising:

[0016] At least one processor;

[0017] and memory that is communicatively connected to at least one processor;

[0018] The memory stores a computer program that can be executed by at least one processor, which is executed by at least one processor to enable the at least one processor to perform the data processing method of the first aspect described above.

[0019] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing computer instructions for causing a processor to execute the data processing method of the first aspect described above.

[0020] The data processing solution provided in this embodiment of the invention acquires data to be processed, wherein the data to be processed is determined based on raw data collected by the data acquisition edge device from a preset device. The data characteristics of the data to be processed are determined, and based on the data characteristics, it is determined whether the data to be processed has a preset problem. If so, a target algorithm is determined based on the data characteristics. The preset problem includes at least one of data anomalies, data missing data, and data noise. The target algorithm is then used to process the data to be processed to obtain target data. By adopting the above technical solution, the data to be processed is first acquired, then the problem of the data to be processed is determined based on the data characteristics, and an algorithm for processing the problem is selected based on the data characteristics, i.e., a target algorithm is determined. Finally, the target algorithm is used to process the data to be processed to obtain target data. This approach, by targeting different data characteristics in the data to be processed, avoids the problem of poor data processing results caused by a single processing method. It can adapt to different types of preset devices and can effectively process large amounts and diverse types of data at the park level.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of a data processing method provided according to Embodiment 1 of the present invention;

[0024] Figure 2 This is a schematic diagram of the structure of a data acquisition edge device according to Embodiment 1 of the present invention;

[0025] Figure 3 This is a flowchart of a data processing method provided according to Embodiment 2 of the present invention;

[0026] Figure 4 This is a schematic diagram of the structure of a data processing apparatus according to Embodiment 3 of the present invention;

[0027] Figure 5 This is a schematic diagram of the structure of a data acquisition edge device according to Embodiment 4 of the present invention. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. In the description of this invention, unless otherwise stated, "a plurality of" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist; for example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0030] Example 1

[0031] Figure 1 This is a flowchart illustrating a data processing method according to Embodiment 1 of the present invention. This embodiment is applicable to processing data acquired from a preset device, which may be, for example, a power supply device and / or a power consumption device. The method can be executed by a data processing device, which can be implemented in hardware and / or software. This data processing device can be configured in a data acquisition edge device, which may consist of two or more physical entities, or it may consist of a single physical entity.

[0032] like Figure 1 As shown, the data processing method provided in Embodiment 1 of the present invention specifically includes the following steps:

[0033] S101. Obtain data to be processed, wherein the data to be processed is determined based on the raw data collected by the data acquisition edge device from the preset device.

[0034] Specifically, the data acquisition edge device can be located in the middle of the three-layer structure of the energy management system, namely the transmission layer. Its main function is to connect the upper application layer and the lower device layer. The data acquisition edge device can have north-south interfaces. The south-facing interface can acquire raw data and send control commands, while the north-facing interface can process the raw data, upload it, and receive application layer control commands. Figure 2 This is a schematic diagram of the structure of a data acquisition edge device, such as... Figure 2 As shown, the driver encapsulation layer of the data acquisition edge device can include peripheral modules and protocol stack modules. Peripheral modules can encapsulate interfaces according to the operating objects, such as universal asynchronous transceivers and network ports, thereby providing a unified interface for data processing modules. The protocol stack module can encapsulate protocols such as Modbus RTU, Modbus TCP, IEC 104, DL / T 645, and CJ / T 188. The functional module layer of the data acquisition edge device can include acquisition modules and forwarding modules. The acquisition module can detect and connect to the device in real time through ports, perform periodic and / or irregular data acquisition from lower-level devices, and modify the device's measurement point data after receiving instructions from the host computer. The forwarding module can realize data relay in the workstation background, completing the reception and forwarding of data within the acquisition unit, responding to background request commands, and forwarding data to other modules within the acquisition unit. The data processing layer of the data acquisition edge device can include a data storage module and a data processing module. The data storage module, after receiving data from the acquisition module, saves the relevant measurement point data to a preset storage area according to business logic. The data processing module periodically retrieves data from the preset storage area in segments, processes the data from the relevant measurement points, and then passes the processed data to the forwarding module. The data processed by the data processing module can be in a first-in, first-out (FIFO) data queue format.

[0035] In this embodiment, after the data acquisition edge device collects data from a preset device, such as an electrical appliance and / or a power generation device, it can first store the data in a preset storage area of ​​the data acquisition edge device. When the data needs to be processed, it can be read from the preset storage area. This data is the data to be processed. The original data can be understood as the data collected from the preset device that has not been processed.

[0036] S102. Determine the data characteristics of the data to be processed, and determine whether there is a preset problem in the data to be processed based on the data characteristics. If there is, determine the target algorithm according to the data characteristics. The preset problem includes at least one of data anomaly, data missing, and data noise.

[0037] In this embodiment, due to the instability of the preset equipment, such as electrical equipment and / or power generation equipment, the unprocessed data, i.e., the data to be processed read from the preset storage area, usually has some problems, such as data anomalies, data missing, and data noise. Therefore, the data characteristics of the data to be processed can be determined first, and then the above problems can be judged based on the data characteristics. If they exist, different algorithms, i.e., target algorithms, can be selected according to different data characteristics to solve the problems in the data to be processed. Among them, multiple problems can exist in the data to be processed at the same time. Data anomalies can include data values ​​that are too large or too small. Data missing can be understood as the data in the data to be processed being incomplete and missing. Data noise can be understood as the presence of interfering data in the data to be processed that deviates from the expected data value.

[0038] S103. Process the data to be processed using the target algorithm to obtain the target data.

[0039] In this embodiment, once the target algorithm is determined, the data to be processed can be processed using the target algorithm to obtain the processed data, i.e., the target data. As mentioned above, the target data can be transmitted to the forwarding module of the data acquisition edge device. The above steps 101-103 can be implemented by the processing module in the data acquisition edge device.

[0040] The data processing method provided in this embodiment of the invention acquires data to be processed, wherein the data to be processed is determined based on raw data collected by the data acquisition edge device from a preset device. The data characteristics of the data to be processed are determined, and based on the data characteristics, it is determined whether the data to be processed has a preset problem. If so, a target algorithm is determined according to the data characteristics. The preset problem includes at least one of data anomalies, data missing data, and data noise. The target algorithm is then used to process the data to be processed to obtain target data. In this embodiment of the invention, the data acquisition edge device first collects data to be processed from the preset device, then determines the problem of the data to be processed based on its data characteristics, and selects an algorithm to process the problem based on the data characteristics, i.e., determines the target algorithm. Finally, the target algorithm is used to process the data to obtain target data. This method specifically determines the algorithm for processing the data to be processed based on different data characteristics, avoiding the problem of poor data processing results caused by a single processing method. It can adapt to different types of preset devices and effectively process large amounts and diverse types of data at the park level.

[0041] Optionally, determining the data characteristics of the data to be processed, and judging whether the data to be processed has a preset problem based on the data characteristics, and if so, determining a target algorithm based on the data characteristics, includes: determining whether the data characteristics of the data to be processed have a first data characteristic; if the first data characteristic exists, judging whether the data to be processed has a data anomaly based on the data anomaly judgment characteristic in the first data characteristic, wherein the first data characteristic includes the data anomaly judgment characteristic and a first algorithm selection characteristic; if the data anomaly exists, determining a first target algorithm from a preset algorithm library based on the first algorithm selection characteristic; wherein processing the data to be processed using the target algorithm to obtain target data includes: processing the data to be processed using the first target algorithm to obtain first target data, wherein the first target algorithm belongs to the target algorithm, and the first target data belongs to the target data. The advantage of this setting is that by determining the data characteristics of the data to be processed, it is possible to identify whether the data to be processed has a data anomaly problem. When a data anomaly problem exists, the most suitable algorithm is selected from the algorithm library to handle the data anomaly problem.

[0042] Optionally, determining the data characteristics of the data to be processed, and judging whether the data to be processed has a preset problem based on the data characteristics, and if so, determining a target algorithm based on the data characteristics, includes: determining whether the data characteristics of the data to be processed have a second data characteristic; if the second data characteristic exists, judging whether the data to be processed has missing data based on the data missing judgment characteristic in the second data characteristic, wherein the second data characteristic includes the data missing judgment characteristic and the second algorithm selection characteristic; if the data is missing, determining a second target algorithm from a preset algorithm library based on the second algorithm selection characteristic; wherein, processing the data to be processed using the target algorithm to obtain target data includes: processing the data to be processed using the second target algorithm to obtain second target data, wherein the second target algorithm belongs to the target algorithm, and the second target data belongs to the target data. The advantage of this setup is that after handling the data anomaly problem of the data to be processed, it is still possible to identify whether there is a data missing problem based on the obtained data characteristics of the data to be processed. When a data missing problem exists, the most suitable algorithm is selected from the algorithm library to handle the data missing problem.

[0043] Optionally, determining the data characteristics of the data to be processed, and judging whether the data to be processed has a preset problem based on the data characteristics, and if so, determining the target algorithm based on the data characteristics, includes: determining whether the data characteristics of the data to be processed have a third data characteristic; if the third data characteristic exists, judging whether the data to be processed has data noise based on the data noise judgment characteristic in the third data characteristic, wherein the third data characteristic includes the data noise judgment characteristic and the third algorithm selection characteristic; if the data noise exists, determining a third target algorithm from a preset algorithm library based on the third algorithm selection characteristic; wherein, processing the data to be processed using the target algorithm to obtain target data includes: processing the data to be processed using the third target algorithm to obtain third target data, wherein the third target algorithm belongs to the target algorithm, and the third target data belongs to the target data. The advantage of this setting is that after handling the problems of data anomalies and data missingness, it is also possible to identify whether there is still a data noise problem based on the data characteristics of the obtained data to be processed. When the data noise problem exists, the most suitable algorithm is selected from the algorithm library to handle the data noise problem. Through three processing steps, the validity of the data is guaranteed.

[0044] Example 2

[0045] Figure 3 This is a flowchart of a data processing method provided in Embodiment 2 of the present invention. The technical solution of this embodiment is further optimized based on the above-mentioned optional technical solutions, and provides a specific method for processing data, such as... Figure 3 As shown, this embodiment is applicable to situations where the data to be processed may have multiple preset problems, namely data anomalies, data missing, and data noise, and these problems are judged and processed sequentially. Specifically, according to the order of processing, the data obtained after the previous processing can be used as the data to be processed in the next processing step.

[0046] like Figure 3 As shown, the data processing method provided in Embodiment 2 of the present invention specifically includes the following steps:

[0047] S201. Obtain the data to be processed.

[0048] Optionally, before acquiring the data to be processed, the method further includes: acquiring raw data from a preset device and acquiring the communication protocol used by the preset device; determining whether the communication protocol is consistent with a preset target communication protocol; if inconsistent, converting the raw data into data to be processed and storing the data to be processed in a preset storage area, wherein the data to be processed conforms to the communication rules in the preset target communication protocol. The advantage of this configuration is that it ensures that the data acquisition edge device is compatible with different communication protocols, enabling the data acquisition edge device to acquire data from the vast majority of electrical devices and power supply equipment on the market.

[0049] Specifically, before acquiring the data to be processed, raw data can be collected from electrical equipment and / or power generation equipment, and the corresponding communication protocol can be determined. Then, it is determined whether the communication protocol is consistent with a pre-set communication protocol, i.e., whether it is consistent with a preset target communication protocol. If they are consistent, the raw data can be written to a preset storage area. If they are inconsistent, the communication protocol of the raw data can be converted to the preset target communication protocol. Specifically, the communication format followed by the raw data can be changed to make the modified raw data conform to the communication rules in the preset target communication protocol, thereby obtaining the data to be processed and writing it to the preset storage area. The preset target communication protocol is generally set to a relatively common communication protocol, such as CJ / T 188.

[0050] Furthermore, storing the data to be processed in a preset storage area includes: determining whether the data to be processed is encrypted; if so, decrypting the data to be processed to obtain decrypted data to be processed, and storing the decrypted data to be processed in the preset storage area. The advantage of this setup is that decryption ensures the validity of the encrypted data acquired by the data acquisition edge device.

[0051] Specifically, due to different customer needs, some raw data may be confidential. This raw data is usually encrypted. The encrypted raw data will be marked in fixed data bits, such as set to 1. The encryption method will also be marked in other fixed data bits. Therefore, before storing the data to be processed into the preset storage area, it can be determined whether the data to be processed is encrypted. If so, the data to be processed can be decrypted first. After obtaining the decrypted data to be processed, it can be stored into the preset storage area. If not, the data to be processed can be directly stored into the preset storage area.

[0052] Optionally, the data to be processed retrieved from the preset storage area can also be encrypted to obtain encrypted data, thereby ensuring data security and confidentiality. The encryption method is not limited here, such as encryption using the SM4.0 cryptographic algorithm.

[0053] S202. Determine whether the data feature to be processed has the first data feature. If it exists, proceed to step 203; if it does not exist, proceed to step 205.

[0054] Specifically, data features of the data to be processed can be extracted to determine whether the data feature is a predefined data feature, i.e., the first data feature. If the data feature matches the predefined data feature, step 203 can be executed; otherwise, step 205 can be executed. Since multiple data sets containing different data features can exist for the same data problem, and different data processing algorithms are generally best suited to different data features, the first data feature can be set based on the data features most suitable for the data processing algorithm and the features corresponding to potential problems with the data.

[0055] S203. Based on the data anomaly judgment feature in the first data feature, determine whether there is a data anomaly in the data to be processed. If there is a data anomaly, proceed to step 204. If there is no data anomaly, proceed to step 205.

[0056] The first data feature includes the data anomaly judgment feature and the first algorithm selection feature.

[0057] Specifically, data features can include features for determining whether there are data problems, such as data anomaly detection features, as well as algorithm selection features for selecting the target algorithm, such as first algorithm selection features. Based on the data anomaly detection features, it can be determined whether there are data anomaly problems in the data to be processed.

[0058] Optionally, determining whether the data to be processed has data anomalies based on the data anomaly judgment features in the first data features includes: determining whether the data to be processed conforms to a first preset data distribution; determining a data anomaly judgment algorithm from multiple algorithms in a preset algorithm library that judge data anomalies based on the judgment result; and using the data anomaly judgment algorithm to determine a preset range; determining whether the value of the data to be processed exceeds the preset range, wherein the value of the data to be processed belongs to the data anomaly judgment features; if so, then determining that the data to be processed has data anomalies. The advantage of this setting is that by setting a preset range, abnormal data can be quickly identified from a large amount of data to be processed, and setting the corresponding preset range based on the data distribution characteristics of the data to be processed makes the setting of the preset range more reasonable, thereby improving the accuracy of identifying abnormal data.

[0059] For example, if the data to be processed is wind speed data, the first preset data distribution is a normal distribution. The algorithms in the preset algorithm library for judging data anomalies include the box plot method and the 3σ method. If the data to be processed follows a normal distribution, the data anomaly judgment algorithm can be determined to be the 3σ method. If the data to be processed does not follow a normal distribution, the data anomaly judgment algorithm can be determined to be the box plot method. If the normal range of data determined by the box plot method or the 3σ method is, i.e., the preset range is greater than 0 and less than 63, 0 is the upper limit of abnormal data and 63 is the lower limit of abnormal data, when the value of the data to be processed exceeds the preset range, such as when the data to be processed contains the value 70, it can be determined that the data to be processed is abnormal.

[0060] S204. Select features according to the first algorithm, determine the first target algorithm from the preset algorithm library, use the first target algorithm to process the data to be processed, obtain the first target data, and determine the first target data as the data to be processed.

[0061] Wherein, the first target algorithm belongs to the target algorithm, and the first target data belongs to the target data.

[0062] Specifically, for each data problem, a preset algorithm library stores multiple algorithms, which can contain algorithms for handling multiple different data problems. Based on the first algorithm selection feature, a first target algorithm can be determined from the preset algorithm library. The first target algorithm is the algorithm that can solve the data anomaly problem. The data obtained after the data to be processed by the first target algorithm is the first target data.

[0063] Optionally, determining the first target algorithm from a preset algorithm library based on the first algorithm selection feature includes: determining whether the data to be processed belongs to a first preset data type, wherein the first algorithm selection feature includes the data type of the data to be processed; if it belongs, determining a first data anomaly handling algorithm from multiple algorithms in the preset algorithm library that handle data anomalies; if it does not belong, determining a second data anomaly handling algorithm from multiple algorithms in the preset algorithm library that handle data anomalies, wherein both the first data anomaly handling algorithm and the second data anomaly handling algorithm belong to the first target algorithm. The advantage of this configuration is that determining the data type of the data to be processed allows for the accurate selection of the optimal algorithm from multiple algorithms for handling data anomalies to address the data anomaly problem.

[0064] For example, if the data to be processed is current data, the first preset data type is power data, such as current and voltage data, and the second preset data type is process data, such as temperature, pressure, and flow rate data, then if the data to be processed belongs to the first preset data type, outliers in the data to be processed can be replaced with the average value of the data to be processed within a set time period. When the data to be processed belongs to the second preset data type, outliers in the data to be processed can be replaced with the valid value of the data to be processed at a set time. Here, the time corresponding to the set time period and the set time are both earlier than the time corresponding to the outlier data. The above method of processing the data to be processed corresponds to the processing method of the algorithm for handling data anomalies in the preset algorithm library.

[0065] S205. Determine whether the data features of the data to be processed have a second data feature. If they do, proceed to step 206; otherwise, proceed to step 208.

[0066] Specifically, as mentioned above, by determining whether the features of the data to be processed are consistent with the preset second data features, it can be determined whether the second data features exist in the data to be processed.

[0067] S206. Based on the data missing judgment feature in the second data feature, determine whether there is data missing in the data to be processed. If there is data missing, proceed to step 207. If there is no data missing, proceed to step 208.

[0068] The second data feature includes the data missing judgment feature and the second algorithm selection feature.

[0069] Specifically, the second data feature in this step may include a data missing feature for determining whether the data is missing and a second algorithm selection feature for selecting the target algorithm.

[0070] Optionally, determining whether the data to be processed is missing based on the data missing judgment feature in the second data feature includes: determining whether the timestamps of the data to be processed are discontinuous, and / or whether the number of data to be processed is less than a first preset number, wherein the time displayed by the timestamps of the data to be processed and / or the number of data to be processed are data missing judgment features; if so, it is determined that the data to be processed is missing. The advantage of this setting is that by further determining whether the data to be processed is continuous or the number of data to be processed, it is possible to accurately determine whether the data to be processed is missing.

[0071] For example, if the data to be processed is wind speed data, and the wind turbine, i.e. the power generation equipment, collects wind speed data once every 10 milliseconds, and the data acquisition edge device obtains raw data once every 10 seconds, then the number of raw data collected every 10 seconds should be 1000. Therefore, the first preset number can be set to 1000. If the timestamps of the data to be processed are not continuous, i.e. the interval between timestamps does not meet 10 milliseconds, and / or the number of data to be processed is less than 1000, then it can be determined that there is data missing in the data to be processed. If the timestamps of the data to be processed are continuous and the number of data to be processed is 1000, then it can be determined that there is no data missing in the data to be processed.

[0072] S207. Based on the features selected by the second algorithm, determine the second target algorithm from the preset algorithm library, use the second target algorithm to process the data to be processed, obtain the second target data, and determine the second target data as the data to be processed.

[0073] Wherein, the second target algorithm belongs to the target algorithm, and the second target data belongs to the target data.

[0074] Optionally, determining the second target algorithm from a preset algorithm library based on the second algorithm selection features includes: determining whether the quantity of data to be processed is less than a second preset quantity, the ratio of the difference between the first preset quantity and the quantity of data to be processed to the first preset quantity is less than a preset ratio, and the data to be processed follows a second preset data distribution. The second algorithm selection features include the data distribution type followed by the data to be processed, the quantity of data to be processed, and the ratio of the difference between the first preset quantity and the quantity of data to be processed to the first preset quantity. If yes, a first data missing processing algorithm is determined from multiple algorithms in the preset algorithm library that handle data missing data; if no, a second data missing processing algorithm is determined from multiple algorithms in the preset algorithm library that handle data missing data. Both the first and second data missing processing algorithms are included in the second target algorithm. The advantage of this configuration is that by further judging the characteristics of the data to be processed, the optimal algorithm can be quickly determined from multiple algorithms capable of handling data missing data.

[0075] For example, if the data to be processed is wind speed data, the second preset quantity is 2000, the preset ratio is 20%, the second preset data distribution is a concentrated distribution, and the algorithms in the preset algorithm library for handling missing data include mean imputation and regression imputation, then if the number of data to be processed is 900, which is less than the second preset quantity, the difference between the number of data to be processed (900) and the first preset quantity (1000) is 10% of the first preset quantity (1000), which is less than the preset ratio of 20%, and the data to be processed follows a concentrated distribution, then the first data missing processing algorithm for the data to be processed can be determined to be mean imputation.

[0076] Optionally, if the characteristics corresponding to the first data missing processing algorithm mentioned above are not met, but the data to be processed follows a normal distribution and multiple auxiliary variables can be determined from the data to be processed, then the second data missing processing algorithm for the data to be processed can be determined to be the regression imputation method. In this case, there may be independent variables in the data to be processed that can predict the missing data, and the auxiliary variables can be understood as values ​​that can indirectly reflect the size of the missing data, such as the average value of the data to be processed.

[0077] S208. Determine whether there is a third data feature in the data features to be processed. If there is a third data feature, proceed to step 209. If there is no third data feature, end the process.

[0078] S209. Based on the data noise judgment feature in the third data feature, determine whether there is data noise in the data to be processed. If there is data noise, proceed to step 210. If there is no data noise, end the process.

[0079] The third data feature includes the data noise judgment feature and the third algorithm selection feature.

[0080] Specifically, the second data feature in this step may include a data noise judgment feature for determining whether the data has data noise and a third algorithm selection feature for selecting the target algorithm.

[0081] Optionally, determining whether the data to be processed contains data noise based on the data noise judgment feature in the third data feature includes: determining whether there are harmonics in the data waveform diagram corresponding to the data to be processed, wherein the data waveform diagram corresponding to the data to be processed belongs to the data noise judgment feature; if so, it is determined that the data to be processed contains data noise. The advantage of this setting is that by judging whether there are harmonics in the data waveform diagram, it is intuitively determined whether there is data noise in the data to be processed.

[0082] For example, software such as MATLAB can be used to plot the waveform of the data to be processed. The presence of harmonics in the waveform can be identified manually or by a pre-set algorithm. If harmonics are present, it can be determined that the data to be processed contains noise.

[0083] S210. Based on the features selected by the third algorithm, determine the third target algorithm from the preset algorithm library, use the third target algorithm to process the data to be processed, and obtain the third target data.

[0084] The third target algorithm belongs to the target algorithm, and the third target data belongs to the target data.

[0085] Optionally, determining the third target algorithm from a preset algorithm library based on the third algorithm selection features includes: determining whether the sampling period of the data to be processed is less than a preset period and the pure time delay constant of the data to be processed is greater than a preset constant, wherein the third algorithm selection features include the sampling period of the data to be processed and the pure time delay constant of the data to be processed; if yes, then determining a first data noise processing algorithm from multiple algorithms for processing data noise in the preset algorithm library; if no, then determining a second data noise processing algorithm from multiple algorithms for processing data noise in the preset algorithm library, wherein both the first data noise processing algorithm and the second data noise processing algorithm belong to the third target algorithm. The advantage of this setup is that by judging the sampling period and the pure time delay constant, the optimal algorithm for solving the data noise problem in the data to be processed can be accurately determined.

[0086] For example, if the data to be processed is wind speed data, the preset period is 15 seconds to collect the data, the preset constant is 10 seconds, and the preset algorithm library includes data noise processing algorithms such as weighted recursive average filtering, first-order low-pass filtering, and amplitude limiting and de-jitter filtering algorithms, then if the sampling period of the data to be processed is once every 10 seconds, which is less than once every 15 seconds, and the pure time lag constant of the data to be processed is 20 seconds, which is greater than the preset constant of 10 seconds, then the first data noise processing algorithm for the data to be processed can be determined to be the weighted recursive average filtering algorithm.

[0087] Optionally, if the characteristics corresponding to the first data noise processing algorithm mentioned above are not met, but the sampling period of the data to be processed is once every 20 seconds, which is greater than the preset period of once every 15 seconds, and the amount of high-frequency disturbance noise data in the data to be processed is greater than the first preset value, then the second data noise processing algorithm of the data to be processed can be determined to be a first-order low-pass filtering algorithm.

[0088] Optionally, if the characteristics corresponding to the first and second data noise processing algorithms mentioned above are not met, but the numerical changes of the data to be processed are slow, such as if the data to be processed is temperature data and there is impulsive interference, then the third data noise processing algorithm for the data to be processed can be determined to be an amplitude limiting and de-jitter filtering algorithm. The third data noise processing algorithm also belongs to the third target algorithm.

[0089] Optionally, after obtaining the target data, the method further includes: determining whether the preset target communication protocol used by the data acquisition edge device is consistent with the communication protocol used by the preset device; if they are inconsistent, converting the instruction data of the data acquisition edge into data to be sent; if they are consistent, determining the instruction data as data to be sent; encrypting the data to be sent to obtain encrypted data to be sent, and sending the encrypted data to be sent to the preset device, wherein the data to be sent conforms to the communication rules in the communication protocol used by the preset device.

[0090] Specifically, the communication protocol used by the target data in the data acquisition edge device (i.e., the preset target communication protocol) can be converted into a communication protocol used by the preset device. This yields data to be sent that matches the communication protocol used by the preset device. Then, based on business requirements, such as data confidentiality requirements, it can be determined whether the data to be sent needs to be encrypted. If so, the data to be sent can be encrypted. After obtaining the encrypted data to be sent, it can be sent to the preset device to achieve bidirectional communication between the preset device and the data acquisition edge device. The encryption algorithm can be the SM4 cryptographic algorithm, etc. The instruction data can be, for example, instructions issued by the data acquisition edge device itself, or issued by a cloud platform connected to the data acquisition edge device and forwarded by the data acquisition edge device, used to control or configure the preset device.

[0091] The data processing method provided in this embodiment of the invention involves a data acquisition edge device first collecting data to be processed from a preset device. Then, based on the data characteristics of the data to be processed, it is determined whether there are problems such as data anomalies, data missing, and data noise in the data to be processed. Based on the data characteristics, the optimal algorithm for processing the problem is selected, that is, the target algorithm for each data problem is determined, and the target algorithm is used to process the data to be processed, thus obtaining the target data. By utilizing different data characteristics, the optimal algorithm can be selected from multiple algorithms for processing the same data problem. Furthermore, by making multiple judgments on the possible data problems, the correctness of the data is further guaranteed.

[0092] Example 3

[0093] Figure 4This is a schematic diagram of a data processing apparatus provided in Embodiment 3 of the present invention. Figure 4 As shown, the device includes: a data acquisition module 301, a target algorithm determination module 302, and a data processing module 303, wherein:

[0094] A data acquisition module is used to acquire data to be processed, wherein the data to be processed is determined based on the raw data acquired by the data acquisition edge device from a preset device;

[0095] The target algorithm determination module is used to determine the data characteristics of the data to be processed, and to determine whether there is a preset problem in the data to be processed based on the data characteristics. If there is, the target algorithm is determined according to the data characteristics. The preset problem includes at least one of data anomaly, data missing, and data noise.

[0096] The data processing module is used to process the data to be processed using the target algorithm to obtain the target data.

[0097] The data processing apparatus provided in this embodiment of the invention first collects data to be processed from a preset device using a data acquisition edge device. Then, it determines the problem of the data to be processed based on the data characteristics of the data to be processed, and selects an algorithm to process the problem based on the data characteristics, i.e., determines the target algorithm. Finally, it processes the data to be processed using the target algorithm to obtain the target data. By targeting different data characteristics in the data to be processed, the algorithm for processing the data to be processed is determined in a targeted manner, avoiding the problem of poor data processing effect caused by a single processing method. It can adapt to different types of preset devices and effectively process a large amount and variety of data at the park level.

[0098] Optionally, the target algorithm determination module includes:

[0099] The first feature determination unit is used to determine whether the data features of the data to be processed exist as a first data feature;

[0100] A data anomaly judgment unit is used to determine whether the data to be processed has a data anomaly based on the data anomaly judgment feature in the first data feature if the judgment result returned by the first feature judgment unit is that the first data feature exists. The first data feature includes the data anomaly judgment feature and the first algorithm selection feature.

[0101] The first target algorithm determination unit is used to determine the first target algorithm from a preset algorithm library based on the first algorithm selection feature when the judgment result returned by the data anomaly judgment unit indicates that the data anomaly exists.

[0102] Optionally, the data processing module includes:

[0103] A first processing unit is configured to process the data to be processed using the first target algorithm to obtain first target data, wherein the first target algorithm belongs to the target algorithm and the first target data belongs to the target data.

[0104] Optionally, determining whether the data to be processed has data anomalies based on the data anomaly judgment features in the first data features includes: determining whether the data to be processed conforms to a first preset data distribution; determining a data anomaly judgment algorithm from multiple algorithms in a preset algorithm library that judge data anomalies based on the judgment result; and determining a preset range using the data anomaly judgment algorithm; determining whether the value of the data to be processed exceeds the preset range, wherein the value of the data to be processed belongs to the data anomaly judgment features; if so, determining that the data to be processed has data anomalies.

[0105] The step of determining a first target algorithm from a preset algorithm library based on the first algorithm selection feature includes: determining whether the data to be processed belongs to a first preset data type, wherein the first algorithm selection feature includes the data type of the data to be processed; if it belongs to the data type, determining a first data anomaly handling algorithm from a plurality of algorithms for handling data anomalies in the preset algorithm library; if it does not belong to the data type, determining a second data anomaly handling algorithm from a plurality of algorithms for handling data anomalies in the preset algorithm library, wherein both the first data anomaly handling algorithm and the second data anomaly handling algorithm belong to the first target algorithm.

[0106] Optionally, the target algorithm determination module may also include:

[0107] The second feature determination unit is used to determine whether the data features of the data to be processed have a second data feature;

[0108] A data missing judgment unit is used to determine whether the data to be processed has data missing based on the data missing judgment feature in the second data feature if the judgment result returned by the second feature judgment unit is that the second data feature exists. The second data feature includes the data missing judgment feature and the second algorithm selection feature.

[0109] The second target algorithm determination unit is used to determine the second target algorithm from a preset algorithm library based on the second algorithm selection feature when the judgment result returned by the data missing judgment unit indicates that the data is missing.

[0110] Optionally, the data processing module may also include:

[0111] The second processing unit is used to process the data to be processed using the second target algorithm to obtain second target data, wherein the second target algorithm belongs to the target algorithm and the second target data belongs to the target data.

[0112] Optionally, determining whether the data to be processed has missing data based on the data missing judgment feature in the second data feature includes: determining whether the timestamps of the data to be processed are discontinuous, and / or whether the number of the data to be processed is less than a first preset number, wherein the time displayed by the timestamps of the data to be processed and / or the number of the data to be processed belong to the data missing judgment feature; if so, it is determined that the data to be processed has missing data.

[0113] The step of determining a second target algorithm from a preset algorithm library based on the second algorithm selection features includes: determining whether the number of data to be processed is less than a second preset number, the ratio of the difference between the first preset number and the number of data to be processed to the first preset number is less than a preset ratio, and the data to be processed follows a second preset data distribution. The second algorithm selection features include the data distribution type followed by the data to be processed, the number of data to be processed, and the ratio of the difference between the first preset number and the number of data to be processed to the first preset number. If yes, a first data missing processing algorithm is determined from multiple algorithms in the preset algorithm library that handle data missing data. If no, a second data missing processing algorithm is determined from multiple algorithms in the preset algorithm library that handle data missing data. Both the first data missing processing algorithm and the second data missing processing algorithm are included in the second target algorithm.

[0114] Optionally, the target algorithm determination module may also include:

[0115] The third feature determination unit is used to determine whether the data features of the data to be processed have a third data feature;

[0116] A data noise judgment unit is used to determine whether the data to be processed has data noise based on the data noise judgment feature in the third data feature if the judgment result returned by the third feature judgment unit is that the third data feature exists. The third data feature includes the data noise judgment feature and the third algorithm selection feature.

[0117] The third target algorithm determination unit is used to determine the third target algorithm from the preset algorithm library based on the third algorithm selection feature if the judgment result returned by the data noise judgment is that the data noise exists, wherein the third target algorithm belongs to the target algorithm.

[0118] Optionally, the data processing module may also include:

[0119] The third processing unit is used to process the data to be processed using the third target algorithm to obtain the third target data, wherein the third target algorithm belongs to the target algorithm and the third target data belongs to the target data.

[0120] Optionally, determining whether the data to be processed has data noise based on the data noise judgment feature in the third data feature includes: determining whether there are harmonics in the data waveform diagram corresponding to the data to be processed, wherein the data waveform diagram corresponding to the data to be processed belongs to the data noise judgment feature; if so, it is determined that the data to be processed has data noise.

[0121] The step of determining a third target algorithm from a preset algorithm library based on the third algorithm selection features includes: determining whether the sampling period of the data to be processed is less than a preset period and the pure time lag constant of the data to be processed is greater than a preset constant, wherein the third algorithm selection features include the sampling period of the data to be processed and the pure time lag constant of the data to be processed; if yes, then determining a first data noise processing algorithm from multiple algorithms for processing data noise in the preset algorithm library; if no, then determining a second data noise processing algorithm from multiple algorithms for processing data noise in the preset algorithm library, wherein both the first data noise processing algorithm and the second data noise processing algorithm belong to the third target algorithm.

[0122] Optionally, the device may also include:

[0123] The data acquisition module is used to collect raw data from a preset device and to obtain the communication protocol used by the preset device;

[0124] The protocol determination module is used to determine whether the communication protocol is consistent with the preset target communication protocol;

[0125] The raw data processing module is used to convert the raw data into data to be processed if the judgment result returned by the protocol judgment module is inconsistent, and to store the data to be processed in a preset storage area, wherein the data to be processed conforms to the communication rules in the preset target communication protocol.

[0126] Optionally, storing the data to be processed into a preset storage area includes: determining whether the data to be processed is encrypted data; if so, performing a decryption operation on the data to be processed to obtain decrypted data to be processed, and storing the decrypted data to be processed into the preset storage area.

[0127] Optionally, the device may also include:

[0128] The data to be sent determination module is used to determine, after the target data is obtained, whether the preset target communication protocol used by the data acquisition edge device is consistent with the communication protocol used by the preset device. If they are inconsistent, the instruction data of the data acquisition edge is converted into data to be sent. If they are consistent, the instruction data is determined as data to be sent.

[0129] The data forwarding module is used to encrypt the data to be sent, obtain encrypted data to be sent, and send the encrypted data to be sent to the preset device, wherein the data to be sent conforms to the communication rules in the communication protocol used by the preset device.

[0130] The data processing apparatus provided in the embodiments of the present invention can execute the data processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the method.

[0131] Example 4

[0132] Figure 5 A schematic diagram of a data acquisition edge device 40, which can be used to implement embodiments of the present invention, is shown. The data acquisition edge device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0133] like Figure 5 As shown, the data acquisition edge device 40 includes at least one processor 41 and a memory, such as a read-only memory (ROM) 42 or a random access memory (RAM) 43, communicatively connected to the at least one processor 41. The memory stores computer programs executable by the at least one processor. The processor 41 can perform various appropriate actions and processes based on the computer program stored in the ROM 42 or loaded from storage unit 48 into the RAM 43. The RAM 43 can also store various programs and data required for the operation of the data acquisition edge device 40. The processor 41, ROM 42, and RAM 43 are interconnected via a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0134] Multiple components in the data acquisition edge device 40 are connected to the I / O interface 45, including: an input unit 46, such as a keyboard, mouse, etc.; an output unit 47, such as various types of displays, speakers, etc.; a storage unit 48, such as a disk, optical disk, etc.; and a communication unit 49, such as a network card, modem, wireless transceiver, etc. The communication unit 49 allows the data acquisition edge device 40 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0135] Processor 41 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 41 performs the various methods and processes described above, such as data processing methods.

[0136] In some embodiments, the data processing method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 48. In some embodiments, part or all of the computer program may be loaded and / or installed on the data acquisition edge device 40 via ROM 42 and / or communication unit 49. When the computer program is loaded into RAM 43 and executed by processor 41, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, processor 41 may be configured to perform the data processing method by any other suitable means (e.g., by means of firmware).

[0137] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0138] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0139] The computer equipment provided above can be used to execute the data processing method provided in any of the above embodiments, and has corresponding functions and beneficial effects.

[0140] Example 5

[0141] In the context of this invention, the computer-readable storage medium may be a tangible medium, and the computer-executable instructions, when executed by a computer processor, are used to perform a method of data processing applied to a data acquisition edge device, the method comprising:

[0142] Acquire data to be processed, wherein the data to be processed is determined based on the raw data collected by the data acquisition edge device from a preset device;

[0143] The data characteristics of the data to be processed are determined, and based on the data characteristics, it is determined whether the data to be processed has a preset problem. If it does, a target algorithm is determined according to the data characteristics. The preset problem includes at least one of data anomaly, data missing, and data noise.

[0144] The target algorithm is used to process the data to be processed to obtain the target data.

[0145] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by, or in conjunction with, an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0146] The computer equipment provided above can be used to execute the data processing method provided in any of the above embodiments, and has corresponding functions and beneficial effects.

[0147] It is worth noting that in the embodiments of the above-mentioned data processing device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy distinction between each other and are not used to limit the scope of protection of the present invention.

[0148] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A data processing method, characterized in that, The method, applied to data acquisition edge devices, includes: Acquire data to be processed, wherein the data to be processed is determined based on the raw data collected by the data acquisition edge device from a preset device; The data characteristics of the data to be processed are determined, and based on the data characteristics, it is determined whether the data to be processed has a preset problem. If it does, a target algorithm is determined according to the data characteristics. The preset problem includes at least one of data anomaly, data missing, and data noise. The target algorithm is used to process the data to be processed to obtain the target data; The process of determining the data characteristics of the data to be processed, judging whether the data to be processed has a preset problem based on the data characteristics, and if so, determining the target algorithm based on the data characteristics includes: Determine whether the data features of the data to be processed contain a second data feature; If the second data feature exists, then based on the data missing judgment feature in the second data feature, it is determined whether the data to be processed has data missing, wherein the second data feature includes the data missing judgment feature and the second algorithm selection feature; If the data is missing, then according to the second algorithm selection feature, the second target algorithm is determined from the preset algorithm library; The step of processing the data to be processed using the target algorithm to obtain target data includes: processing the data to be processed using the second target algorithm to obtain second target data, wherein the second target algorithm belongs to the target algorithm and the second target data belongs to the target data; The step of determining whether the data to be processed has missing data based on the data missing judgment feature in the second data feature includes: Determine whether the timestamps of the data to be processed are discontinuous, and / or whether the number of data to be processed is less than a first preset number, wherein the timestamps of the data to be processed and / or the number of data to be processed are data missing judgment features; If so, then it is determined that the data to be processed is missing. The step of selecting features based on the second algorithm and determining the second target algorithm from a preset algorithm library includes: Determine whether the number of data to be processed is less than a second preset number, the ratio of the difference between the first preset number and the number of data to be processed to the first preset number is less than a preset ratio, and the data to be processed follows a second preset data distribution. The second algorithm selection features include the data distribution type followed by the data to be processed, the number of data to be processed, and the ratio of the difference between the first preset number and the number of data to be processed to the first preset number. If yes, then a first data missing processing algorithm is determined from multiple algorithms in the preset algorithm library for processing the missing data; if no, then a second data missing processing algorithm is determined from multiple algorithms in the preset algorithm library for processing the missing data, wherein both the first data missing processing algorithm and the second data missing processing algorithm belong to the second target algorithm.

2. The method according to claim 1, characterized in that, The process of determining the data characteristics of the data to be processed, judging whether the data to be processed has a preset problem based on the data characteristics, and if so, determining the target algorithm based on the data characteristics includes: Determine whether the data features of the data to be processed contain the first data feature; If the first data feature exists, then based on the data anomaly judgment feature in the first data feature, it is determined whether the data to be processed has a data anomaly, wherein the first data feature includes the data anomaly judgment feature and the first algorithm selection feature; If the data anomaly exists, then based on the feature selection of the first algorithm, the first target algorithm is determined from the preset algorithm library; The step of processing the data to be processed using the target algorithm to obtain target data includes: processing the data to be processed using the first target algorithm to obtain first target data, wherein the first target algorithm belongs to the target algorithm and the first target data belongs to the target data.

3. The method according to claim 2, characterized in that, The step of determining whether the data to be processed has data anomalies based on the data anomaly judgment features in the first data features includes: Determine whether the data to be processed conforms to a first preset data distribution. Based on the determination result, select a data anomaly judgment algorithm from multiple algorithms in a preset algorithm library that determine the data anomaly, and use the data anomaly judgment algorithm to determine a preset range. Determine whether the value of the data to be processed exceeds the preset range, wherein the value of the data to be processed belongs to the data anomaly judgment feature; If so, then it is determined that the data to be processed contains data anomalies; The step of selecting features based on the first algorithm and determining the first target algorithm from a preset algorithm library includes: Determine whether the data to be processed belongs to a first preset data type, wherein the first algorithm selection feature includes the data type of the data to be processed; If it belongs to the target algorithm, a first data anomaly handling algorithm is determined from multiple algorithms in the preset algorithm library that handle the data anomaly. If it does not belong to the target algorithm, a second data anomaly handling algorithm is determined from multiple algorithms in the preset algorithm library that handle the data anomaly. Both the first data anomaly handling algorithm and the second data anomaly handling algorithm belong to the first target algorithm.

4. The method according to claim 1, characterized in that, The process of determining the data characteristics of the data to be processed, judging whether the data to be processed has a preset problem based on the data characteristics, and if so, determining the target algorithm based on the data characteristics includes: Determine whether the data features of the data to be processed contain a third data feature; If the third data feature exists, then based on the data noise judgment feature in the third data feature, it is determined whether the data to be processed has data noise, wherein the third data feature includes the data noise judgment feature and the third algorithm selection feature; If the data noise exists, then according to the features selected by the third algorithm, a third target algorithm is determined from the preset algorithm library; The step of processing the data to be processed using the target algorithm to obtain target data includes: processing the data to be processed using the third target algorithm to obtain third target data, wherein the third target algorithm belongs to the target algorithm and the third target data belongs to the target data.

5. The method according to claim 4, characterized in that, The step of determining whether the data to be processed contains data noise based on the data noise judgment feature in the third data feature includes: Determine whether there are harmonics in the data waveform diagram corresponding to the data to be processed, wherein the data waveform diagram corresponding to the data to be processed belongs to the data noise judgment feature; If so, then it is determined that the data to be processed contains data noise; The step of selecting features based on the third algorithm and determining the third target algorithm from a preset algorithm library includes: Determine whether the sampling period of the data to be processed is less than a preset period and the pure time delay constant of the data to be processed is greater than a preset constant, wherein the third algorithm selection features include the sampling period of the data to be processed and the pure time delay constant of the data to be processed; If yes, then a first data noise processing algorithm is determined from multiple algorithms in the preset algorithm library for processing the data noise; if no, then a second data noise processing algorithm is determined from multiple algorithms in the preset algorithm library for processing the data noise, wherein both the first data noise processing algorithm and the second data noise processing algorithm belong to the third target algorithm.

6. The method according to claim 1, characterized in that, Before acquiring the data to be processed, the process also includes: Collect raw data from a preset device and obtain the communication protocol used by the preset device; Determine whether the communication protocol is consistent with the preset target communication protocol; If there is a discrepancy, the original data is converted into data to be processed and stored in a preset storage area, wherein the data to be processed conforms to the communication rules in the preset target communication protocol.

7. The method according to claim 6, characterized in that, The step of storing the data to be processed into a preset storage area includes: Determine whether the data to be processed is encrypted. If so, decrypt the data to be processed to obtain the decrypted data to be processed, and store the decrypted data to be processed in a preset storage area.

8. The method according to claim 1, characterized in that, After obtaining the target data, the process also includes: Determine whether the preset target communication protocol used by the data acquisition edge device is consistent with the communication protocol used by the preset device. If they are inconsistent, convert the instruction data of the data acquisition edge into data to be sent. If they are consistent, determine the instruction data as data to be sent. The data to be sent is encrypted to obtain encrypted data to be sent, and the encrypted data to be sent to the preset device, wherein the data to be sent conforms to the communication rules in the communication protocol used by the preset device.

9. A data processing apparatus, characterized in that, Configured on a data acquisition edge device, the device includes: A data acquisition module is used to acquire data to be processed, wherein the data to be processed is determined based on the raw data acquired by the data acquisition edge device from a preset device; The target algorithm determination module is used to determine the data characteristics of the data to be processed, and to determine whether there is a preset problem in the data to be processed based on the data characteristics. If there is, the target algorithm is determined according to the data characteristics. The preset problem includes at least one of data anomaly, data missing, and data noise. The data processing module is used to process the data to be processed using the target algorithm to obtain the target data; The target algorithm determination module further includes: The second feature determination unit is used to determine whether the data features of the data to be processed have a second data feature; A data missing judgment unit is used to determine whether the data to be processed is missing based on the data missing judgment feature in the second data feature if the second data feature exists, wherein the second data feature includes the data missing judgment feature and the second algorithm selection feature; The second target algorithm determination unit is used to determine the second target algorithm from a preset algorithm library if the data is missing, based on the second algorithm selection feature. The data processing module further includes: The second processing unit is used to process the data to be processed using the second target algorithm to obtain second target data, wherein the second target algorithm belongs to the target algorithm and the second target data belongs to the target data; The data missing determination unit is specifically used to: determine whether the timestamps of the data to be processed are discontinuous, and / or whether the number of the data to be processed is less than a first preset number, wherein the timestamps of the data to be processed and / or the number of the data to be processed are data missing determination features; if so, it is determined that the data to be processed has data missing. The second target algorithm determining unit is specifically used to: determine whether the quantity of data to be processed is less than a second preset quantity, the ratio of the difference between the first preset quantity and the quantity of data to be processed to the first preset quantity is less than a preset ratio, and the data to be processed follows a second preset data distribution, wherein the second algorithm selection features include the data distribution type followed by the data to be processed, the quantity of data to be processed, and the ratio of the difference between the first preset quantity and the quantity of data to be processed to the first preset quantity; if yes, then determine a first data missing processing algorithm from multiple algorithms in a preset algorithm library for processing data missing; if no, then determine a second data missing processing algorithm from multiple algorithms in a preset algorithm library for processing data missing, wherein both the first data missing processing algorithm and the second data missing processing algorithm belong to the second target algorithm.

10. A data acquisition edge device, characterized in that, The data acquisition edge device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor to enable the at least one processor to perform the data processing method according to any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the data processing method of any one of claims 1-8.

Images