A method and device for data cleaning processing, electronic equipment, and storage medium

Abstract

The present application discloses a method and device for data cleaning and processing, an electronic device, and a storage medium, wherein the method for data cleaning and processing includes: obtaining data from an original data source; converting the data from the original data source into data to be processed with a unified caliber according to a preset conversion rule; cleaning the data to be processed according to a pre-configured logical process arrangement and data logic configuration through a stream computing model to obtain result data; analyzing whether there is data in the result data that does not meet the corresponding expected result based on pre-configured alarm information and verification information; if it is analyzed that there is no data in the result data that does not meet the corresponding expected result, outputting the result data to a target table; and visually displaying the result data in the target table. Thus, a simple and accurate data cleaning and processing method is realized.

Description

Technical Field

[0001] The present application relates to the technical field of data cleaning, and in particular to a method and device for data cleaning processing, an electronic device, and a storage medium. Background Art

[0002] In the current actual industrial production process, many industries will collect production information of equipment, which will generate a large amount of data in different structural forms. These data often need further cleaning and processing, such as eliminating redundant data, supplementing missing data, etc. The cleaned data can be visualized to meet the needs of enterprise data analysis and decision-making.

[0003] The existing method for cleaning data mainly configures corresponding data cleaning rules for different data sources, and compiles corresponding scripts based on the corresponding data cleaning rules to realize data cleaning.

[0004] However, the existing cleaning process is a whole process. If there is a problem, the entire cleaning process needs to be reconfigured and compiled, which is too cumbersome. In addition, different data sources are configured with different data rules, which not only takes a lot of time and cost, but also is quite inconvenient when the data rules need to be adjusted. Summary of the invention

[0005] Based on the above-mentioned deficiencies in the prior art, the present application provides a method and device for data cleaning processing, an electronic device, and a storage medium to solve the problem that the existing methods are too cumbersome.

[0006] In order to achieve the above objectives, this application provides the following technical solutions:

[0007] The first aspect of the present application provides a method for data cleaning processing, comprising:

[0008] Get data from the original data source;

[0009] Convert the data of the original data source into the data to be processed in a unified manner according to the preset conversion rules;

[0010] The data to be processed is cleaned according to the pre-configured logical process arrangement and data logic configuration through the stream computing model to obtain result data;

[0011] Based on the pre-configured alarm information and verification information, analyzing whether there is data in the result data that does not meet the corresponding expected result;

[0012] If it is analyzed that there is no data in the result data that does not meet the corresponding expected result, the result data is output to the target table;

[0013] The result data in the target table is visually displayed.

[0014] Optionally, in the above-mentioned data cleaning method, the step of obtaining data from an original data source includes:

[0015] Collect data from multiple data sources;

[0016] The data of each of the data sources is stored according to a preset storage format to obtain multiple original data sources.

[0017] Optionally, in the above-mentioned data cleaning method, converting the data of the original data source into data to be processed with a unified caliber according to a preset conversion rule includes:

[0018] Filtering the original data source to be processed from the multiple original data sources;

[0019] Determining the data type of the data of the original data source to be processed according to the metadata of the data of the original data source to be processed;

[0020] According to the pre-established mapping relationship between the data types of the various fields of the original data sources to be processed and the various fields of the target data sources, the data types of the data of the various original data sources to be processed are converted to obtain unified data to be processed.

[0021] Optionally, in the above-mentioned data cleaning method, after converting the data of the original data source into the data to be processed with a unified caliber according to the preset conversion rule, it also includes:

[0022] The data to be processed is locally persisted.

[0023] Optionally, in the above-mentioned data cleaning method, before the data to be processed is cleaned according to the pre-configured logical process arrangement and data logic configuration through the stream computing model to obtain the result data, it also includes:

[0024] In response to a configuration operation of a user, configure the logic flow arrangement, the data logic configuration, and the alarm information of the stream computing model;

[0025] Trigger any process node of the flow computing model, clean the debugging data according to the pre-configured logical process arrangement and data logic configuration, and obtain the cleaning result of any process node;

[0026] Displaying the cleaning result of any process node, and analyzing whether the cleaning result of any process node meets the corresponding expected result based on the alarm information and the verification information;

[0027] If the cleaning result of any of the process nodes does not meet the corresponding expected result, an alarm prompt information is generated based on the cleaning result of any of the process nodes, and the alarm prompt information is visualized.

[0028] A second aspect of the present application provides a data cleaning processing device, comprising:

[0029] An acquisition unit, used for acquiring data from an original data source;

[0030] A conversion unit, used to convert the data of the original data source into data to be processed with a unified caliber according to a preset conversion rule;

[0031] A data processing model is used to clean the data to be processed according to a pre-configured logical process arrangement and data logic configuration through a stream computing model to obtain result data;

[0032] A data monitoring unit, used to analyze whether there is data in the result data that does not meet the corresponding expected result based on pre-configured alarm information and verification information;

[0033] an output unit, configured to output the result data to a target table if it is analyzed that there is no data in the result data that does not satisfy the corresponding expected result;

[0034] A visualization unit is used to visualize the result data in the target table.

[0035] Optionally, in the above-mentioned data cleaning processing device, the acquisition unit includes:

[0036] A collection unit, used for collecting data from multiple data sources;

[0037] The storage unit is used to store the data of each of the data sources according to a preset storage format to obtain multiple original data sources.

[0038] Optionally, in the above-mentioned data cleaning processing device, the conversion unit includes:

[0039] A screening unit, used for screening the original data source to be processed from the plurality of original data sources;

[0040] a type determination unit, configured to determine a data type of the data of the original data source to be processed according to metadata of the data of the original data source to be processed;

[0041] The conversion subunit is used to convert the data type of the data of each of the original data sources to be processed according to the pre-established mapping relationship between the data types of each field of the original data source to be processed and the target data source, so as to obtain data to be processed with unified caliber.

[0042] Optionally, in the above-mentioned data cleaning processing device, it also includes:

[0043] The persistence unit is used to perform local persistence processing on the data to be processed.

[0044] Optionally, in the above-mentioned data cleaning processing device, it also includes:

[0045] A configuration unit, configured to respond to a configuration operation of a user and configure the logic flow arrangement, the data logic configuration, and the alarm information of the stream computing model;

[0046] A debugging unit, used to trigger any process node of the flow computing model, clean the debugging data according to the pre-configured logical process arrangement and data logic configuration, and obtain a cleaning result of any process node;

[0047] An analysis unit, configured to display the cleaning result of any process node, and analyze whether the cleaning result of any process node meets the corresponding expected result based on the alarm information and the verification information;

[0048] The prompt unit is used to generate an alarm prompt information based on the cleaning result of any process node if the cleaning result of any process node does not meet the corresponding expected result, and visualize the alarm prompt information.

[0049] A third aspect of the present application provides an electronic device, including:

[0050] Memory and processor;

[0051] Wherein, the memory is used to store programs;

[0052] The processor is used to execute the program, and when the program is executed, it is specifically used to implement the data cleaning processing method as described in any one of the above items.

[0053] A fourth aspect of the present application provides a computer storage medium for storing a computer program, which, when executed, is used to implement a data cleaning method as described in any one of the above.

[0054] The present application provides a method for data cleaning and processing, which obtains data from the original data source and then converts the data from the original data source into data to be processed with a unified caliber according to preset conversion rules, thereby simplifying the process of analyzing the data type during program execution. In addition, the data to be processed is cleaned according to the pre-configured logical process arrangement and data logic configuration through the stream computing model to obtain the result data. Thus, the data cleaning is achieved through stream computing, and the stream computing is divided into process nodes to clean the data, and only a certain process node can be adjusted. Then, based on the pre-configured alarm information and verification information, the result data is analyzed to see if there is data that does not meet the corresponding expected results. If it is analyzed that there is no data that does not meet the corresponding expected results in the result data, the result data is output to the target table, thereby ensuring the accuracy of the result. Finally, the result data in the target table is visualized. Thus, a simple and accurate data cleaning and processing method is realized. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0056] Figure 1 A flowchart of a data cleaning method provided in an embodiment of the present application;

[0057] Figure 2 A flowchart of a method for obtaining data from an original data source provided in an embodiment of the present application;

[0058] Figure 3 A flowchart of a method for converting data provided in an embodiment of the present application;

[0059] Figure 4 A flow chart of a method for configuring a stream computing model provided in an embodiment of the present application;

[0060] Figure 5 A schematic diagram of the structure of a data cleaning device provided in an embodiment of the present application;

[0061] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0063] In this application, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0064] The present application embodiment provides a method for data cleaning processing, such as Figure 1 As shown, the following steps are included:

[0065] S101, obtaining data from an original data source.

[0066] Specifically, in the embodiment of the present application, it is first necessary to access the data source and then process the data of the data source.

[0067] Optionally, data from one or more original data sources may be obtained from a collector or manually input, wherein the data from each original data source may be in a different format.

[0068] like Figure 2 As shown, a specific implementation of step S101 includes the following steps:

[0069] S201. Collect data from multiple data sources.

[0070] Specifically, corresponding data may be collected from multiple data sources that generate data.

[0071] S202: Store the data from each data source according to a preset storage format to obtain multiple original data sources.

[0072] The data from each data source is stored in a preset storage format. For example, data of the same type is stored in the same field, etc., to facilitate subsequent data type mapping and other processing.

[0073] S102: Convert the data of the original data source into data to be processed with a unified caliber according to preset conversion rules.

[0074] It should be noted that since the formats of data from different original data sources or the same data source may not be unified, making it inconvenient to uniformly process them, in an embodiment of the present application, preset conversion rules are pre-set to convert the data from the original data source into data to be processed with a unified caliber according to the preset conversion rules.

[0075] Specifically, the preset conversion rules may include conversion rules corresponding to different types of data, so that different types of data can be converted respectively.

[0076] Optionally, in another embodiment of the present application, after executing step S102, the following may be further performed: performing local persistence processing on the data to be processed.

[0077] Therefore, in the embodiment of the present application, the access to the data source includes four parts: acquisition, storage, access and local persistence of the data source.

[0078] Optionally, in another embodiment of the present application, the original data source includes multiple data sources. Accordingly, in an implementation of step S102 in the embodiment of the present application, as follows: Figure 3 As shown, including:

[0079] S301: Filter out original data sources to be processed from multiple original data sources.

[0080] It should be noted that in order to avoid the need for different docking data sources, all original data sources are often docked. However, the data to be processed varies according to different business requirements, so it is necessary to first filter out the original data sources to be processed from multiple original data sources according to the requirements.

[0081] S302: Determine the data type of the data of the original data source to be processed according to the metadata of the data of the original data source to be processed.

[0082] The metadata is data describing the data, that is, descriptive information of the data set information resource. In the embodiment of the present application, the data type of the data is converted according to the data type of the data, so the data type of the data of the original data source to be processed is first determined according to the metadata of the data.

[0083] S303: According to the pre-established mapping relationship between the data types of the fields of the original data sources to be processed and the fields of the target data sources, the data types of the data of the original data sources to be processed are converted to obtain unified data to be processed.

[0084] In the embodiment of the present application, a mapping relationship between the data types of each field of each original data source and each field of the target data source is pre-established, wherein the target data source refers to a data source storing the generated data to be processed.

[0085] The established mapping relationship is to process data in different record formats or on different storage media through conversion rules to form a data source with a unified caliber.

[0086] Optionally, for structured data, such as MySql, Orcal, etc., data type conversion rules are adopted, such as the conversion of MySql data types in Table 1.

[0087] Table 1

[0088] The data type of the original data source Conversion rules The data type of the target data source CHAR … String Date … Date BOOL … Boolean

[0089] For semi-structured data, such as JSON and XML, a visual operation interface is used to serialize and display the semi-structured data. Through the user's specification of the data type of the data field, a mapping is formed between the data type of the data in the target data source and the data type of the data in the original data source.

[0090] S103. Clean the data to be processed according to the pre-configured logical process arrangement and data logic configuration through the stream computing model to obtain result data.

[0091] It should be noted that in the implementation of this application, the flow computing model needs to be configured in advance, and then the flow computing model is used to perform flow computing on the processing to be processed. Specifically, the logical process arrangement, data logic configuration, and alarm information need to be configured.

[0092] It should also be noted that when the stream computing model cleans the data to be processed, it usually needs to go through multiple process links, such as filtering, sorting, and missing value processing of the data. Therefore, in order to ensure the accuracy of the results, it is also possible to debug each process node of the stream computing model when configuring the stream computing model, that is, in the embodiment of the present application, the stream computing model processing includes data logic process arrangement, data logic configuration, alarm information configuration, and data process debugging.

[0093] Optionally, in another embodiment of the present application, a method for configuring a stream computing model is provided, such as Figure 4 As shown, including:

[0094] S401. Respond to the user's configuration operation and configure the logical process arrangement, data logic configuration, and alarm information of the flow computing model.

[0095] The essence of data logic configuration is the processing rules of the data to be processed. Data process orchestration refers to the process of achieving data cleaning through a series of logical assumptions of the user for the target data source with unified caliber. It specifically includes three parts: data input, data rule processing orchestration and data output.

[0096] Among them, data input refers to selecting a target data source with a unified caliber obtained through metadata access and processing, and then transferring the input target data source to a specified process environment after field name duplication is performed, and processing the data to be processed using corresponding processing rules. Optionally, the input target data source can be one or more target data sources of different types.

[0097] It should be noted that data rule processing refers to cleaning the input data to be processed according to certain processing rules and transferring it to the next process node for processing. Processing rules refer to the way of data cleaning, such as filtering, sorting, and missing value processing of data.

[0098] The alarm information configuration is mainly used to manage the data quality and security of the input data to be processed through the configured data logic flow and logic configuration to generate the data results, using the configured alarm information and the system's internal data verification mechanism.

[0099] S402, triggering any process node of the flow computing model, cleaning the debugging data according to the pre-configured logical process arrangement and data logic configuration, obtaining the cleaning result of any process node and displaying the cleaning result of any process node.

[0100] Specifically, during the data flow debugging process, any node debugging operation is triggered to process the debugging data in real time, and the cleaning result of the process node can be returned in real time through WebSocket, and then the request result is visualized to provide debuggers.

[0101] S403: Based on the alarm information and the verification information, analyze whether the cleaning result of any process node meets the corresponding expected result.

[0102] Among them, the alarm information includes custom thresholds, upper and lower limits, precision, etc. The verification information has rules including verification methods, etc.

[0103] Optionally, in order to allow the user to understand the current alarm information and verification information, the alarm information and verification information may also be displayed on the user interface.

[0104] It should be noted that if the cleaning result of any process node does not meet the corresponding expected result, step S404 is executed.

[0105] S404: Generate warning prompt information based on the cleaning result of any process node, and visualize the warning prompt information.

[0106] Optionally, the warning information may include the analysis result in step S403, as well as specific error data, so that the user can accurately understand the existing problem and adjust the process node. After adjusting the process node, subsequent debugging can be continued from the process node until the final result reaches the expected effect.

[0107] Optionally, the result data obtained by executing step S103 may include only the final cleaning result, that is, only the input of the last process node, and may also include the input of each process node.

[0108] S104: Based on the pre-configured alarm information and verification information, whether there is data in the analysis result data that does not meet the corresponding expected result.

[0109] In order to ensure the accuracy of the final displayed result data, it is necessary to analyze whether there is data in the result data that does not meet the corresponding expected results based on the pre-configured alarm information and verification information.

[0110] If the result data does not contain data that does not meet the corresponding expected result, then step S105 is executed. Figure 4 In the manner shown, the process nodes corresponding to the expected results that do not meet the corresponding results are adjusted.

[0111] S105. Output the result data to the target table.

[0112] Specifically, the result data is output to the target table to be stored in a specified format, so that charts in different formats can be generated for display as needed.

[0113] S106. Visually display the result data in the target table.

[0114] Optionally, the result data in the target table is visualized, specifically in the form of reports, tables, charts, etc.

[0115] A method for data cleaning and processing provided by an embodiment of the present application obtains data from an original data source, and then converts the data from the original data source into data to be processed with a unified caliber according to a preset conversion rule, thereby not needing to configure corresponding data rules for different data sources, simplifying the configuration process. In addition, the data to be processed is cleaned according to the pre-configured logical process arrangement and data logic configuration through a stream computing model to obtain result data. Thus, data cleaning is achieved through stream computing, and stream computing is divided into process nodes to clean data, and adjustments can be made only for a certain process node. Then, based on the pre-configured alarm information and verification information, the result data is analyzed to see if there is data that does not meet the corresponding expected result. If it is analyzed that there is no data that does not meet the corresponding expected result in the result data, the result data is output to a target table, thereby ensuring the accuracy of the result. Finally, the result data in the target table is visualized. Thus, a simple and accurate data cleaning and processing method is realized.

[0116] Another embodiment of the present application provides a data cleaning processing device, such as Figure 5 As shown, including:

[0117] The acquisition unit 501 is used to acquire data from an original data source.

[0118] The conversion unit 502 is used to convert the data of the original data source into the data to be processed with a unified caliber according to the preset conversion rules.

[0119] The data processing model 503 is used to clean the data to be processed according to the pre-configured logical process arrangement and data logic configuration through the stream computing model to obtain result data.

[0120] The data monitoring unit 504 is used to analyze whether there is data in the result data that does not meet the corresponding expected result based on the pre-configured alarm information and verification information.

[0121] The output unit 505 is used to output the result data to the target table if it is analyzed that there is no data that does not meet the corresponding expected result in the result data.

[0122] The visualization unit 506 is used to visualize the result data in the target table.

[0123] Optionally, in the data cleaning processing device provided in another embodiment of the present application, the acquiring unit includes:

[0124] The collection unit is used to collect data from multiple data sources.

[0125] The storage unit is used to store the data of each data source according to a preset storage format to obtain multiple original data sources.

[0126] Optionally, in the data cleaning processing device provided in another embodiment of the present application, the conversion unit includes:

[0127] The screening unit is used to screen out the original data source to be processed from multiple original data sources.

[0128] The type determination unit is used to determine the data type of the data of the original data source to be processed according to the metadata of the data of the original data source to be processed.

[0129] The conversion subunit is used to convert the data type of the data of each original data source to be processed according to the pre-established mapping relationship between the data types of each field of each original data source to be processed and each field of the target data source, so as to obtain data to be processed with unified caliber.

[0130] Optionally, in the data cleaning processing device provided in another embodiment of the present application, the device further includes:

[0131] The persistence unit is used to perform local persistence processing on the data to be processed.

[0132] Optionally, in the data cleaning processing device provided in another embodiment of the present application, the device further includes:

[0133] The configuration unit is used to respond to the user's configuration operation and configure the logical process arrangement, data logic configuration, and alarm information of the stream computing model.

[0134] The debugging unit is used to trigger any process node of the flow computing model, clean the debugging data according to the pre-configured logical process arrangement and data logic configuration, and obtain the cleaning result of any process node.

[0135] The analysis unit is used to display the cleaning result of any process node and analyze whether the cleaning result of any process node meets the corresponding expected result based on the alarm information and verification information.

[0136] The prompt unit is used to generate an alarm prompt information based on the cleaning result of any process node if the cleaning result of any process node does not meet the corresponding expected result, and visualize the alarm prompt information.

[0137] It should be noted that the specific working process of each unit provided in the above embodiments of the present application can refer to the corresponding steps in the above method embodiments, and will not be repeated here.

[0138] Another embodiment of the present application provides an electronic device, such as Figure 6 As shown, including:

[0139] Memory 601 and processor 602 .

[0140] The memory 601 is used to store programs.

[0141] The processor 602 is used to execute the program stored in the memory 601, and when the program is executed, it is specifically used to implement the data cleaning processing method provided in any one of the above embodiments.

[0142] It should be noted that the specific implementation process can refer to the specific steps of the data cleaning method provided in the above embodiments, which will not be repeated here.

[0143] Another embodiment of the present application provides a computer storage medium for storing a computer program. When the computer program is executed, it is used to implement the data cleaning processing method provided in any of the above embodiments.

[0144] Computer storage media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0145] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0146] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for data cleaning processing, characterized in that: include: Get data from the original data source; Convert the data of the original data source into the data to be processed in a unified manner according to the preset conversion rules; The data to be processed is cleaned according to the pre-configured logical process arrangement and data logic configuration through the stream computing model to obtain result data; Based on the pre-configured alarm information and verification information, analyzing whether there is data in the result data that does not meet the corresponding expected result; If it is analyzed that there is no data in the result data that does not meet the corresponding expected result, the result data is output to the target table; Visually display the result data in the target table; Wherein, the process of cleaning the data to be processed according to the pre-configured logical process arrangement and data logic configuration through the stream computing model to obtain the result data also includes: In response to a configuration operation of a user, configure the logic flow arrangement, the data logic configuration, and the alarm information of the stream computing model; Trigger any process node of the flow computing model, clean the debugging data according to the pre-configured logical process arrangement and data logic configuration, and obtain the cleaning result of any process node; Displaying the cleaning result of any process node, and analyzing whether the cleaning result of any process node meets the corresponding expected result based on the alarm information and the verification information; If the cleaning result of any of the process nodes does not meet the corresponding expected result, an alarm prompt information is generated based on the cleaning result of any of the process nodes, and the alarm prompt information is visualized.

2. The method according to claim 1, characterized in that: The obtaining of data from the original data source includes: Collect data from multiple data sources; The data of each of the data sources is stored in a preset storage format to obtain multiple original data sources.

3. The method according to claim 2, characterized in that The step of converting the data of the original data source into the data to be processed in a unified manner according to the preset conversion rules includes: Filtering the original data source to be processed from the multiple original data sources; Determining the data type of the data of the original data source to be processed according to the metadata of the data of the original data source to be processed; According to the pre-established mapping relationship between the data types of the various fields of the original data sources to be processed and the various fields of the target data sources, the data types of the data of the various original data sources to be processed are converted to obtain unified data to be processed.

4. The method according to claim 1, characterized in that After converting the data of the original data source into the data to be processed with unified caliber according to the preset conversion rules, the method further includes: The data to be processed is locally persisted.

5. A data cleaning device, characterized in that: include: An acquisition unit, used for acquiring data from an original data source; A conversion unit, used to convert the data of the original data source into data to be processed with a unified caliber according to a preset conversion rule; A data processing model is used to clean the data to be processed according to a pre-configured logical process arrangement and data logic configuration through a stream computing model to obtain result data; A data monitoring unit, used to analyze whether there is data in the result data that does not meet the corresponding expected result based on pre-configured alarm information and verification information; An output unit, configured to output the result data to a target table if it is analyzed that there is no data in the result data that does not meet the corresponding expected result; A visualization unit, used for visually displaying the result data in the target table; A configuration unit, configured to respond to a configuration operation of a user and configure the logic flow arrangement, the data logic configuration, and the alarm information of the stream computing model; A debugging unit, used to trigger any process node of the flow computing model, clean the debugging data according to the pre-configured logical process arrangement and data logic configuration, and obtain the cleaning result of any process node; An analysis unit, configured to display the cleaning result of any process node, and analyze whether the cleaning result of any process node meets the corresponding expected result based on the alarm information and the verification information; The prompt unit is used to generate an alarm prompt information based on the cleaning result of any process node if the cleaning result of any process node does not meet the corresponding expected result, and visualize the alarm prompt information.

6. The device according to claim 5, characterized in that The conversion unit comprises: A screening unit, used for screening out the original data source to be processed from the plurality of the original data sources; a type determination unit, configured to determine a data type of the data of the original data source to be processed according to metadata of the data of the original data source to be processed; The conversion subunit is used to convert the data type of the data of each of the original data sources to be processed according to the pre-established mapping relationship between the data types of each field of the original data source to be processed and the target data source, so as to obtain data to be processed with unified caliber.

7. An electronic device, characterized in that: include: Memory and processor; Wherein, the memory is used to store programs; The processor is used to execute the program, and when the program is executed, it is specifically used to implement the data cleaning processing method as described in any one of claims 1 to 4.

8. A computer storage medium, characterized in that Used to store a computer program, which, when executed, is used to implement the data cleaning method as described in any one of claims 1 to 4.

Images