Data cleaning method and system based on large model and electronic equipment

Abstract

The invention provides a data cleaning method and system based on a large model and electronic equipment. The method is realized by cleaning an assembly line framework, the core is that an acquisition service, a cleaning service and a storage service which are completely independent and run in parallel are abstractly packaged, and the operation is efficiently managed by a container. Acquisition services need to adapt to source data of different data sources, different reading mechanisms are adopted, different containers are started by injecting different reading mechanisms, and different reading services are isolated. And the cleaning service is responsible for executing a data cleaning strategy in the cleaning link to perform data cleaning on the source data and screening invalid data in the source data. And the write-in service injects the target data generated after cleaning into a write-in service container. Through the operation process, full automation of the cleaning process can be realized without manual operation, so that the data cleaning efficiency can be improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data cleaning method, system and electronic equipment based on a large model. Background Art

[0002] With the rapid development of big data technology and big models, the scale and complexity of data are increasing. Data cleaning refers to the process of re-examining and verifying the original data, with the purpose of deleting duplicate information, correcting erroneous data, and cleaning invalid data, thereby improving the quality of data. As an important part of big data processing and big model data engineering, its importance is self-evident.

[0003] Currently, the existing data cleaning technology is implemented through a cleaning framework module, and the data cleaning framework module is usually implemented through manual operation.

[0004] However, data cleaning through manual operations cannot fully automate the cleaning process, which results in low efficiency of data cleaning. Summary of the invention

[0005] The embodiments of the present invention provide a data cleaning method, system and electronic device based on a large model, which can improve the efficiency of data cleaning.

[0006] In a first aspect, an embodiment of the present invention provides a data cleaning method based on a large model, comprising:

[0007] A1: Collect source data to be processed and inject it into the collection service container, wherein the source data to be processed corresponds to at least one data source, and source data from different sources corresponds to different data reading mechanisms;

[0008] A2: Based on the adapted data reading mechanism, the controller Operator is used to control the reading of the source data to be processed in the collection service container;

[0009] A3: When the reading of the source data to be processed is completed, the source data to be processed is cleaned based on a preset data cleaning strategy to filter invalid data in the source data to be processed and generate target data, wherein the invalid data includes duplicate data and erroneous data, and the data cleaning strategy includes: at least one of sampling cleaning, full cleaning, and batch multiple rounds of cleaning;

[0010] A4: When the source data to be processed is cleaned, the target data is injected and stored into the write service container, and the process returns to step A1.

[0011] Preferably,

[0012] Before the source data to be processed is collected and injected into the collection service container, the method further includes:

[0013] Pre-set cleaning line configuration information, authority management information, cleaning task management information and life cycle management information;

[0014] The massive data cleaning module is used to write and call the available service interface API and orchestrate the cleaning pipeline tasks, wherein the available services include: Spark service, Hive service, Flink service, Kafka service and Hadoop service. The Spark service is used to process big data distributed parallel computing analysis and structured data, the Hive service is used to provide big data warehouse services, the Flink service is used to provide a real-time stream computing framework, the Kafka service is used to provide distributed messaging middleware, and the Hadoop service is used to provide distributed computing, distributed file system and NoSQL column database basic services in the Hadoop ecosystem.

[0015] Preferably,

[0016] The A3 includes:

[0017] Injecting the read source data to be processed and the data cleaning strategy into different cleaning service containers; when the controller Operator monitors that the reading of the source data to be processed is completed, controlling to start the cleaning service container, performing data cleaning on the source data to be processed based on the data cleaning strategy, so as to filter out invalid data in the source data to be processed, and generate target data, wherein the invalid data includes duplicate data and erroneous data, and the data cleaning strategy includes: at least one of sampling cleaning, full cleaning, and batch multi-round cleaning;

[0018] When the controller Operator detects an abnormality in the cleaning process, the cleaning service container is restarted from the current process processing node to complete all cleaning tasks of the source data to be processed; the processing status of each of the process processing nodes is recorded and stored in the log record table.

[0019] Preferably,

[0020] When the source data to be processed is cleaned, after the target data is injected and stored into the write service container, the method further includes:

[0021] Determine the data comparison difference between the source data to be processed and the target data by using a cleaning monitoring management module;

[0022] Based on the data comparison difference, construct a difference analysis graph, wherein the difference analysis graph is used to reflect the data difference before and after the source data to be processed is cleaned;

[0023] The data cleaning quality of the processed source data is determined according to the first formula and a preset quality assessment percentage threshold, wherein the first formula is:

[0024]

[0025] Among them, K is the current quality assessment percentage, N is the amount of processed source data, and A is the total amount of source data.

[0026] In a second aspect, an embodiment of the present invention provides a data cleaning system based on a large model, the system comprising: a cleaning framework module;

[0027] The cleaning framework module is used to perform:

[0028] A1: Collect source data to be processed and inject it into the collection service container, wherein the source data to be processed corresponds to at least one data source, and source data from different sources corresponds to different data reading mechanisms;

[0029] A2: Based on the adapted data reading mechanism, the controller Operator is used to control the reading of the source data to be processed in the collection service container;

[0030] A3: When the reading of the source data to be processed is completed, the source data to be processed is cleaned based on a preset data cleaning strategy to filter invalid data in the source data to be processed and generate target data, wherein the invalid data includes duplicate data and erroneous data, and the data cleaning strategy includes: at least one of sampling cleaning, full cleaning, and batch multiple rounds of cleaning;

[0031] A4: When the source data to be processed is cleaned, the target data is injected and stored into the write service container, and the process returns to step A1.

[0032] Preferably,

[0033] Before the cleaning framework module, it further includes: an information configuration module and a mass data cleaning module; the information configuration module is used to pre-set cleaning pipeline configuration information, authority management information, cleaning task management information and life cycle management information;

[0034] The massive data cleaning module is used to write and call the available service interface API and orchestrate the cleaning pipeline tasks, wherein the available services include: Spark service, Hive service, Flink service, Kafka service and Hadoop service. The Spark service is used to process big data distributed parallel computing analysis and structured data, the Hive service is used to provide big data warehouse services, the Flink service is used to provide a real-time stream computing framework, the Kafka service is used to provide distributed messaging middleware, and the Hadoop service is used to provide distributed computing, distributed file system and NoSQL column database basic services in the Hadoop ecosystem.

[0035] Preferably,

[0036] The cleaning framework module is also used to execute:

[0037] Injecting the read source data to be processed and the data cleaning strategy into different cleaning service containers; when the controller Operator monitors that the reading of the source data to be processed is completed, controlling to start the cleaning service container, performing data cleaning on the source data to be processed based on the data cleaning strategy, so as to filter out invalid data in the source data to be processed and generate target data;

[0038] When the controller Operator detects an abnormality in the cleaning process, the cleaning service container is restarted from the current process processing node to complete all cleaning tasks of the source data to be processed; the processing status of each of the process processing nodes is recorded and stored in the log record table.

[0039] Preferably,

[0040] After the cleaning framework module, it further includes: a cleaning monitoring management module

[0041] The cleaning monitoring management module is used to execute:

[0042] Determine the data comparison difference between the source data to be processed and the target data;

[0043] Based on the data comparison difference, construct a difference analysis graph, wherein the difference analysis graph is used to reflect the data difference before and after the source data to be processed is cleaned;

[0044] The data cleaning quality of the processed source data is determined according to the first formula and a preset quality assessment percentage threshold, wherein the first formula is:

[0045]

[0046] Among them, K is the current quality assessment percentage, N is the amount of processed source data, and A is the total amount of source data.

[0047] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: at least one memory and at least one processor;

[0048] The at least one memory is used to store a machine-readable program;

[0049] The at least one processor is used to call the machine-readable program to execute any method described in the first aspect of claim 1.

[0050] In a fourth aspect, an embodiment of the present invention provides a computer-readable medium, on which computer instructions are stored. When the computer instructions are executed by a processor, the processor executes any one of the methods described in the first aspect.

[0051] The embodiment of the present invention provides a data cleaning method, system and electronic device based on a large model. The method is implemented through a cleaning pipeline framework, and the core of the cleaning pipeline framework is to abstractly encapsulate the acquisition service, cleaning service and storage service that are completely independent and run in parallel, and are efficiently managed and run by containers. The acquisition service needs to adapt to the source data from different data sources, adopt different data reading mechanisms, start different containers by injecting different reading mechanisms, and isolate different reading services. The cleaning service is responsible for executing the data cleaning strategy in the cleaning link to clean the source data and filter out invalid data. The write service injects the target data generated after cleaning into the write service container. The Operator controller can notify each node service execution in an asynchronous manner, and when the data reading is monitored, the cleaning service container is started to clean the data. When the data cleaning is completed, the data writing service is asynchronously called, and the data reading service is notified to read the data again. Through the above process, the cleaning process can be fully automated without manual operation, thereby improving the efficiency of data cleaning. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0053] Figure 1 It is a flow chart of a data cleaning method based on a large model provided by one embodiment of the present invention;

[0054] Figure 2is a flow chart of another data cleaning method based on a large model provided by an embodiment of the present invention;

[0055] Figure 3 is a schematic diagram of a data cleaning system based on a large model provided by an embodiment of the present invention;

[0056] Figure 4 is a schematic diagram of another large model-based data cleaning system provided by an embodiment of the present invention;

[0057] Figure 5 It is a schematic diagram of another large model-based data cleaning system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0059] like Figure 1 As shown, an embodiment of the present invention provides a data cleaning method based on a large model, which may include the following steps:

[0060] Step 101: Collect source data to be processed and inject it into a collection service container, wherein the source data to be processed corresponds to at least one data source, and source data from different sources corresponds to different data reading mechanisms;

[0061] Step 102: Based on the adapted data reading mechanism, the controller Operator is used to control the reading of the source data to be processed in the collection service container;

[0062] Step 103: After the source data to be processed is read, the source data to be processed is cleaned based on a preset data cleaning strategy to filter invalid data in the source data to be processed and generate target data, wherein the invalid data includes duplicate data and erroneous data, and the data cleaning strategy includes: at least one of sampling cleaning, full cleaning, and batch multi-round cleaning;

[0063] Step 104: When the source data to be processed is cleaned, the target data is injected and stored into the write service container, and the process returns to step 101.

[0064] In an embodiment of the present invention, a data cleaning method based on a large model is provided. The method is implemented through a cleaning pipeline framework, and the core of the cleaning pipeline framework is to abstractly encapsulate the acquisition service, cleaning service and storage service that are completely independent and run in parallel, and are efficiently managed and run by containers. The acquisition service needs to adapt to the source data from different data sources, adopt different data reading mechanisms, start different containers by injecting different reading mechanisms, and isolate different reading services. The cleaning service is responsible for executing the data cleaning strategy in the cleaning link to clean the source data and filter out invalid data. The write service injects the target data generated after cleaning into the write service container. The Operator controller can notify each node service execution in an asynchronous manner, and when the data reading is monitored to end, the cleaning service container is started to clean the data. When the data cleaning is completed, the data writing service is asynchronously called, and the data reading service is notified to read the data again. Through the above process, the cleaning process can be fully automated without manual operation, thereby improving the efficiency of data cleaning.

[0065] In order to meet the needs of massive and large-scale data cleaning, in one embodiment of the present invention, before the source data to be processed is collected and injected into the collection service container in step 101, the above embodiment further includes: presetting cleaning pipeline configuration information, authority management information, cleaning task management information and life cycle management information;

[0066] The massive data cleaning module is used to write and call the available service interface API and orchestrate the cleaning pipeline tasks, wherein the available services include: Spark service, Hive service, Flink service, Kafka service and Hadoop service. The Spark service is used to process big data distributed parallel computing analysis and structured data, the Hive service is used to provide big data warehouse services, the Flink service is used to provide a real-time stream computing framework, the Kafka service is used to provide distributed messaging middleware, and the Hadoop service is used to provide distributed computing, distributed file system and NoSQL column database basic services in the Hadoop ecosystem.

[0067] In an embodiment of the present invention, the massive data cleaning module can realize highly parallel massive cleaning tasks based on the IEMR storage and computing base and its Hadoop ecosystem. Therefore, the massive data cleaning module can be used to write and call the service interface API and orchestrate the cleaning pipeline tasks, automatically execute parallel data cleaning tasks on large-scale distributed cluster nodes, reduce manual intervention, and thus greatly improve the capacity and efficiency of data cleaning parallel processing, meet the massive and large-scale data cleaning, and continuously supply and supplement the source data of the large model platform.

[0068] Specifically, in order to achieve full automation of the cleaning process, the cleaning pipeline configuration information (for example, the configuration information of each submodule such as source data acquisition, sampling cleaning, full cleaning and cleaning data storage, etc.) can be pre-set, which can be implemented through the cleaning pipeline framework module, which can be written in Java language and the operating environment can be a Linux containerized system. In order to handle the collaborative work requirements of multiple roles, the authority management information can be pre-set (for example, the cleaning developer is responsible for the design and optimization of the data cleaning strategy, the data quality inspector is responsible for the inspection and feedback of the data cleaning results, and the data manager is responsible for the scheduling and management monitoring of the data cleaning task). By setting different user permissions to meet the cleaning work collaboration of various roles such as cleaning developers, data quality inspectors and data managers, the cleaning task management information (for example, the order of processing each task) can be pre-set. Through reasonable authority control and task allocation, efficient collaboration between different roles can be achieved, and the work efficiency of data cleaning can be improved. In addition, it is also necessary to set the life cycle management information, including: the cycle, number, start and end time of data cleaning, etc., so as to finally achieve full automation of the entire data cleaning process through the configuration of the above series of information.

[0069] In order to realize full automation of the cleaning process, in one embodiment of the present invention, step 103 in the above embodiment performs data cleaning on the source data to be processed based on a preset data cleaning strategy when the reading of the source data to be processed is completed, so as to filter out invalid data in the source data to be processed and generate target data, which may include the following steps:

[0070] Injecting the read source data to be processed and the data cleaning strategy into different cleaning service containers; when the controller Operator monitors that the reading of the source data to be processed is completed, controlling to start the cleaning service container, performing data cleaning on the source data to be processed based on the data cleaning strategy, so as to filter out invalid data in the source data to be processed, and generating target data;

[0071] When the controller Operator detects an abnormality in the cleaning process, the cleaning service container is restarted from the current process processing node to complete all cleaning tasks of the source data to be processed; the processing status of each of the process processing nodes is recorded and stored in the log record table.

[0072] In an embodiment of the present invention, a data set can be used to collect, display and store the source data to be processed. The sampled data set can be used to cache, display and spot check the sampled data. The controller Operator executes the data cleaning strategy to clean the data. When a cleaning anomaly is detected, the data is cleaned again from the current node to complete the cleaning of the source data. The data cleaning strategy is a strategy library, which generally needs to go through the processes of development, testing, online, and even failure and delisting. Cleaning strategy libraries in various fields and industries can be continuously developed, customized and accumulated. Compared with the fixed data cleaning strategies of the prior art, this solution can flexibly adjust the data cleaning strategy according to actual needs, continuously improve the efficiency and effect of data cleaning, and ultimately meet the data quality requirements of the large model platform.

[0073] In order to realize visualization of the cleaning process, in one embodiment of the present invention, in the above embodiment, when the cleaning of the source data to be processed is completed in step 104, after the target data is injected and stored into the write service container, the following further comprises:

[0074] Determine the data comparison difference between the source data to be processed and the target data by using a cleaning monitoring management module;

[0075] Based on the data comparison difference, construct a difference analysis graph, wherein the difference analysis graph is used to reflect the data difference before and after the source data to be processed is cleaned;

[0076] The data cleaning quality of the processed source data is determined according to the first formula and a preset quality assessment percentage threshold, wherein the first formula is:

[0077]

[0078] Wherein, K is the current quality assessment percentage, N is the amount of processed source data, and A is the total amount of source data. In an embodiment of the present invention, the cleaning monitoring management module can be used to implement multi-role cleaning collaboration and data cleaning management, including visual management of the pipeline, visual comparison and difference inspection of the source data before and after cleaning (for example, the display of the difference analysis chart can highlight the data comparison difference between the source data to be processed and the target data) and cleaning quality statistical analysis monitoring (for example, the current quality assessment percentage is greater than the quality assessment percentage threshold, reflecting that the current data cleaning quality is not high). By comparing the source data before and after cleaning and analyzing and feeding back the historical data cleaning results, the data cleaning strategy can be continuously optimized and adjusted. Through statistical analysis methods, problems and bottlenecks in the data cleaning process can be found, and then the data cleaning strategy can be optimized and adjusted according to actual conditions.

[0079] like Figure 2To more clearly illustrate the technical solution and advantages of the present invention, a data cleaning method based on a large model is provided in detail below in accordance with an embodiment of the present invention, which may specifically include the following steps:

[0080] Step 201: pre-setting cleaning line configuration information, authority management information, cleaning task management information and life cycle management information;

[0081] Step 202: Use the massive data cleaning module to write and call the available service interface API and arrange and schedule the cleaning pipeline task, where the available services include: Spark service, Hive service, Flink service, Kafka service and Hadoop service. Spark service is used to process big data distributed parallel computing analysis and structured data, Hive service is used to provide big data warehouse service, Flink service is used to provide real-time stream computing framework, Kafka service is used to provide distributed messaging middleware, and Hadoop service is used to provide distributed computing, distributed file system and NoSQL column database basic services in the Hadoop ecosystem;

[0082] Step 203: Collect the source data to be processed and inject it into the collection service container, wherein the source data to be processed corresponds to at least one data source, and source data from different sources corresponds to different data reading mechanisms; Step 204: Based on the adapted data reading mechanism, use the controller Operator to control the reading of the source data to be processed in the collection service container;

[0083] Specifically, the data reading rules involve reading speed, size, order, etc. If the data file is too large, it needs to be split. If there is a time sequence between the data, it must also be read in order.

[0084] Step 205: injecting the read source data to be processed and the data cleaning strategy into different cleaning service containers, wherein the data cleaning strategy includes: at least one of sampling cleaning, full cleaning and batch multi-round cleaning;

[0085] Step 206: When the controller Operator monitors that the reading of the source data to be processed is completed, the cleaning service container is controlled to start, and the source data to be processed is cleaned based on the data cleaning strategy to filter invalid data in the source data to be processed and generate target data, wherein the invalid data includes duplicate data and error data;

[0086] Specifically, the cleaning framework module can be used to enrich the data cleaning strategy library, support pipeline chain data collection, sampling cleaning, full cleaning, batch multi-round cleaning and data sampling, etc. This modular design can flexibly adjust the data cleaning strategy according to actual needs, continuously improve the efficiency and effect of data cleaning, and ultimately meet the data quality requirements of the large model platform.

[0087] Step 207: When the controller Operator detects that an exception occurs in the cleaning process, the cleaning service container is restarted from the current process processing node to complete all cleaning tasks for the source data to be processed;

[0088] Step 208: Record the processing status of each process processing node and store it in the log record table;

[0089] Step 209: When the source data to be processed is cleaned, the target data is injected and stored into the write service container, and the process returns to step 203 .

[0090] Specifically, the Operator controller is based on asynchronous decoupling considerations, and each service is decoupled. The Operator controller can notify each process processing node service execution in an asynchronous manner, and start the cleaning service container to clean the data after listening to the end of data reading. When the data cleaning is completed, Operato will asynchronously call the data writing service and notify the data reading service to read the data again. In this way, the resources of each service have a gap to release, reducing service pressure. In addition, it is also possible to further improve the quality of data cleaning through manual operation and monitoring, such as interface display, configuration, and operation of source data reading, data cleaning, and storage of cleaned data, and provide monitoring information of the results of these processes, through both manual and automated methods.

[0091] Step 210: using the cleaning monitoring management module to determine the data comparison difference between the source data to be processed and the target data;

[0092] Step 211: constructing a difference analysis graph based on the data comparison difference, wherein the difference analysis graph is used to reflect the data difference before and after the source data to be processed is cleaned;

[0093] Step 212: Determine the data cleaning quality of the processed source data according to the first formula and a preset quality assessment percentage threshold;

[0094] Specifically, the first formula is: Among them, K is the current quality assessment percentage, N is the amount of processed source data, and A is the total amount of source data.

[0095] like Figure 3As shown, an embodiment of the present invention provides a data cleaning system based on a large model, the system comprising: a cleaning framework module 301;

[0096] The cleaning framework module 301 is used to perform:

[0097] A1: Collect source data to be processed and inject it into the collection service container, wherein the source data to be processed corresponds to at least one data source, and source data from different sources corresponds to different data reading mechanisms;

[0098] A2: Based on the adapted data reading mechanism, the controller Operator is used to control the reading of the source data to be processed in the collection service container;

[0099] A3: When the reading of the source data to be processed is completed, the source data to be processed is cleaned based on a preset data cleaning strategy to filter invalid data in the source data to be processed and generate target data, wherein the invalid data includes duplicate data and erroneous data, and the data cleaning strategy includes: at least one of sampling cleaning, full cleaning, and batch multiple rounds of cleaning;

[0100] A4: When the source data to be processed is cleaned, the target data is injected and stored into the write service container, and the process returns to step A1.

[0101] based on Figure 3 A data cleaning system based on a large model is shown in Figure 4 As shown, in one embodiment of the present invention, before the cleaning framework module 301, it further includes: an information configuration module 302 and a mass data cleaning module 303;

[0102] The information configuration module 302 is used to pre-set cleaning line configuration information, authority management information, cleaning task management information and life cycle management information;

[0103] The massive data cleaning module 303 is used to write and call the available service interface API and orchestrate the cleaning pipeline tasks, wherein the available services include: Spark service, Hive service, Flink service, Kafka service and Hadoop service. The Spark service is used to process big data distributed parallel computing analysis and structured data, the Hive service is used to provide big data warehouse services, the Flink service is used to provide a real-time stream computing framework, the Kafka service is used to provide distributed messaging middleware, and the Hadoop service is used to provide distributed computing, distributed file system and NoSQL column database basic services in the Hadoop ecosystem.

[0104] like Figure 4As shown, in one embodiment of the present invention, the cleaning framework module 301 is also used to perform:

[0105] Injecting the read source data to be processed and the data cleaning strategy into different cleaning service containers; when the controller Operator monitors that the reading of the source data to be processed is completed, controlling to start the cleaning service container, performing data cleaning on the source data to be processed based on the data cleaning strategy, so as to filter out invalid data in the source data to be processed and generate target data;

[0106] When the controller Operator detects an abnormality in the cleaning process, the cleaning service container is restarted from the current process processing node to complete all cleaning tasks of the source data to be processed; the processing status of each of the process processing nodes is recorded and stored in the log record table.

[0107] based on Figure 4 A data cleaning system based on a large model is shown in Figure 5 As shown, after the cleaning framework module 301, it further includes: a cleaning monitoring management module 304;

[0108] The cleaning monitoring management module 304 is used to perform:

[0109] Determine the data comparison difference between the source data to be processed and the target data;

[0110] Based on the data comparison difference, construct a difference analysis graph, wherein the difference analysis graph is used to reflect the data difference before and after the source data to be processed is cleaned;

[0111] The data cleaning quality of the processed source data is determined according to the first formula and a preset quality assessment percentage threshold, wherein the first formula is:

[0112]

[0113] Among them, K is the current quality assessment percentage, N is the amount of processed source data, and A is the total amount of source data.

[0114] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on a data cleaning system based on a large model. In other embodiments of the present invention, a data cleaning system based on a large model may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0115] The information interaction, execution process and other contents between the units in the above-mentioned device are based on the same concept as the embodiment of the method of the present invention. For specific contents, please refer to the description in the embodiment of the method of the present invention, and no further description is given here.

[0116] An embodiment of the present invention further provides an electronic device, comprising: at least one memory and at least one processor;

[0117] at least one memory for storing a machine-readable program;

[0118] At least one processor is used to call a machine-readable program to execute a data cleaning method based on a large model in any embodiment of the present invention.

[0119] An embodiment of the present invention further provides a computer-readable medium on which computer instructions are stored. When the computer instructions are executed by a processor, the processor executes a large model-based data cleaning method in any embodiment of the present invention.

[0120] Specifically, a system or device equipped with a storage medium can be provided, on which software program code that implements the functions of any of the above-mentioned embodiments is stored, and a computer (or CPU or MPU) of the system or device can be enabled to read and execute the program code stored in the storage medium.

[0121] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0122] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer by a communication network.

[0123] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0124] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0125] Each embodiment of the present invention has at least the following beneficial effects:

[0126] 1. In an embodiment of the present invention, a data cleaning method based on a large model is provided. The method is implemented through a cleaning pipeline framework, and the core of the cleaning pipeline framework is to abstractly encapsulate the acquisition service, cleaning service and storage service that are completely independent and run in parallel, and are efficiently managed and run by containers. The acquisition service needs to adapt to source data from different data sources, adopt different data reading mechanisms, start different containers by injecting different reading mechanisms, and isolate different reading services. The cleaning service is responsible for executing the data cleaning strategy in the cleaning link to clean the source data and filter out invalid data. The write service injects the target data generated after cleaning into the write service container. The Operator controller can notify each node service execution in an asynchronous manner. When the data reading is monitored to be completed, the cleaning service container is started to clean the data. When the data cleaning is completed, the data writing service is asynchronously called, and the data reading service is notified to read the data again. Through the above process, the cleaning process can be fully automated without manual operation, thereby improving the efficiency of data cleaning;

[0127] 2. In the embodiment of the present invention, the massive data cleaning module can realize highly parallel massive cleaning tasks based on the IEMR storage and computing base and its Hadoop ecosystem. Therefore, the massive data cleaning module can be used to write and call service interface APIs and arrange and schedule cleaning pipeline tasks, automatically execute parallel data cleaning tasks on large-scale distributed cluster nodes, reduce manual intervention, and thus greatly improve the parallel processing capability and efficiency of data cleaning, meet the needs of massive and large-scale data cleaning, and continuously supply and supplement the source data of the large model platform;

[0128] 3. In the embodiment of the present invention, the data set can be used to collect, display and store the source data to be processed, and the sample data set can be used to cache, display and spot check records of the sample data. The controller Operator executes the data cleaning strategy to clean the data, and re-cleans the data from the current node when a cleaning anomaly is detected, so as to complete the cleaning of all the source data. The data cleaning strategy is a strategy library, which generally needs to go through the processes of development, testing, online, and even failure and delisting. It can continuously develop, customize and accumulate cleaning strategy libraries in various fields and industries. Compared with the fixed data cleaning strategy of the prior art, this solution can flexibly adjust the data cleaning strategy according to actual needs, continuously improve the efficiency and effect of data cleaning, and finally meet the data quality requirements of the large model platform.

[0129] It should be noted that not all steps and modules in the above-mentioned processes and system structure diagrams are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The system structure described in the above-mentioned embodiments can be a physical structure or a logical structure, that is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or some components in multiple independent devices may be implemented together.

[0130] In the above embodiments, the hardware unit can be realized by mechanical means or electrical means. For example, a hardware unit can include permanent dedicated circuits or logic (such as special processors, FPGA or ASIC) to complete the corresponding operation. The hardware unit can also include programmable logic or circuits (such as general-purpose processors or other programmable processors), which can be temporarily set by software to complete the corresponding operation. Concrete implementation (mechanical means or dedicated permanent circuits or temporarily set circuits) can be determined based on cost and time considerations.

[0131] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A data cleaning method based on a large model, characterized in that: The method includes: A1: Collect source data to be processed and inject it into the collection service container, wherein the source data to be processed corresponds to at least one data source, and source data from different sources corresponds to different data reading mechanisms; A2: Based on the adapted data reading mechanism, the controller Operator is used to control the reading of the source data to be processed in the collection service container; A3: When the reading of the source data to be processed is completed, the source data to be processed is cleaned based on a preset data cleaning strategy to filter invalid data in the source data to be processed and generate target data, wherein the invalid data includes duplicate data and erroneous data, and the data cleaning strategy includes: at least one of sampling cleaning, full cleaning, and batch multiple rounds of cleaning; A4: When the source data to be processed is cleaned, the target data is injected and stored into the write service container, and the process returns to step A1.

2. The method according to claim 1, characterized in that Before the source data to be processed is collected and injected into the collection service container, the method further includes: Pre-set cleaning line configuration information, authority management information, cleaning task management information and life cycle management information; The massive data cleaning module is used to write and call the available service interface API and orchestrate the cleaning pipeline tasks, wherein the available services include: Spark service, Hive service, Flink service, Kafka service and Hadoop service. The Spark service is used to process big data distributed parallel computing analysis and structured data, the Hive service is used to provide big data warehouse services, the Flink service is used to provide a real-time stream computing framework, the Kafka service is used to provide distributed messaging middleware, and the Hadoop service is used to provide distributed computing, distributed file system and NoSQL column database basic services in the Hadoop ecosystem.

3. The method according to claim 1, characterized in that The A3 includes: Injecting the read source data to be processed and the data cleaning strategy into different cleaning service containers; When the controller Operator monitors that the reading of the source data to be processed is completed, the cleaning service container is controlled to be started, and the source data to be processed is cleaned based on the data cleaning strategy to filter invalid data in the source data to be processed and generate target data, wherein the invalid data includes duplicate data and error data, and the data cleaning strategy includes: at least one of sampling cleaning, full cleaning, and batch multi-round cleaning; When the controller Operator detects an abnormality in the cleaning process, the cleaning service container is restarted from the current process processing node to complete all cleaning tasks of the source data to be processed; The processing status of each of the process processing nodes is recorded and stored in a log record table.

4. The method according to any one of claims 1 to 3, characterized in that: When the source data to be processed is cleaned, after the target data is injected and stored into the write service container, the method further includes: Determine the data comparison difference between the source data to be processed and the target data by using a cleaning monitoring management module; Based on the data comparison difference, construct a difference analysis graph, wherein the difference analysis graph is used to reflect the data difference before and after the source data to be processed is cleaned; The data cleaning quality of the processed source data is determined according to the first formula and a preset quality assessment percentage threshold, wherein the first formula is: Among them, K is the current quality assessment percentage, N is the amount of processed source data, and A is the total amount of source data.

5. The data cleaning system based on the large model is characterized by: The system includes: a cleaning frame module; The cleaning framework module is used to perform: A1: Collect source data to be processed and inject it into the collection service container, wherein the source data to be processed corresponds to at least one data source, and source data from different sources corresponds to different data reading mechanisms; A2: Based on the adapted data reading mechanism, the controller Operator is used to control the reading of the source data to be processed in the collection service container; A3: When the reading of the source data to be processed is completed, the source data to be processed is cleaned based on a preset data cleaning strategy to filter invalid data in the source data to be processed and generate target data, wherein the invalid data includes duplicate data and erroneous data, and the data cleaning strategy includes: at least one of sampling cleaning, full cleaning, and batch multiple rounds of cleaning; A4: When the source data to be processed is cleaned, the target data is injected and stored into the write service container, and the process returns to step A1.

6. The system according to claim 5, characterized in that Before the cleaning framework module, it further includes: an information configuration module and a mass data cleaning module; The information configuration module is used to pre-set cleaning line configuration information, authority management information, cleaning task management information and life cycle management information; The massive data cleaning module is used to write and call the available service interface API and orchestrate the cleaning pipeline tasks, wherein the available services include: Spark service, Hive service, Flink service, Kafka service and Hadoop service. The Spark service is used to process big data distributed parallel computing analysis and structured data, the Hive service is used to provide big data warehouse services, the Flink service is used to provide a real-time stream computing framework, the Kafka service is used to provide distributed messaging middleware, and the Hadoop service is used to provide distributed computing, distributed file system and NoSQL column database basic services in the Hadoop ecosystem.

7. The system according to claim 5, characterized in that The cleaning framework module is also used to execute: Injecting the read source data to be processed and the data cleaning strategy into different cleaning service containers; When the controller Operator monitors that the reading of the source data to be processed is completed, the cleaning service container is controlled to be started, and the source data to be processed is cleaned based on the data cleaning strategy to filter invalid data in the source data to be processed and generate target data, wherein the invalid data includes duplicate data and error data, and the data cleaning strategy includes: at least one of sampling cleaning, full cleaning, and batch multi-round cleaning; When the controller Operator detects an abnormality in the cleaning process, the cleaning service container is restarted from the current process processing node to complete all cleaning tasks of the source data to be processed; The processing status of each of the process processing nodes is recorded and stored in a log record table.

8. The system according to any one of claims 5 to 7, characterized in that: After the cleaning framework module, it further includes: a cleaning monitoring management module, the cleaning monitoring management module is used to execute: Determine the data comparison difference between the source data to be processed and the target data; Based on the data comparison difference, construct a difference analysis graph, wherein the difference analysis graph is used to reflect the data difference before and after the source data to be processed is cleaned; The data cleaning quality of the processed source data is determined according to the first formula and a preset quality assessment percentage threshold, wherein the first formula is: Among them, K is the current quality assessment percentage, N is the amount of processed source data, and A is the total amount of source data.

9. An electronic device, characterized in that include: at least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is configured to call the machine-readable program to execute the method according to any one of claims 1 to 4.

10. A computer readable medium, characterized in that The computer readable medium stores computer instructions, which, when executed by a processor, cause the processor to perform any one of the methods of claims 1 to 4.

Images