Data cleaning method and device based on user-defined scheduling
By generating data cleaning rules through custom scheduling parameters, the lack of flexibility and specificity in existing data cleaning methods is solved, thus achieving both flexibility and efficiency in data cleaning.
Patent Information
- Application Number
- CN202610008252.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-05
- Publication Date
- 2026-02-27
AI Technical Summary
Existing data cleaning methods lack flexibility and specificity, resulting in inefficient data cleaning.
By obtaining user-inputted custom scheduling parameters, data cleaning rules are generated, data cleaning is performed using the custom scheduling parameters, and task status is obtained and logged in real time.
It improves the flexibility and efficiency of data cleaning, adapts to the data cleaning needs of different scenarios, and breaks the limitations of the traditional fixed data cleaning model.
Smart Images
Figure CN121579467A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a data cleaning method and apparatus based on custom scheduling. Background Technology
[0002] In the era of big data, data comes from a wide range of sources and comes in various formats. The data often contains redundancy, errors, inconsistencies, and other problems. This poor-quality data can seriously affect the accuracy and reliability of data analysis results. Therefore, data cleaning has become a key step in the data processing workflow.
[0003] Most existing data cleaning methods employ fixed cleaning rules and scheduling strategies, lacking flexibility and specificity. This results in data cleaning being neither flexible nor efficient.
[0004] In conclusion, improving the flexibility and efficiency of data cleaning is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0005] In view of this, this application provides a data cleaning method and apparatus based on custom scheduling, which aims to improve the flexibility and efficiency of data cleaning.
[0006] Firstly, this application provides a data cleaning method based on custom scheduling, including:
[0007] Get user-inputted custom scheduling parameters;
[0008] Data cleaning rules are generated based on the custom scheduling parameters;
[0009] Using the aforementioned data cleaning rules, the data to be cleaned is cleaned to obtain cleaned data;
[0010] The cleaned data is written back to the database.
[0011] Optionally, the custom scheduling parameters include node range parameters, and the generation of data cleaning rules based on the custom scheduling parameters includes:
[0012] The node range parameters are parsed to obtain the node range for data cleaning;
[0013] The data cleaning rules are generated based on the node range of the data cleaning process.
[0014] Optionally, the custom scheduling parameters include time range parameters, and the step of generating data cleaning rules based on the custom scheduling parameters includes:
[0015] The time range parameter is parsed to obtain the time range for data cleaning;
[0016] The data cleaning rules are generated based on the time range of the data cleaning process.
[0017] Optionally, the method further includes:
[0018] The task status of data cleaning is acquired in real time, and the task status is recorded in the log.
[0019] Optionally, before generating data cleaning rules based on the custom scheduling parameters, the method further includes:
[0020] The custom scheduling parameters are verified to obtain the verified custom scheduling parameters; the verification includes format checking and conflict detection.
[0021] The process of generating data cleaning rules based on the custom scheduling parameters includes:
[0022] The data cleaning rules are generated based on the verified custom scheduling parameters.
[0023] Secondly, this application provides a data cleaning apparatus based on custom scheduling, comprising:
[0024] The acquisition module is used to acquire user-inputted custom scheduling parameters;
[0025] The generation module is used to generate data cleaning rules based on the custom scheduling parameters;
[0026] The data cleaning module is used to clean the data to be cleaned using the data cleaning rules, and obtain cleaned data.
[0027] The write-back module is used to write the cleaned data back to the database.
[0028] Optionally, the custom scheduling parameters include node range parameters, and the generation module is specifically used for:
[0029] The node range parameters are parsed to obtain the node range for data cleaning;
[0030] The data cleaning rules are generated based on the node range of the data cleaning process.
[0031] Optionally, the custom scheduling parameters include time range parameters, and the generation module is specifically used for:
[0032] The time range parameter is parsed to obtain the time range for data cleaning;
[0033] The data cleaning rules are generated based on the time range of the data cleaning process.
[0034] Optionally, the device further includes:
[0035] The logging module is used to obtain the task status of data cleaning in real time and record the task status in the log.
[0036] Optionally, the device further includes:
[0037] The verification module is used to verify the custom scheduling parameters and obtain the verified custom scheduling parameters; the verification includes format checking and conflict detection.
[0038] The generation module is specifically used for:
[0039] The data cleaning rules are generated based on the verified custom scheduling parameters.
[0040] Thirdly, embodiments of this application provide a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements a data cleaning method based on custom scheduling as described in any of the embodiments of the first aspect of this application.
[0041] Fourthly, embodiments of this application provide a computer-readable storage medium storing instructions that, when executed on a terminal device, cause the terminal device to perform a data cleaning method based on custom scheduling as described in any of the embodiments of the first aspect of this application.
[0042] This application provides a data cleaning method based on custom scheduling. When executing the method, user-inputted custom scheduling parameters are first obtained. Then, data cleaning rules are generated based on these parameters. Next, the data to be cleaned is cleaned using these rules, resulting in cleaned data. Finally, the cleaned data is written back to the database. This method, by allowing users to customize the data cleaning rules, breaks the limitations of traditional fixed data cleaning models, enabling flexible configuration of data cleaning rules to adapt to different data cleaning needs in various scenarios, thus improving the flexibility and efficiency of data cleaning. Attached Figure Description
[0043] To more clearly illustrate the technical solutions in this embodiment or the prior art, the drawings used in the description of the embodiment or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0044] Figure 1 A flowchart illustrating a data cleaning method based on custom scheduling provided in this application embodiment;
[0045] Figure 2 A schematic diagram of the structure of a system for executing a data cleaning method based on a custom schedule, provided in an embodiment of this application;
[0046] Figure 3 A schematic diagram of a data cleaning device based on custom scheduling provided in an embodiment of this application;
[0047] Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation
[0048] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. This application provides a data cleaning method and apparatus based on custom scheduling, relating to the field of data processing technology. The above are merely examples and do not limit the application field of the methods and apparatus provided in this application.
[0049] In the era of big data, data comes from a wide range of sources and comes in various formats. The data often contains redundancy, errors, inconsistencies, and other problems. This poor-quality data can seriously affect the accuracy and reliability of data analysis results. Therefore, data cleaning has become a key step in the data processing workflow.
[0050] Most existing data cleaning methods employ fixed cleaning rules and scheduling strategies, lacking flexibility and specificity. This results in data cleaning being neither flexible nor efficient.
[0051] The inventors, through research, proposed the technical solution of this application. First, they obtain user-inputted custom scheduling parameters. Then, based on these parameters, they generate data cleaning rules. Next, they use these rules to clean the data to be cleaned, obtaining the cleaned data. Finally, the cleaned data is written back to the database. This approach, by allowing users to customize data cleaning rules, breaks the limitations of traditional fixed data cleaning models, enabling flexible configuration of data cleaning rules to adapt to different data cleaning needs in various scenarios, thus improving the flexibility and efficiency of data cleaning.
[0052] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only a part of the embodiments of the present application, and not all of them. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present application. It should be noted that, for ease of description, only the parts related to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features in the embodiments of the present application can be combined with each other.
[0053] See Figure 1 , Figure 1 A flowchart of a data cleaning method based on custom scheduling provided in this application embodiment includes:
[0054] First, it should be noted that the data cleaning method based on custom scheduling provided in this application can be derived from, for example... Figure 2 The system shown is used to execute this. Figure 2 This application provides a schematic diagram of the structure of a system for executing a data cleaning method based on a custom schedule. It is a web-based distributed system using Tomcat as the web server, Spring Boot for the backend service, and Vue for the frontend interface, integrated with a big data platform (such as a Dataworks cluster). The client (Vue frontend) runs in the user's browser, providing a configuration interface for scheduling parameters, task monitoring, and result display. The server (Spring Boot backend) is deployed on Tomcat, handling business logic, scheduling execution, and data cleaning tasks. The big data platform (external system), such as the Dataworks cluster, executes the actual data cleaning job. The database stores scheduling configurations, task logs, and metadata.
[0055] S101: Obtain user-inputted custom scheduling parameters.
[0056] Users input custom scheduling parameters through the Vue frontend interface. These custom scheduling parameters include, but are not limited to, node range parameters and time range parameters. Node range parameters can limit the range of cleaned data, for example, downstream nodes after the transaction record cleansing node. Time range parameters can limit the time range of the cleaned data. The frontend sends the node range parameters to the Spring Boot backend via an HTTP API.
[0057] S102: Generate data cleaning rules based on custom scheduling parameters.
[0058] The Spring Boot backend scheduling engine module receives node range parameters and validates them. Validation methods include, but are not limited to, format checks and conflict detection. Then, based on the validated custom scheduling parameters, data cleaning rules are generated. Specifically, the range conditions are first parsed to generate data filtering rules, and then time triggers are set, based on cron expressions or custom intervals.
[0059] S103: Use data cleaning rules to clean the data to be cleaned and obtain the cleaned data.
[0060] A monitoring time trigger automatically initiates the task when a specified time interval is reached. The backend sends a cleaning job request to the big data platform via a REST API, along with range parameters. The scheduling engine automatically splits the task into multiple sub-tasks according to the backend configuration and supports task reruns.
[0061] The data cleaning executor of a big data platform (such as a Dataworks cluster) receives requests and performs actual cleaning, that is, filtering data according to the scope.
[0062] S104: Write the cleaned data back to the database.
[0063] Write the cleaned data back to the database and update the status.
[0064] In addition, task status (such as success, failure, and progress) can be collected in real time and logged. The Vue frontend obtains status updates from the backend via polling or WebSocket and displays task progress, cleaning results, and alarm information on the interface. Users can adjust parameters at any time, and the system dynamically reschedules tasks.
[0065] In the embodiments provided in this application, user-inputted custom scheduling parameters are first obtained. Then, data cleaning rules are generated based on these custom scheduling parameters. Next, the data to be cleaned is cleaned using these data cleaning rules, resulting in cleaned data. Finally, the cleaned data is written back to the database. This approach, by allowing users to customize and configure data cleaning rules, breaks the limitations of traditional fixed data cleaning models, enabling flexible configuration of data cleaning rules to adapt to different data cleaning needs in various scenarios, thus improving the flexibility and efficiency of data cleaning.
[0066] The above are some specific implementations of the data cleaning method based on custom scheduling provided in the embodiments of this application. Based on this, this application also provides a corresponding apparatus. The apparatus provided in the embodiments of this application will be described below from the perspective of functional modularity.
[0067] See Figure 3 , Figure 3 This application provides a schematic diagram of a data cleaning device based on custom scheduling, the data cleaning device 300 based on custom scheduling, comprising:
[0068] The acquisition module 310 is used to acquire user-inputted custom scheduling parameters;
[0069] The generation module 320 is used to generate data cleaning rules based on the custom scheduling parameters;
[0070] Data cleaning module 330 is used to clean the data to be cleaned using the data cleaning rules to obtain cleaned data;
[0071] The write-back module 340 is used to write the cleaned data back to the database.
[0072] Optionally, the custom scheduling parameters include node range parameters, and the generation module 320 is specifically used for:
[0073] The node range parameters are parsed to obtain the node range for data cleaning;
[0074] The data cleaning rules are generated based on the node range of the data cleaning process.
[0075] Optionally, the custom scheduling parameters include time range parameters, and the generation module 320 is specifically used for:
[0076] The time range parameter is parsed to obtain the time range for data cleaning;
[0077] The data cleaning rules are generated based on the time range of the data cleaning process.
[0078] Optionally, the device 300 further includes:
[0079] The logging module is used to obtain the task status of data cleaning in real time and record the task status in the log.
[0080] Optionally, the device 300 further includes:
[0081] The verification module is used to verify the custom scheduling parameters and obtain the verified custom scheduling parameters; the verification includes format checking and conflict detection.
[0082] The generation module 320 is specifically used for:
[0083] The data cleaning rules are generated based on the verified custom scheduling parameters.
[0084] This application also provides corresponding devices and computer storage media for implementing the solutions provided in this application.
[0085] like Figure 4 As shown, computer device 01 is represented in the form of a general-purpose computing device. The components of computer device 01 may include, but are not limited to: one or more processors or processor units 03, system memory 08, and bus 04 connecting different system components (including system memory 08 and processor unit 03).
[0086] Bus 04 represents one or more of several bus architectures, including memory buses or memory controllers, peripheral buses, graphics acceleration ports, processors, or local buses using any of the various bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.
[0087] Computer device 01 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by computer device 01, including volatile and non-volatile media, removable and non-removable media.
[0088] System memory 08 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 09 and / or cache memory 10. Computer device 01 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 11 may be used to read and write non-removable, non-volatile magnetic media (…). Figure 4 Not shown; usually referred to as a "hard drive"). Although Figure 4 As not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 04 via one or more data media interfaces. System memory 08 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present invention.
[0089] A program / utility 12 having a set (at least one) of program modules 13 may be stored, for example, in system memory 08. Such program modules 13 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 13 typically perform the functions and / or methods described in the embodiments of the present invention.
[0090] Computer device 01 can also communicate with one or more external devices 02 (e.g., keyboard, pointing device, display 07, etc.), and with one or more devices that enable a user to interact with the computer device 01, and / or with any device that enables the computer device 01 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed through input / output (I / O) interface 06. Furthermore, computer device 01 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) through network adapter 05. Figure 4 As shown, network adapter 05 communicates with other modules of computer device 01 via bus 04. It should be understood that, although... Figure 4 As not shown in the diagram, it can be used in conjunction with computer device 01 with other hardware and / or software modules, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0091] The processor unit 03 executes various functional applications and data processing by running programs stored in the system memory 08, such as implementing a data cleaning method based on custom scheduling provided in the embodiments of this application.
[0092] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0093] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that all or part of the steps in the methods of the above embodiments can be implemented by means of software plus a general-purpose hardware platform. Based on this understanding, the technical solution of this application can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as a read-only memory (ROM) / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, a server, or a network communication device such as a router) to execute the methods described in various embodiments or some parts of the embodiments of this application.
[0094] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.
[0095] The above description is merely an exemplary implementation of this application and is not intended to limit the scope of protection of this application.
Claims
1. A data cleaning method based on custom scheduling, characterized in that, include: Get user-inputted custom scheduling parameters; Data cleaning rules are generated based on the custom scheduling parameters; Using the aforementioned data cleaning rules, the data to be cleaned is cleaned to obtain cleaned data; The cleaned data is written back to the database.
2. The method according to claim 1, characterized in that, The custom scheduling parameters include node range parameters, and the generation of data cleaning rules based on the custom scheduling parameters includes: The node range parameters are parsed to obtain the node range for data cleaning; The data cleaning rules are generated based on the node range of the data cleaning process.
3. The method according to claim 1, characterized in that, The custom scheduling parameters include time range parameters, and the generation of data cleaning rules based on the custom scheduling parameters includes: The time range parameter is parsed to obtain the time range for data cleaning; The data cleaning rules are generated based on the time range of the data cleaning process.
4. The method according to claim 1, characterized in that, The method further includes: The task status of data cleaning is acquired in real time, and the task status is recorded in the log.
5. The method according to claim 1, characterized in that, Before generating data cleaning rules based on the custom scheduling parameters, the method further includes: The custom scheduling parameters are verified to obtain the verified custom scheduling parameters; the verification includes format checking and conflict detection. The process of generating data cleaning rules based on the custom scheduling parameters includes: The data cleaning rules are generated based on the verified custom scheduling parameters.
6. A data cleaning device based on custom scheduling, characterized in that, include: The acquisition module is used to acquire user-inputted custom scheduling parameters; The generation module is used to generate data cleaning rules based on the custom scheduling parameters; The data cleaning module is used to clean the data to be cleaned using the data cleaning rules, and obtain cleaned data. The write-back module is used to write the cleaned data back to the database.
7. The method according to claim 6, characterized in that, The custom scheduling parameters include node range parameters, and the generation module is specifically used for: The node range parameters are parsed to obtain the node range for data cleaning; The data cleaning rules are generated based on the node range of the data cleaning process.
8. The apparatus according to claim 6, characterized in that, The custom scheduling parameters include time range parameters, and the generation module is specifically used for: The time range parameter is parsed to obtain the time range for data cleaning; The data cleaning rules are generated based on the time range of the data cleaning process.
9. A computer device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the data cleaning method based on a custom schedule as described in any one of claims 1-5.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a terminal device, cause the terminal device to perform the data cleaning method based on a custom schedule as described in any one of claims 1-5.