Dirty data processing methods, devices, electronic equipment and storage media

Abstract

This disclosure relates to a dirty data processing method, apparatus, electronic device, and storage medium. The method includes: detecting whether a dirty data collection parameter exists in the configuration parameters of a data synchronization task; in response to the existence of a dirty data collection parameter, loading a target collection plugin based on the target plugin type configured in the dirty data collection parameter; and using the target collection plugin, employing an asynchronous thread to listen to a blocking queue to read and store dirty data from the blocking queue, wherein the blocking queue is used to store dirty data generated during the execution of the data synchronization task. By adopting the solution of this disclosure, when a dirty data collection parameter exists in the configuration parameters, the corresponding target collection plugin is loaded, and dirty data collection is implemented using an asynchronous thread and a blocking queue. This decouples the dirty data collection process from the data synchronization process, ensuring that dirty data collection does not affect the normal operation of data synchronization and thus guaranteeing the efficiency of data synchronization.

Description

Technical Field

[0001] This disclosure relates to the field of data synchronization technology, and in particular to a dirty data processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the continuous development of information technology and the increasing emphasis on the value of data, more and more enterprises and institutions have realized that the "data silos" phenomenon, caused by decentralized business processes, lack of interoperability, and lack of data sharing, poses significant challenges to data analysis, utilization, and mining. The first step in solving the "data silos" problem is to aggregate the data from various separate business systems or departments, establishing a unified data collection, processing, and sharing center. One of the data processing methods used in this process is data synchronization technology.

[0003] In practical applications of data synchronization technology, data from different networks, operating platforms, databases, data formats, and applications is extracted from the source end to the target end. During the insertion of data into the target end's data table, issues such as primary key conflicts, field length mismatches, and data type format mismatches may prevent data from being written correctly from the source end. This data that cannot be correctly written to the target end is called "dirty data." The occurrence of dirty data leads to data not being correctly stored in the database, resulting in deviations in subsequent data analysis and processing. Furthermore, the subsequent processing of this dirty data increases system maintenance costs.

[0004] Currently, there are two common solutions for handling dirty data generated during data synchronization in related technologies: one is to discard the dirty data directly without processing, which will lead to data loss and affect the accuracy of subsequent data processing, statistics and analysis; the other is to pause subsequent data writing operations and collect the dirty data when dirty data is detected. In this solution, the dirty data collection process is strongly coupled and related to the data synchronization process, which will also affect the efficiency and performance of data synchronization. Summary of the Invention

[0005] In order to solve the above-mentioned technical problems, or at least partially solve the above-mentioned technical problems, at least one embodiment of the present disclosure provides a dirty data processing method, apparatus, electronic device and storage medium.

[0006] In a first aspect, this disclosure provides a dirty data processing method, including:

[0007] Check if there are dirty data collection parameters in the configuration parameters of the data synchronization task;

[0008] In response to the presence of dirty data collection parameters, the target collection plugin is loaded based on the target plugin type configured in the dirty data collection parameters;

[0009] The target collection plugin employs an asynchronous thread to listen to a blocking queue, in order to read and store dirty data from the blocking queue, wherein the blocking queue is used to store dirty data generated during the execution of the data synchronization task.

[0010] Secondly, this disclosure provides a dirty data processing apparatus, comprising:

[0011] The detection module is used to detect whether there are dirty data collection parameters in the configuration parameters of the data synchronization task;

[0012] A loading module is used to load a target collection plugin based on the target plugin type configured in the dirty data collection parameters in response to the existence of dirty data collection parameters;

[0013] The dirty data collection module is used to read and store dirty data from the blocking queue by using an asynchronous thread to listen to the blocking queue through the target collection plugin. The blocking queue is used to store dirty data generated during the execution of the data synchronization task.

[0014] Thirdly, this disclosure provides an electronic device, including: a processor and a memory; the processor executes a dirty data processing method as described in any of the first aspects by invoking a program or instructions stored in the memory.

[0015] Fourthly, this disclosure provides a computer-readable storage medium storing a program or instructions that cause a computer to perform any of the dirty data processing methods provided in the first aspect.

[0016] Fifthly, this disclosure provides a computer program product, including a computer program, wherein the computer program, when executed by a processor, implements any of the dirty data processing methods provided in the first aspect.

[0017] The technical solution provided in this disclosure has at least the following advantages compared with the prior art:

[0018] In this embodiment, by detecting whether a dirty data collection parameter exists in the configuration parameters of the data synchronization task, and in response to the presence of such a parameter, a target collection plugin is loaded based on the target plugin type configured in the dirty data collection parameter. Then, the target collection plugin uses an asynchronous thread to listen to a blocking queue to read and store dirty data generated during the execution of the data synchronization task. By employing this scheme, when a dirty data collection parameter exists in the configuration parameters, the corresponding target collection plugin is loaded, and dirty data collection is implemented using an asynchronous thread and a blocking queue. This decouples the dirty data collection process from the data synchronization process, ensuring that dirty data collection does not affect the normal operation of data synchronization and thus guaranteeing the efficiency of data synchronization. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0020] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A schematic flowchart of a dirty data processing method provided in an embodiment of this disclosure;

[0022] Figure 2 This diagram illustrates the display of dirty data information in a visual dirty data management interface.

[0023] Figure 3 This is a schematic diagram of a dirty data details page as an exemplary embodiment of this disclosure;

[0024] Figure 4 This is a schematic block diagram of a dirty data processing apparatus as an exemplary embodiment of the present disclosure. Detailed Implementation

[0025] To better understand the above-described objectives, features, and advantages of this disclosure, the present disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. It is understood that the described embodiments are only some, not all, of the embodiments of this disclosure. The specific embodiments described herein are merely for explaining this disclosure and not for limiting it. Unless otherwise specified, the embodiments of this disclosure and the features within them can be combined with each other. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure are within the scope of protection of this disclosure.

[0026] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0027] If dirty data generated during data synchronization is discarded without processing, it will lead to data loss, which will affect the accuracy of subsequent data processing, statistics and analysis. Therefore, the current practice is to collect the dirty data.

[0028] In related technologies, the common practice for dirty data collection is as follows: during data synchronization, dirty data is detected every time data is inserted. Once dirty data is detected, subsequent data insertion operations are paused, and then dirty data collection begins. In other words, the existing dirty data collection operation is synchronized with the data insertion operation. This synchronous collection method tightly couples the dirty data collection process with the data synchronization process. Collecting dirty data can block the insertion of other data, thus affecting the efficiency and performance of data synchronization. In addition, the synchronous dirty data collection method is not flexible enough, cannot achieve pluggable dirty data collection, and has poor scalability, as it cannot specify different dirty data collection methods through configuration parameters.

[0029] To address the aforementioned issues, this disclosure provides a method for handling dirty data. By detecting the presence of a dirty data collection parameter in the configuration parameters of the data synchronization task, and in response to the presence of such a parameter, a target collection plugin is loaded based on the target plugin type configured in the dirty data collection parameter. Then, the target collection plugin uses an asynchronous thread to listen to a blocking queue to read and store dirty data generated during the execution of the data synchronization task. The blocking queue is used to store dirty data generated during the execution of the data synchronization task. By employing this scheme, when a dirty data collection parameter is present in the configuration parameters, the corresponding target collection plugin is loaded, and dirty data collection is implemented using an asynchronous thread and a blocking queue. This decouples the dirty data collection process from the data synchronization process, ensuring that dirty data collection does not affect the normal operation of data synchronization and thus guaranteeing the efficiency of data synchronization.

[0030] Furthermore, the scheme disclosed herein separates the dirty data collection and data synchronization processes. The corresponding target collection plugin is loaded for dirty data collection by configuring the dirty data collection parameters in the configuration parameters. This allows the dirty data collection function to be enabled and disabled through the configuration parameters, making the dirty data collection plugin pluggable and extensible. Moreover, the dirty data collection is implemented in the form of a plugin, which has low code intrusion, ensures data security, and is suitable for data synchronization frameworks such as DataX.

[0031] Figure 1 This is a flowchart illustrating a dirty data processing method provided in an embodiment of the present disclosure. The dirty data processing method can be executed by a dirty data processing device provided in the embodiment of the present disclosure. The dirty data processing device can be implemented using software and / or hardware, and can generally be integrated into an electronic device for execution. The electronic device can be any device with data synchronization capability.

[0032] like Figure 1 As shown, the dirty data processing method provided in this embodiment may include the following steps:

[0033] Step 101: Check if there are dirty data collection parameters in the configuration parameters of the data synchronization task.

[0034] Typically, before performing a data synchronization task, it is necessary to configure the relevant configuration parameters of the data synchronization task. These parameters may include, but are not limited to, source-side basic information, target-side basic information, field mapping information, and scheduling strategy information. Source-side basic information may include, but is not limited to, task name, task directory, task description, source data source, and source table. Target-side basic information may include, but is not limited to, target data source, data integration method, and primary key conflict policy. Field mapping information may include, but is not limited to, the field mapping relationship between the source table and the target table. Scheduling strategy information may include, but is not limited to, scheduling type, scheduling date, scheduling start time, synchronization strategy, whether to limit flow, whether to retry on failure, the limit for triggering automatic task termination due to dirty data, and timeout.

[0035] In this embodiment of the disclosure, to enable and disable the dirty data collection function, a dirty data collection parameter is added to the configuration parameters. If dirty data collection is required in the data synchronization task, the dirty data collection parameter can be configured in the configuration parameters. If dirty data collection is not required in the data synchronization task, the dirty data collection parameter can be omitted, and only the relevant parameters of the data synchronization task need to be configured. Therefore, in this embodiment of the disclosure, the presence or absence of a dirty data collection parameter in the configuration parameters of the data synchronization task can be used to determine whether the dirty data collection function needs to be enabled.

[0036] Step 102: In response to the existence of dirty data collection parameters, load the corresponding target collection plugin based on the target collection type configured in the dirty data collection parameters.

[0037] In this embodiment, when a dirty data collection parameter is detected in the configuration parameters, it indicates that the dirty data collection function needs to be enabled. Then, based on the target collection type configured in the dirty data collection parameter, a dirty data collection plugin matching the target collection type (referred to as the target collection plugin in this embodiment) is loaded, thereby enabling the dirty data collection function. If no dirty data collection parameter is detected in the configuration parameters, it indicates that the dirty data collection function does not need to be enabled. In this case, the dirty data collection plugin is not loaded, thus disabling the dirty data collection function. By configuring or not configuring the dirty data collection parameter to load or not load the dirty data collection plugin, the dirty data collection function is designed as a pluggable plugin, resulting in low code invasiveness and strong extensibility.

[0038] Optionally, a DirtyPluginManager can be set up in the electronic device to detect whether dirty data collection parameters exist in the configuration parameters and decide whether to load the dirty data collection-related plugins based on the detection results. When DirtyPluginManager detects that dirty data collection parameters have been configured, it will extract the configured target collection type from the dirty data collection parameters, and then use the class loader to load the corresponding target collection plugin according to the target collection type, thereby enabling the dirty data collection function.

[0039] For example, the `collectorType` parameter can be set to configure the type of dirty data collection. Assume the available dirty data collection types include Log collection and ElasticSearch collection. These two dirty data collection methods correspond to two plugin classes: Log collection corresponds to the `LogPluginCollector` plugin, and ElasticSearch collection corresponds to the `ElasticSearchPluginCollector` plugin. Both plugins are subclasses of the asynchronous thread abstract class `DirtyPluginCollector`. `DirtyPluginManager` starts `DirtyPluginCollector`, using asynchronous multithreading to achieve asynchronous dirty data collection. `DirtyPluginCollector` is an abstract class that implements the `Runnable` interface. This type of class is called a thread helper class, and it only defines one `run()` method, which can be used to implement multithreading. Because `DirtyPluginCollector` is an abstract class, when `DirtyPluginManager` loads the `DirtyPluginCollector` class, it actually loads its corresponding subclass, namely `LogPluginCollector` or `ElasticSearchPluginCollector`.

[0040] If the dirty data collection parameter is configured with `collectorType=Log`, then `DirtyPluginManager` will load the corresponding `LogPluginCollector` plugin as the target collection plugin to complete the asynchronous dirty data collection process. If the dirty data collection parameter is configured with `collectorType=ElasticSearch`, then `DirtyPluginManager` will load the corresponding `ElasticSearchPluginCollector` plugin as the target collection plugin to complete the asynchronous dirty data collection process. If the dirty data collection parameter does not have a specific value for `collectorType` (i.e., `collectorType=empty`), then by default, dirty data will be output to `Log`, and the `LogPluginCollector` plugin will be loaded as the target collection plugin to complete the asynchronous dirty data collection process.

[0041] It should be noted that the embodiments of this disclosure support the extension of dirty data collection types. If you want to extend the collection type, for example, to collect dirty data to storage media such as Kafka or Hadoop Distributed File System (HDFS), you only need to extend the corresponding dirty data collection plugin. Through the corresponding collection plugin, dirty data can be collected to the corresponding storage media, which has strong scalability and flexibility.

[0042] Step 103: Using the target collection plugin, an asynchronous thread listens to the blocking queue to read dirty data from the blocking queue and store it. The blocking queue is used to store dirty data generated during the execution of the data synchronization task.

[0043] Since data synchronization tasks can be configured with multiple degrees of parallelism, each parallel subtask may generate dirty data during the data synchronization process. In order to collect the dirty data generated in different parallel subtasks in a unified manner, this embodiment of the disclosure uses an asynchronous thread plus a blocking queue to collect dirty data. The asynchronous thread is a dirty data collection plugin, and a blocking queue LinkedBlockingQueue is defined in the asynchronous thread class to store dirty data.

[0044] In this embodiment of the disclosure, after loading the target collection plugin, an asynchronous thread can be used to listen to the blocking queue to collect dirty data.

[0045] Specifically, the asynchronous dirty data collection process is as follows: After a data synchronization task (or parallel subtask) generates dirty data during the synchronization process, it is collected by the electronic device. For example, the electronic device collects dirty data through DirtyPluginManager and saves the dirty data to the blocking queue in the target collection plugin. The run() method in the target collection plugin continuously listens to the blocking queue, continuously reads dirty data from the blocking queue, and stores the read dirty data.

[0046] Different collection types load different collection plugins, and the storage location of dirty data read from the blocking queue is also different.

[0047] For example, if the configured target collection type is Log collection, then the loaded target collection plugin is LogPluginCollector, and dirty data read from the blocking queue is output to the program's console and printed to the log. As another example, if the configured target collection type is ElasticSearch collection, then the loaded target collection plugin is ElasticSearchPluginCollector, and dirty data read from the blocking queue is stored through the distributed full-text search engine ElasticSearch. This storage method is more conducive to subsequent data querying and processing.

[0048] In this embodiment of the disclosure, by using an asynchronous thread and a queue, the data synchronization process is decoupled from the dirty data collection process. When dirty data is generated during the data synchronization process, it does not affect the continued insertion of subsequent data. The generated dirty data will be collected and processed by the asynchronous thread, which improves the efficiency and performance of data insertion.

[0049] The dirty data processing method of this disclosure detects whether a dirty data collection parameter exists in the configuration parameters of the data synchronization task. In response to the presence of a dirty data collection parameter, a target collection plugin is loaded based on the target plugin type configured in the dirty data collection parameter. Then, the target collection plugin uses an asynchronous thread to listen to a blocking queue to read and store dirty data generated during the execution of the data synchronization task. By employing the scheme of this disclosure, when a dirty data collection parameter exists in the configuration parameters, the corresponding target collection plugin is loaded, and dirty data collection is implemented using an asynchronous thread and a blocking queue. This decouples the dirty data collection process from the data synchronization process, ensuring that dirty data collection does not affect the normal operation of data synchronization and thus guaranteeing the efficiency of data synchronization.

[0050] As mentioned above, the dirty data collection methods disclosed in this paper mainly include Log collection and Elasticsearch collection, and other collection methods can be extended according to needs. However, for dirty data generated during data synchronization, it is necessary to accurately collect and properly process the dirty data. Therefore, to facilitate the management, display, and processing of dirty data, in one optional implementation of this paper, dirty data is collected by outputting it to Elasticsearch, while outputting it to Log is an extension solution that can be flexibly switched to when dirty data management, display, and processing are not required. Thus, the target collection type configured in the dirty data collection parameters can be Elasticsearch collection, and the loaded target collection plugin is the Elasticsearch collection plugin (i.e., the aforementioned ElasticsearchPluginCollector). When collecting generated dirty data, the Elasticsearch collection plugin can use an asynchronous thread to listen to the blocking queue to read dirty data (the exclusive dirty data is called the target dirty data) from the blocking queue, and then store the target dirty data in Elasticsearch. Elasticsearch is an open-source, highly scalable, distributed full-text search engine that can store and retrieve data in near real-time. In this embodiment, dirty data is collected and stored in Elasticsearch, which facilitates subsequent querying and processing of the dirty data.

[0051] To facilitate the management of dirty data generated during data synchronization, when the configured dirty data collection method is Elasticsearch, dirty data information can be saved to Elasticsearch. This allows users to easily view relevant information about the dirty data through a visual management interface that can be queried from Elasticsearch. The dirty data information to be saved to Elasticsearch may include, but is not limited to, task identifiers, task instance IDs, exception information that generated the dirty data, and inserted data items. Therefore, in one optional embodiment of this disclosure, the dirty data processing method may further include: obtaining the dirty data information stored in Elasticsearch and displaying the dirty data information in a visual dirty data management interface.

[0052] The information that can be displayed in the visual dirty data management interface may include, but is not limited to, task identifier, task instance ID, task name, target table name, run date, task start / end time, amount of data read from the source, amount of data written to the target, amount of dirty data, etc.

[0053] In this embodiment of the disclosure, dirty data generated during data synchronization can be stored in Elasticsearch. The dirty data information to be saved in Elasticsearch may include, but is not limited to, the task identifier, task instance ID, task name, target table name, run date, task start / end time, amount of data read from the source, amount of data written to the target, amount of dirty data, exception information that generated dirty data, and data items of dirty data. The electronic device can obtain the stored dirty data information from Elasticsearch and display the obtained dirty data information item by item in the visual dirty data management interface.

[0054] For example, Figure 2 This diagram illustrates the display of dirty data information in a visual dirty data management interface, such as... Figure 2 As shown, the visual dirty data management interface includes multiple display items such as task identifier (i.e., task ID), task name, task instance ID, source table, target table, run date, start time, end time, and amount of data read. Figure 2 Each row of data in the interface represents a piece of dirty data, that is, the dirty data information corresponding to a data synchronization task. The visual dirty data management interface also provides operation items for dirty data, including data replenishment operation 1 and close operation 2. A data replenishment operation and a close operation are set after each piece of dirty data information. Users can click data replenishment operation 1 to replenish the dirty data generated in the corresponding data synchronization task to the target data table, and can click close operation 2 to delete the corresponding dirty data information in the visual dirty data management interface. Figure 2 As shown, the visual dirty data management interface also provides batch data replenishment operation 3 and batch shutdown operation 4, allowing users to simultaneously perform data replenishment or shutdown operations on dirty data generated in multiple data synchronization tasks. Figure 2 As shown, the visual dirty data management interface also provides a task search box 5, where users can filter dirty data information generated in the corresponding task by entering the task ID or task name.

[0055] In this embodiment of the disclosure, dirty data information stored in ElasticSearch is obtained and displayed in a visual dirty data management interface, thereby realizing the visual display of relevant information about dirty data.

[0056] Furthermore, in an optional embodiment of this disclosure, the dirty data information includes a task identifier. Users can also view information such as abnormal dirty data and inserted data items by clicking on the task identifier displayed in the visual dirty data management interface. Therefore, the dirty data processing method of this disclosure further includes: in response to receiving a user's click operation on the target task identifier displayed in the visual dirty data management interface, displaying abnormal information about dirty data generated during the execution of the target data synchronization task corresponding to the target task identifier on the dirty data details page.

[0057] Users can click on the task identifier in any dirty data information displayed in the visual dirty data management interface to query the abnormal information of the corresponding dirty data. The task identifier clicked by the user is the target task identifier.

[0058] For example, the clickable task identifier in the visual dirty data management interface can be set to a different color than other information items. For instance, the task ID can be displayed in blue to indicate its clickability, while other information items are displayed in black. Users can click the task ID in the dirty data information, and in response to this click, they will be taken to the dirty data details page, as shown below. Figure 3 As shown, the dirty data details page displays abnormal information about dirty data generated during the execution of the target data synchronization task corresponding to the target task identifier clicked by the user.

[0059] In this embodiment of the disclosure, in response to receiving a user's click operation on the target task identifier displayed in the visual dirty data management interface, the abnormal information of dirty data generated during the execution of the target data synchronization task corresponding to the target task identifier is displayed on the dirty data details page. Thus, the query and visualization of the abnormal information of dirty data is realized, so that the user can quickly locate the reason for the data writing failure based on the abnormal information, and then make subsequent adjustments to the table structure or data accordingly.

[0060] Furthermore, in an optional embodiment of this disclosure, in order to improve the efficiency of processing dirty data and simplify the dirty data processing flow, this embodiment of the disclosure also designs a dirty data replenishment function, that is, to rewrite the data items of the collected dirty data into the target end data table. Since dirty data is mostly generated due to primary key conflicts, mismatched field lengths, mismatched data types, etc. on the target end, the exception information is usually field type conversion error, field length mismatch, etc. Therefore, in this embodiment of the disclosure, after displaying the exception information of dirty data generated during the execution of the target data synchronization task corresponding to the target task identifier in the dirty data details page, the table parameters of the target end data table corresponding to the target data synchronization task can also be obtained. The table parameters may include, but are not limited to, field type, field length, etc., and the table parameters are displayed to the user. The user can adjust the table parameters based on the displayed exception information. For example, when the exception information is field length mismatch, the field length in the table parameters is modified to be no less than the field length of the first dirty data. When the exception information is field type conversion error, the field type in the table parameters is modified to be consistent with the field type of the first dirty data. Next, the electronic device can receive the user's adjustment operation on the table parameters, and in response to the adjustment operation, modify the table parameters of the target data table, generate the target data table after the table parameters are adjusted, and then use the ElasticSearch read plugin to read the target dirty data corresponding to the target task identifier from ElasticSearch, and write the target dirty data into the target data table after the table parameters are adjusted.

[0061] It's understandable that the target data table corresponding to the target data synchronization task, i.e., the data table on the target end configured in the data synchronization task's configuration information to be synchronized to, is the data table on the target end to which data is written when a data write failure results in dirty data. For example, in a certain data synchronization task, if a data write fails to write a piece of data to data table 'a' on the target end, then that piece of data becomes dirty data. Data table 'a' is the target data table corresponding to that data synchronization task. If the user clicks on the task identifier of this data synchronization task in the visual dirty data management interface, then the task identifier of that data synchronization task is the target task identifier.

[0062] Since the dirty data has already been collected and stored in Elasticsearch, the dirty data replenishment function designed in this embodiment uses Elasticsearch as the source for data synchronization, and the target data table remains the same. Using Elasticsearch read plugins provided by synchronization frameworks such as DataX, the dirty data items stored in Elasticsearch are extracted and added to the target data table. This method enables simple and fast dirty data processing, efficiently ensuring data integrity during data synchronization.

[0063] In this embodiment of the disclosure, the user adjusts the table parameters of the target data table based on the abnormal information of the dirty data stored in Elasticsearch, thereby obtaining the target data table with adjusted table parameters. Then, the dirty data stored in Elasticsearch is written into the target data table with adjusted table parameters. This realizes the dirty data replenishment function, which can rewrite the generated dirty data, simplify the dirty data processing process, and improve the efficiency of dirty data processing.

[0064] To prevent data omissions during dirty data collection, in one optional embodiment of this disclosure, the quantity of synchronized data processed can also be checked. Therefore, in this embodiment, the dirty data processing method of this disclosure can further include: in response to the completion of the data synchronization task, obtaining the total number of data entries read from the source data table during the data synchronization process, obtaining the first number of data entries written to the target data table during the data synchronization process, and obtaining the second number of dirty data entries read from the blocking queue; then, in response to the fact that the sum of the first and second counts is inconsistent with the total number of entries, providing a data omission prompt to the user.

[0065] In practical applications, three counters can be designed to count the number of different types of data. These counters include: a read counter for counting the number of data entries read from the source table during data synchronization; a write counter for counting the number of data entries successfully written to the target table; and a dirty data counter for counting the number of dirty data entries read from the blocking queue. During data synchronization, after each batch of data is read from the source table, the read counter increments the number of read data entries to count the total amount of data read (i.e., the total number of data entries read). After data is successfully inserted during synchronization, the write counter increments the number of data entries in each batch of data successfully inserted into the target table to count the total amount of successfully inserted data (i.e., the number of data entries successfully written to the target table, referred to as the first count in this embodiment). After the dirty data collection plugin collects a dirty data item, that is, after reading a dirty data item from the blocking queue, the dirty data counter increments the current count value by 1 to count the total amount of dirty data collected (that is, the number of dirty data items read from the blocking queue, referred to as the second count in this embodiment).

[0066] Next, after the data synchronization task is completed, the electronic device can obtain the count values ​​of the three counters mentioned above. The count value obtained from the read counter is the total number of data entries read from the source data table during this data synchronization process. The count value obtained from the write counter is the first count of data successfully written to the target data table during this data synchronization process. The count value obtained from the dirty data counter is the second count of dirty data read from the blocking queue. Then, the sum of the first count and the second count is calculated and compared with the total count mentioned above. If the two are inconsistent, it is determined that data omission has occurred during the data synchronization process. At this time, a data omission prompt can be given to the user to remind them that data omission has occurred during this data synchronization process.

[0067] Optionally, the count values ​​collected by the three counters mentioned above can be saved to the database for subsequent querying and verification. When storing, the task identifier of this data synchronization task can be used as the unique identifier of the count value.

[0068] To implement the above embodiments, this disclosure also provides a dirty data processing apparatus.

[0069] Figure 4 This is a schematic block diagram of a dirty data processing apparatus according to an exemplary embodiment of the present disclosure, such as Figure 4 As shown, the dirty data processing device 40 includes: a detection module 410, a loading module 420, and a dirty data collection module 430.

[0070] Among them, the detection module 410 is used to detect whether there are dirty data collection parameters in the configuration parameters of the data synchronization task;

[0071] Loading module 420 is used to load a target collection plugin based on the target plugin type configured in the dirty data collection parameters in response to the existence of dirty data collection parameters;

[0072] The dirty data collection module 430 is used to read and store dirty data from the blocking queue by using an asynchronous thread to listen to the blocking queue through the target collection plugin, wherein the blocking queue is used to store dirty data generated during the execution of the data synchronization task.

[0073] Optionally, if the target collection type is ElasticSearch collection (a distributed full-text search engine), then the target collection plugin is an ElasticSearch collection plugin; the dirty data collection module 430 is further used for:

[0074] The ElasticSearch collection plugin uses an asynchronous thread to listen to the blocking queue in order to read target dirty data from the blocking queue.

[0075] The target dirty data is stored in Elasticsearch.

[0076] Further optionally, the dirty data processing device 40 further includes:

[0077] The first acquisition module is used to acquire the dirty data information corresponding to each dirty data item stored in the ElasticSearch.

[0078] The first display module is used to display the dirty data information in the visual dirty data management interface.

[0079] Optionally, the dirty data processing device 40 further includes:

[0080] The second display module is used to respond to the user's click operation on the first dirty data information displayed in the visual dirty data management interface, and to display the abnormal information of the first dirty data corresponding to the first dirty data information on the dirty data details page.

[0081] Optionally, the dirty data processing device 40 further includes:

[0082] The second acquisition module is used to acquire the table parameters of the target data table to be written to the first dirty data;

[0083] The generation module is used to receive the user's adjustment operation on the table parameters based on the abnormal information, and in response to the adjustment operation, generate the target data table after the table parameters are adjusted.

[0084] The read / write module is used to read the first dirty data from Elasticsearch using the Elasticsearch read plugin and write it into the target data table after the table parameters are adjusted.

[0085] Optionally, the dirty data processing device 40 further includes:

[0086] The data verification module is used to, in response to the completion of the data synchronization task, obtain the total number of data entries read from the source data table during the data synchronization process; obtain the first number of data entries written to the target data table during the data synchronization process; obtain the second number of dirty data entries read from the blocking queue; and, in response to the fact that the sum of the first and second data entries is inconsistent with the total number of data entries, provide a data omission prompt to the user.

[0087] The dirty data processing apparatus provided in this disclosure can execute any dirty data processing method applicable to electronic devices provided in this disclosure, and has the corresponding functional modules and beneficial effects for executing the method. Content not described in detail in the apparatus embodiments of this disclosure can be referred to the description in any method embodiment of this disclosure.

[0088] This disclosure also provides an electronic device, including a processor and a memory; the processor executes the steps of the dirty data processing method embodiments described above by calling programs or instructions stored in the memory. To avoid repetition, these steps will not be repeated here.

[0089] This disclosure also provides a non-transitory computer-readable storage medium that stores a program or instructions that cause a computer to perform the steps of the dirty data processing method described in the foregoing embodiments. To avoid repetition, these steps will not be repeated here.

[0090] This disclosure also provides a computer program product, including a computer program, wherein, when executed by a processor, the computer program implements the steps of the various embodiments of the dirty data processing method as described in the foregoing embodiments.

[0091] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0092] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for processing dirty data, characterized in that, The method includes: Check if there are dirty data collection parameters in the configuration parameters of the data synchronization task; In response to the presence of dirty data collection parameters, the corresponding target collection plugin is loaded based on the target collection type configured in the dirty data collection parameters; The target collection plugin uses an asynchronous thread to listen to the blocking queue, reads dirty data from the blocking queue and stores it. The blocking queue is used to store dirty data generated during the execution of the data synchronization task. The asynchronous thread plus the blocking queue is used to collect dirty data generated in different parallel subtasks in a unified manner, and decouples the data synchronization process from the dirty data collection process. Wherein, if the target collection type is ElasticSearch collection (a distributed full-text search engine), then the target collection plugin is an ElasticSearch collection plugin; Furthermore, the step of using the target collection plugin to asynchronously listen to the blocking queue, read dirty data from the blocking queue, and store it includes: The ElasticSearch collection plugin uses an asynchronous thread to listen to the blocking queue in order to read target dirty data from the blocking queue. Store the target dirty data in Elasticsearch; The method further includes: Retrieve dirty data information stored in the ElasticSearch; The dirty data information is displayed in a visual dirty data management interface, wherein the dirty data information includes a task identifier; In response to receiving a user's click on the target task identifier displayed in the visual dirty data management interface, the dirty data details page displays abnormal information about dirty data generated during the execution of the target data synchronization task corresponding to the target task identifier.

2. The dirty data processing method according to claim 1, characterized in that, The method further includes: Obtain the table parameters of the target data table corresponding to the target data synchronization task; Receive the user's adjustment operation on the table parameters based on the abnormal information, and in response to the adjustment operation, generate the target data table after the table parameters are adjusted; Using the ElasticSearch read plugin, the target dirty data corresponding to the target task identifier is read from the ElasticSearch, and the target dirty data is written into the target end data table after the table parameters are adjusted.

3. The dirty data processing method according to any one of claims 1-2, characterized in that, The method further includes: In response to the completion of the data synchronization task, obtain the total number of data entries read from the source data table during the data synchronization process; Get the first record number of data written to the target data table during the data synchronization process; Obtain the second count of dirty data read from the blocked queue; In response to the fact that the sum of the first number and the second number is inconsistent with the total number of numbers, a data omission prompt is given to the user.

4. A dirty data processing device, characterized in that, The device includes: The detection module is used to detect whether there are dirty data collection parameters in the configuration parameters of the data synchronization task; A loading module is used to load a target collection plugin based on the target plugin type configured in the dirty data collection parameters in response to the existence of dirty data collection parameters; The dirty data collection module is used to listen to the blocking queue using an asynchronous thread through the target collection plugin, so as to read dirty data from the blocking queue and store it. The blocking queue is used to store dirty data generated during the execution of the data synchronization task. The method of using the asynchronous thread and the blocking queue is used to collect dirty data generated in different parallel subtasks in a unified manner, and to decouple the data synchronization process from the dirty data collection process. Wherein, if the target collection type is ElasticSearch collection (a distributed full-text search engine), then the target collection plugin is an ElasticSearch collection plugin; The dirty data collection module is also used for: The ElasticSearch collection plugin uses an asynchronous thread to listen to the blocking queue in order to read target dirty data from the blocking queue. Store the target dirty data in Elasticsearch; The device further includes: The first acquisition module is used to acquire the dirty data information corresponding to each dirty data item stored in the ElasticSearch. The first display module is used to display the dirty data information in a visual dirty data management interface, wherein the dirty data information includes a task identifier; The second display module is used to respond to a user's click operation on the target task identifier displayed in the visual dirty data management interface, and to display abnormal information of dirty data generated during the execution of the target data synchronization task corresponding to the target task identifier on the dirty data details page.

5. The dirty data processing apparatus according to claim 4, characterized in that, The device further includes: The data verification module is used to, in response to the completion of the data synchronization task, obtain the total number of data entries read from the source data table during the data synchronization process; obtain the first number of data entries written to the target data table during the data synchronization process; obtain the second number of dirty data entries read from the blocking queue; and, in response to the fact that the sum of the first and second data entries is inconsistent with the total number of data entries, provide a data omission prompt to the user.

6. An electronic device, characterized in that, include: Processor and memory; The processor executes the dirty data processing method as described in any one of claims 1-3 by calling the program or instructions stored in the memory.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program or instructions that cause a computer to perform the dirty data processing method as described in any one of claims 1-3.

Images