Data processing method and related equipment

Abstract

The invention provides a data processing method and related equipment. The method comprises the following steps: in response to a received task scheduling instruction about a target task, reading to-be-compared first data and second data related to the target task in a paging manner based on a preset configuration rule; comparing the first data with the second data to obtain difference data between the first data and the second data; judging whether the difference data is within a preset range or not; responding to the condition that the difference data is not in the preset range, and determining that the first data is inconsistent with the second data; and recording abnormal information about the inconsistency.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing, and in particular to a data processing method and related equipment. Background Art

[0002] With the widespread application of information technology, a large number of users use Internet services to meet various needs such as shopping, catering, travel, and entertainment. The stability of online information products and platforms is particularly important, among which the stability and correctness of business data are the key. At the same time, online information products and platforms still need to iterate functions or complete technical reconstruction. When iterating functions or reconstructing systems, in order to ensure the correctness of online information products and platform data, the industry generally adopts the following standard processes: Phase 1, launch new functions, retain old functions, double-write new and old function data, read and use old functions and old data, and compare new and old data; Phase 2, grayscale process, double-write new and old function data, and gradually use new functions and new data; Phase 3, remove old function code and data. The existing data comparison and analysis process is usually implemented in iterative / refactoring code, and no unified solution has been settled. It has the following shortcomings: (1) Low efficiency. The data comparison and analysis function needs to be implemented in each iteration / refactoring work, and the efficiency of repeated work is low. (2) Unreliable quality. To achieve a comprehensive, reliable and stable data analysis and comparison function, high requirements are placed on R&D personnel. Defective functions of this type will affect the comparison work at the least and the normal operation of the business system at the worst. (3) Complex operation. If problems with the analysis function are found during the data comparison and analysis process, the problem needs to be fixed and the system is back online. The process is complicated and affects the stability of the system. (4) The notification mechanism is inefficient. When differences are found in the data comparison and analysis, the system that implements the data comparison and analysis function alone usually only records abnormal logs. R&D personnel have weak perception and fail to discover problems in a timely manner. Summary of the invention

[0003] The present disclosure proposes a data processing method and related equipment to solve technical problems such as low data processing efficiency and poor quality of processing results to a certain extent.

[0004] In a first aspect, the present disclosure provides a data processing method, comprising:

[0005] In response to receiving a task scheduling instruction regarding a target task, reading first data and second data to be compared related to the target task in pages based on a preset configuration rule;

[0006] Comparing the first data with the second data to obtain difference data between the first data and the second data;

[0007] Determining whether the difference data is within a preset range;

[0008] In response to the difference data not being within the preset range, determining that the first data is inconsistent with the second data;

[0009] Exception information regarding the inconsistency is logged.

[0010] In a second aspect of the present disclosure, a data processing device is provided, comprising:

[0011] A data reading module, configured to, in response to receiving a task scheduling instruction regarding a target task, read first data and second data to be compared related to the target task in pages based on a preset configuration rule;

[0012] a comparison and analysis module, configured to compare the first data with the second data to obtain difference data between the first data and the second data; and to determine whether the difference data is within a preset range; and in response to the difference data not being within the preset range, determine that the first data is inconsistent with the second data;

[0013] The difference processing module is used to record the abnormal information about the inconsistency.

[0014] In a third aspect of the present disclosure, an electronic device is provided, comprising one or more processors, a memory; and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, and the programs include instructions for executing the method described in the first aspect.

[0015] According to a fourth aspect of the present disclosure, a non-volatile computer-readable storage medium containing a computer program is provided. When the computer program is executed by one or more processors, the processors execute the method described in the first aspect.

[0016] In a fifth aspect of the present disclosure, a computer program product is provided, comprising computer program instructions, which, when executed on a computer, enable the computer to execute the method described in the first aspect.

[0017] From the above, it can be seen that the data processing method and related equipment provided by the present disclosure can efficiently realize data comparison requirements by configuring comparison rules without coding. The comparison results obtained are of reliable quality, and various data comparison requirements are comprehensively sorted out and analyzed, and superior resources are concentrated to create high-quality unified data comparison tools. When the data comparison and analysis function needs to be adjusted, there is no need to redeploy the service. The configuration can be modified to take effect in real time. The flow control properties can be configured according to the data volume and the characteristics of the comparison work to ensure the normal operation of the business system. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 A schematic diagram of a data processing architecture according to an embodiment of the present disclosure.

[0020] Figure 2 The figure is a schematic diagram of the hardware structure of an exemplary electronic device according to an embodiment of the present disclosure.

[0021] Figure 3 The figure is a flowchart of the data processing method according to the embodiment of the present disclosure.

[0022] Figure 4 A schematic diagram of data reading according to an embodiment of the present disclosure.

[0023] Figure 5 It is a schematic diagram of data comparison of an embodiment of the present disclosure.

[0024] Figure 6 Schematic diagram of a data processing device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0026] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0027] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0028] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0029] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0030] Figure 1 A schematic diagram of a data processing architecture of an embodiment of the present disclosure is shown. Figure 1 , the data processing architecture 100 may include a server 110, a terminal 120, and a network 130 providing a communication link. The server 110 and the terminal 120 may be connected via a wired or wireless network 130. The server 110 may be an independent physical server, or a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, security services, and CDN.

[0031] The terminal 120 may be implemented in hardware or software. For example, when the terminal 120 is implemented in hardware, it may be various electronic devices having a display screen and supporting page display, including but not limited to smart phones, tablet computers, e-book readers, laptop portable computers, and desktop computers, etc. When the terminal 120 device is implemented in software, it may be installed in the electronic devices listed above; it may be implemented as multiple software or software modules (such as software or software modules used to provide distributed services), or it may be implemented as a single software or software module, which is not specifically limited here.

[0032] It should be noted that the data processing method provided in the embodiment of the present application can be executed by the terminal 120 or by the server 110. It should be understood that Figure 1 The number of terminals, networks and servers in the embodiment is only for illustration and is not intended to limit the number of terminals, networks and servers. Any number of terminals, networks and servers may be provided as required.

[0033] Figure 2FIG. 2 shows a schematic diagram of the hardware structure of an exemplary electronic device 200 provided in an embodiment of the present disclosure. Figure 2 As shown, the electronic device 200 may include: a processor 202, a memory 204, a network module 206, a peripheral interface 208 and a bus 210. The processor 202, the memory 204, the network module 206 and the peripheral interface 208 are connected to each other in communication within the electronic device 200 through the bus 210.

[0034] Processor 202 may be a central processing unit (CPU), a neural network processor (NPU), a microcontroller (MCU), a programmable logic device, a digital signal processor (DSP), an application specific integrated circuit (ASIC), or one or more integrated circuits. Processor 202 may be used to perform functions related to the technology described in this disclosure. In some embodiments, processor 202 may also include multiple processors integrated into a single logical component. For example, Figure 2 As shown, processor 202 may include a plurality of processors 202a, 202b, and 202c.

[0035] The memory 204 may be configured to store data (eg, instructions, computer code, etc.). Figure 2 As shown, the data stored in the memory 204 may include program instructions (e.g., program instructions for implementing the data processing method of the embodiment of the present disclosure) and data to be processed (e.g., the memory may store configuration files of other modules, etc.). The processor 202 may also access the program instructions and data stored in the memory 204, and execute the program instructions to operate on the data to be processed. The memory 204 may include a volatile storage device or a non-volatile storage device. In some embodiments, the memory 204 may include a random access memory (RAM), a read-only memory (ROM), an optical disk, a magnetic disk, a hard disk, a solid-state drive (SSD), a flash memory, a memory stick, etc.

[0036] The network module 206 can be configured to provide communication with other external devices to the electronic device 200 via a network. The network can be any wired or wireless network capable of transmitting and receiving data. For example, the network can be a wired network, a local wireless network (e.g., Bluetooth, WiFi, near field communication (NFC) etc.), a cellular network, the Internet or a combination thereof. It is understood that the type of network is not limited to the above specific examples. In some embodiments, the network module 306 can include any number of network interface controllers (NICs), radio frequency modules, transceivers, modems, routers, gateways, adapters, cellular network chips, etc., in any combination.

[0037] The peripheral interface 208 can be configured to connect the electronic device 200 to one or more peripheral devices to achieve information input and output. For example, the peripheral devices can include input devices such as a keyboard, a mouse, a touch pad, a touch screen, a microphone, and various sensors, and output devices such as a display, a speaker, a vibrator, and an indicator light.

[0038] The bus 210 can be configured to transmit information between various components of the electronic device 200 (e.g., the processor 202, the memory 204, the network module 206, and the peripheral interface 208), such as an internal bus (e.g., a processor-memory bus), an external bus (USB port, PCI-E bus), etc.

[0039] It should be noted that, although the architecture of the electronic device 200 only shows the processor 202, the memory 204, the network module 206, the peripheral interface 208 and the bus 210, in the specific implementation process, the architecture of the electronic device 200 may also include other components necessary for normal execution. In addition, it can be understood by those skilled in the art that the architecture of the electronic device 200 may also only include the components necessary for implementing the embodiments of the present disclosure, and does not necessarily include all the components shown in the figure.

[0040] With the widespread application of information technology, a large number of users use Internet services to meet various needs such as shopping, catering, travel, and entertainment. The stability of online information products and platforms is particularly important, among which the stability and correctness of business data are the key. At the same time, online information products and platforms still need to iterate functions or complete technical reconstruction. How to efficiently achieve the above goals at the same time has brought huge challenges to technical personnel. When iterating functions or reconstructing systems, in order to ensure the correctness of online information products and platform data, the industry generally adopts the following standard processes: Phase 1, launch new functions, retain old functions, double-write new and old function data, read and use old functions and old data, and compare new and old data; Phase 2, grayscale process, double-write new and old function data, and gradually use new functions and new data; Phase 3, remove old function code and data. When implementing the above process in the industry, including data comparison and analysis, R&D personnel code on demand to support the above functions each time, and there is no unified solution. The existing data comparison and analysis process is usually implemented in iterative / refactoring code, and no unified solution has been precipitated. It has the following shortcomings: (1) Low efficiency. Each iteration / refactoring work needs to implement the data comparison and analysis function, which leads to low efficiency of repeated work; (2) Unreliable quality. To achieve a comprehensive, reliable and stable data analysis and comparison function, high requirements are placed on R&D personnel. Such defective functions may affect the comparison work at best and the normal operation of the business system at worst; (3) Complex operation. If problems with the analysis function are found during the data comparison and analysis process, the problem needs to be fixed and the system needs to be put back online. The process is complicated and affects the stability of the system; (4) The notification mechanism is inefficient. When differences are found in the data comparison and analysis, the system that implements the data comparison and analysis function alone usually only records abnormal logs. R&D personnel have weak perception and fail to find problems in time. Therefore, how to improve the efficiency and quality of data analysis and comparison has become a technical problem that needs to be solved urgently.

[0041] In view of this, the embodiments of the present disclosure provide a data processing method and related equipment. By configuring comparison rules, data comparison requirements can be efficiently realized without coding work. The obtained comparison results are of reliable quality, and various data comparison requirements are comprehensively sorted out and analyzed, and superior resources are concentrated to create high-quality unified data comparison tools. When the data comparison and analysis function needs to be adjusted, there is no need to redeploy the service. By modifying the configuration, it can take effect in real time. The flow control properties can be configured according to the data volume and the characteristics of the comparison work to ensure the normal operation of the business system.

[0042] See also Figure 3 , Figure 3 The schematic flow chart of the data processing method according to the embodiment of the present disclosure is shown. The data processing method according to the embodiment of the present disclosure can be deployed on the server side. Figure 3 In the data processing method 300, the data processing method 300 may further include the following steps.

[0043] In step S310 , in response to receiving a task scheduling instruction regarding a target task, first data and second data to be compared related to the target task are read in pages based on a preset configuration rule.

[0044] Among them, the task scheduling instruction may refer to an instruction issued by the system or user to trigger the execution of a specific task. For example, it is hoped that a data comparison task will be automatically run every morning to check whether the data of the previous day is abnormal. At this time, the scheduling system will issue a task scheduling instruction at the specified time. The preset configuration rule may refer to a pre-defined rule for guiding the specific steps of task execution. For example, the preset configuration rule may include the order of data reading, the conditions for comparison, the format of the output result, etc. For example, the rule may stipulate that 100 data are read for comparison each time. Paging reading may refer to reading data one by one according to a specified number or batch, rather than reading all data at once. For example, assuming that there are 1,000 data in the storage, and the preset configuration rule stipulates that 100 data are read each time, then the system will read all data in 10 times. The target task may refer to a task that needs to be performed, such as data processing, analysis or other operations, or comparing two sets of new and old data to check the consistency and integrity of the data. The first data and the second data to be compared may refer to two sets of data that need to be compared. For example: the first data may be old data, and the second data is new data, and the two sets of data are compared to confirm the consistency of the new and old data. For another example, the storage time of the first data and the second data may be the same or the storage time may be slightly different. In this case, by comparing the first data and the second data, it is possible to determine whether there is a problem in the generation logic of the second data.

[0045] Specifically, the user wants to regularly compare the sales data of the previous day with the data of the current day to check whether there are any anomalies. At the preset time, the task scheduling system can automatically issue a task scheduling instruction to trigger the data comparison task. The preset configuration rules stipulate that 100 pieces of data are read for comparison each time, and the data are read in the order of storage time. The system reads 100 pieces of data for comparison each time according to the preset configuration rules until all the data is read. Among them, the comparison can be performed after reading a page of new and old data each time. While comparing, the next page of new and old data can be read to improve the comparison efficiency.

[0046] In some embodiments, the task scheduling instruction is a single-point scheduling task instruction, and the first data and the second data are read in pages based on the preset configuration rule in direct response to the single-point scheduling task instruction.

[0047] In some embodiments, the task scheduling instruction is a full-scale scheduling task instruction, then a subtask corresponding to the full-scale scheduling task instruction is obtained based on the preset configuration rule, and the first data and the second data are read in pages for the subtask based on the preset configuration rule.

[0048] Among them, after the data comparison tool receives the scheduling command issued by the task scheduling service, if it is a single-node scheduling task, it will be executed directly. If it is a full-volume scheduling, it will obtain the data comparison and analysis subtask from the configuration management service. If it can be obtained, the sub-process will be executed. If the sub-task cannot be obtained, the process will be terminated. For example, the comparison task is divided into 5 subtasks, there are 6 computing nodes, and one node cannot obtain the task, so it will be idle and not execute the task. . Specifically, when the data comparison tool receives the scheduling command issued by the task scheduling service, it first determines whether it is a single-node scheduling task; if so, it will immediately start to execute the task. On the contrary, if it is a scheduling task involving the full amount of data, the tool will further request the configuration management service to obtain a specific list of data comparison and analysis subtasks; once these subtasks are successfully obtained, the data comparison tool will execute each sub-process in turn to complete the comparison and analysis of the entire full amount of data. This process not only ensures the flexibility of task execution, but also improves the efficiency and accuracy of large-scale data processing.

[0049] In some embodiments, reading the first data and the second data to be compared in pages based on a preset configuration rule includes:

[0050] Determine whether the reading rate exceeds a preset rate;

[0051] In response to the reading rate exceeding the preset rate, the first data and the second data are continued to be read after waiting for a preset time period.

[0052] Among them, it is determined whether the data reading rate exceeds the limit. If it exceeds the limit, wait for the configured time. Since the tool is deployed on the business node and directly reads the production database, the data reading and processing rate is controlled to prevent affecting the normal operation of the business system and ensure system stability. Specifically, since the time consumption of data reading and processing is related to the data storage, query, and processing methods, the amount of data within a certain period of time can be set to limit the flow. For example, 100,000 data are read in 1 minute, and the data reading and comparative analysis processing are completed in 40 seconds. Then, every 10 seconds, the flow limiting tool can be asked whether it can start reading the next batch of data.

[0053] In some embodiments, reading the first data and the second data to be compared in pages based on a preset configuration rule further includes:

[0054] In response to the reading rate not exceeding the preset rate, determining whether the first data and the second data are in the same database;

[0055] In response to the first data and the second data being in the same database, determining whether the first data and the second data are in the same data table;

[0056] In response to the first data and the second data being in the same data table, acquiring the first data and the second data based on a single-table paging query;

[0057] In response to the first data and the second data not being in the same data table, the first data and the second data are acquired based on a multi-table associated paging query.

[0058] Among them, it is determined whether the data to be compared exists in the same database. If the data to be compared does not exist in the same database, a page of new and old data to be compared and analyzed is read separately. The purpose of paging is to control the amount of data read at a time and reduce the impact on the business. The paging size can be configured. If the data to be compared is in the same database, the new and old data pages to be compared and analyzed are matched by associated attributes to obtain each pair of new and old data to be compared and analyzed. When the new and old data are in the same database, there are two situations. One is that the new and old data exist in the same table, and each data row queried by paging that meets the conditions contains the new and old data to be compared and analyzed; the other is that the new and old data exist in different tables. Through the associated attributes of two or more tables, the new and old data to be compared and analyzed can be queried by paging.

[0059] In some embodiments, method 300 further includes:

[0060] In response to the first data and the second data not being in the same database, the first data and the second data are respectively acquired from corresponding databases.

[0061] Specifically, see Figure 4 , Figure 4 A schematic diagram of data reading according to an embodiment of the present disclosure is shown. When the data comparison tool performs a task, it first determines whether the data reading rate exceeds the preset limit; if it exceeds the limit, it waits for the configured time before continuing to ensure that the normal operation of the business system is not affected by the excessively high reading rate, thereby ensuring the stability of the system. Next, the tool checks whether the data to be compared is located in the same database; if not, it reads a page of new and old data pages to be compared and analyzed respectively, and controls the amount of data read at a time through paged reading to further reduce the impact on the business, and the page size can be configured according to actual conditions. If the data is located in the same library, it is further distinguished according to the specific situation: if the new and old data exist in the same table, the new and old data to be compared and analyzed can be obtained directly through paged query; if the new and old data exist in different tables, paged query is performed through associated attributes, and the new and old data pairs to be compared and analyzed can also be obtained. Finally, these new and old data pairs are compared and analyzed to find inconsistent data and ensure the consistency and accuracy of the data. This process not only effectively controls the rate of data reading and processing, but also ensures the efficient and stable operation of the system through flexible data reading strategies.

[0062] In step S320, the first data and the second data are compared to obtain difference data between the first data and the second data.

[0063] Among them, the new and old data to be compared are read according to the preset configuration rules. The data can be read in paging mode and read under the flow control rate limit. After each page of new and old data is read, the data comparison and analysis begins. After processing or while processing, the data to be processed continues to be read. It is determined whether there is data to be compared and analyzed. If not, the data to be compared is read. If no data is read, the comparison and analysis work is completed. If there is data to be compared, the data to be compared and analyzed is processed cyclically to compare and analyze the new and old data to be compared and analyzed, and inconsistent data is found, such as Figure 5 As shown, Figure 5 A schematic diagram of data comparison according to an embodiment of the present disclosure is shown.

[0064] In some embodiments, comparing the first data and the second data includes:

[0065] Determining whether the first data and the second data are multiple data;

[0066] In response to the first data and the second data being multiple pieces of data, the first data and the second data are merged and compared.

[0067] The new and old data can be one or more data. Determine whether the data to be compared and analyzed needs to be merged. If the new and old data are multiple data, they need to be merged. To merge the data to be analyzed, the tool provides data merging methods such as numbers, characters, and amounts. The business system can also expand the merging methods as needed.

[0068] Specifically, when the new and old data to be compared and analyzed contain one or more data, the system will first determine whether the data to be compared and analyzed need to be merged. If the new and old data contain multiple data, they need to be merged to ensure the consistency and accuracy of the data comparison. For example, the system first checks whether the data to be compared and analyzed contains multiple data. If there is only one data item, it does not need to be merged. If the new and old data contain multiple data items, they need to be merged into a unified format for comparison and analysis. The tool provides a variety of data merging methods, including merging methods for numbers, characters, amounts, etc., and also supports business systems to expand more merging methods according to specific needs. For example, digital merging: If the data to be compared contains fields of digital type (such as order amount), the system can merge these digital fields into a unified digital type to facilitate numerical comparison. Character merging: If the data to be compared contains fields of character type (such as order status), the system can merge these character fields into a unified character type to facilitate string comparison. Amount merging: If the data to be compared contains fields of amount type (such as order amount), the system can merge these amount fields into a unified amount type and compare the amounts. It should be understood that in addition to the standard merge methods provided by the tool, the business system can also extend more merge methods according to specific needs. For example, if there are specific data types or formats in the business system, these special types of data can be processed through custom merge methods. For example, if there are date and time type fields (such as order time) in the business system, these date and time fields can be merged into a unified date and time type through custom merge methods, and time comparisons can be performed. In this way, the system can ensure that different types of data are correctly integrated before comparative analysis, thereby improving the accuracy and efficiency of data comparison. Different types of data to be compared are merged into a unified format to ensure the consistency of data comparison. Through standard merge methods and custom extension methods, the system can handle various types of data, improving the accuracy and efficiency of data comparison. The business system can extend the merge method according to specific needs, increasing the flexibility and adaptability of the system.

[0069] In step S330, it is determined whether the difference data is within a preset range.

[0070] Among them, a preset range is set based on the preset configuration rule to determine the tolerance difference, and it is judged whether the calculated difference is within the preset range. If it is not within the preset range, it means that the difference is large, and it is determined that the data to be processed is inconsistent; if the difference is within the preset range, it is small, and it is determined that the data to be processed is consistent, and the next set of new and old data to be compared is continued to be obtained and compared.

[0071] In step S340, in response to the difference data not being within the preset range, it is determined that the first data is inconsistent with the second data.

[0072] Among them, the new and old data items are compared. If the difference is within an acceptable range, the data is considered consistent, otherwise the data is considered inconsistent.

[0073] In step S350, abnormal information about the inconsistency is recorded.

[0074] Among them, inconsistent data processing, recording and comparing abnormal logs, reporting statistical data, and notifying technical personnel on demand based on configuration. The notification method currently supports SMS and instant messaging tools.

[0075] In some embodiments, method 300 further includes at least one of the following:

[0076] Reporting the buried data about the abnormal information;

[0077] Alternatively, a corresponding target object is prompted for the abnormal information.

[0078] Specifically, when data comparison and analysis finds inconsistent data, these data need to be processed. This may include recording specific inconsistent information, such as the source of the data, the specific content of the inconsistency, etc. The inconsistent data found and its related information are recorded in a log file for subsequent analysis and tracking. The relevant statistical information of the inconsistent data found can also be reported to the monitoring system or data analysis platform for further analysis and monitoring. According to the preset configuration rules, decide whether to notify the target object, that is, the relevant staff, and select the appropriate notification method, such as SMS, instant messaging, email, etc.

[0079] It can be seen that compared with the comparison function provided in an independent system mode in the prior art, the method according to the embodiment of the present disclosure adopts a toolkit mode to introduce it into the business service, and does not require additional computing resources. It also provides general data comparison and analysis capabilities and provides a set of general configuration specifications. Users can write configuration files according to the standards to meet comparison requirements. The configuration files are stored through the microservice configuration management service, which can support real-time modification and real-time effectiveness. While providing standard and comprehensive data comparison functions, business system technicians can expand the functions of key links, including data reading, data merging, data comparison and other links. In addition, a variety of notification methods are adopted. When it is found that the data comparison difference is large in the prior art, only abnormal data is stored. While recording the comparison anomaly, this solution can notify through SMS, instant messaging software, etc. The notification mechanism is more efficient. In addition, by reporting the buried data, it can be connected to a visual monitoring platform similar to Grafana to achieve efficient monitoring and management.

[0080] It should be noted that the method of the embodiment of the present disclosure can be performed by a single device, such as a computer or a server. The method of the present embodiment can also be applied in a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present disclosure, and the multiple devices will interact with each other to complete the described method.

[0081] It should be noted that the above describes some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0082] Based on the same technical concept, corresponding to any of the above-mentioned embodiments and methods, the present disclosure also provides a data processing device, see Figure 6 , the data processing device comprises:

[0083] A data reading module, configured to, in response to receiving a task scheduling instruction regarding a target task, read first data and second data to be compared related to the target task in pages based on a preset configuration rule; wherein the storage time of the first data is before the storage time of the second data;

[0084] a comparison and analysis module, configured to compare the first data with the second data to obtain difference data between the first data and the second data; and to determine whether the difference data is within a preset range; and in response to the difference data not being within the preset range, determine that the first data is inconsistent with the second data;

[0085] The difference processing module is used to record the abnormal information about the inconsistency.

[0086] Specifically, the data processing device can be applied to the microservice architecture, and can also meet the requirements of the traditional single service architecture through appropriate adjustments. The overall architecture diagram of the technical solution is as follows: Figure 1As shown. The disclosed embodiment meets the data comparison requirements of system iteration / reconstruction in the form of a toolkit, mainly considering that such requirements are one-time or phased, and the use of a toolkit is more flexible. For systems with iteration / reconstruction, a toolkit in the form of a jar package is introduced, and the data comparison and analysis capabilities provided by the toolkit can be used after simple configuration. This tool currently supports comparative analysis of relational databases, non-relational databases, and cached data. Relational databases include MySQL, Oracle, etc., non-relational databases include MongoDB, HBase, etc., and cache services include Redis, Memcached, etc. The lightweight implementation of this toolkit relies on the rich and powerful basic capabilities provided by the basic platform, and the basic services it relies on include task scheduling services, configuration management services, log services, data tracking services, monitoring services, instant messaging services, and SMS services. The core modules contained in the data comparison toolkit according to the disclosed embodiment include:

[0087] The task control module is responsible for receiving the call commands of the task scheduling service. The task scheduling modes mainly include single-node scheduling and full scheduling. The full scheduling mode can be used for large data volume comparison and analysis scenarios. All computing nodes of the deployment toolkit can participate in data comparison and analysis, thereby enhancing the comparison and analysis capabilities for large data volumes.

[0088] The configuration reading module is responsible for reading configuration data from the configuration management service in real time. When the configuration data is updated, it supports real-time reading of the updated data. The read configuration data is used to control the entire data comparison and analysis process.

[0089] The data reading module supports the reading, filtering, and association of comparative analysis data. In terms of reading mode, it supports both full and incremental modes. In terms of filtering mode, in the full task scheduling mode, it provides data splitting and filtering methods based on database sub-libraries and sub-tables, business fields, etc., and also provides an extension mode. The business system can customize the splitting mode of the data set to be compared and analyzed. In terms of association mode, it supports a variety of different association modes, such as different columns of a single table and multi-table association in the same database, and multi-table association in different databases.

[0090] Traffic control module. Since the tool is deployed on the business node and directly reads the production database, the traffic control module controls the reading rate of the data reading module to prevent affecting the normal operation of the business system and ensure system stability.

[0091] The comparative analysis module is responsible for comparative analysis of new and old data. It supports comparative analysis of two or two groups of data, supports commonly used data aggregation and data comparison methods, and provides extended interfaces to support business system customized data aggregation and comparison methods.

[0092] The difference processing module records the abnormal comparison logs for the data determined to be different by the comparison and analysis module, reports the statistical data, and notifies the technical staff as needed based on the configuration. The notification method currently supports SMS and instant messaging tools.

[0093] For the convenience of description, the above device is described by dividing it into various modules according to its functions. Of course, when implementing the present disclosure, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0094] The device of the above embodiment is used to implement the corresponding data processing method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0095] Based on the same technical concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the data processing method described in any of the above embodiments.

[0096] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0097] The computer instructions stored in the storage medium of the above embodiments are used to enable the computer to execute the data processing method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0098] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Based on the concept of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present disclosure as described above, which are not provided in detail for the sake of simplicity.

[0099] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present disclosure difficult to understand, the known power / ground connections to the integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present disclosure difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present disclosure will be implemented (that is, these details should be fully within the scope of understanding of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present disclosure, it is apparent to those skilled in the art that the embodiments of the present disclosure can be implemented without these specific details or with changes in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0100] Although the present disclosure has been described in conjunction with specific embodiments of the present disclosure, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0101] The embodiments of the present disclosure are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A data processing method, comprising: In response to receiving a task scheduling instruction regarding a target task, reading first data and second data to be compared related to the target task in pages based on a preset configuration rule; Comparing the first data with the second data to obtain difference data between the first data and the second data; Determining whether the difference data is within a preset range; In response to the difference data not being within the preset range, determining that the first data is inconsistent with the second data; Exception information regarding the inconsistency is logged.

2. The method according to claim 1, wherein Reading the first data and the second data to be compared in pages based on a preset configuration rule includes: Determine whether the reading rate exceeds a preset rate; In response to the reading rate exceeding the preset rate, the first data and the second data are continued to be read after waiting for a preset time period.

3. The method according to claim 2, wherein: The method further comprises: reading the first data and the second data to be compared in pages based on the preset configuration rules; In response to the reading rate not exceeding the preset rate, determining whether the first data and the second data are in the same database; In response to the first data and the second data being in the same database, determining whether the first data and the second data are in the same data table; In response to the first data and the second data being in the same data table, acquiring the first data and the second data based on a single-table paging query; In response to the first data and the second data not being in the same data table, the first data and the second data are acquired based on a multi-table associated paging query.

4. The method according to claim 3, further comprising: In response to the first data and the second data not being in the same database, the first data and the second data are respectively acquired from corresponding databases.

5. The method according to claim 1, wherein: Comparing the first data with the second data includes: Determining whether the first data and the second data are multiple data; In response to the first data and the second data being multiple pieces of data, the first data and the second data are merged and compared.

6. The method according to claim 1, wherein: If the task scheduling instruction is a single-point scheduling task instruction, directly respond to the single-point scheduling task instruction and read the first data and the second data in pages based on the preset configuration rule; Alternatively, if the task scheduling instruction is a full-task scheduling instruction, a subtask corresponding to the full-task scheduling instruction is acquired based on the preset configuration rule, and the first data and the second data are read in pages for the subtask based on the preset configuration rule.

7. The method according to claim 1, further comprising at least one of the following: Reporting the buried data about the abnormal information; Alternatively, a corresponding target object is prompted for the abnormal information.

8. A data processing device, comprising: A data reading module, configured to, in response to receiving a task scheduling instruction regarding a target task, read first data and second data to be compared related to the target task in pages based on a preset configuration rule; A comparison and analysis module, used for comparing the first data with the second data to obtain difference data between the first data and the second data; and determining whether the difference data is within a preset range; and in response to the difference data not being within the preset range, determining that the first data is inconsistent with the second data; The difference processing module is used to record the abnormal information about the inconsistency.

9. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the program. 10 . A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method according to claim 1 .

Images