Data processing method, data processing system, device, equipment and medium

Abstract

The invention discloses a data processing method, a data processing system, a data processing device, equipment and a medium, and relates to the technical field of computers. The method comprises the following steps: receiving consumption data sent by a disaster recovery end message queue platform, the consumption data being determined by the disaster recovery end message queue platform based on a data consumption request sent by a data receiving end, and the consumption data comprising a data identifier; determining a data processing operation for processing the consumption data based on the data identifier and a business data table corresponding to the data consumption request; and executing data processing operation on the consumption data in the business data table. According to the technical scheme provided by the invention, the abnormal data can be accurately identified in disaster recovery switching, the time sequence and integrity of the data in the service data table can be ensured, and the data consistency and service stability of the distributed system are ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and in particular to a data processing method, a data processing system, an apparatus, an equipment and a medium. BACKGROUND

[0002] In a distributed system, a message queue cluster becomes a core component for data transmission between a producer and multiple downstream consumers by virtue of a publish-subscribe mode. To guarantee service continuity, a disaster recovery architecture of a production end and a disaster recovery end is usually adopted, that is, the production end provides services in normal times, and the disaster recovery end is switched to in case of an exception of the production end. However, in the disaster recovery switching process, due to problems such as node failure in the message queue cluster and failure of consumption progress synchronization, message disorder (such as reversal of the time sequence of data) and repetition (such as multiple deliveries of the same data) are easily caused, which destroys the timeliness, integrity and sequence of data. Downstream consumers need to rely on accurate and ordered data to support business logic, and it is difficult for the prior art to cope with such data abnormalities caused by disaster recovery switching, which easily causes business processing errors. Therefore, how to solve the problem of data abnormalities caused in the disaster recovery switching process has become a key technical challenge to guarantee the data consistency and business stability of the distributed system. SUMMARY

[0003] The present application provides a data processing method, a data processing system, an apparatus, an equipment and a medium, which can accurately identify abnormal data in disaster recovery switching, can ensure the time sequence and integrity of data in a business data table, and guarantees the data consistency and business stability of the distributed system.

[0004] In a first aspect, the present application provides a data processing method applied to a data receiving end connected with a disaster recovery end message queue platform, which comprises the following steps:

[0005] receiving consumption data sent by the disaster recovery end message queue platform, the consumption data being determined by the disaster recovery end message queue platform based on a data consumption request sent by the data receiving end, and the consumption data comprising a data identifier;

[0006] determining a data processing operation for processing the consumption data based on the data identifier and a business data table corresponding to the data consumption request;

[0007] executing the data processing operation on the consumption data in the business data table.

[0008] Further, the data identifier at least includes a data timestamp and a data primary key, and the business data table at least includes a data master table and a data detail table; and the determining, based on the data identifier and the data consumption request, of a data processing operation for processing the consumption data, includes: querying whether the data primary key exists in the data master table; if the data primary key does not exist, determining that the data processing operation is an addition operation; if the data primary key exists, extracting a historical consumption detail sub-table corresponding to the data primary key from the data detail table; determining, based on the historical consumption detail sub-table, whether the data timestamp satisfies a preset abnormal condition; if the preset abnormal condition is satisfied, determining that the data processing operation is an abnormal processing operation; and if the preset abnormal condition is not satisfied, determining that the data processing operation is a modification operation.

[0009] Further, the preset abnormal condition at least includes one of: the data timestamp is consistent with a timestamp of historical data in the historical consumption detail sub-table; and compared with the timestamp of the historical data, the data timestamp is not the closest time to a current time.

[0010] Further, in a case where the data processing operation is the abnormal processing operation, the performing, in the business data table, of the data processing operation on the consumption data includes: determining a corresponding abnormal type based on the preset abnormal condition; if the abnormal type is data duplication, determining that the abnormal processing operation is discarding the consumption data; and if the abnormal type is data out-of-order, determining a modification position of the consumption data in the data detail table based on the data timestamp, and determining that the abnormal processing operation is writing detail data corresponding to the consumption data at the modification position.

[0011] Further, in a case where the data processing operation is the addition operation, the performing, in the business data table, of the data processing operation on the consumption data includes: obtaining current state data of the data primary key; determining the current state data and the data timestamp as master information of the consumption data, and adding the master information in the data master table; and adding detail data corresponding to the consumption data in the data detail table.

[0012] Further, in a case where the data processing operation is the modification operation, the performing, in the business data table, of the data processing operation on the consumption data includes: writing detail data corresponding to the consumption data in the data detail table.

[0013] In a second aspect, the present application provides a data processing system, comprising a data production end, a production end message queue platform, a disaster recovery end message queue platform and a data receiving end, wherein the data receiving end is connected with the disaster recovery end message queue platform and the production end message queue platform respectively, and the data production end is connected with the disaster recovery end message queue platform and the production end message queue platform respectively.

[0014] In a third aspect, the present application provides a data processing device integrated in a data receiving end, wherein the data receiving end is connected with a disaster recovery end message queue platform, and the device comprises:

[0015] a data receiving module, configured to receive consumption data sent by the disaster recovery end message queue platform, wherein the consumption data is determined by the disaster recovery end message queue platform based on a data consumption request sent by the data receiving end, and the consumption data comprises a data identifier;

[0016] an operation determining module, configured to determine a data processing operation for processing the consumption data based on the data identifier and a business data table corresponding to the data consumption request;

[0017] a data processing module, configured to execute the data processing operation on the consumption data in the business data table.

[0018] In a fourth aspect, the present application provides an electronic device, comprising at least one processor, and a memory connected with the at least one processor in communication, wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the data processing method according to any of the embodiments of the present application.

[0019] In a fifth aspect, the present application provides a computer readable storage medium, which stores computer instructions, and the computer instructions are used to enable a processor to execute the data processing method according to any of the embodiments of the present application.

[0020] To solve the defects of the prior art in the background art, the embodiments of the present application provide a data processing method, and the execution of the method can bring the following beneficial effects: in the disaster recovery switching scene, the present application can quickly judge whether the data is duplicated or out of order or other abnormalities through the association verification of the data identifier and the business data table, can accurately identify abnormal data, and avoids directly writing abnormal data into the business data table. Different data processing operations are performed according to the identified abnormal types, to ensure the time sequence and integrity of the data in the business data table. Even if the data is abnormal due to the failure of offset synchronization at the disaster recovery end, the present application can quickly recover the data accuracy after disaster recovery switching through the abnormal verification and automatic error correction of the data identifier, and reduce the cost of manual intervention. The data abnormality problem in disaster recovery switching can be effectively solved, and the data consistency and business stability of the distributed system are guaranteed.

[0021] It should be noted that the above computer instructions can be stored in whole or in part on a computer readable storage medium. The computer readable storage medium can be packaged together with the processor of the data processing device, or can be packaged separately from the processor of the data processing device, and the present application does not limit this.

[0022] The description of the second aspect, the third aspect, …, and the fifth aspect in the present application can refer to the detailed description of the first aspect; and the beneficial effects of the description of the second aspect, the third aspect, …, and the fifth aspect can refer to the beneficial effect analysis of the first aspect, which will not be described here.

[0023] It should be understood that the contents described in this part are not intended to identify the key or important features of the embodiments of the present application, nor are they used to limit the scope of the present application. Other features of the present application will become apparent through the following description.

[0024] It can be understood that before using the technical solutions disclosed in the embodiments of the present application, the type, use range and use scene of the personal information involved in the present application should be informed to the user and the authorization of the user should be obtained through appropriate means according to relevant laws and regulations. BRIEF DESCRIPTION OF DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creating any creative labor on the basis of these drawings.

[0026] Figure 1 A flowchart of a data processing method provided by the embodiments of the present application is shown in the figure;

[0027] Figure 2A framework schematic diagram of a data processing system provided by an embodiment of the present application is shown in FIG. 1.

[0028] Figure 3 A structure schematic diagram of a data processing apparatus provided by an embodiment of the present application is shown in FIG. 2.

[0029] Figure 4 A block diagram of an electronic device for implementing a data processing method according to an embodiment of the present application is shown in FIG. 3. DETAILED DESCRIPTION

[0030] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work should fall within the scope of protection of the present application.

[0031] It should be noted that the terms "first", "second", "target" and "original" and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include", "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to the process, method, product or device.

[0032] Figure 1 A flowchart of a data processing method provided by an embodiment of the present application is shown in FIG. 4. The embodiment can be applied to a scenario in which a data receiving end performs abnormality checking on received consumption data and processes abnormal consumption data during a disaster recovery switching process. The data processing method provided by the embodiment is applied to a data receiving end, which is connected to a message queue platform of a disaster recovery end. The data processing method can be executed by a data processing apparatus provided by an embodiment of the present application. The apparatus can be implemented in software and / or hardware, and integrated in an electronic device that executes the method.

[0033] Referring to FIG. 4, the method of the embodiment includes but is not limited to the following steps: Figure 1

[0034] ​S110, receive the consumption data sent by the disaster end message queue platform, the consumption data being determined by the disaster end message queue platform based on the data consumption request sent by the data receiving end, and the consumption data including a data identifier.

[0035] The message queue platform, as a core component of a distributed system, can provide message publishing and subscribing functions, and is responsible for sending information received from a producer to multiple consumers according to a subscription relationship. The message queue platform can be divided into a production end message queue platform and a disaster end message queue platform. The disaster end message queue platform is a backup system of the production end message queue platform, and stores message data synchronized with the production end message queue platform, so as to guarantee business continuity. The production end message queue platform provides services under normal circumstances, and the disaster end message queue platform provides services in the case of an abnormal situation.

[0036] The data receiving end can be a message data receiver, that is, a consumer of the message queue, which is a downstream system (such as a business server, a data processing service, etc.) that receives and processes message data, and is responsible for initiating a consumption request and processing returned consumption data. The data consumption request can be an instruction sent by the consumer to the message queue platform for obtaining data, including business types, consumption ranges, and the like. The consumption data can be specific data filtered by the message queue platform according to the data consumption request and processed by the consumer, which is business information (such as order data, state change records, etc.) previously published to the queue by the producer. The data identifier can be unique identification information embedded in the consumption data, such as an identifier (ID).

[0037] In the embodiment of the present application, when the production end message queue platform cannot provide services due to an abnormality, the system switches to the disaster end message queue platform to continue providing services. At this time, when the data receiving end (that is, the downstream system) needs to process data, it will actively initiate a data consumption request (usually including the queue / partition identifier of the consumption, the expected amount of data to be obtained, the last consumption offset, and the like) to the disaster end message queue platform. After receiving the request, the disaster end message queue platform will filter the consumption data that meets the conditions from the messages stored by itself according to the parameters in the request (such as the range of unconsumed data, queue rules), and push these data to the data receiving end. Since the disaster end needs to maintain data consistency with the production end, the consumption data pushed will include a data identifier (such as a primary key, a timestamp, etc.).

[0038] S120, determine a data processing operation for processing the consumption data based on a business data table corresponding to the data identifier and the data consumption request.

[0039] The business data table can be a database table storing core data of a certain type of business, and is a target storage carrier to which the consumption data is finally written or updated, and the structure thereof is strongly associated with business logic. For example, a user information table and an order status table. The data processing operation is a specific operation performed on the consumption data, such as adding, updating, discarding, deleting, and the like.

[0040] In an optional embodiment, after the data receiving end obtains the consumption data, the data identifier contained in the consumption data is first extracted, and the business data table corresponding to the consumption data, i.e., the target database table to which the consumption data is finally written or updated, is determined by the business identifier in the data consumption request or a preset rule. Subsequently, by comparing the data identifier with the existing data records in the business data table (such as checking whether the data exists), the state of the current consumption data is determined to be new data (to be added), existing data to be updated (to be modified), repeated data (to be ignored), or out-of-order data (to be processed after sorting by timestamp), and the corresponding data processing operation (such as adding, updating, discarding, etc.) is determined. The core of this process is to associate and verify the consumption data and the target business data table through the data identifier, to ensure that the data processing operation complies with the business rules, such as avoiding repeated storage and ensuring the time sequence correctness of state updates.

[0041] In another optional embodiment, the data identifier includes at least a data timestamp and a data primary key, and the business data table includes at least a data master table and a data detail table. The data primary key is a field that uniquely identifies a piece of data and is used to locate the data record in the master table. The timestamp and the data primary key are used to distinguish different data and assist in determining the uniqueness and time sequence of the data, and are the core basis for processing repeated and out-of-order data. The data master table can be a table storing core basic data, such as user basic information and commodity basic information, and the data primary key is the core index of the data master table. The data detail table can be a table storing detailed records associated with the data of the master table, such as user order details and commodity transaction details, and usually contains multiple sub-records associated with the same primary key.

[0042] Specifically, based on the data identifier and the business data table corresponding to the data consumption request, the data processing operation for processing the consumption data is determined, including: querying whether the data primary key exists in the data master table; if the data primary key does not exist, determining that the data processing operation is an adding operation; if the data primary key exists, extracting a historical consumption detail sub-table corresponding to the data primary key from the data detail table; determining whether the data timestamp satisfies a preset abnormal condition based on the historical consumption detail sub-table; if the preset abnormal condition is satisfied, determining that the data processing operation is an abnormal processing operation; and if the preset abnormal condition is not satisfied, determining that the data processing operation is a modification operation.

[0043] First, the data primary key is extracted from the consumption data, and it is queried in the data primary table whether the data primary key exists. The core of this step is to judge whether the consumption data is new data. If the data primary key does not exist, it means that the consumption data is first appeared, and the processing operation is directly determined as an addition operation, such as: executing an INSERT statement to write the master table and the associated detail table. If the data primary table exists, it means that the consumption data already exists, and further verification of its update rationality is required. At this time, the historical consumption detail sub-table corresponding to the data primary key is extracted from the data detail table, that is, the sub-table recording the historical update records of the consumption data, including the timestamp of each update, field change and other information. The data timestamp is extracted from the consumption data, and compared with the latest timestamp recorded in the historical consumption detail sub-table to judge whether it meets the preset abnormal condition (such as the current timestamp is earlier than the latest historical timestamp, that is, the disorder occurs; or the timestamp is repeated and the data content is consistent, that is, the repetition occurs). If the abnormal condition is met, the abnormal processing operation is executed, such as: skipping the repeated data, temporarily storing the disordered data for rearrangement, recording the abnormal log and the like; if the abnormal condition is not met (that is, the timestamp is the latest and the data is normally updated), the processing operation is determined as a modification operation, such as: executing an UPDATE statement to update the corresponding records of the master table and the detail table.

[0044] Optionally, the preset abnormal condition at least includes one of: the data timestamp is consistent with the timestamp of the historical data in the historical consumption detail sub-table; compared with the timestamp of the historical data, the data timestamp is not the nearest time to the current time.

[0045] S130, performing a data processing operation on the consumption data in the business data table.

[0046] In the embodiment of the application, when the data is added or modified in the data master table, the historical record in the data detail table is updated synchronously, forming a complete chain that the data master table reflects the current state and the data detail table retains the change track, supporting business traceability and audit, and can guarantee the consistency of the whole link data.

[0047] In an embodiment, in the case that the data processing operation is an abnormal processing operation, performing a data processing operation on the consumption data in the business data table includes: first determining the corresponding abnormal type based on the preset abnormal condition, for example, if the data timestamp is consistent with the timestamp of the historical data in the historical consumption detail sub-table, then the abnormal type is data duplication; if compared with the timestamp of the historical data, the data timestamp is not the nearest time to the current time, then the abnormal type is data disorder.

[0048] In the case of data duplication, if the data identifier is completely consistent with the existing record in the business data table, and the timestamp and content are not updated, the abnormal processing operation is determined to be discarded and the data is not written or updated to the business data table, only the log is marked as duplication, avoiding repeated data occupying storage or causing business logic errors, such as repeated order creation.

[0049] In the case of data disorder, if the timestamp of the current consumption data is earlier than the latest record with the same primary key in the business data table, the data that should have appeared first arrives later, then based on the data timestamp, the modification position of the consumption data in the data detail table is determined, and the abnormal processing operation is determined to be writing the detail data corresponding to the consumption data at the modification position. Specifically, by comparing the data timestamp with the timestamp of the historical record in the data detail table, the modification position of the disordered data in the time sequence is determined (such as inserting the current data with timestamp T2 between the records with timestamps T1 and T3), and then the detail data is written at the modification position, that is, the disordered data is inserted into the corresponding position of the detail table in the order of timestamp, ensuring the time sequence integrity of the data in the detail table and providing correct historical track for subsequent business query or backtracking.

[0050] In another embodiment, in the case of data processing operation being an addition operation (i.e. the data primary key does not exist in the data master table), the consumption data is executed in the business data table, including: obtaining the current state data of the data primary key, i.e. the core business state information corresponding to the data primary key, such as the name of a new user, the account status, or the amount of a new order, the creation state, etc.; determining the current state data and the data timestamp as the master information of the consumption data, and adding the master information in the data master table; for the detailed business information contained in the consumption data, the detail data corresponding to the consumption data can be added in the data detail table. These detail data are associated with the data master table through the data primary key, forming a complete data chain of "master + detail", ensuring that the business data can not only query the core state through the data master table, but also trace the detailed information through the data detail table.

[0051] In yet another embodiment, in the case of data processing operation being a modification operation (i.e. the data primary key exists in the master table, and the timestamp is verified to be a normal update without disorder or duplication), the consumption data is executed in the business data table, including: extracting the detailed information related to the current modification (such as the specific payment method and amount of the order status changed from "to be paid" to "paid") from the consumption data, and writing the detail data corresponding to the consumption data in the data detail table.

[0052] It should be noted that the modification operation usually does not directly cover the existing historical record in the data detail table, but a new detail data containing the latest timestamp is added to form an incremental record. Both the complete change track of the data (for tracing history) and the latest state identified by the timestamp are retained. At the same time, the data master table is updated to the current latest state, ensuring that the data master table always reflects the current situation of the data, and the data detail table records the details of each modification.

[0053] For example, the data master table and the data detail table are shown in the following table:

[0054] Table 1 Data Master Table

[0055]

[0056]

[0057] Table 2 Data Detail Table

[0058] Data primary key Other information 1 Other information 2 ...... Status Data timestamp A00001 XXX XXXXX ...... 1 20250101125901 A00001 XXX XXXXX ...... 2 20250101135902 A00001 XXX XXXXX ...... 3 20250101145903 ...... ...... ...... ...... ...... ......

[0059] The technical scheme provided by the embodiment receives the consumption data sent by the disaster recovery end message queue platform, the consumption data is determined by the disaster recovery end message queue platform based on the data consumption request sent by the data receiving end, and the consumption data includes a data identifier; based on the data identifier and the business data table corresponding to the data consumption request, a data processing operation for processing the consumption data is determined; and the data processing operation is performed on the consumption data in the business data table. In the disaster recovery switching scenario, the association verification of the data identifier and the business data table can quickly judge whether the data is duplicated or out of order, etc. Abnormal data can be accurately identified to avoid directly writing abnormal data into the business data table. Different data processing operations are performed according to the identified abnormal type to ensure the time sequence and integrity of the data in the business data table. Even if the disaster recovery end causes data anomalies due to offset synchronization failure, the scheme can verify and automatically correct the anomalies through the data identifier, so that the system quickly recovers data accuracy after disaster recovery switching and reduces the cost of manual intervention. The data anomaly problem in disaster recovery switching can be effectively solved, and the data consistency and business stability of the distributed system are guaranteed.

[0060] Figure 2 A framework diagram of a data processing system provided by the embodiment of the application is provided. The embodiment of the application is based on the above-mentioned embodiment and explains and describes the data transmission process in detail. The main purpose of the application is an implementation method for processing data anomalies caused by message queue cluster anomalies based on a data identifier. In combination with Figure 2The data processing system comprises a data production end, a production end message queue platform, a disaster recovery end message queue platform and a data receiving end, wherein the data receiving end is connected with the disaster recovery end message queue platform and the production end message queue platform through a domain name resolution component, and the data production end is connected with the disaster recovery end message queue platform and the production end message queue platform through a domain name resolution component.

[0061] The data production end generates data and publishes the data to the message queue platform. When sending, the sending time stamp of the current message, the data primary key, the latest information of the current primary key and the modification information of this time are specified. The data primary key can be used to determine the allocation to a specific partition after calculation.

[0062] The message queue platform provides publishing and subscribing functions, wherein the producer performs message publishing registration, and the consumer performs subscribing registration, including consumer identification, subscribed producer and the like. The message queue platform comprises a production end control center, a consumption end control center, a partition for storing data, a producer data processing module, a consumer data processing module, a data synchronization module, a data leveling module and an offset synchronization module.

[0063] The processing logic of the message queue platform can specifically comprise: 1, production end control center processing logic: producer permission verification, after the permission check is passed, the data synchronization and the producer data processing logic are initiated. 2, data synchronization: after the production end receives the data and passes the permission check, the disaster recovery end service availability is detected, if the disaster recovery end service is unavailable, the data synchronization is not performed, otherwise a copy of the data message is copied and directly sent to the disaster recovery end, so as to ensure that the original data received from the producer is consistent between the production end and the disaster recovery end. If the disaster recovery fails, the production end is directly returned to failure, which is regarded as receiving message failure (this way can prevent the production from having data and the disaster recovery from having no data after the production is abnormal, so as to avoid inconsistency). 3, producer data processing: after receiving the data sent by the producer, the partition is calculated according to the producer identification and the message primary key, and then the message is delivered to the specified partition.

[0064] The data consumer and the consumer control center belong to a long connection handshake state. The data consumer initiates a data consumption request to the message queue platform. If there is no data consumption in the queue, it is directly returned. If there is data in the queue, it is transferred to the next step for processing by the message queue platform. The processing logic of the message queue platform can further include: 1. Consumer control center processing logic: consumer permission verification, partition judgment, parameter acquisition (pre-buried configuration information, such as the maximum number of one-time delivery configuration), submission of consumption progress and triggering of offset synchronization service. 2. Offset synchronization: When a batch of data is requested for data consumption, the offset of the last batch of successfully consumed data is submitted. The production end message queue platform synchronizes the offset to the disaster recovery end message queue platform, so that the consumption progress of the disaster recovery end and the production end are kept in alignment. 3. Consumer data processing: According to the data consumption request, the data in the specified partition belonging to the specified data amount of the consumer is returned.

[0065] Further, for the scenario of realizing order through the message queue and matching the database record table, Redis memory data storage can be considered to reduce the database input / output processing time and be more efficient in running efficiency.

[0066] When there is consumable data in the queue, the data receiving end obtains multiple data from the message queue platform at a time according to the pre-buried configuration information. After the repeated and out-of-order checking processing service, the main file data (i.e. core data) and detailed data are realized into the database through the data update service. Through the business processing service, business logic processing can be performed and the data can be stored in the database.

[0067] When the production end message queue platform is abnormal and switches to the disaster recovery end message queue platform, the disaster recovery end message queue platform normally provides services to the outside. According to the established emergency disaster recovery drilling scheme, the switch is performed. After the production abnormality is recovered, the data is leveled first, and then the production is switched back to provide services.

[0068] During the production and disaster recovery switching process, out-of-order and repetition may occur. Here, only two scenarios are listed as an illustration, and the specific scenarios are as follows:

[0069] Scenario one: first, the data of primary key A00001 is modified three times, in which the state field is changed from 1 to 2, then from 2 to 3, and finally from 3 to 4. Since the primary key (A00001) is the same, the three times are sent to the same partition (partition 1 as an example) in the message queue platform. As shown in Table 3:

[0070] Table 3: Data diagram of message queue platform

[0071]

[0072]

[0073] Second step, consumer 1 applies N pieces of data at a time, including 1->2 and 2->3, and after consumption, submits the consumption progress to the production end message queue platform, and prepares to consume the next batch of data including 3->4. At this time, the production end message queue platform is abnormal, and the offset synchronization disaster recovery fails.

[0074] Third step, consumer 1 connects to the disaster recovery end message queue platform at this time, and originally wants to take 3->4, but due to the failure of the synchronization offset in the last step, the disaster recovery end message queue platform considers that the data taken this time is the same as the data taken in the second step. Therefore, it continues to issue N pieces of data (including 1->2 and 2->3). At this time, for consumer 1, 1->2 is repeated consumption, and 2->3 is repeated consumption. Therefore, the situation of repeated consumption occurs.

[0075] Fourth step, the downstream consumer performs data anomaly check and performs exception handling.

[0076] Scenario two: first step, the data of primary key A00002 is modified once, in which the state field is changed from 1 to 2. Since the primary key is the same, it is sent to the same partition (take partition 1 as an example) in the message queue platform.

[0077] Second step, two things happen at the same time during the disaster recovery switching: (1) the producer's state is modified again at this time, and the state field is changed from 2 to 3. At this time, the producer directly sends to the disaster recovery end message queue platform, and the disaster recovery end message queue platform receives the modification of 2->3. (2) Due to the influence of network delay, system jam, etc. of the production end message queue platform, the disaster recovery end message queue platform receives 1->2 synchronized from the production end message queue platform; therefore, in the disaster recovery end message queue platform, it is possible that 2->3 will be given priority over 1->2, which is out of order, as shown in Table 4 below:

[0078] Table 4: Data diagram of message queue platform

[0079]

[0080] Third step, the downstream consumer performs data anomaly check and performs exception handling. Then, business processing is performed.

[0081] The application realizes producer publishing and consumer subscribing through a message queue, sends data generated by the producer to downstream consumers through a message queue platform, which can reduce the number of directly associated downstream systems, improve data synchronization performance, and minimize the performance pressure of upstream and downstream systems. The message queue platform architecture adopts a production end and disaster recovery end disaster recovery high-availability architecture, and provides data synchronization, offset synchronization, data leveling and other mechanisms, which improves system stability and enhances system risk resistance. The downstream consumer supports data duplication and disorder processing mechanisms to prevent business impact and reduce system operation risks, thereby achieving the goal of accurate, complete and orderly data, efficient, stable and safe system operation.

[0082] Figure 3 A structural diagram of a data processing device provided by an embodiment of the application is shown in Figure 3 The device 300 is integrated in a data receiving end connected with a disaster recovery end message queue platform, and can include:

[0083] A data receiving module 310 is configured to receive consumption data sent by the disaster recovery end message queue platform, wherein the consumption data is determined by the disaster recovery end message queue platform based on a data consumption request sent by the data receiving end, and the consumption data includes a data identifier.

[0084] An operation determining module 320 is configured to determine a data processing operation for processing the consumption data based on the data identifier and a business data table corresponding to the data consumption request.

[0085] A data processing module 330 is configured to execute the data processing operation on the consumption data in the business data table.

[0086] In an embodiment, the data identifier includes at least a data timestamp and a data primary key, and the business data table includes at least a data master table and a data detail table.

[0087] Further, the data processing module 330 can be specifically configured to query whether the data primary key exists in the data master table, determine the data processing operation as an addition operation if the data primary key does not exist, extract a historical consumption detail sub-table corresponding to the data primary key from the data detail table if the data primary key exists, determine whether the data timestamp meets a preset abnormal condition based on the historical consumption detail sub-table, determine the data processing operation as an abnormal processing operation if the data timestamp meets the preset abnormal condition, and determine the data processing operation as a modification operation if the data timestamp does not meet the preset abnormal condition.

[0088] In an embodiment, the preset abnormal condition comprises at least one of: the data timestamp is consistent with a timestamp of historical data in the historical consumption detail subtable; and the data timestamp is not the closest to a current time compared with the timestamp of the historical data.

[0089] Further, the data processing module 330 can be specifically configured to: in a case where the data processing operation is an abnormal processing operation, determine a corresponding abnormal type based on the preset abnormal condition; if the abnormal type is data duplication, determine that the abnormal processing operation is to discard the consumption data; and if the abnormal type is data out-of-order, determine a modification position of the consumption data in the data detail table based on the data timestamp, and determine that the abnormal processing operation is to write the detail data corresponding to the consumption data at the modification position.

[0090] Further, the data processing module 330 can be specifically configured to: in a case where the data processing operation is an addition operation, acquire current state data of the data primary key; determine the current state data and the data timestamp as primary file information of the consumption data, and add the primary file information in the data primary file table; and add the detail data corresponding to the consumption data in the data detail table.

[0091] Further, the data processing module 330 can be specifically configured to: in a case where the data processing operation is a modification operation, write the detail data corresponding to the consumption data in the data detail table.

[0092] The data processing apparatus provided in the embodiment can be applied to the data processing method provided in any of the above embodiments, and has corresponding functions and beneficial effects.

[0093] Figure 4 is a block diagram of an electronic device that implements a data processing method according to an embodiment of the present application. The electronic device 10 is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices (e.g., headsets, glasses, watches, etc.), and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present application described and / or claimed in this document.

[0094] As Figure 4As shown, the electronic device 10 includes at least one processor 11, and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., communicatively connected to the at least one processor 11, where the memory stores a computer program executable by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0095] Various components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc., an output unit 17, such as various types of displays, a speaker, etc., a storage unit 18, such as a magnetic disk, an optical disk, etc., and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0096] The processor 11 can be various general and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as a data processing method.

[0097] In some embodiments, the data processing method can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded onto the RAM 13 and executed by the processor 11, one or more steps of the data processing method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the data processing method by any other appropriate means, such as by means of firmware.

[0098] The various embodiments of the systems and techniques described above can be implemented in digital electronic circuitry, integrated circuitry, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system on a chip systems (SOCs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0099] Computer programs used to implement the methods of the application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer program

[0100] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store computer programs for use by or in connection with an instruction execution system, apparatus, or device. Computer-readable storage media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0101] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0102] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0103] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. A server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0104] Note that the above merely describes preferred embodiments of the present application and the principles of the technology applied. Those skilled in the art will understand that the present application is not limited to the specific embodiments described here, and that various obvious changes, re-adjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present application. For example, those skilled in the art can use the above-mentioned various forms of flow, reorder, add or delete steps; can perform the steps described in the present application in parallel, sequentially or in different order, as long as the desired results of the technical solutions of the present application can be achieved, which is not limited herein.

[0105] The foregoing detailed description has not been limited to specific details regarding the application. It should be apparent that various modifications, combinations, sub-combinations, and alternatives can be made to the disclosed embodiments. Any modifications, changes, and improvements that come within the spirit and scope of the application should be considered as falling within the scope of the application.

Claims

1. A data processing method, characterized by, The method is applied to a data receiving end connected with a disaster recovery end message queue platform, and comprises the following steps: Receiving consumption data sent by the disaster recovery end message queue platform, wherein the consumption data is determined by the disaster recovery end message queue platform based on a data consumption request sent by the data receiving end, and the consumption data comprises a data identifier; Determining a data processing operation for processing the consumption data based on the data identifier and a business data table corresponding to the data consumption request; Executing the data processing operation on the consumption data in the business data table.

2. The data processing method according to claim 1, characterized in that, The data identifier comprises at least a data timestamp and a data primary key, and the business data table comprises at least a data master table and a data detail table; The method of determining the data processing operation for processing the consumption data based on the data identifier and the business data table corresponding to the data consumption request comprises the following steps: Querying whether the data primary key exists in the data master table; If the data primary key does not exist, determining that the data processing operation is an addition operation; If the data primary key exists, extracting a historical consumption detail sub-table corresponding to the data primary key from the data detail table; Determining whether the data timestamp meets a preset abnormal condition based on the historical consumption detail sub-table; If the preset abnormal condition is met, determining that the data processing operation is an abnormal processing operation; If the preset abnormal condition is not met, determining that the data processing operation is a modification operation.

3. The data processing method according to claim 2, characterized in that, The preset abnormal condition comprises at least one of the following: The data timestamp is consistent with a timestamp of historical data in the historical consumption detail sub-table; Compared with the timestamp of the historical data, the data timestamp is not the closest time to the current time.

4. The data processing method according to claim 2, characterized in that, In the case that the data processing operation is an abnormal processing operation, the method of executing the data processing operation on the consumption data in the business data table comprises the following steps: Determining a corresponding abnormal type based on the preset abnormal condition; If the abnormal type is data duplication, determining that the abnormal processing operation is to discard the consumption data; If the abnormal type is data out-of-order, determining a modification position of the consumption data in the data detail table based on the data timestamp, and determining that the abnormal processing operation is to write detail data corresponding to the consumption data at the modification position.

5. The data processing method according to claim 2, characterized in that, In the case that the data processing operation is an addition operation, the method of executing the data processing operation on the consumption data in the business data table comprises the following steps: Obtaining current state data of the data primary key; Determining the current state data and the data timestamp as master file information of the consumption data, and adding the master file information in the data master table; Adding detail data corresponding to the consumption data in the data detail table.

6. The data processing method according to claim 2, characterized in that, In the case that the data processing operation is a modification operation, the method of executing the data processing operation on the consumption data in the business data table comprises the following steps: Writing detail data corresponding to the consumption data in the data detail table.

7. A data processing system, characterized by The data production end, the production end message queue platform, the disaster recovery end message queue platform, and a data receiving end for executing the data processing method according to any one of claims 1 to 6 are connected with each other.

8. A data processing apparatus, characterized by, The device is integrated in the data receiving end connected with the disaster recovery end message queue platform, and comprises: a data receiving module configured to receive consumption data sent by the disaster recovery end message queue platform, the consumption data being determined by the disaster recovery end message queue platform based on a data consumption request sent by the data receiving end, and the consumption data comprising a data identifier; an operation determining module configured to determine a data processing operation for processing the consumption data based on the data identifier and a business data table corresponding to the data consumption request; a data processing module configured to execute the data processing operation on the consumption data in the business data table.

9. An electronic device, comprising: The electronic device comprises: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores a computer program executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the data processing method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for enabling the processor to execute the data processing method according to any one of claims 1 to 6 when executed.

Images