Data processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the application provides a kind of data processing method, device, electronic equipment and storage medium, it is related to computer technical field.Data processing method includes: scanning the first time corresponding to current to-be-processed data;From the first time, receive update data, store the received multiple update data in temporary object, and the receiving time stage of each update data is marked as first stage;When the full data is obtained by scanning for the to-be-processed data corresponding to the first time, determine the second time, the receiving time stage of each update data received after the second time is marked as second stage;From the second time, based on the receiving time stage of each update data stored in temporary object, the stored multiple update data is merged with full data, and reconstruction data is generated.The data processing method of the application can effectively guarantee the completion of data reconstruction task.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a data processing method, apparatus, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] With the development of Internet technology, distributed database technology has become increasingly mature. More and more users are using distributed databases as production databases to perform large-scale data calculations and achieve flexible and elastic data storage.

[0003] Most user services run in the cloud, and as user data grows, data reconstruction is frequently required when using distributed databases. Currently, for data reconstruction tasks, such as adding indexes, the approach is to scan the data to be processed and then catch up with the logs as the scan is complete. However, during peak business periods, the rate of log generation may exceed the rate of catching up, potentially preventing the data reconstruction from being completed. Summary of the Invention

[0004] This application provides a data processing method, apparatus, electronic device, and storage medium. The technical solutions provided by this application are as follows:

[0005] On one hand, embodiments of this application provide a data processing method, the method comprising:

[0006] Receive data reconstruction instructions and scan the data to be processed corresponding to the current first moment based on the data reconstruction instructions;

[0007] Starting from the first moment, update data is received, and the received update data is stored in a temporary object. The receiving time stage of each update data is marked as the first stage.

[0008] When the scan of the data to be processed corresponding to the first moment is completed and the full amount of data is obtained, the current second moment is determined, and the receiving time stage of each updated data received after the second moment is marked as the second stage.

[0009] Starting from the second moment, based on the receiving time stage of each updated data stored in the temporary object, the stored multiple updated data are merged with the full data to generate reconstructed data.

[0010] In some possible implementations, based on the reception time phase of each updated data stored in the temporary object, the stored multiple updated data are merged with the full data to generate reconstructed data, including:

[0011] For each updated data point among the stored updated data, perform the following operations to generate reconstructed data:

[0012] If the data update is received in the first phase, the updated data will be merged into the full dataset.

[0013] If the data update reception time phase is the second phase, then skip the data update.

[0014] In some possible implementations, if the data update reception phase is the first phase, then merging the update data into the full dataset includes:

[0015] If the data update is received in the first phase, then the updated data will be treated as data to be merged.

[0016] Lock the data to be merged;

[0017] Check if there is any new updated data in the temporary object that is updating the data. Based on the detection result, merge the locked data to be merged into the full data.

[0018] In some possible implementations, the locked data to be merged is merged into the full dataset based on the detection results, including:

[0019] If the detection result indicates that there is new updated data for the updated data in the temporary object, and the receiving time stage of the new updated data is the second stage, then skip the data to be merged.

[0020] If the detection result indicates that there is no new updated data for the updated data in the temporary object, then the locked data to be merged will be merged into the full data.

[0021] In some possible implementations, the received multiple update data are stored in a temporary object, including:

[0022] For each updated data, if the updated data is deleted data, then the deletion instruction for the deleted data is marked to obtain the marked deletion instruction;

[0023] The updated data corresponding to the mark deletion instruction is stored in a temporary object.

[0024] In some possible implementations, the locked data to be merged is merged into the full dataset, including:

[0025] If the data to be merged corresponds to the mark-delete instruction, then modify the mark-delete instruction to a delete instruction;

[0026] The modified data corresponding to the deletion command will be merged into the full dataset.

[0027] In some possible implementations, a deletion instruction for deleting data is marked to obtain a marked deletion instruction, including:

[0028] Inserting data carrying a deletion identifier into the deletion instruction yields a marked deletion instruction.

[0029] In some possible implementations, based on the reception time phase of each updated data stored in the temporary object, the stored multiple updated data are merged with the full data to generate reconstructed data, including:

[0030] Divide the update data stored in the temporary object into at least two batches of update data;

[0031] Iterate through each batch of updated data to determine the receiving time phase for each batch of updated data.

[0032] The updated data is merged with the full data based on the receiving time of each batch of updated data to generate reconstructed data.

[0033] On the other hand, embodiments of this application provide a data processing apparatus, which includes:

[0034] The receiving module is used to receive data reconstruction instructions and scan the data to be processed corresponding to the current first moment based on the data reconstruction instructions;

[0035] The storage module is used to receive updated data from the first moment, store the received updated data in a temporary object, and mark the receiving time stage of each updated data as the first stage;

[0036] The marking module is used to determine the current second time when the full data is obtained after scanning the data to be processed corresponding to the first time, and to mark the reception time stage of each updated data received after the second time as the second stage.

[0037] The merging module is used to merge multiple updated data stored in the temporary object with the full data, starting from the second time point, based on the reception time stage of each updated data stored in the temporary object, to generate reconstructed data.

[0038] In some possible implementations, when the merging module receives each update data stored in the temporary object, it merges the stored multiple update data with the full data to generate reconstructed data. Specifically, this is used for:

[0039] For each updated data point among the stored updated data, perform the following operations to generate reconstructed data:

[0040] If the data update is received in the first phase, the updated data will be merged into the full dataset.

[0041] If the data update reception time phase is the second phase, then skip the data update.

[0042] In some possible implementations, when the merging module merges the updated data into the full dataset if the data reception time phase is the first phase, it is specifically used for:

[0043] If the data update is received in the first phase, then the updated data will be treated as data to be merged.

[0044] Lock the data to be merged;

[0045] Check if there is any new updated data in the temporary object that is updating the data. Based on the detection result, merge the locked data to be merged into the full data.

[0046] In some possible implementations, when the merging module merges the locked data to be merged into the full dataset based on the detection results, it specifically performs the following functions:

[0047] If the detection result indicates that there is new updated data for the updated data in the temporary object, and the receiving time stage of the new updated data is the second stage, then skip the data to be merged.

[0048] If the detection result indicates that there is no new updated data for the updated data in the temporary object, then the locked data to be merged will be merged into the full data.

[0049] In some possible implementations, when the storage module stores the received multiple update data into a temporary object, it specifically performs the following functions:

[0050] For each updated data, if the updated data is deleted data, then the deletion instruction for the deleted data is marked to obtain the marked deletion instruction;

[0051] The updated data corresponding to the mark deletion instruction is stored in a temporary object.

[0052] In some possible implementations, the merging module, when merging locked data to be merged into the full dataset, specifically performs the following functions:

[0053] If the data to be merged corresponds to the mark-delete instruction, then modify the mark-delete instruction to a delete instruction;

[0054] The modified data corresponding to the deletion command will be merged into the full dataset.

[0055] In some possible implementations, when the storage module marks a deletion instruction for deleted data and receives a marked deletion instruction, it specifically performs the following:

[0056] Inserting data carrying a deletion identifier into the deletion instruction yields a marked deletion instruction.

[0057] In some possible implementations, the merging module, during the reception phase of each update data stored in the temporary object, merges the stored multiple update data with the full data to generate reconstructed data, specifically for:

[0058] Divide the update data stored in the temporary object into at least two batches of update data;

[0059] Iterate through each batch of updated data to determine the receiving time phase for each batch of updated data.

[0060] The updated data is merged with the full data based on the receiving time of each batch of updated data to generate reconstructed data.

[0061] On the other hand, embodiments of this application also provide an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method provided in any optional embodiment of this application.

[0062] On the other hand, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in any optional embodiment of this application.

[0063] On the other hand, embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the methods provided in any optional embodiment of this application.

[0064] The beneficial effects of the technical solution provided in this application are as follows:

[0065] By storing updated data in a temporary object and marking the receiving time stages of the updated data, the first stage is defined as the period from the start of scanning to the completion of scanning and the start of merging, and the period after the second stage is defined as the second stage. The merging operation between the updated data and the data to be processed is performed according to the different receiving time stages of the updated data. During peak business periods, hot data will inevitably generate corresponding data for the second stage in the temporary object. Since the receiving time stages of this part of the data have been marked, the backfilling of this part of the data can be skipped by marking it. This effectively avoids the situation where the data merging speed cannot keep up with the update speed due to the large number of transactions on hot data, and can effectively ensure the completion of the data reconstruction task.

[0066] Furthermore, by marking the receiving time stage of each updated data, updated data in the first receiving time stage needs to be merged, while data in the second receiving time stage is skipped. In this way, only the data that needs to be merged needs to be merged in the second stage, while the data that does not need to be merged is filtered out, reducing the amount of data to be merged. Even during peak business periods and when there is a large amount of hot data, the merging efficiency can be effectively improved. It can even achieve the advantage that the more hot data there is, the fewer rounds of temporary index data are needed.

[0067] Furthermore, by detecting whether there is new updated data in the temporary object, the locked data to be merged is merged into the full data based on the detection result. This can prevent new updated data that was not scanned before from being generated when locking the data to be merged. It can further filter out data that does not need to be merged, thereby improving the efficiency of data reconstruction. Attached Figure Description

[0068] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below.

[0069] Figure 1 This is a schematic diagram of an existing index reconstruction scheme.

[0070] Figure 2 This is a schematic diagram illustrating the application environment of the data processing method provided in the embodiments of this application;

[0071] Figure 3 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0072] Figure 4 A schematic diagram of a data reconstruction scheme provided in this application;

[0073] Figure 5 A schematic diagram of an index reconstruction scheme provided in this application;

[0074] Figure 6 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0075] Figure 7 This is a schematic diagram of the structure of an electronic device to which this application applies. Detailed Implementation

[0076] The embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the embodiments described below with reference to the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions of the embodiments of this application.

[0077] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the terms “comprising” and “including” as used in embodiments of this application mean that the corresponding feature can be implemented as the presented feature, information, data, step, operation, element, and / or component, but do not exclude implementation as other features, information, data, step, operation, element, component, and / or combinations thereof supported by the art. It should be understood that when we say that an element is “connected” or “coupled” to another element, the one element can be directly connected or coupled to the other element, or it can mean that the one element and the other element establish a connection relationship through an intermediate element. Furthermore, “connected” or “coupled” as used herein can include wireless connection or wireless coupling. The term “and / or” as used herein indicates at least one of the items defined by the term; for example, “A and / or B” can be implemented as “A,” or as “B,” or as “A and B.” When describing multiple (two or more) items, if the relationship between the multiple items is not explicitly defined, the multiple items can refer to one, several or all of the multiple items. For example, the description of "parameter A includes A1, A2, A3" can be implemented as parameter A includes A1 or A2 or A3, or it can be implemented as parameter A includes at least two of the three items A1, A2 and A3.

[0078] With the rise of the internet, distributed database technology has become a primary scenario for users to utilize databases, with an increasing number of users using distributed databases as their production databases. Simultaneously, as most user operations run in the cloud, and as user data grows, traditional DDL execution methods require stopping user operations, which users are increasingly unwilling to accept. Therefore, the ability to modify the schema online is becoming an increasingly important capability indicator for distributed databases. The schema is a collection of database objects.

[0079] Meanwhile, with the increasing volume of user data, DDL tasks requiring data reconstruction, such as adding indexes, may take a long time to complete even with online schema changes, hindering the improvement of user services. Therefore, improving the execution efficiency of online DDL has become another issue that requires continuous optimization. For example, adding an index to a 10TB or 100TB table for online users may take a very long time, which also hinders the improvement of both user experience and business.

[0080] Traditional offline DDL implementations complete DDL by adding a Metadata Lock (MDL) to the schema. This method requires blocking user services, which is basically unusable in cloud environments and some scenarios where user services have high timeliness. In particular, blocking services during peak user service periods is unacceptable.

[0081] The approach combining full data logging with log tracking involves selecting a timestamp as a starting point during the DDL data reconstruction phase, scanning schema data, and creating a new schema for full reconstruction. After the full reconstruction is complete, logs are rewritten starting from the timestamp to catch up with user business logic. Once a certain catch-up threshold is reached, the DDL schema is locked to complete log catching up, and the old and new schemas are switched to complete the DDL operation. This approach still requires locking schema objects, which can amplify failures in distributed systems. DDL operations require double the storage space, which can become a bottleneck with increasingly large datasets. Asynchronous recycling of old schemas also requires significant storage engine resources. During peak business periods, the rate of log generation may exceed the catch-up rate, preventing DDL tasks from completing or necessitating blocking user business logic to finish the DDL task.

[0082] Online DDL solutions can complete DDL operations on the same schema. Furthermore, due to the online implementation, the solution avoids using locks to block user business operations and ensures data modifications on the schema without data anomalies during the entire DDL execution. However, the backfill task and user business operations on the same object may interfere with each other, competing for computation and storage resources. Ensuring the correct order of operations on the same object requires certain handling. Operating on the same object limits the resource utilization of the distributed cluster, easily leading to resource hotspots and insufficient resource utilization. DDL execution has a significant impact on user business operations, and backfilling data also requires transaction commits, which may cause data processing conflicts with user business operations. Resolving these conflicts can severely degrade the performance of the backfill task.

[0083] like Figure 1 As shown in the diagram, arrow 2 represents the process of preparing the full index starting at a certain point in time, and arrow 1 represents the incremental part of the index synchronized by user businesses. Through existing solutions, it was found that different methods are currently used to implement the two parts of data mentioned above for index creation. Therefore, we can refer to these two parts of index data as: the full index creation and the incremental index creation.

[0084] Based on the preceding description, the task of rebuilding the index can be viewed as completing a full operation plus merging an incremental operation. If these two stages can be decoupled on the basis of the online DDL solution, then greater flexibility in the execution of DDL tasks will be achieved. However, a core idea of ​​online DDL is to complete the full and incremental operations on a single object, avoiding the merging process, because once merging is required, locks may need to be introduced for synchronization.

[0085] The technical solution of this application introduces a new method that retains the advantages of online DDL while also fully expanding the space for DDL task optimization: it can minimize the impact of DDL execution on user business, that is, it basically does not affect the execution of user business; it achieves the goal of retaining the advantages of online DDL while significantly improving the efficiency of DDL task execution; this solution can make fuller use of various resources in the cluster to achieve an optimal resource utilization and DDL task execution effect.

[0086] The following description of several optional embodiments illustrates the technical solutions provided in this application and the technical effects produced by these solutions. It should be noted that the following embodiments can be referenced, borrowed from, or combined with each other. Identical terms, similar features, and similar implementation steps in different embodiments will not be repeated.

[0087] Figure 2 This is a schematic diagram illustrating the application environment of the data processing method provided in this application embodiment. The application environment may include a data reconstruction server 201 and a client 202. The data reconstruction server 201 receives a data reconstruction instruction sent by the client 202, and scans the data to be processed corresponding to the current first moment based on the data reconstruction instruction; starting from the first moment, it receives updated data, stores the received multiple updated data in a temporary object, and marks the reception time stage of each updated data as a first stage; when the scanning of the data to be processed corresponding to the first moment is completed and the full data is obtained, it determines the current second moment, and marks the reception time stage of each updated data received after the second moment as a second stage; starting from the second moment, based on the reception time stage of each updated data stored in the temporary object, it merges the stored multiple updated data with the full data to generate reconstructed data.

[0088] The above scenario describes a data processing method executed by a data reconstruction server. In other application scenarios, the data processing method could also be executed by a terminal. Those skilled in the art will understand that the above scenario is merely an example and does not limit the application scenarios of the data processing method described in this application.

[0089] Those skilled in the art will understand that a server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server or server cluster that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0090] The client (also referred to as a user terminal or user device) can be a smartphone, tablet, laptop, desktop computer, smart voice interaction device (e.g., smart speaker), wearable electronic device (e.g., smartwatch), in-vehicle terminal, smart home appliance (e.g., smart TV), AR / VR device, etc., but is not limited thereto. The client and server can be connected directly or indirectly through wired or wireless communication, which is not limited herein.

[0091] Figure 3 This illustration shows a flowchart of a data processing method provided in an embodiment of this application. This method can be executed by a server, such as... Figure 2 As shown, taking a server as the executing entity as an example, the data processing method provided in this application may include the following steps:

[0092] Step S301: Receive data reconstruction instruction, and scan the data to be processed corresponding to the current first moment based on the data reconstruction instruction.

[0093] Specifically, the timestamp at which the scan begins can be taken as the first moment.

[0094] In the specific implementation process, the scanning process of the data to be processed can also be called full data backfill. The scanning task for the data to be processed can be carried out in an independent backend object. During the full data backfill process, there is no modification to the front-end user business, that is, no data update is involved.

[0095] Step S302: Starting from the first moment, receive update data, store the received update data in a temporary object, and mark the receiving time stage of each update data as the first stage.

[0096] The updated data may include modifications to the data to be processed.

[0097] Specifically, to receive updated data and record modifications to the front-end business logic, a temporary object can be allocated to store the updated data.

[0098] In the specific implementation process, the time period before the scanning of the data to be processed is completed from the first moment can be regarded as the first stage. The updated data is marked, and the receiving time period is marked as the first stage. The first stage can also be called the backfilling stage.

[0099] Step S303: When the scan of the data to be processed corresponding to the first moment is completed and the full amount of data is obtained, the current second moment is determined, and the reception time stage of each updated data received after the second moment is marked as the second stage.

[0100] The time period between the first moment and the second moment can be defined as the first stage, and the time period after the second moment can be defined as the second stage.

[0101] Specifically, when the full data is obtained after the scan is completed, i.e. when the full backfill is completed, the system continues to receive updated data. At this time, the receiving time phase for the updated data changes from the first phase to the second phase.

[0102] Step S304: Starting from the second moment, based on the receiving time stage of each updated data stored in the temporary object, the stored multiple updated data are merged with the full data to generate reconstructed data.

[0103] Specifically, when the scan is complete, which is the second moment, the updated data and the full data need to be merged. The second stage can also be called the merging stage.

[0104] It is important to note that during the merging phase, the temporary object continues to receive updated data. Therefore, the merging process needs to be performed based on the different receiving time stages marked by each updated data stored in the temporary object. The specific merging process will be further explained in detail below.

[0105] In the above embodiments, by storing the updated data in a temporary object and marking the receiving time stages of the updated data, the first stage is defined as the period from the start of scanning to the completion of scanning and the start of merging, and the period after the second stage is defined as the second stage. The merging operation between the updated data and the data to be processed is performed according to the different receiving time stages of the updated data. During peak business periods, hot data will inevitably generate corresponding data for the second stage in the temporary object. Since the receiving time stages of this part of the data have been marked, the backfilling of this part of the data can be skipped by marking, thereby effectively avoiding the merging not keeping up with the updated data due to the large number of transactions on the hot data, and effectively ensuring the completion of the data reconstruction task.

[0106] The following will further illustrate the process of merging the stored updated data with the full data according to different receiving time stages of the updated data, with reference to the embodiments.

[0107] In some possible implementations, step S304, based on the reception time phase of each update data stored in the temporary object, merges the stored multiple update data with the full data to generate reconstructed data, including:

[0108] For each updated data point among the stored updated data, perform the following operations to generate reconstructed data:

[0109] (1) If the receiving time of the updated data is the first stage, then the updated data is merged into the full data.

[0110] Specifically, if any updated data is received in the first stage, that is, during the backfilling stage, it means that the data was not updated in the second stage, that is, the merging stage. In this case, the updated data needs to be merged into the full data.

[0111] (2) If the data update reception time stage is the second stage, then skip the data update.

[0112] Specifically, if any updated data is received in the second stage, that is, during the merging stage, it means that this updated data has already been synchronized to the full data and can be ignored. In other words, this updated data does not need to be synchronized to the full data again.

[0113] For each updated data stored in the temporary object, determine the receiving time stage of the updated data. If the receiving time stage is the first stage, then merge the updated data into the full data to generate reconstructed data; if the receiving time stage is the second stage, then skip the updated data.

[0114] In the above embodiments, by marking the receiving time stage of each updated data, the updated data in the first receiving time stage needs to be merged, while the data in the second receiving time stage is skipped. In this way, only the data that needs to be merged needs to be merged in the second stage, and the data that does not need to be merged is filtered out, which reduces the amount of data to be merged. Even during peak business periods and when there is a large amount of hot data, the merging efficiency can be effectively improved. It can even achieve the advantage that the more hot data there is, the fewer temporary indexes are needed.

[0115] In some possible implementations, if the data update reception phase is the first phase, then merging the update data into the full dataset may include:

[0116] a. If the data update reception time period is the first stage, then the updated data will be treated as data to be merged.

[0117] Specifically, if the update data is received in the first stage, that is, during the backfilling stage, it means that the data was not updated during the merging stage. In this case, this part of the update data needs to be merged into the full data, that is, the update data is treated as the data to be merged.

[0118] b. Lock the data to be merged.

[0119] Specifically, the data to be merged is locked, meaning the data to be merged remains unchanged and is not modified.

[0120] c. Detect whether there is any new updated data in the temporary object that is updating the data. Based on the detection result, merge the locked data to be merged into the full data.

[0121] Specifically, at this point, the temporary object is checked to see if there is any new update data generated in the second phase for this update data. This step is mainly to prevent new update data that was not scanned before but was generated when locking the data to be merged. At this point, this part of the update data no longer needs to be merged.

[0122] Specifically, merging locked data into the full dataset based on the detection results can include:

[0123] c1. If the detection result indicates that there is new updated data for the updated data in the temporary object, and the receiving time stage of the new updated data is the second stage, then skip the data to be merged.

[0124] c2. If the detection result indicates that there is no new updated data for the updated data in the temporary object, then the locked data to be merged will be merged into the full data.

[0125] In the specific implementation process, if new updated data is detected in the second phase, it means that the latest updated data has been synchronized to the full data and can be ignored; if no data is found to be generated in the second phase, then since the data to be merged has been locked, it proves that the updated data needs to be merged into the full data.

[0126] like Figure 4 As shown, for each updated data stored in the temporary object, the receiving time stage of the updated data is determined; if the receiving time stage is the second stage, the updated data is skipped; if the receiving time stage is the first stage, the updated data is treated as data to be merged; the data to be merged is locked; it is determined whether there is any new updated data in the temporary object; if so, the data to be merged is skipped; if not, the locked data to be merged is merged into the full data.

[0127] In the above embodiments, by detecting whether there is new updated data for the updated data in the temporary object, the locked data to be merged is merged into the full data based on the detection result. This can prevent new updated data from being generated when locking the data to be merged, even if it was not scanned before. This can further filter out data that does not need to be merged, thereby improving the data reconstruction efficiency.

[0128] In some possible implementations, step S304, based on the reception time phase of each update data stored in the temporary object, merges the stored multiple update data with the full data to generate reconstructed data, including:

[0129] (1) Divide the update data stored in the temporary object into at least two batches of update data;

[0130] (2) Iterate through each batch of updated data in turn to determine the receiving time stage of each batch of updated data;

[0131] (3) Based on the receiving time stage of each batch of updated data, merge the updated data with the full data to generate reconstructed data.

[0132] Specifically, multiple updated data in the temporary object can be divided into multiple batches and processed one batch at a time. For each updated data in each batch, the receiving time stage of the updated data is determined. The updated data is then merged with the full data according to the receiving time stage of each updated data as described above. The specific merging method has been described in detail in the above embodiments and will not be repeated here.

[0133] In some possible implementations, step S302, which stores the received multiple update data into a temporary object, may include:

[0134] (1) For each updated data, if the updated data is deleted data, the deletion instruction for the deleted data is marked to obtain the marked deletion instruction.

[0135] Among them, deleted data can be data that has undergone a deletion operation.

[0136] Specifically, the deletion command for deleting data is marked to obtain the marked deletion command, which may include:

[0137] Inserting data carrying a deletion identifier into the deletion instruction yields a marked deletion instruction.

[0138] (2) Store the update data corresponding to the mark deletion instruction in a temporary object.

[0139] In practice, by inserting data carrying a deletion identifier into the deletion instruction, the deleted data can be prevented from being cleaned up before the data is merged in the first or second stage; in addition, during the merging stage, the iterator can read the deleted data.

[0140] In some possible implementations, merging the locked data to be merged into the full dataset may include:

[0141] If the data to be merged corresponds to the mark-delete instruction, then modify the mark-delete instruction to a delete instruction;

[0142] The modified data corresponding to the deletion command will be merged into the full dataset.

[0143] Specifically, if the data to be merged is delete data, the delete instruction for the delete data needs to be modified into an actual delete operation, that is, modified into a delete instruction, and then merged into the full index.

[0144] The data processing method of this application stores updated data in a temporary object and marks the receiving time stages of the updated data. The first stage is defined as the period from the start of scanning to the completion of scanning and the start of merging, and the period after the second time is defined as the second stage. The merging operation between the updated data and the data to be processed is performed according to the different receiving time stages of the updated data. During peak business periods, hot data will inevitably generate corresponding data for the second stage in the temporary object. Since the receiving time stages of this part of the data have been marked, the backfilling of this part of the data can be skipped by marking it. This effectively avoids the situation where the data merging speed cannot keep up with the update speed due to the large number of transactions on hot data, and can effectively ensure the completion of the data reconstruction task.

[0145] Furthermore, by marking the receiving time stage of each updated data, updated data in the first receiving time stage needs to be merged, while data in the second receiving time stage is skipped. In this way, only the data that needs to be merged needs to be merged in the second stage, while the data that does not need to be merged is filtered out, reducing the amount of data to be merged. Even during peak business periods and when there is a large amount of hot data, the merging efficiency can be effectively improved. It can even achieve the advantage that the more hot data there is, the fewer rounds of temporary index data are needed.

[0146] Furthermore, by detecting whether there is new updated data in the temporary object, the locked data to be merged is merged into the full data based on the detection result. This can prevent new updated data that was not scanned before from being generated when locking the data to be merged. It can further filter out data that does not need to be merged, thereby improving the efficiency of data reconstruction.

[0147] To more clearly illustrate the data processing method of this application, the following will provide further explanation in conjunction with the accompanying drawings and examples.

[0148] Taking the data to be processed as an index as an example, such as Figure 5 As shown, in one example, the data processing method of this application may include:

[0149] 1) Receive the index to be processed, scan the index corresponding to the current first moment, and then enter the first stage (backfilling stage);

[0150] 2) Start receiving incremental indexes (updating data) from the first moment, store the received incremental indexes in a temporary object, and mark the receiving time stage of each incremental index as the first stage;

[0151] 3) When the scan of the index to be processed corresponding to the first moment is completed and the full index is obtained, the current second moment is determined, and the second stage (merging stage) is entered at this time;

[0152] 4) The reception time phase of each incremental index received after the second time point is marked as the second phase;

[0153] 5) Traverse the incremental indexes stored in the temporary object to obtain the receiving time stage of the incremental index of the temporary object;

[0154] 6) If the data was generated in the second stage, it means it has already been synchronized to the full index and can be ignored; if the data was generated in the first stage, it means the data was not updated in the second stage, and step 7) needs to be executed.

[0155] 7) If the incremental index is in the first receiving phase, then treat the incremental index as data to be merged; lock the data to be merged.

[0156] 8) Check if there is new updated data in the temporary object. If the detection result is that there is new updated data in the temporary object and the receiving time stage of the new updated data is the second stage, then skip the data to be merged.

[0157] 9) If the detection result indicates that there is no new updated data for the updated data in the temporary object, then the locked data to be merged will be merged into the full data.

[0158] 10) After the merge is completed, the index creation is finished and the index status is changed to normal; at the same time, the temporary index data is asynchronously cleaned up.

[0159] In the example above, the first stage (backfill stage) can also be called the backfill stage, and the second stage (merge stage) can also be called the merge stage.

[0160] In the example above, after decoupling the full index data construction phase from user business, the conflict between the DDL backfill period and business execution is reduced to 0.

[0161] When business is always at its peak, hot data will inevitably generate corresponding data in the merge phase of the temporary index. Therefore, this solution can skip the backfilling of this part of the data in the merge phase, thereby avoiding the fact that there are many transactions on hot data and the merge cannot keep up with the newly added data, which would prevent the entire DDL task from being completed.

[0162] Only necessary data is merged, filtering out records that don't need merging (hot data has already been updated back into the index during the merging phase, so it doesn't need to be merged), improving merging efficiency. This allows for the advantage of having less data in the temporary index to merge as there is more hot data. In extreme cases, if the data in the temporary index is all hot data, the merging process can be completed by scanning the temporary index only once.

[0163] For locks on records in the data to be merged, because the granularity is relatively fine, if the batch of data to be backfilled is not large, it will not have a significant impact on user business. At the same time, since the records locked during backfilling are modified in the backfill stage and have no subsequent operations, this part of the data can be considered relatively cold data, and locking them will not have a significant impact on the business. Moreover, if the data batch is relatively small, the locking time is almost negligible.

[0164] This solution does not require twice the storage space of the index; the only difference in space usage is the addition of records for the delete portion in the temporary object.

[0165] like Figure 6 As shown, in some possible embodiments, a data processing apparatus is provided, comprising:

[0166] The receiving module 601 is used to receive data reconstruction instructions and scan the data to be processed corresponding to the current first moment based on the data reconstruction instructions;

[0167] Storage module 602 is used to receive update data from the first moment, store the received update data in a temporary object, and mark the receiving time stage of each update data as the first stage;

[0168] The marking module 603 is used to determine the current second time when the full data is obtained after scanning the data to be processed corresponding to the first time, and to mark the reception time stage of each updated data received after the second time as the second stage.

[0169] The merging module 604 is used to merge multiple updated data stored in the temporary object with the full data, starting from the second moment, based on the receiving time stage of each updated data stored in the temporary object, to generate reconstructed data.

[0170] In some possible implementations, when the merging module 604 merges the stored multiple update data with the full data to generate reconstructed data during the reception phase of each update data stored in the temporary object, it is specifically used for:

[0171] For each updated data point among the stored updated data, perform the following operations to generate reconstructed data:

[0172] If the data update is received in the first phase, the updated data will be merged into the full dataset.

[0173] If the data update reception time phase is the second phase, then skip the data update.

[0174] In some possible implementations, when the merging module 604 merges the updated data into the full data if the received data reception phase is the first phase, it is specifically used for:

[0175] If the data update is received in the first phase, then the updated data will be treated as data to be merged.

[0176] Lock the data to be merged;

[0177] Check if there is any new updated data in the temporary object that is updating the data. Based on the detection result, merge the locked data to be merged into the full data.

[0178] In some possible implementations, when merging the locked data to be merged into the full dataset based on the detection results, the merging module 604 is specifically used for:

[0179] If the detection result indicates that there is new updated data for the updated data in the temporary object, and the receiving time stage of the new updated data is the second stage, then skip the data to be merged.

[0180] If the detection result indicates that there is no new updated data for the updated data in the temporary object, then the locked data to be merged will be merged into the full data.

[0181] In some possible implementations, when storing the received multiple update data into a temporary object, the storage module 602 is specifically used for:

[0182] For each updated data, if the updated data is deleted, then the deletion instruction for the deleted data is marked to obtain the marked deletion instruction;

[0183] The update data corresponding to the mark deletion instruction is stored in a temporary object.

[0184] In some possible implementations, when merging locked data to be merged into the full dataset, the merging module 604 is specifically used for:

[0185] If the data to be merged corresponds to the mark-delete instruction, then modify the mark-delete instruction to a delete instruction;

[0186] The modified data corresponding to the deletion command will be merged into the full dataset.

[0187] In some possible implementations, when the storage module 602 marks a deletion instruction for deleted data and obtains a marked deletion instruction, it specifically performs the following:

[0188] Inserting data carrying a deletion identifier into the deletion instruction yields a marked deletion instruction.

[0189] In some possible implementations, when the merging module 604 merges the stored multiple update data with the full data to generate reconstructed data during the reception phase of each update data stored in the temporary object, it is specifically used for:

[0190] Divide the update data stored in the temporary object into at least two batches of update data;

[0191] Iterate through each batch of updated data to determine the receiving time phase for each batch of updated data.

[0192] The updated data is merged with the full data based on the receiving time of each batch of updated data to generate reconstructed data.

[0193] The data processing apparatus of this application stores updated data in a temporary object and marks the receiving time stages of the updated data. The first stage is defined as the period from the start of scanning to the completion of scanning and the start of merging, and the period after the second stage is defined as the second stage. The merging operation between the updated data and the data to be processed is performed according to the different receiving time stages of the updated data. During peak business periods, hot data will inevitably generate corresponding data for the second stage in the temporary object. Since the receiving time stages of this part of the data have been marked, the backfilling of this part of the data can be skipped by marking, thereby effectively avoiding the situation where the data merging speed cannot keep up with the update speed due to the large number of transactions on hot data, and effectively ensuring the completion of the data reconstruction task.

[0194] Furthermore, by marking the receiving time stage of each updated data, updated data in the first receiving time stage needs to be merged, while data in the second receiving time stage is skipped. In this way, only the data that needs to be merged needs to be merged in the second stage, while the data that does not need to be merged is filtered out, reducing the amount of data to be merged. Even during peak business periods and when there is a large amount of hot data, the merging efficiency can be effectively improved. It can even achieve the advantage that the more hot data there is, the fewer rounds of temporary index data are needed.

[0195] Furthermore, by detecting whether there is new updated data in the temporary object, the locked data to be merged is merged into the full data based on the detection result. This can prevent new updated data that was not scanned before from being generated when locking the data to be merged. It can further filter out data that does not need to be merged, thereby improving the efficiency of data reconstruction.

[0196] The apparatus in this application embodiment can execute the method provided in this application embodiment, and the implementation principle is similar. The actions performed by each module in the apparatus of each embodiment of this application correspond to the steps in the method of each embodiment of this application. For detailed functional descriptions of each module of the apparatus, please refer to the descriptions in the corresponding methods shown above, which will not be repeated here.

[0197] This application provides an electronic device, including a memory, a processor, and a computer program stored in the memory. When the processor executes the computer program stored in the memory, it can implement the method in any optional embodiment of this application.

[0198] Figure 7 A schematic diagram of the structure of an electronic device to which an embodiment of the present invention applies is shown, such as... Figure 7 As shown, the electronic device can be a server or a user terminal, and it can be used to implement the methods provided in any embodiment of the present invention.

[0199] like Figure 7 As shown, the electronic device 700 may primarily include at least one processor 701. Figure 7 The diagram shows components such as a memory 702, a communication module 703, and an input / output interface 704. Optionally, these components can be connected and communicate with each other via a bus 705. It should be noted that... Figure 7 The structure of the electronic device 700 shown is merely illustrative and does not constitute a limitation on the electronic devices to which the methods provided in the embodiments of this application are applicable.

[0200] The memory 702 can be used to store operating systems and applications, etc. The applications can include computer programs that implement the methods shown in the embodiments of the present invention when invoked by the processor 701, and can also include programs for implementing other functions or services. The memory 702 can be ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices that can store information and computer programs, or it can be EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disk storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited thereto.

[0201] Processor 701 is connected to memory 702 via bus 705 and implements corresponding functions by calling application programs stored in memory 702. Processor 701 can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the present invention disclosure. Processor 701 can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0202] Electronic device 700 can connect to a network via communication module 703 (which may include, but is not limited to, components such as a network interface) to communicate with other devices (such as user terminals or servers) through the network and achieve data interaction, such as sending data to or receiving data from other devices. Communication module 703 may include wired network interfaces and / or wireless network interfaces, meaning the communication module may include at least one of wired or wireless communication modules.

[0203] Electronic device 700 can connect to required input / output devices, such as keyboards and display devices, via input / output interface 704. Electronic device 700 itself may have a display device, and other display devices can also be connected externally via interface 704. Optionally, storage devices, such as hard drives, can also be connected via interface 704 to store data from electronic device 700, retrieve data from storage devices, or store data from storage devices into memory 702. It is understood that input / output interface 704 can be a wired interface or a wireless interface. Depending on the actual application scenario, the device connected to input / output interface 704 can be a component of electronic device 700 or an external device connected to electronic device 700 when needed.

[0204] The bus 705 used to connect the components may include a pathway for transmitting information between the components. The bus 705 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Depending on its function, the bus 705 can be classified as an address bus, data bus, control bus, etc.

[0205] Optionally, in the solution provided by the embodiments of the present invention, the memory 702 can be used to store a computer program that executes the solution of the present invention, and the processor 701 runs the computer program. When the processor 701 runs the computer program, it implements the operation of the method or apparatus provided by the embodiments of the present invention.

[0206] Based on the same principle as the method provided in the embodiments of this application, the embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the corresponding content of the aforementioned method embodiments.

[0207] This application also provides a computer program product, which includes a computer program that, when executed by a processor, can implement the corresponding content of the aforementioned method embodiments.

[0208] It should be noted that the terms "first," "second," "third," "fourth," "1," "2," etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in a sequence other than that shown in the figures or text.

[0209] It should be understood that although arrows indicate various operation steps in the flowcharts of this application's embodiments, the order in which these steps are implemented is not limited to the order indicated by the arrows. Unless explicitly stated herein, in some implementation scenarios of this application's embodiments, the implementation steps in each flowchart can be executed in other orders as required. Furthermore, some or all steps in each flowchart, based on the actual implementation scenario, may include multiple sub-steps or multiple stages. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage can also be executed at different times. In scenarios where execution times differ, the execution order of these sub-steps or stages can be flexibly configured according to requirements, and this application's embodiments do not limit this.

[0210] The above description is only an optional implementation method for some implementation scenarios of this application. It should be noted that for those skilled in the art, other similar implementation methods based on the technical concept of this application without departing from the technical concept of this application also fall within the protection scope of the embodiments of this application.

Claims

1. A data processing method, characterized in that, include: Receive a data reconstruction instruction for a DDL task, and scan the data to be processed corresponding to the current first moment based on the data reconstruction instruction; Starting from the first moment, update data is received, and the received update data is stored in a temporary object, and the receiving time stage of each update data is marked as the first stage; When the scan of the data to be processed corresponding to the first time point is completed and the full amount of data is obtained, the current second time point is determined, and the reception time stage of each updated data received after the second time point is marked as the second stage. Starting from the second moment, based on the receiving time stage of each updated data stored in the temporary object, the stored multiple updated data are merged with the full data to skip the backfilling of updated data marked as the second receiving time stage, and reconstructed data is generated. The temporary object is independent of the object in the full data. The step of storing multiple received update data into a temporary object includes: if the update data is deleted data, then marking the deletion instruction for the deleted data to obtain a marked deletion instruction, and storing the update data corresponding to the marked deletion instruction into the temporary object; The generation of reconstructed data includes: if the updated data is deleted data and the receiving time stage of the updated data is the first stage, then the updated data is locked; if there is no new updated data for the updated data in the temporary object, then the locked updated data is merged into the full data; wherein, if the updated data corresponds to a marked deletion instruction, then the marked deletion instruction is modified into a deletion instruction, and the modified updated data corresponding to the deletion instruction is merged into the full data.

2. The data processing method according to claim 1, characterized in that, The step of merging multiple stored update data with the full data based on the receiving time stage of each update data stored in the temporary object to generate reconstructed data includes: For each updated data in the stored set of data, perform the following operations to generate the reconstructed data: If the receiving time phase of the updated data is the first phase, then the updated data is merged into the full data. If the receiving time stage of the updated data is the second stage, then the updated data is skipped.

3. The data processing method according to claim 2, characterized in that, If the receiving time phase of the updated data is the first phase, then merging the updated data into the full data includes: If the receiving time stage of the updated data is the first stage, then the updated data will be regarded as data to be merged. Lock the data to be merged; Detect whether there is any new updated data for the updated data in the temporary object, and merge the locked data to be merged into the full data based on the detection result.

4. The data processing method according to claim 3, characterized in that, The step of merging the locked data to be merged into the full dataset based on the detection results includes: If the detection result indicates that there is new updated data for the updated data in the temporary object, and the receiving time stage of the new updated data is the second stage, then the data to be merged is skipped. If the detection result indicates that there is no new updated data for the updated data in the temporary object, then the locked data to be merged is merged into the full data.

5. The data processing method according to claim 4, characterized in that, The step of storing the received multiple update data into a temporary object includes: For each piece of updated data, if the updated data is deleted data, then the deletion instruction for the deleted data is marked to obtain a marked deletion instruction; The updated data corresponding to the marker deletion instruction is stored in the temporary object.

6. The data processing method according to claim 5, characterized in that, The step of merging the locked data to be merged into the full data includes: If the data to be merged corresponds to the mark deletion instruction, then the mark deletion instruction is modified to the deletion instruction; The modified data to be merged, corresponding to the deletion instruction, is merged into the full data.

7. The data processing method according to claim 5, characterized in that, The step of marking the deletion instruction for the deleted data to obtain a marked deletion instruction includes: The deletion instruction is obtained by inserting data carrying a deletion identifier into the deletion instruction.

8. The data processing method according to any one of claims 1 to 7, characterized in that, The step of merging multiple stored update data with the full data based on the receiving time stage of each update data stored in the temporary object to generate reconstructed data includes: The update data stored in the temporary object is divided into at least two batches of update data; Iterate through each batch of updated data to determine the receiving time phase for each batch of updated data. The updated data is merged with the full data based on the receiving time stage of each batch of updated data to generate reconstructed data.

9. A data processing apparatus, characterized in that, include: The receiving module is used to receive data reconstruction instructions for DDL tasks and scan the data to be processed corresponding to the current first moment based on the data reconstruction instructions; The storage module is used to receive update data starting from the first moment, store the received update data in a temporary object, and mark the receiving time stage of each update data as the first stage; The marking module is used to determine the current second time when the full data is obtained after scanning the data to be processed corresponding to the first time, and to mark the reception time stage of each updated data received after the second time as the second stage; The merging module is used to merge multiple updated data stored in the temporary object with the full data based on the receiving time stage of each updated data stored in the temporary object, starting from the second time moment, so as to skip the backfilling of updated data marked as the second stage of receiving time stage and generate reconstructed data. The temporary object is independent of the object in the full data. The step of storing multiple received update data into a temporary object includes: if the update data is deleted data, then marking the deletion instruction for the deleted data to obtain a marked deletion instruction, and storing the update data corresponding to the marked deletion instruction into the temporary object; The generation of reconstructed data includes: if the updated data is deleted data and the receiving time stage of the updated data is the first stage, then the updated data is locked; if there is no new updated data for the updated data in the temporary object, then the locked updated data is merged into the full data; wherein, if the updated data corresponds to a marked deletion instruction, then the marked deletion instruction is modified into a deletion instruction, and the modified updated data corresponding to the deletion instruction is merged into the full data.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the data processing method according to any one of claims 1-8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the data processing method according to any one of claims 1-8.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the data processing method according to any one of claims 1-8.

Images