Dynamic updating method and device, computer device, readable storage medium and program product

Abstract

The application relates to a dynamic updating method and device, computer equipment, a readable storage medium and a program product. The method comprises the following steps: receiving a target subtask sent by a scheduler; the target subtask is generated by the scheduler in a data cluster after splitting an updating task and distributing the updating task according to the load conditions of each processing node; determining incremental data according to the target subtask, comparing the incremental data with historical data in a database, and determining a target field which has changed; and generating updating data based on the target field, and writing the updating data into the database. The method can improve the data processing efficiency and the system response capability.

Description

Technical Field

[0001] This application relates to the field of database technology, and in particular to a dynamic update method, apparatus, computer equipment, readable storage medium, and program product. Background Technology

[0002] With the development of distributed database technology and database technology, more and more business systems are beginning to adopt data cluster architecture to process high-frequency dynamic data in real time. Compared with traditional disk databases, databases have the characteristics of fast response speed and low read / write latency, and are therefore widely used in scenarios such as real-time status synchronization, online business processing, and high-concurrency data updates.

[0003] In related technologies, when data in a data source changes, the processing nodes in the data cluster typically perform update operations directly on the corresponding data records. For example, upon receiving a data update request, each processing node reads the corresponding data record and writes the new data into the database to complete the data synchronization update.

[0004] However, current dynamic update methods typically overwrite the entire row of data directly. Even if only a few fields change, it will trigger the repeated writing of the entire row of data, resulting in a large number of invalid write operations and increasing system resource consumption. At the same time, existing technologies lack a dynamic task allocation mechanism based on the load of processing nodes, which can easily lead to some processing nodes being overloaded while other processing nodes are idle, thereby reducing the overall update efficiency of the data cluster and the data processing capacity in high-concurrency scenarios. Summary of the Invention

[0005] Therefore, it is necessary to provide a dynamic update method, apparatus, computer equipment, readable storage medium, and program product that can improve data processing efficiency and system responsiveness in response to the above-mentioned technical problems.

[0006] Firstly, this application provides a dynamic update method, including:

[0007] Receive the target subtask sent by the scheduler; the target subtask is generated by the scheduler in the data cluster after splitting the update task and allocating it according to the load of each processing node.

[0008] Incremental data is determined based on the target sub-task, and the incremental data is compared with historical data in the database to determine the target fields that have changed;

[0009] Updated data is generated based on the target field, and the updated data is written into the database.

[0010] In one embodiment, the target subtask is generated by the scheduler after allocating initial subtasks based on the load of each processing node. The initial subtask is obtained by the scheduler splitting the update task in the data update instruction according to the table structure changes in the data update instruction after receiving the data update instruction.

[0011] In one embodiment, comparing the incremental data with historical data in the database to determine the target field that has changed includes:

[0012] The metadata of hot fields whose historical access frequency exceeds a preset threshold is preloaded into the memory cache;

[0013] Based on the field metadata, candidate fields corresponding to the incremental data are determined;

[0014] The incremental data is compared with the data of the candidate fields, and the candidate fields whose field values ​​have changed are selected as the target fields.

[0015] In one embodiment, writing the updated data into the database includes:

[0016] The updated data for the same primary key will be merged, and the merged updated data will be used as an update transaction.

[0017] If one of the updated data writes to the database fails in the update transaction, the updated data already written to the database in the update transaction will be restored to the corresponding historical data.

[0018] In one embodiment, before writing the updated data to the database, the method further includes:

[0019] Write the transaction identifier corresponding to the update transaction, the target field, the historical data corresponding to the target field, and the update data into the pre-log; if the update data fails to be written to the database, restore the historical data of the corresponding field.

[0020] In one embodiment, the method further includes:

[0021] Count the frequency of the updated data;

[0022] When the frequency is lower than a preset threshold, the updated data for the same primary key will be merged.

[0023] When the frequency exceeds the preset threshold, the updated data is written to the database in batches.

[0024] Secondly, this application also provides a dynamic update device, comprising:

[0025] The receiving module is used to receive the target subtask sent by the scheduler; the target subtask is generated by the scheduler in the data cluster after splitting the update task and allocating it according to the load of each processing node.

[0026] The data determination module is used to determine incremental data based on the target sub-task, and compare the incremental data with historical data in the database to determine the target field that has changed.

[0027] The update module is used to generate update data based on the target field and write the update data into the database.

[0028] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method in any of the above embodiments.

[0029] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method in any of the above embodiments.

[0030] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the methods in any of the above embodiments.

[0031] The aforementioned dynamic update method, apparatus, computer equipment, readable storage medium, and program product involve a scheduler that first breaks down the update task into sub-tasks based on the update task and generates corresponding target sub-tasks based on the load of each processing node. This allows for dynamic allocation of update tasks according to the resource usage of different processing nodes, improving the load balancing and parallel processing capabilities of the data cluster. Next, each processing node determines the corresponding incremental data based on the target sub-task and compares this incremental data with historical data in the database to identify the changed target fields. Update data is generated only for the changed target fields, avoiding duplicate updates to unchanged fields and reducing system resource overhead from invalid writes. Finally, update data is generated based on the target fields and written to the database, thus achieving field-level dynamic updates and improving data processing efficiency and real-time system response capabilities in high-frequency data update scenarios. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 A flowchart illustrating a dynamic update method;

[0034] Figure 2 This is a structural block diagram of a dynamic update device in one embodiment;

[0035] Figure 3 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0037] In one embodiment, such as Figure 1 As shown, a dynamic update method is provided. This embodiment illustrates the method applied to a terminal, but it is understood that the method can also be applied to a server, and to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0038] Step 102: Receive the target subtask sent by the scheduler; the target subtask is generated by the scheduler in the data cluster after splitting the update task and allocating it according to the load of each processing node.

[0039] During the dynamic difference capture phase, the system captures field-level change data in real time by parsing the binlog logs of the source database or application-layer instrumentation information, and extracts the corresponding field identifiers, old values, new values, and timestamps as difference data. Then, the scheduler generates corresponding update tasks based on the difference data. Each update task identifies the update range of the corresponding field data and the update operation to be performed.

[0040] Optionally, the target task is generated after the scheduler receives the data update instruction, splits the update task in the data update instruction according to the table structure changes in the data update instruction, obtains the initial subtasks, and then allocates the initial subtasks according to the load of the processing nodes.

[0041] After determining the target subtask, the scheduler sends the target subtask to the corresponding processing node.

[0042] Step 104: Determine incremental data based on the target sub-task, and compare the incremental data with historical data in the database to determine the target field that has changed.

[0043] Optionally, when generating target subtasks, the scheduler assigns a corresponding data range, field range, or primary key range to each target subtask. After receiving a target subtask, each processing node reads the corresponding new or changed data according to the data range of the target subtask and uses the new or changed data as incremental data.

[0044] Incremental data is used to characterize data content that has changed compared to historical data, such as new values ​​of fields, update times of fields, or change states of fields.

[0045] Next, each processing node compares the incremental data with historical data in the database to determine the target field that has actually changed. Specifically, the processing node first identifies the corresponding field based on the field identifier in the incremental data, then reads the historical data of the corresponding field in the database, and compares the field values ​​in the incremental data with the field values ​​in the historical data field by field; when the field values ​​are inconsistent, the corresponding field is determined to be the target field.

[0046] For example, suppose the historical data in the database is: User ID: 1001; Contact Number: 13800000000; Status: Online. The incremental data corresponding to the current target subtask is: User ID: 1001; Contact Number: 13900000000; Status: Online. At this point, the processing node determines through field-level comparison that the data corresponding to the "Contact Number" field has changed, while the data corresponding to the "Status" field has not changed. Therefore, the "Contact Number" field is determined as the target field.

[0047] Step 106: Generate updated data based on the target field and write the updated data into the database.

[0048] Optionally, after determining the target field, the processing node extracts the corresponding field value from the incremental data and generates update data by combining the field identifier, the new field value, and the corresponding primary key. This update data represents the data content that needs to be written to the database.

[0049] Specifically, the processing node generates corresponding field update instructions based on the updated data, and writes the updated data into the database based on the field update instructions, thereby completing the field-level dynamic update.

[0050] In the above embodiment, the scheduler first splits the update task according to its own task and generates corresponding target subtasks based on the load of each processing node. This allows for dynamic allocation of update tasks according to the resource usage of different processing nodes, improving load balancing and parallel processing capabilities in the data cluster. Next, each processing node determines the corresponding incremental data based on the target subtask and compares it with historical data in the database to identify the changed target fields. Update data is generated only for the changed target fields, avoiding duplicate updates to unchanged fields and reducing system resource overhead from invalid writes. Finally, update data is generated based on the target fields and written to the database, thus achieving field-level dynamic updates and improving data processing efficiency and real-time system response capabilities in high-frequency data update scenarios.

[0051] In one embodiment, the target subtask is generated by the scheduler after allocating the initial subtasks based on the load of each processing node. The initial subtask is obtained by the scheduler splitting the update task in the data update instruction according to the table structure changes in the data update instruction after receiving the data update instruction.

[0052] Optionally, when splitting update tasks, the scheduler first determines the task scope corresponding to each updated data based on the field range, table range, or primary key range in the data update instruction; then, the scheduler performs hash mapping based on the field identifier, table identifier, or primary key identifier corresponding to each task scope, and divides the update task into multiple non-overlapping initial subtasks according to the hash mapping result.

[0053] Optionally, when allocating initial subtasks, the scheduler obtains the load information of each processing node in real time and dynamically allocates multiple initial subtasks based on the load information of each processing node. The load information includes at least one of CPU utilization, memory utilization, and network I / O utilization.

[0054] For example, when the CPU utilization of the first processing node is lower than a preset threshold, the scheduler increases the number of initial subtasks allocated to the first processing node; when the CPU utilization of the second processing node is higher than the preset threshold, the scheduler reduces the number of initial subtasks allocated to the second processing node, or reduces the number of concurrent update threads corresponding to the second processing node.

[0055] Optionally, the scheduler may also set corresponding task priorities for each update task based on the urgency or business level of the update task, and prioritize the allocation of update tasks with higher priorities.

[0056] In the above embodiments, by splitting the update task, multiple processing nodes can execute the corresponding data update operations in parallel, thereby improving the data processing efficiency in high-concurrency scenarios. Furthermore, by dynamically allocating the initial sub-tasks according to the load of each processing node, the load balancing capability and overall update efficiency in the data cluster can be improved.

[0057] In one embodiment, comparing incremental data with historical data in the database to determine the target field that has changed includes: preloading the metadata of hot fields with historical access frequencies higher than a preset threshold into the memory cache; determining candidate fields corresponding to the incremental data based on the field metadata; comparing the incremental data with the data of the candidate fields, and taking the candidate fields whose field values ​​have changed as the target fields.

[0058] The preset threshold is used to characterize the minimum requirement for field access frequency, and it can be set according to system operating status, historical access count, or preset business rules.

[0059] Optionally, the system will periodically count the number of times each field is accessed within a preset time range, and identify fields with access counts exceeding a preset threshold as hot fields. Subsequently, the system will load the field metadata, such as the field identifier, field type, and field mapping relationship corresponding to the hot fields, into the memory cache to reduce the overhead of repeated parsing in the subsequent field identification process.

[0060] Optionally, after receiving incremental data, the system determines the candidate fields corresponding to the incremental data based on the field identifiers and field mapping relationships in the field metadata. Then, it reads the historical data corresponding to the candidate fields from the database and compares the field values ​​in the incremental data with those in the historical data field by field; when field values ​​are inconsistent, the corresponding candidate field is determined as the target field.

[0061] For example, field metadata includes at least one of field identifier, field name, field type, and field mapping relationship. For instance, the field metadata records that: field identifier "F1" corresponds to the "Contact Number" field; and field identifier "F2" corresponds to the "Status" field.

[0062] The current incremental data includes: field identifier "F1", field value "13900000000"; field identifier "F2", field value "online".

[0063] Based on the field mapping relationships in the field metadata, the candidate field corresponding to field identifier "F1" is determined to be the "Contact Number" field, and the candidate field corresponding to field identifier "F2" is the "Status" field. Then, the system reads historical data from the database: Contact Number: 13800000000; Status: Online.

[0064] The system compares the field values ​​in the incremental data with those in the historical data. Since the field value for "Contact Number" changed from "13800000000" to "13900000000", while the field value for "Status" remained unchanged, the system identifies the "Contact Number" field as the target field.

[0065] In the above embodiments, by preloading the field metadata corresponding to hot fields into the memory cache, the overhead of repeatedly parsing field mapping relationships during incremental data processing can be reduced, thereby improving the identification efficiency of candidate fields and the data processing efficiency in high-frequency data update scenarios.

[0066] In one embodiment, writing the updated data to the database includes: merging the updated data for the same primary key and using the merged updated data as an update transaction; when one of the updated data in the update transaction fails to be written to the database, restoring the updated data already written to the database in the update transaction to the corresponding historical data.

[0067] Since data corresponding to the same primary key usually belongs to the same data record, merging the update data for the same primary key can reduce the number of times the same data record is written repeatedly and reduce the transaction overhead caused by multiple independent updates.

[0068] Furthermore, for update data with an update frequency lower than a preset threshold, the system can also merge multiple update data in batches before performing the write operation, in order to reduce the system resource consumption caused by frequent writes.

[0069] In this embodiment, an update transaction refers to a group of update operations corresponding to the same primary key. When any update data in an update transaction fails to be written to the database, the system will restore the update data that has already been written to the database in the update transaction to the corresponding historical data, so as to ensure that all update data in the update transaction is successfully written to the database or none of them are written to the database.

[0070] Optionally, this embodiment supports both batch update mode and single-field update mode. Batch update mode is suitable for low-frequency scenarios involving changes to multiple fields; single-field update mode is suitable for high-frequency scenarios involving changes to a single field.

[0071] Furthermore, after a successful update transaction, the system generates a corresponding version snapshot. This version snapshot includes at least one of a timestamp or an incrementing version number. The system can use this version snapshot to backtrack to historical data states and, in the event of concurrent update conflicts, determine the corresponding data version based on the version number, thereby improving consistency and traceability during the data update process.

[0072] In the above embodiments, by merging update data for the same primary key, the transaction overhead caused by duplicate writes can be reduced; furthermore, by performing data updates based on update transactions and restoring the corresponding historical data when writes fail, the consistency and reliability of the data update process can be improved.

[0073] In one embodiment, before writing the updated data to the database, the method further includes: writing the transaction identifier corresponding to the update transaction, the target field, the historical data corresponding to the target field, and the updated data to a pre-log; the pre-log is used to recover the historical data of the corresponding field when the update data fails to be written to the database.

[0074] In high-concurrency data update scenarios, there may be abnormal system interruptions, node failures, or partial data writing failures during the update process. Therefore, this embodiment will record the corresponding transaction information and historical data in advance before performing data updates, so as to restore the data state before the update in case of data writing failure.

[0075] Furthermore, in a cluster environment, the system coordinates cross-node transactions through a distributed two-phase commit protocol: In the first phase, the scheduler sends a readiness request to all processing nodes, and each processing node performs a local pre-commit and locks resources; if all nodes are ready, the scheduler issues a commit instruction in the second phase, otherwise it issues a recovery instruction to enable each processing node to restore the historical data of the corresponding fields, so as to ensure the consistency of cross-node transactions.

[0076] Finally, to achieve data synchronization between storage layers, after the in-memory database transaction is committed, the difference operations in the write-ahead log will be asynchronously synchronized to the disk database to ensure that the data in the in-memory database and the disk database achieve eventual consistency.

[0077] In the above embodiments, by pre-recording transaction information and historical data, the historical data of the corresponding field can be restored when the update fails, thereby improving the reliability and consistency of the data update process.

[0078] In one embodiment, the method further includes: counting the frequency of data updates; merging update data for the same primary key when the frequency is lower than a preset threshold; and writing update data into the database in batches when the frequency is higher than the preset threshold.

[0079] Optionally, the system will count the number of updates corresponding to the updated data within a preset time range, and use the number of updates per unit time as the frequency of data updates.

[0080] Specifically, when the frequency is lower than the preset threshold, it indicates that the current data update frequency is low. In this case, the system will merge the update data for the same primary key before performing the write operation to reduce the transaction overhead caused by frequent writes. When the frequency is higher than the preset threshold, it indicates that the current data update frequency is high. In this case, the system will write the update data in batches according to the preset quantity to reduce the amount of data written at one time and improve the real-time performance of data updates.

[0081] In the above embodiments, by dynamically adjusting the data writing method according to the frequency of data updates, both the processing efficiency in low-frequency update scenarios and the real-time data update in high-frequency update scenarios can be taken into account.

[0082] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0083] Based on the same inventive concept, this application also provides a dynamic update apparatus for implementing the dynamic update method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more dynamic update apparatus embodiments provided below can be found in the limitations of the dynamic update method described above, and will not be repeated here.

[0084] In one exemplary embodiment, such as Figure 2 As shown, a dynamic update device is provided, comprising: a receiving module 100, a data determining module 200, and an update module 300, wherein:

[0085] The receiving module 100 is used to receive the target subtask sent by the scheduler; the target subtask is generated by the scheduler in the data cluster after splitting the update task and allocating it according to the load of each processing node.

[0086] The data determination module 200 is used to determine incremental data based on the target sub-task, and compare the incremental data with historical data in the database to determine the target fields that have changed.

[0087] The update module 300 is used to generate update data based on the target field and write the update data into the database.

[0088] In one embodiment, the target subtask in the receiving module is generated by the scheduler after allocating the initial subtask based on the load of each processing node. The initial subtask is obtained by the scheduler splitting the update task in the data update instruction according to the table structure changes in the data update instruction after receiving the data update instruction.

[0089] In one embodiment, the data determining module 200 includes:

[0090] The pre-loading unit is used to preload the metadata of hot fields with historical access frequencies higher than a preset threshold into the memory cache.

[0091] The field determination unit is used to determine the candidate fields corresponding to incremental data based on the field metadata.

[0092] The comparison unit is used to compare incremental data with the data of candidate fields and to select the candidate field whose field value has changed as the target field.

[0093] In one embodiment, the update module 300 includes:

[0094] The first merging unit is used to merge update data for the same primary key and use the merged update data as an update transaction.

[0095] The rollback unit is used to restore the updated data that has already been written to the database in an update transaction to the corresponding historical data when an update transaction fails to write updated data to the database.

[0096] In one embodiment, the above-mentioned apparatus further includes a log generation module, which includes:

[0097] The write log unit is used to write the transaction identifier, target field, historical data corresponding to the target field, and updated data of the update transaction to the pre-log; if the update data writing to the database fails, the historical data of the corresponding field is restored.

[0098] In one embodiment, the update module 300 includes:

[0099] Frequency statistics unit, used to count the frequency of data updates.

[0100] The second merging unit is used to merge update data for the same primary key when the frequency is lower than a preset threshold.

[0101] The batch write unit is used to write updated data to the database in batches when the frequency exceeds a preset threshold.

[0102] Each module in the aforementioned dynamic update device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0103] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 3 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores and updates data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. The computer program, when executed by the processor, implements a dynamic update method.

[0104] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0105] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method in any of the above embodiments.

[0106] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method in any of the above embodiments.

[0107] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the method in any of the above embodiments.

[0108] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0109] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0110] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A dynamic update method, characterized in that, The method is applied to each processing node in a data cluster; the method includes: Receive the target subtask sent by the scheduler; the target subtask is generated by the scheduler in the data cluster after splitting the update task and allocating it according to the load of each processing node. Incremental data is determined based on the target sub-task, and the incremental data is compared with historical data in the database to determine the target fields that have changed; Updated data is generated based on the target field, and the updated data is written into the database.

2. The method according to claim 1, characterized in that, The target subtask is generated by the scheduler after allocating the initial subtasks based on the load of each processing node. The initial subtask is obtained by the scheduler after receiving the data update instruction and splitting the update task in the data update instruction according to the table structure changes in the data update instruction.

3. The method according to claim 1, characterized in that, The step of comparing the incremental data with historical data in the database to determine the target field that has changed includes: The metadata of hot fields whose historical access frequency exceeds a preset threshold is preloaded into the memory cache; Based on the field metadata, candidate fields corresponding to the incremental data are determined; The incremental data is compared with the data of the candidate fields, and the candidate fields whose field values ​​have changed are selected as the target fields.

4. The method according to claim 1, characterized in that, The step of writing the updated data into the database includes: The updated data for the same primary key will be merged, and the merged updated data will be used as an update transaction. If one of the updated data writes to the database fails in the update transaction, the updated data already written to the database in the update transaction will be restored to the corresponding historical data.

5. The method according to claim 4, characterized in that, Before writing the updated data into the database, the method further includes: The transaction identifier corresponding to the update transaction, the target field, the historical data corresponding to the target field, and the update data are written to the pre-log; if the update data fails to be written to the database, the pre-log restores the historical data of the corresponding field.

6. The method according to claim 4, characterized in that, The method further includes: Count the frequency of the updated data; When the frequency is lower than a preset threshold, the updated data for the same primary key will be merged. When the frequency exceeds the preset threshold, the updated data is written to the database in batches.

7. A dynamic updating device, characterized in that, The device includes: The receiving module is used to receive the target subtask sent by the scheduler; the target subtask is generated by the scheduler in the data cluster after splitting the update task and allocating it according to the load of each processing node. The data determination module is used to determine incremental data based on the target sub-task, and compare the incremental data with historical data in the database to determine the target field that has changed; The update module is used to generate update data based on the target field and write the update data into the database.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images