A data updating method, an updating node, an electronic device, and a storage medium
By employing a soft deletion mechanism and a phased data update approach during the data update process, the problem of low data synchronization efficiency was solved, achieving efficient data synchronization and service stability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- TRAVELSKY TECHNOLOGY LIMITED
- Filing Date
- 2022-08-23
- Publication Date
- 2026-05-29
AI Technical Summary
In existing technologies, the data update methods result in low data synchronization efficiency, affecting the normal provision of services.
A soft deletion mechanism is used to process expired data, generate current and historical database change file sets, and push them to the computing node cluster for updates at different time periods, thereby reducing the amount of data modification and improving synchronization efficiency.
By employing a soft-delete mechanism and a phased data update approach, data synchronization efficiency was improved, avoiding any impact on the normal provision of services.
Smart Images
Figure CN115344590B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data update technology, and in particular to a data update method, update node, electronic device, and storage medium. Background Technology
[0002] The latest fare calculation systems for airline tickets and other fares have abandoned the shared memory approach in their underlying architecture, instead using a memory-mapped database as the carrier of fare and rule data. This database is essentially a set of memory-mapped files. Update nodes update the data and send the changed memory-mapped files to the computing cluster, allowing the cluster to provide services using the updated data.
[0003] The current data update method primarily involves the updating node updating its local memory-mapped database upon receiving new data. During the update process, expired data is directly deleted from the memory-mapped database, and data to be replaced is removed. Then, an update file is generated based on the changed data and sent to the computing cluster for updating.
[0004] However, removing data directly from the memory-mapped database will require the data to be reordered, resulting in a large number of data changes. Ultimately, when synchronizing from the memory-mapped database to the computing cluster, a large number of data modifications need to be completed, leading to low data synchronization efficiency and potentially affecting the normal provision of services. Summary of the Invention
[0005] In view of the shortcomings of the prior art, this application provides a data update method, update node, electronic device and storage medium to solve the problem of low update efficiency in the prior art.
[0006] To achieve the above objectives, this application provides the following technical solution:
[0007] The first aspect of this application provides a data update method applied to an update node, wherein the update node includes a current update node and historical update nodes, and the data update method includes:
[0008] Each update node receives the original update data in real time.
[0009] When the update node receives the original update data, it updates the original update data to its local memory-mapped database and performs soft deletion on the expired data in the local memory-mapped database; wherein, the data that has been soft deleted is in a state that cannot be used as current computing data;
[0010] The current update node generates a current database change file set based on the original update data and pushes the current database change file set to the compute node cluster to trigger each compute node in the compute node cluster to update its local current database using the current database change file set;
[0011] If the current period is a data maintenance period, the historical update node generates a historical database change file set based on the original update data and pushes the historical database change file set to the computing node cluster to trigger each computing node in the computing node cluster to update its local historical database using the historical database change file set.
[0012] Optionally, the above data update method further includes:
[0013] If the current period is a data maintenance period, the historical update node will perform hard deletion of the expired data that was soft-deleted in its memory-mapped database;
[0014] The historical update node generates a second historical database change file set based on the expired data that has undergone hard deletion.
[0015] The historical update node pushes the second historical database change file set to the computing node cluster to trigger each computing node in the computing node cluster to update its local historical database using the second historical database change file set.
[0016] Optionally, the above data update method further includes:
[0017] If the current node is in the data maintenance period, it will perform hard deletion of the expired data that was soft-deleted in its memory-mapped database;
[0018] The current update node generates a second current database change file set based on the expired data that has undergone hard deletion.
[0019] The current update node pushes the second current database change file set to the compute node cluster to trigger each compute node in the compute node cluster to update its local historical database using the second current database change file set.
[0020] Optionally, in the above data update method, the step of soft-deleting expired data in the local memory-mapped database includes:
[0021] Set the expiration time in the valid time range corresponding to the expired data in the memory-mapped database from the maximum value to the update cycle time corresponding to the original update data.
[0022] Optionally, in the above data update method, the step of hard-deleting the expired data that has been soft-deleted in its own memory-mapped database includes:
[0023] The expired data that has been soft-deleted in its own memory-mapped database is deleted from the memory-mapped database.
[0024] A second aspect of this application provides an update node, characterized in that it includes:
[0025] Current update node and historical update nodes;
[0026] The current update node is used to receive raw update data in real time. Upon receiving the raw update data, it updates the raw update data to the local memory-mapped database, performs soft deletion on expired data in the local memory-mapped database, generates a current database change file set based on the raw update data, and pushes the current database change file set to the compute node cluster to trigger each compute node in the compute node cluster to update its local current database using the current database change file set. The data that has been soft-deleted is in a state where it cannot be used as current compute data.
[0027] The historical update node is used to receive the original update data in real time. When the original update data is received, the node updates the original update data to the local memory-mapped database and performs soft deletion on the expired data in the local memory-mapped database. When the data maintenance period is in progress, the node generates a historical database change file set based on the original update data and pushes the historical database change file set to the computing node cluster to trigger each computing node in the computing node cluster to update its local historical database using the historical database change file set.
[0028] Optionally, in the above-mentioned update nodes, the historical update node is further used for:
[0029] During the current data maintenance period, the expired data that has been soft-deleted in its memory-mapped database will be hard-deleted.
[0030] Based on the expired data that has undergone hard deletion, a second set of historical database change files is generated;
[0031] The second set of historical database change files is pushed to the compute node cluster to trigger each compute node in the compute node cluster to update its local historical database using the second set of historical database change files.
[0032] Optionally, in the above-described update nodes, the current update node is further used for:
[0033] During the current data maintenance period, the expired data that has been soft-deleted in its memory-mapped database will be hard-deleted.
[0034] Based on the expired data that has undergone hard deletion, a second current database change file set is generated;
[0035] The second current database change file set is pushed to the compute node cluster to trigger each compute node in the compute node cluster to update its local historical database using the second current database change file set.
[0036] A third aspect of this application provides an electronic device, comprising:
[0037] One or more processing devices;
[0038] A memory on which one or more programs are stored;
[0039] When the one or more programs are executed by the one or more processing devices, the one or more processing devices implement the data update method as described in any of the preceding claims.
[0040] A fourth aspect of this application provides a computer storage medium, characterized in that it is used to store a program, which, when executed, is used to implement the data update method as described in any one of the preceding claims.
[0041] This application provides a data update method applied to update nodes, which include a current update node and historical update nodes. Both update nodes receive raw update data in real time and update their local memory-mapped databases upon receiving the raw update data. Then, expired data in the local memory-mapped database is soft-deleted. The soft-deleted data is in a state where it can no longer be used for current computation; that is, it's equivalent to simply setting the expired data to no longer be used for current computation, rather than directly deleting it from the database, thus reducing the amount of data modified. Next, the current update node, with its relatively shorter data retention time, generates a current database change file set based on the raw update data and pushes this set to the compute node cluster. Since the current update node's database contains less data, the size of the change files can be reduced, thereby improving data synchronization efficiency. Each compute node in the compute node cluster uses the current database change file set to update its local current database, ensuring the current data is up-to-date. For historical update nodes, a historical database change file set is generated based on the original update data only when the data maintenance period is in progress. This historical database change file set is then pushed to the compute node cluster for updates, thus avoiding impacting data update efficiency and consequently affecting service provision when the service is being provided normally. Attached Figure Description
[0042] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0043] Figure 1 A flowchart illustrating a data update method provided in an embodiment of this application;
[0044] Figure 2 A flowchart illustrating a method for historical update nodes to maintain and process data, provided in an embodiment of this application;
[0045] Figure 3 A flowchart illustrating a method for the current update node to maintain and process data, provided in an embodiment of this application;
[0046] Figure 4 A schematic diagram of an update node architecture provided in an embodiment of this application;
[0047] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0048] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.
[0049] In this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0050] This application provides a data update method applied to update nodes. The update nodes include current update nodes and historical update nodes. For example... Figure 1 As shown, the data update method provided in this application includes the following steps:
[0051] S101 and the update node receive the original update data in real time.
[0052] In this embodiment, three main types of nodes are included: current update nodes, historical update nodes, and compute nodes. Current update nodes and historical update nodes are primarily used for data update management. Compute nodes primarily provide services based on the data pushed by the two update nodes.
[0053] It should be noted that when stored data needs to be updated, i.e., when the original data is being updated, since both the current update node and the historical update node have local memory-mapped databases, these databases need to be updated as well. Therefore, the original update data is pushed to both update nodes simultaneously, i.e., to both the current update node and the historical update node. Accordingly, the current update node and the historical update node receive the pushed original update data in real time, and upon receiving the original update data, they respectively execute step S102.
[0054] Data is typically updated periodically, meaning a new batch of original update data is pushed out at regular intervals. This original update data is usually a compressed file containing compressed text. The name of the original update data can include the specific date and time, making it easy to record its effective date and manage its validity period. The name can also include a version number; for example, it could be named "20211206_140000.zip," indicating that this is the batch of original update data from 2 PM on December 6, 2021, while the next batch would typically be "20211206_150000.zip."
[0055] S102. When the update node receives the original update data, it updates the original update data to the local memory-mapped database and performs soft deletion on the expired data in the local memory-mapped database.
[0056] Data that has been soft-deleted is no longer usable for current calculations. Expired data refers to data that contains the latest information in the original update data and needs to be replaced.
[0057] Specifically, for the current update node and the historical update node, after receiving the original update data, both nodes update the original update data to their own local memory-mapped database.
[0058] For example, the original update data is "20211206_140000.zip". The data version in the local memory-mapped database of the current update node is "20211205_060000-20211206_130000", meaning that the current database ranges from "20211205_060000" to "20211206_130000". The data version in the local memory-mapped database of the historical update node is "20181205_060000-20211206_130000". After the current update node and the historical update node update the original update data to their local memory-mapped databases, the updated versions of the two update nodes are "0211205_060000-20211206_140000" and "20181205_060000-20211206_140000" respectively.
[0059] It's important to note that soft deletion doesn't mean removing data outright, but rather ceasing its use in current calculations. This means that when calculating ticket prices or related data, the data cannot be used; it's equivalent to deleting it from the current calculation. Directly deleting data would require reordering, necessitating extensive data modifications during memory-to-disk synchronization, leading to inefficiency. Soft deletion, on the other hand, ensures only a small amount of data needs to be synchronized. However, it's also worth noting that while soft-deleted data can no longer be used for current calculations, it can still be used as historical data. For example, when calculating historical ticket prices or other historical data within the data's validity period, the data can still be used—another advantage of soft deletion.
[0060] Specifically, in this application embodiment, a specific implementation method for soft-deleting expired data in a local memory-mapped database includes:
[0061] Set the expiration time in the valid time range corresponding to the expired data in the memory-mapped database from the maximum value to the update cycle time corresponding to the original updated data.
[0062] It should be noted that in this embodiment, a corresponding data record is generated for each version of the data. This record is a basic element constituting the memory-mapped database and is stored in various containers, such as data groups and hash tables. This data record contains the valid data range, which includes the start and end times. The start time is usually the date in the data name, while the end time is initially set to a maximum value.
[0063] Therefore, in this embodiment, filtering out expired data does not refer to determining data that has exceeded the valid data range, because the expiration time of the valid data range is initially a maximum value. Instead, the expired data filtered out in this embodiment refers to data in the received original update data that includes new data to replace it. For example, for a ticket price rule for a certain membership level, when the original update data includes a new ticket price rule for that membership level, the ticket price rule that originally existed in the memory-mapped database will be determined as expired data, and it needs to be soft-deleted.
[0064] In this embodiment, soft deletion involves changing the expiration date in the valid time range of expired data in the memory-mapped database from its maximum value to the update cycle time corresponding to the original updated data. This is equivalent to changing the date to the time in the name of the current original updated data, which is the generation time of the current original updated data. Since the system checks whether the current time is within the valid time range of the data before it is used, and if it is not, the data can no longer be used, thus achieving soft deletion of the data.
[0065] Of course, this is just one possible way; data can also be marked as unavailable by means of identification or other methods.
[0066] S103. The current update node generates a current database change file set based on the original update data and pushes the current database change file set to the compute node cluster to trigger each compute node in the compute node cluster to update the local current database using the current database change file set.
[0067] As shown in step S102, the data in the memory-mapped database of the current updating node is consistent with the data in the memory-mapped database of the historical updating nodes. However, it should be noted that the validity period of the data in the memory-mapped database of the current updating node is much shorter than that of the data in the memory-mapped database of the historical updating nodes. For example, the validity period for the current updating node is typically one week, while that for the historical updating nodes is typically three years. Therefore, for the current updating node, the file containing the updated data that needs to be synchronized to the compute node cluster should be smaller, thereby effectively improving the efficiency of data synchronization and avoiding impact on the normally provided services.
[0068] Therefore, in this embodiment, after each update of the original update data to the database, only the current update node, based on the original update data and the data originally stored in its own memory-mapped database, determines the changed data and uses this data to generate a current database change file set. Then, the current database change file set is pushed to the compute node cluster to trigger each compute node in the cluster to update its local current database using the current database change file set.
[0069] Optionally, the current update node can push the current database change file set to the entire computing cluster at once via multicast.
[0070] S104. Determine whether the current period is a data maintenance period.
[0071] It should be noted that each compute node includes a current database for the current update node and a historical database for previous update nodes. To enable faster data synchronization and avoid impacting normal service provision, each update of the original data only updates the current database in the compute node based on the current database change file set generated by the current update node. The historical database also needs to be updated; however, considering the long validity period of data in the memory-mapped database of the historical update nodes, the update data volume is large, and the update efficiency is relatively slow. Therefore, updates are only performed during the data maintenance period to avoid impacting the service. Therefore, in this embodiment, it is determined in real time whether the current period is for data maintenance; if it is determined that the current period is for data maintenance, step S105 is executed.
[0072] S105. The historical update node generates a historical database change file set based on the original update data and pushes the historical database change file set to the compute node cluster to trigger each compute node in the compute node cluster to update its local historical database using the historical database change file set.
[0073] It should be noted that the process of step S105 is the same as that of step S103, only the object being executed and the database being updated are different, so it will not be described again here.
[0074] Optionally, in another embodiment of this application, when the data is currently in a data maintenance period, the historical update node can further perform data maintenance processing. For example... Figure 2 As shown in the embodiment of this application, a method for historical update nodes to maintain and process data includes:
[0075] S201. If the current period is a data maintenance period, the historical update node will perform hard deletion of the expired data that has been soft-deleted in its own memory-mapped database.
[0076] It should be noted that directly deleting data requires modifying a large amount of data, affecting data update efficiency. Therefore, during normal service provision, only expired data is soft-deleted. However, soft-deleted data still consumes resources. Therefore, in this embodiment, during the data maintenance period, soft-deleted expired data is hard-deleted, meaning the expired data is removed from the database and no longer exists there.
[0077] Therefore, the specific implementation method for hard deleting expired data that has been soft-deleted in its own memory-mapped database is as follows:
[0078] Remove expired data that has been soft-deleted from its own memory-mapped database.
[0079] S202. The historical update node generates a second set of historical database change files based on expired data that has undergone hard deletion.
[0080] Since the data in the memory-mapped database of the historical update node has changed after the expired data is hard deleted, it needs to be synchronized to the historical database of the compute node. Therefore, based on the expired data that has been hard deleted, a second historical database change file set is generated, and then step S203 is executed.
[0081] S203. The historical update node pushes the second historical database change file set to the computing node cluster to trigger each computing node in the computing node cluster to update its local historical database using the second historical database change file set.
[0082] Similarly, during the current data maintenance period, the current update node can also further maintain and process the data. For example... Figure 3 As shown in the embodiment of this application, a method for the current update node to maintain and process data includes:
[0083] S301. If the current data maintenance period is in effect, the current update node will perform hard deletion on the expired data that has been soft-deleted in its own memory-mapped database.
[0084] S302. The current update node generates a second current database change file set based on the expired data that has undergone hard deletion.
[0085] S303. The current updating node pushes the second current database change file set to the compute node cluster to trigger each compute node in the compute node cluster to update its local historical database using the second current database change file set.
[0086] It should be noted that the specific implementation methods and principles of steps S301 to S303 are consistent with those of steps S201 to S203. Therefore, the specific implementation methods of steps S301 to S303 can be referred to the implementation methods of steps S201 to S203, and will not be repeated here.
[0087] This application provides a data update method applied to update nodes, which include a current update node and historical update nodes. Both update nodes receive raw update data in real time and update their local memory-mapped databases upon receiving the raw update data. Then, expired data in the local memory-mapped database is soft-deleted. The soft-deleted data is in a state where it cannot be used as current computation data; that is, it is only that the expired data is no longer used as current computation data, rather than being directly deleted from the database, thus reducing the amount of data modified. Then, the current update node, with a relatively short data retention time, generates a current database change file set based on the raw update data and pushes the current database change file set to the compute node cluster. Since the amount of data in the database of each update node is relatively small, the size of the change files can be reduced, thereby improving data synchronization efficiency. Each compute node in the compute node cluster uses the current database change file set to update its local current database, thus ensuring the update of the current data. For historical update nodes, a historical database change file set is generated based on the original update data only when the data maintenance period is in progress. This historical database change file set is then pushed to the compute node cluster for updates, thus avoiding impacting data update efficiency and consequently affecting service provision when the service is being provided normally.
[0088] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, nodes, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0089] Although the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous.
[0090] Another embodiment of this application provides an update node, such as Figure 4 As shown, it includes:
[0091] Current update node 401 and historical update node 402.
[0092] The current update node 401 is used to receive raw update data in real time. When it receives raw update data, it updates the raw update data to the local memory-mapped database, performs soft deletion on expired data in the local memory-mapped database, generates a current database change file set based on the raw update data, and pushes the current database change file set to the compute node cluster to trigger each compute node in the compute node cluster to update the local current database using the current database change file set.
[0093] Data that has been soft-deleted is in a state where it cannot be used as current calculation data.
[0094] The historical update node 402 is used to receive raw update data in real time. When it receives raw update data, it updates the raw update data to the local memory-mapped database and performs soft deletion on expired data in the local memory-mapped database. When the data maintenance period is in progress, it generates a historical database change file set based on the raw update data and pushes the historical database change file set to the compute node cluster to trigger each compute node in the compute node cluster to update its local historical database using the historical database change file set.
[0095] Optionally, in another embodiment of this application, the historical update node is further used for:
[0096] During the current data maintenance period, expired data that has been soft-deleted in its own memory-mapped database will be hard-deleted.
[0097] A second set of historical database change files is generated based on expired data that has undergone hard deletion.
[0098] The second set of historical database change files is pushed to the compute node cluster to trigger each compute node in the cluster to update its local historical database using the second set of historical database change files.
[0099] Optionally, in another embodiment of this application, the current update node is further used for:
[0100] During the current data maintenance period, expired data that has been soft-deleted in its own memory-mapped database will be hard-deleted.
[0101] Based on expired data that has undergone hard deletion, generate a second set of current database change files;
[0102] The second current database change file set is pushed to the compute node cluster to trigger each compute node in the compute node cluster to update its local historical database using the second current database change file set.
[0103] It should be noted that the specific working process of each unit provided in the above embodiments of this application can be referred to the corresponding steps in the above method embodiments, and will not be repeated here.
[0104] Furthermore, the functions of each unit described in the above embodiments can be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that can be used include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SOCs), Complex Programmable Logic Devices (CPLDs), etc.
[0105] Another embodiment of this application provides an electronic device, such as... Figure 5 The diagram illustrates a structural schematic of an electronic device 500 suitable for implementing embodiments of the present disclosure. The electronic device in these embodiments may include, but is not limited to, electronic devices such as desktop computers, laptops, tablets, and vehicle terminals.
[0106] like Figure 5 As shown, the electronic device 500 includes one or more processing devices 501, such as a central processing unit, a graphics processing unit, etc., and a memory 502 on which one or more programs are stored. When one or more programs are executed by one or more processing devices 501, the one or more processing devices 501 implement the data update method as provided in any of the above embodiments.
[0107] Alternatively, the electronic device may also include other components, see also [reference needed]. Figure 5 The processing device 501, read-only memory 502, and random access memory 503 are interconnected via bus 504. Input / output (I / O) interface 505 is also connected to bus 504. Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays, speakers, vibrators, etc.; storage devices 508 including, for example, magnetic tape, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although... Figure 5 An electronic device 500 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.
[0108] Another embodiment of this application provides a computer storage medium for storing a program, which, when executed, implements the data update method as described in any of the preceding claims.
[0109] Computer storage media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.
[0110] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0111] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A data update method, characterized in that, Applied to update nodes, the update nodes include current update nodes and historical update nodes, the data update method includes: Each update node receives the original update data in real time. When the update node receives the original update data, it updates the original update data to its local memory-mapped database and performs soft deletion on the expired data in the local memory-mapped database; wherein, the data that has been soft deleted is in a state that cannot be used as current computing data; The current update node generates a current database change file set based on the original update data and pushes the current database change file set to the compute node cluster to trigger each compute node in the compute node cluster to update its local current database using the current database change file set; If the current period is a data maintenance period, the historical update node generates a historical database change file set based on the original update data and pushes the historical database change file set to the computing node cluster to trigger each computing node in the computing node cluster to update its local historical database using the historical database change file set.
2. The method according to claim 1, characterized in that, Also includes: If the current period is a data maintenance period, the historical update node will perform hard deletion of the expired data that was soft-deleted in its memory-mapped database; The historical update node generates a second historical database change file set based on the expired data that has undergone hard deletion. The historical update node pushes the second historical database change file set to the computing node cluster to trigger each computing node in the computing node cluster to update its local historical database using the second historical database change file set.
3. The method according to claim 1, characterized in that, Also includes: If the current node is in the data maintenance period, it will perform hard deletion of the expired data that was soft-deleted in its memory-mapped database; The current update node generates a second current database change file set based on the expired data that has undergone hard deletion. The current update node pushes the second current database change file set to the compute node cluster to trigger each compute node in the compute node cluster to update its local historical database using the second current database change file set.
4. The method according to claim 1, characterized in that, The step of soft-deleting expired data in the local memory-mapped database includes: Set the expiration time in the valid time range corresponding to the expired data in the memory-mapped database from the maximum value to the update cycle time corresponding to the original update data.
5. The method according to any one of claims 2 or 3, characterized in that, The step of performing hard deletion on the expired data that has been soft-deleted in its own memory-mapped database includes: The expired data that has been soft-deleted in its own memory-mapped database is deleted from the memory-mapped database.
6. An update node, characterized in that, include: Current update node and historical update nodes; The current update node is used to receive raw update data in real time. Upon receiving the raw update data, it updates the raw update data to the local memory-mapped database, performs soft deletion on expired data in the local memory-mapped database, generates a current database change file set based on the raw update data, and pushes the current database change file set to the compute node cluster to trigger each compute node in the compute node cluster to update its local current database using the current database change file set. The data that has been soft-deleted is in a state where it cannot be used as current compute data. The historical update node is used to receive the original update data in real time. When the original update data is received, the node updates the original update data to the local memory-mapped database and performs soft deletion on the expired data in the local memory-mapped database. When the data maintenance period is in progress, the node generates a historical database change file set based on the original update data and pushes the historical database change file set to the computing node cluster to trigger each computing node in the computing node cluster to update its local historical database using the historical database change file set.
7. The update node according to claim 6, characterized in that, The historical update node is also used for: During the current data maintenance period, the expired data that has been soft-deleted in its memory-mapped database will be hard-deleted. Based on the expired data that has undergone hard deletion, a second set of historical database change files is generated; The second set of historical database change files is pushed to the compute node cluster to trigger each compute node in the compute node cluster to update its local historical database using the second set of historical database change files.
8. The update node according to claim 6, characterized in that, The current update node is also used for: During the current data maintenance period, the expired data that has been soft-deleted in its memory-mapped database will be hard-deleted. Based on the expired data that has undergone hard deletion, a second current database change file set is generated; The second current database change file set is pushed to the compute node cluster to trigger each compute node in the compute node cluster to update its local historical database using the second current database change file set.
9. An electronic device, characterized in that, include: One or more processing devices; A memory on which one or more programs are stored; When the one or more programs are executed by the one or more processing devices, the one or more processing devices implement the data update method as described in any one of claims 1 to 5.
10. A computer storage medium, characterized in that, Used to store a program, which, when executed, is used to implement the data update method as described in any one of claims 1 to 5.