Distributed freight rate data consistency guarantee system and method
By adopting private memory mapping and data state node management in civil aviation freight system, the read and write separation of memory data and delayed loading are achieved, the system performance jitter and consistency problems caused by frequent data updates are solved, and the system performance and response are improved speed.
Patent Information
- Application Number
- CN202411388498.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-30
- Publication Date
- 2025-01-03
AI Technical Summary
When the data is frequently updated, the existing civil aviation freight system needs to be constantly cold-started new service processes, resulting in slowing system response time and causing performance jitter problems, making it difficult to ensure data consistency.
Adopt private memory mapping mechanism to share memory file sets through data update processes and service processes, realize data read and write separation in memory, avoid rewriting the data in use, introduce data state nodes to manage data state, and use request drivers to delay load data to reduce cold start.
It effectively solves the problem of system performance jitter, ensures the consistency of data during transactions, reduces page-missing in the operating system, and improves system performance.
Smart Images
Figure CN119248804A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of civil aviation information technology, and in particular to a distributed freight rate data consistency assurance system and method. Background Art
[0002] The civil aviation freight rate system is a typical data-intensive and computing-intensive system. In terms of freight rate application standards, the freight rate system needs to calculate hundreds of results that can be perceived by users in real time from a billion-level solution space. In a simple transaction, data needs to be accessed frequently, up to 100,000 times. In terms of freight rate data standards, freight rate data covers 5 major categories, 250 subcategories and thousands of characteristic attributes, and there are intricate associations, time sequences and dependencies between data. Each access to data requires multiple indexing and locating of data in a complex mesh data structure.
[0003] In order to optimize system performance, reduce data access times, and improve data indexing efficiency, the freight rate system uses a self-developed in-memory database that supports lock-free direct shared memory access. At the same time, the source data is preprocessed to solidify complex data relationships in the in-memory database. For the freight rate business, data consistency is very important, and different results are not allowed to be returned at the same time and for the same request.
[0004] In order to ensure the consistency of the above complex data, in the prior art, such as Figure 1 As shown in the figure, the freight rate system currently adopts the A and B block idea to make applications and data redundant. The computing engine in application A accesses data A, and the computing engine in application B accesses data B. The computing engines of application A and application B have the same functions, but they access different data. At a certain moment, only one of application A or application B provides services, and the other stops providing services and waits for data updates. The load balancing component decides who will provide services and starts the corresponding service process. Assuming that application A is currently providing services, when there is data update, application A continues to provide services and keeps data A unchanged; at this time, all data B is updated. When data B is updated, the load balancer is notified. The load balancer confirms that all data B has been updated, starts the service process of application B, and switches to application B uniformly. Application B starts to provide services, and finally terminates the service process of application A and stops the service of application A.
[0005] Although this solution can ensure data consistency, when data is updated frequently, it is necessary to constantly cold-start new service processes and load new data. When the new service process accesses the memory database to process transactions, there will be a large number of page faults, resulting in slower system response time and causing system performance jitter problems. Summary of the invention
[0006] To solve the above problems, the present disclosure provides a distributed freight rate data consistency assurance system and method, which adopts a private memory mapping mechanism to solve the problem of system performance jitter when data is frequently updated and ensures the consistency of data during the transaction process.
[0007] This application is implemented through the following technical solutions:
[0008] A distributed freight rate data consistency assurance system, comprising an application node and a data status node;
[0009] The application node includes service processes, physical memory, memory file sets, data update processes, and source data files;
[0010] The data status node includes the data status management process, application node data batch, data update batch and new data loading identification;
[0011] The source data file includes the updated content of the freight rate data, the data batch of this update and the data batch of the previous update;
[0012] The data update process is used to pre-process the source data file and generate a memory file;
[0013] Memory file set and physical memory, used to store memory files;
[0014] New data loading identification, using identification information to identify the loading status of the service process;
[0015] The service process is used to process transaction requests and load memory files according to identification information;
[0016] Application node data batches, used to save the latest data batches loaded by the service process in the corresponding application node;
[0017] Data update batches are used to save the latest updated data batches in the data update process of the corresponding application node;
[0018] The data management process is used to modify the identification information of the new data loading identification according to the content of the application node data batch and the data update batch.
[0019] Furthermore, the service process and the data update process share a memory file set and physical memory.
[0020] Furthermore, the data update process establishes a private mapping with the memory file set; the service process establishes a shared mapping with the memory file set.
[0021] A distributed freight rate data consistency assurance method applicable to the above system includes:
[0022] The data update process obtains the source data file, preprocesses the source data file, and generates a memory file;
[0023] The data update process updates the memory file to the memory file set according to the identification information of the new data loading identification; after the update is completed, the data update batch is modified;
[0024] The service process loads the memory file according to the identification information of the new data loading identifier, and after the loading is completed, modifies the application node data batch;
[0025] The data status management process modifies the identification information of the new data loading identifier according to the modification of the data update batch and the application node data batch to maintain the consistency of the freight rate data.
[0026] Furthermore, the data update process obtains the source data file, preprocesses the source data file, and generates a memory file, including:
[0027] The data update process periodically scans whether there are new source data files. If there are new source data files, the source data files are obtained;
[0028] Determine whether the last updated data batch in the source data file is consistent with the latest data update batch; if they are consistent, preprocess the source data file and generate a memory file; if they are inconsistent, scan and obtain the last updated source data file until the last updated data batch in the source data file is consistent with the latest data update batch, then preprocess the freight rate data in the source data file batch by batch and generate a memory file.
[0029] Furthermore, the source data file is preprocessed and a memory file is generated, including:
[0030] The address space and physical memory of the data update process store the contents of the latest data batch memory file;
[0031] The data update process compares the source data file with the memory file of the latest data batch, extracts the changed files, allocates new memory space in the physical memory, writes the changed files into the new memory space; then combines the changed files with the memory file of the latest data batch to generate a new file;
[0032] The data update process creates a memory file and writes a new file into the memory file.
[0033] Further, the data update process updates the memory file to the memory file set according to the identification information of the new data loading identification; after the update is completed, the data update batch is modified, including:
[0034] The data update process obtains the identification information of the new data loading identification. If the identification information is not loaded, and the application node data batch and the data update batch are in a consistent state, the memory file is updated to the memory file set; otherwise, the identification information is obtained again after waiting for a preset time interval, and the memory file is updated to the memory file set when the identification information is not loaded and the application node data batch and the data update batch are in a consistent state;
[0035] When updating, all files in the memory file set are mapped to the address space of the data update process;
[0036] After the update is completed, modify the data update batch corresponding to this application node according to the data batch of the source data file.
[0037] Further, the service process loads the memory file according to the identification information of the new data loading identifier, and after the loading is completed, the application node data batch is modified, including:
[0038] When the service process processes a transaction, it obtains the identification information of the new data loading identification; if the identification information is to start loading, it starts loading the memory file; if the identification information is not loaded, it waits for a preset time interval and obtains the identification information again, until the identification information is to start loading, it starts loading the memory file;
[0039] After the service process is loaded, the loading results of other service processes in the same application node are obtained; if all loading is completed, the application node data batch corresponding to this application node is modified.
[0040] Furthermore, loading the memory file includes: the loading process traverses the memory files in its address space, releases the files of the same type as the changed file in the address space, and maps the changed file into its address space.
[0041] Furthermore, the data status management process modifies the identification information of the new data loading identification according to the modification of the data update batch and the application node data batch, including:
[0042] When receiving a modification notification of a data update batch, check whether all data update batches are consistent; if not, ignore this modification notification; if consistent, modify the identification information to start loading; at this time, the service process can load the memory file, and the data update process no longer updates the memory file set;
[0043] When receiving modification notification of application node data batch, check whether the application node data batch is consistent; if not, ignore this modification notification; if consistent, but the application node data batch and the data update batch are inconsistent, ignore this modification notification; if consistent, and the application node data batch and the data update batch are consistent, modify the identification information to not loaded; at this time, the service process no longer loads the memory file, and the data update process can update the memory file set.
[0044] Compared with the prior art, the present invention has the following advantages:
[0045] 1. By utilizing the private memory mapping mechanism of the operating system, the read and write separation of data in the memory is realized, which avoids rewriting the data in use and ensures the consistency of data during the transaction process.
[0046] 2. Introduce data status nodes to manage data status, realize the read and write separation of file data, avoid rewriting occupied files, and ensure the consistency of data in storage.
[0047] 3. By adopting the request-driven delayed data loading method, the service process does not need to be cold-started when loading data, so that the service process can cache the non-invalid data, reducing the page faults of the operating system, and can effectively solve the problem of system performance jitter.
[0048] Other features and advantages of the present disclosure will be described in the following description, and partly become apparent from the description, or be understood by implementing the present disclosure. The purpose and other advantages of the present disclosure can be realized and obtained by the structures pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0050] Figure 1 A data flow diagram of the prior art in the background technology is shown;
[0051] Figure 2 A schematic diagram of a distributed freight rate data consistency assurance system is shown;
[0052] Figure 3 A schematic flow chart of a method for ensuring consistency of distributed freight rate data according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0053] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.
[0054] Terminology explanation:
[0055] Application Node: In software architecture and distributed systems, application nodes usually refer to entities that run applications or services, which can be physical servers, virtual machines, or containers. Application nodes are responsible for executing the logic of the application and may interact with other nodes to complete specific tasks.
[0056] Data State Node: In a distributed system, data state nodes can refer to nodes that are dedicated to storing and managing data states. These nodes are responsible for processing data read and write operations and ensuring data consistency and integrity.
[0057] Private Mapping: It is a memory mapping mechanism in Linux that is used to create a memory area that is associated with the original file or other memory area, but modifications to this area will not be reflected back to the original file. In other words, private mapping creates a copy, and any modifications to the mapped area will not affect other processes or the original file.
[0058] Shared Mapping: It is a memory mapping mechanism in Linux that allows multiple processes to share the same memory area. When a process modifies this shared area, all other processes that have mapped the same area will see these modifications. This mechanism is mainly used for inter-process communication (IPC) and shared data.
[0059] Physical Memory: Also known as Main Memory, it refers to the actual hardware components in a computer system that are used to temporarily store data and program instructions. Physical memory is usually volatile, which means that the information stored in it is cleared when the power is turned off. The physical memory of a computer system is usually made up of Random Access Memory (RAM), which is a storage medium that can read and write data quickly.
[0060] Address Space: refers to the range of all memory addresses that a process can access. Each process has its own independent address space, even if multiple processes may share some memory areas.
[0061] like Figure 1 As shown, a distributed freight rate data consistency assurance system according to an embodiment of the present disclosure includes an application node and a data status node.
[0062] The application node includes a service process, a physical memory, a memory file set, a data update process and the source data file;
[0063] The data status node includes a data status management process, an application node data batch, a data update batch and a new data loading identifier.
[0064] In this embodiment, the data status node includes multiple application node data batches and data update batches, the application node data batches and the data update batches correspond one to one, and each pair of the application node data batches and the data update batches corresponds to one application node.
[0065] In this embodiment, the content of the memory file is the file corresponding to the source data file preprocessed by the data update process. Each data batch has a unique batch number, which is used to distinguish memory files of different data batches.
[0066] In this embodiment, multiple service processes and one data update process are set in the same application node. The service process and the data update process share the memory file set and the physical memory; the service process only reads but does not write data, and the data update process can read and write data; the data update process establishes a private mapping with the memory file set; and multiple service processes establish a shared mapping with the memory file set.
[0067] The source data file includes updated content of freight rate data, data batch of the current source data file and data batch of the previous source data file. The freight rate data includes price data, rule data and route data.
[0068] The data update process is used to pre-process the source data file, determine the data batch of the source data file, and generate a memory file;
[0069] The new data loading identifier is used to generate identification information of whether the service process is to be loaded;
[0070] The memory file set and the physical memory are used to store memory files of each data batch;
[0071] The service process is used to process the transaction request and load the memory file according to the identification information;
[0072] The application node data batch is used to store the data batch of the memory file that is most recently loaded by the service process in the application node;
[0073] The data update batch is used to store the data batch corresponding to the latest update of the source data file by the data update process in the application node;
[0074] The data management process is used to rewrite the identification information of the new data loading identifier according to the content of the application node data batch and the data update batch.
[0075] Based on the above system, the embodiment of the present disclosure further provides a distributed freight rate data consistency guarantee method corresponding to the above system, including:
[0076] S1, the data update process obtains the source data file, pre-processes the source data file, and generates a memory file. The main process is as follows:
[0077] S1.1, the data update process periodically scans whether there is a new source data file, and if there is a new source data file, obtains the source data file;
[0078] S1.2, determine whether the data batch last updated in the source data file is consistent with the latest data update batch; if they are consistent, preprocess the freight rate data in the source data file and generate the memory file; if they are inconsistent, scan and obtain the source data file last updated until the data batch last updated in the source data file is consistent with the latest data update batch, and then preprocess the freight rate data in the source data file batch by batch and generate the memory file.
[0079] In this embodiment, the freight rate data in the source data files are preprocessed batch by batch, and the main process is as follows: the source data files are preprocessed according to the data batches, and after processing one source data file, the source data files of the next data batch are processed.
[0080] In this embodiment, the source data file is preprocessed to generate a memory file, and the main process is as follows:
[0081] The address space of the data update process and the physical memory store the contents of the latest data batch memory file;
[0082] The data update process compares the source data file with the memory file of the latest data batch, extracts the changed file, allocates new memory space in the physical memory, writes the changed file into the new memory space; and then combines the changed file with the memory file of the latest data batch to generate a new file;
[0083] The data update process creates a memory file and writes the new file into the memory file.
[0084] S2, the data update process updates the memory file to the memory file set according to the identification information; after the update is completed, the data update batch is modified. The main process is as follows:
[0085] S2.1, the data update process obtains the identification information of the new data loading identifier. If the identification information is not loaded, and the application node data batch and the data update batch are in a consistent state, the memory file is updated to the memory file set; otherwise, the identification information is obtained again after waiting for a preset time interval, until the identification information is not loaded, and the application node data batch and the data update batch are in a consistent state, the memory file is updated to the memory file set.
[0086] In this embodiment, the data update process and the memory file set are privately mapped.
[0087] S2.2, when updating, mapping all files in the memory file set to the address space of the data update process;
[0088] S2.3, after the update is completed, modify the data update batch corresponding to the application node according to the data batch of the source data file.
[0089] S3, the service process loads the memory file according to the identification information of the new data loading identifier; after the loading is completed, the application node data batch is modified. The main process is as follows:
[0090] S3.1, when the service process processes a transaction, it obtains the identification information of the new data loading identifier; if the identification information is to start loading, it starts loading the memory file; if the identification information is not loaded, it waits for a preset time interval and obtains the identification information again, until the identification information is to start loading, it starts loading the memory file.
[0091] When the service process loads the memory file, it traverses the memory files in its address space, releases the files of the same type as the changed file in the address space, and maps the changed file to its address space.
[0092] All the service processes in the same application node map the files in the new memory space into their address spaces through shared mapping.
[0093] S3.2, after the service process is loaded, the loading results of other service processes in the same application node are obtained; if all loading is completed, the application node data batch corresponding to the current application node is modified.
[0094] When multiple service processes in the same application node modify the application node data batch, a mutually exclusive method is used to ensure that only one service process modifies the application node data batch at the same time.
[0095] S4. The data status management process modifies the identification information of the new data loading identification according to the modification of the data update batch and the application node data batch to maintain the consistency of the freight rate data. The main process is as follows:
[0096] When receiving the modification notification of the data update batch, check whether all the data update batches are consistent; if not, ignore this modification notification; if consistent, modify the identification information to start loading; at this time, the service process can load the memory file, and the data update process no longer updates the memory file set.
[0097] When receiving the modification notification of the application node data batch, check whether the application node data batch is consistent; if not, ignore this modification notification; if they are consistent, but the application node data batch and the data update batch are inconsistent, ignore this modification notification; if they are consistent, and the application node data batch and the data update batch are consistent, modify the identification information to not loaded; at this time, the service process no longer loads the memory file, and the data update process can update the memory file set.
[0098] Example
[0099] The calculation system for updating freight rate data is based on the Linux operating system and is developed in C++. The memory mapping uses the memory map technology of Linux, which can map disk files to the address space of the process. The process can read and write file contents by operating C++ objects.
[0100] Assume that the latest data batch of the memory file in the current memory file set is 20240101:120000, where 20240101 represents January 1, 2024, and 120000 represents 12:00:00. All application node data batches and all data update batches in the data status node are 20240101:120000, the tag information of the new data loading identifier is false, and the service process is in the state of no data loading.
[0101] The data update process obtains a new source data file. The data batch updated this time in the source data file is 20240101:130000, and the data batch updated last time is 20240101:120000. The data batch updated last time in the source data file is consistent with the data update batch in the data status node, and the data content in the source data file begins to be preprocessed.
[0102] The identification information of the new data loading identification in the data status node is false, the service process is in the state of not loading data, and the application node data batch and data update batch are both 20240101:120000 at this time, and the memory file set is updated. At this time, the service process has mapped the memory files in the memory file set to its address space in a shared, read-only mode. The data update process maps the memory files in the memory file set into its address space in a private, read-write mode, and then reads the source data file for this update to preprocess the freight rate data.
[0103] The preprocessing process of freight rate data is as follows: if the source data file modifies the memory file in the memory file set, such as modifying the price data. Since the data update process and the memory file set are privately mapped, the Linux system will allocate a new memory space in the physical memory, and write the modified file in the source data file, that is, the price data, into the new memory space. The file in the new memory space is combined with the old file in the physical memory to generate a new memory file; the data update process then updates the new memory file to the memory file set.
[0104] After updating the memory file set, the data update process modifies the data update batch corresponding to the application node in the data status node, changes it to 20240101:130000, and then notifies the data status management process to modify the identification information of the new data loading identifier. The data status management process obtains all data update batch data. If all data update batches are 20240101:130000, the identification information of the new data loading identifier is changed to true, and the service process can load new data.
[0105] When the service process processes a transaction, it checks the identification information of the new data loading identification. If the identification information is true, the service process maps the file in the new memory space to its address space in a shared, read-only mode. If the service process has already mapped a file of the same type, it releases the memory file in the 20240101:120000 batch, that is, releases the price data in the memory file in the 20240101:120000 batch, and remaps the file in the new memory space in the 20240101:130000 batch.
[0106] After the service process completes the memory file mapping, it inquires about the data batches of other service processes in the same application node. If the data batches are all 20240101:130000, the application node data batch corresponding to the application node in the data status node is modified to 20240101:130000, and the data status management process is notified to modify the identification information of the new data loading identification. The data status management process obtains all data update batch data. If all application node data batches are 20240101:130000 and all data update batches are 20240101:130000, the new data loading identification is changed to false, and the service process no longer loads new data.
[0107] Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure.
Claims
1. A distributed freight rate data consistency assurance system, characterized in that: The system includes an application node and a data status node; The application node includes a service process, a physical memory, a memory file set, a data update process and a source data file; The data status node includes a data status management process, an application node data batch, a data update batch, and a new data loading identifier; The source data file includes the updated content of the freight rate data, the data batch of this update and the data batch of the previous update; The data update process is used to pre-process the source data file and generate a memory file; The memory file set and the physical memory are used to store the memory file; The new data loading identifier uses the identification information to identify the loading status of the service process; The service process is used to process the transaction request and load the memory file according to the identification information; The application node data batch is used to store the latest data batch loaded by the service process in the corresponding application node; The data update batch is used to store the latest updated data batch of the data update process corresponding to the application node; The data management process is used to modify the identification information of the new data loading identifier according to the content of the application node data batch and the data update batch.
2. The system according to claim 1, characterized in that The service process and the data update process share the memory file set and the physical memory.
3. The system according to claim 1, characterized in that The data update process establishes a private mapping with the memory file set; the service process establishes a shared mapping with the memory file set.
4. A method for ensuring the consistency of distributed freight rate data applicable to the system of claim 1, characterized in that: The method comprises, The data update process obtains a source data file, preprocesses the source data file, and generates a memory file; The data updating process updates the memory file to the memory file set according to the identification information of the new data loading identification; After the update is completed, modify the data update batch; The service process loads the memory file according to the identification information of the new data loading identifier, and after the loading is completed, modifies the application node data batch; The data status management process modifies the identification information of the new data loading identifier according to the modification of the data update batch and the application node data batch, so as to maintain the consistency of the freight rate data.
5. The method according to claim 4, characterized in that The data update process obtains a source data file, preprocesses the source data file, and generates a memory file, including: The data update process periodically scans whether there is a new source data file, and if there is a new source data file, obtains the source data file; Determine whether the last updated data batch in the source data file is consistent with the latest data update batch; if they are consistent, preprocess the source data file and generate the memory file; if they are inconsistent, scan and obtain the last updated source data file until the last updated data batch in the source data file is consistent with the latest data update batch, then preprocess the freight rate data in the source data file batch by batch, and generate a memory file.
6. The method according to claim 5, characterized in that Preprocess the source data file and generate a memory file, including: The address space of the data update process and the physical memory store the contents of the latest data batch memory file; The data update process compares the source data file with the memory file of the latest data batch, extracts the changed file, allocates new memory space in the physical memory, writes the changed file into the new memory space; and then combines the changed file with the memory file of the latest data batch to generate a new file; The data update process creates a memory file and writes the new file into the memory file.
7. The method according to claim 4, characterized in that The data updating process updates the memory file to the memory file set according to the identification information of the new data loading identification; After the update is completed, modify the data update batch, including: The data update process obtains identification information of the new data loading identification, and if the identification information is not loaded, and the application node data batch and the data update batch are in a consistent state, the memory file is updated to the memory file set; otherwise, the identification information is obtained again after a preset time interval, and the memory file is updated to the memory file set when the identification information is not loaded and the application node data batch and the data update batch are in a consistent state; During the update, all files in the memory file set are mapped to the address space of the data update process; After the update is completed, the data update batch corresponding to the application node is modified according to the data batch of the source data file.
8. The method according to claim 6, characterized in that The service process loads the memory file according to the identification information of the new data loading identifier, and after the loading is completed, modifies the application node data batch, including: When the service process processes a transaction, it obtains the identification information of the new data loading identification; if the identification information is to start loading, it starts loading the memory file; if the identification information is not loaded, it waits for a preset time interval and obtains the identification information again, until the identification information is to start loading, it starts loading the memory file; After the service process is loaded, the loading results of other service processes in the same application node are obtained; if all loadings are completed, the application node data batch corresponding to the current application node is modified.
9. The method according to claim 8, characterized in that The loading of the memory file includes: the loading process traverses the memory files in its address space, releases the files of the same type as the changed file in the address space, and maps the changed file into its address space.
10. The method according to claim 4, characterized in that The data state management process modifies the identification information of the new data loading identification according to the modification of the data update batch and the application node data batch, including: When receiving the modification notification of the data update batch, check whether all the data update batches are consistent; if they are inconsistent, ignore this modification notification; if they are consistent, modify the identification information to start loading; at this time, the service process can load the memory file, and the data update process no longer updates the memory file set; When receiving the modification notification of the application node data batch, check whether the application node data batch is consistent; if not, ignore this modification notification; if they are consistent, but the application node data batch and the data update batch are inconsistent, ignore this modification notification; if they are consistent, and the application node data batch and the data update batch are consistent, modify the identification information to not loaded; at this time, the service process no longer loads the memory file, and the data update process can update the memory file set.