Freight rate data cutting method and system
By constructing a linked list of deleted and active data blocks, determining the movement boundary, and performing physical migration processing, the memory shortage problem caused by the increase in freight rate data volume was solved, and the response speed of the freight rate search system was improved.
Patent Information
- Application Number
- CN202510747336.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-05
- Publication Date
- 2025-10-17
AI Technical Summary
The existing fare search system is experiencing insufficient server memory due to the increase in fare data, which affects service response time and reduces user experience.
By constructing a linked list of deleted data blocks and a linked list of effective data blocks, the movement boundary is determined, and physical migration is performed based on the matching relationship. The original freight rate data is then trimmed, including traversing the linked list, splitting the data blocks, recording the matching relationship, and overwriting the deleted data blocks.
It enables rapid trimming of all freight rate data within the existing processing framework, reducing the data size so that it can be fully loaded into the production server's memory, thereby improving service response speed.
Smart Images
Figure CN120804367A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of domestic freight search, and particularly relates to a freight data pruning method and system. BACKGROUND
[0002] The freight search system is designed as a highly cohesive freight calculation system to realize real-time search of flight tickets, and realizes a memory data storage based on MMA (Memory Mapping). In the search process, all freight data is stored in the local memory of the server, so as to realize fast data acquisition and calculation. Therefore, the actual memory size on the server should be greater than the total sum of all freight data.
[0003] The airlines publish freight data through a front-end freight publishing system. The publishing system pushes the freight data to the freight search system every 5 minutes. The data preprocessing node in the search system receives and analyzes the freight message data, and then performs preprocessing operations such as writing, modifying and deleting on the generated memory data storage. Finally, the data storage is pushed to each search node in the system.
[0004] With the continuous increase of freight data published by airlines in recent years, the total amount of freight data is also increasing. However, the production server memory of the freight search system is limited, and the existing data cannot be loaded into the memory, which seriously affects the response time of the service and reduces the user experience.
[0005] CONTENT
[0006] The purpose of the present application is to provide a freight data pruning method to solve the problems raised in the background.
[0007] To achieve the above purpose, the present application provides the following technical solution: a freight data pruning method, comprising:
[0008] determining the deletion data and the effective data in the original freight data;
[0009] traversing the original freight data to construct a deletion data block linked list with continuous deletion data as deletion data blocks and to construct an effective data block linked list with continuous effective data as effective data blocks;
[0010] determining a moving boundary, traversing the deletion data block linked list in the forward direction to construct a new deletion data block linked list with the moving boundary as the end point, and traversing the effective data block linked list in the reverse direction to construct a new effective data block linked list;
[0011] determining the matching relationship between the deletion data blocks in the new deletion data block linked list and the effective data blocks in the new effective data block linked list;
[0012] Based on the matching relationship between the deletion data blocks in the new deletion data block chain table and the effective data blocks in the new effective data block chain table, the original freight data is physically migrated, and the original freight data after the physical migration is cropped.
[0013] Further, the deletion data in the original freight data is determined by:
[0014] obtaining a screening condition for the deletion data;
[0015] traversing the original freight data, and setting a deletion mark for the freight data meeting the screening condition to determine the deletion data in the original freight data.
[0016] Further, the moving boundary is determined by:
[0017] traversing the deletion data block chain table in the forward direction while traversing the effective data block chain table in the reverse direction, stopping the traversal when the cumulative size of the traversed effective data block is equal to the overall size of the deletion data block chain table, and taking the traversal termination address as the moving boundary.
[0018] Further, the matching relationship between the deletion data in the new deletion data block chain table and the effective data in the new effective data block chain table includes:
[0019] S401: traversing the new deletion data block chain table and the new effective data block chain table in parallel, removing the deletion data block and the effective data block of the same size from the chain table to obtain an updated deletion data block chain table and an updated effective data block chain table, and recording the matching relationship of the deletion data block and the effective data block of the same size;
[0020] S402: traversing the updated deletion data block chain table and the updated effective data block chain table in parallel, filling the effective data block by the deletion data block one by one, and including:
[0021] when the effective data block is larger than the deletion data block, the effective data block is divided into a sub-block A matching the deletion data block and a remaining sub-block B, the matching relationship of the sub-block A and the deletion data block is recorded, and the sub-block B is inserted into the updated effective data block chain table;
[0022] when the deletion data block is larger than the effective data block, the deletion data block is divided into a sub-block C matching the effective data block and a remaining sub-block D, the matching relationship of the sub-block C and the effective data block is recorded, and the sub-block D is inserted into the updated deletion data block chain table;
[0023] repeating S401 and S402 until the new deletion data block chain table is empty.
[0024] Further, the forward traversal of the deletion data block chain table to construct the new deletion data block chain table and the reverse traversal of the effective data block chain table to construct the new effective data block chain table include:
[0025] forward traversing the original link list of the deletion data blocks to the moving boundary, and sorting the deletion data blocks in descending order according to the size of the deletion data blocks to construct a new link list of the deletion data blocks;
[0026] backward traversing the original link list of the effective data blocks to the moving boundary, and sorting the effective data blocks in descending order according to the size of the effective data blocks to construct a new link list of the effective data blocks.
[0027] Further, based on the matching relationship between the deletion data blocks in the new link list of the deletion data blocks and the effective data blocks in the new link list of the effective data blocks, the physical migration processing of the original freight rate data includes:
[0028] obtaining the head and tail addresses of the matched effective data blocks and deletion data blocks, and performing physical migration on the effective data blocks so that the effective data blocks cover the deletion data blocks.
[0029] Further, the pruning of the original freight rate data after the physical migration processing includes:
[0030] adjusting the table tail pointer of the original freight rate data after the physical migration processing to delete invalid data.
[0031] Further, the method further includes:
[0032] modifying the index key value and the address of the physically migrated effective data blocks.
[0033] Further, the method further includes:
[0034] determining the deletion data in the original freight rate data and recording the index key value and the address of the deletion data at the same time; and
[0035] deleting the index key value and the address of the deletion data in the original freight rate data after the physical migration.
[0036] Another aspect of the present application discloses a freight rate data pruning system, comprising:
[0037] a data soft deletion module configured to determine the deletion data and the effective data in the original freight rate data;
[0038] a data analysis module configured to traverse the original freight rate data to construct a deletion data block link list by taking the continuous deletion data as deletion data blocks and to construct an effective data block link list by taking the continuous effective data as effective data blocks;
[0039] determining a moving boundary, traversing the deletion data block link list in the forward direction to construct a new link list of the deletion data blocks, and traversing the effective data block link list in the backward direction to construct a new link list of the effective data blocks;
[0040] Determine the matching relationship between the deletion data block in the new deletion data block chain table and the effective data block in the new effective data block chain table.
[0041] The data physical deletion module is configured to perform physical migration processing on the original freight rate data based on the matching relationship between the deletion data block in the new deletion data block chain table and the effective data block in the new effective data block chain table, and to perform pruning on the original freight rate data after the physical migration processing.
[0042] Compared with the prior art, the beneficial effects of the present application are:
[0043] The present application constructs a deletion data block chain table and an effective data block chain table based on original freight rate data, constructs a new deletion data block chain table and a new effective data block chain table based on a moving boundary, and determines the matching relationship between the deletion data block in the new deletion data block chain table and the effective data block in the new effective data block chain table and performs data pruning, that is, under the existing processing framework, the full freight rate data is quickly pruned in the data preprocessing link to reduce the size of the original freight rate data, and the pruned data can be loaded into the production server memory. BRIEF DESCRIPTION OF DRAWINGS
[0044] Figure 1 The method flowchart of the present application;
[0045] Figure 2 The deletion data determination flowchart;
[0046] Figure 3 The construction schematic diagram of the new deletion data block chain table and the new effective data block chain table;
[0047] Figure 4 The matching flowchart of the deletion data block and the effective data block;
[0048] Figure 5 The system framework diagram. DETAILED DESCRIPTION
[0049] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.
[0050] A freight rate data pruning method, with reference to Figure 1 , comprising:
[0051] S100: Determine the deletion data and the effective data in the original freight rate data;
[0052] S200: traversing the original fare data to continuously delete data as a deletion data block to build a deletion data block linked list and to continuously take effect data as an effective data block to build an effective data block linked list;
[0053] S300: determining a moving boundary, taking the moving boundary as a cut-off point, traversing the deletion data block linked list in a forward direction to build a new deletion data block linked list and traversing the effective data block linked list in a reverse direction to build a new effective data block linked list;
[0054] S400: determining a matching relationship between the deletion data block in the new deletion data block linked list and the effective data block in the new effective data block linked list;
[0055] S500: based on the matching relationship between the deletion data block in the new deletion data block linked list and the effective data block in the new effective data block linked list, performing physical migration processing on the original fare data and performing pruning on the original fare data after the physical migration processing.
[0056] In some embodiments, in step S100, determining the deletion data in the original fare data includes:
[0057] obtaining a screening condition of the deletion data;
[0058] traversing the original fare data, setting a deletion mark for the fare data meeting the screening condition to determine the deletion data in the original fare data.
[0059] Specifically, referring to Figure 2 , the screening condition of the deletion data can be obtained by reading a configuration file, for example, obtaining a dff.ini configuration file under an etc directory, reading an airline list in a cutCarriers configuration item to obtain a pruned airline list, and then traversing all records of a domestic fare original data table NfsxxFare, judging whether the airline field is in the pruned airline list, if so, setting recordTypeId of the record to a deletion state, the record corresponds to deletion data, and marking the deletion data, if not, continuing to traverse, and correspondingly, the record corresponds to effective data.
[0060] In some embodiments, in step S200, the deletion data is represented by a deletion data block, which is composed of a continuous interval of deletion data, for example, referring to Figure 3The deletion of one record in the deletion data block linked list includes the single record 1 data in the NFSXXFARE table and takes the single record 1 data as a single deletion data block in the deletion data block linked list. At this time, the head pointer of the deletion data block corresponding to the one record is the head pointer of the record 1 data, and the tail pointer of the one record is the tail pointer of the record 1 data. The deletion of two records in the deletion data block linked list includes the record 3 and record 4 data in the NFSXXFARE table, that is, the record 3 and record 4 together constitute a single deletion data block in the deletion data block linked list. At this time, the head pointer of the deletion data block corresponding to the two records is the head pointer of the record 3 data, and the tail pointer of the deletion data block corresponding to the two records is the tail pointer of the record 4 data.
[0061] In some embodiments, in S300, the determination of the moving boundary, that is, finding the boundary of the effective data filling the deletion data, specifically, the determination of the moving boundary includes:
[0062] The deletion data block linked list is traversed forwardly while the effective data block linked list is traversed reversely. When the cumulative size of the effective data block and the overall size of the deletion data block linked list are equal, the traversal is stopped, and the traversal termination address is taken as the moving boundary.
[0063] In some embodiments, in S300, the forward traversal of the deletion data block linked list to construct a new deletion data block linked list and the reverse traversal of the effective data block linked list to construct a new effective data block linked list include:
[0064] The original deletion data block linked list is traversed forwardly to the moving boundary, and is sorted in descending order according to the size of the deletion data block to construct a new deletion data block linked list;
[0065] The original effective data block linked list is traversed reversely to the moving boundary, and is sorted in descending order according to the size of the effective data block to construct a new effective data block linked list.
[0066] In some embodiments, in S400, the determination of the matching relationship between the deletion data in the new deletion data block linked list and the effective data in the new effective data block linked list is to find the effective data capable of filling and covering the deletion data. Specifically, referring to Figure 4 , the determination of the matching relationship between the deletion data in the new deletion data block linked list and the effective data in the new effective data block linked list includes:
[0067] S401: The new deletion data block linked list and the new effective data block linked list are traversed in parallel, the deletion data block and the effective data block with the same size are removed from the linked list to obtain an updated deletion data block linked list and an updated effective data block linked list, and the matching relationship of the deletion data block and the effective data block with the same size is recorded;
[0068] S402: traversing the new list of delete data blocks and the new list of effective data blocks in parallel, filling the effective data blocks into the delete data blocks one by one, and including:
[0069] When the effective data block is greater than the delete data block, the effective data block is split into a sub-block A matching the delete data block and a remaining sub-block B, the matching relationship between the sub-block A and the delete data block is recorded, and the sub-block B is inserted into the effective data block update list;
[0070] When the delete data block is greater than the effective data block, the delete data block is split into a sub-block C matching the effective data block and a remaining sub-block D, the matching relationship between the sub-block C and the effective data block is recorded, and the sub-block D is inserted into the delete data update list;
[0071] S401 and S402 are repeatedly executed until the new list of delete data blocks is empty.
[0072] In some embodiments, in step S500, the physical migration processing of the original freight data based on the matching relationship between the delete data blocks in the new list of delete data blocks and the effective data blocks in the new list of effective data blocks includes:
[0073] The start and end addresses of the matched effective data block and delete data block are obtained, and the effective data is physically migrated to cover the delete data block.
[0074] Specifically, the delete data blocks in the new list of delete data blocks form covered data, and the effective data blocks in the new list of effective data blocks form migrated data. Based on the record of the matching relationship, the start and end addresses of the matched delete data and effective data can be obtained by searching, i.e., the start and end addresses of the migrated data and the start and end addresses of the covered data are determined, the data to be deleted is covered using memcpy, and finally the effective data block covers the delete data block.
[0075] The original freight data after the physical migration processing is pruned, including:
[0076] The table tail pointer of the original freight data after the physical migration processing is adjusted to delete invalid data.
[0077] Specifically, after completing the physical migration processing of the original freight data, i.e., after completing the data covering, the data table tail part of the processed data is invalid data, including soft delete data and migrated data, and the data table tail pointer needs to be adjusted to point to the end of the current effective data, thereby deleting the invalid data to realize data pruning.
[0078] In some embodiments, the above method further comprises recording the index key value and address of the deleted data in the original fare data while determining the deleted data, that is, recording the index key value and address of the soft deleted data while setting the record as deleted, facilitating subsequent index deletion. When there are multiple indexes, different index key values are recorded respectively. The domestic fare data table has two indexes KEY_m_cxrCode_m_oriCode_m_destCode and KEY_m_cxrCode_m_oriCode_m_destCode_m_journeyType_m_fareCategory. Taking one record as an example, the index key values are [1E HAK BPL ] and [1E HAK BPL RT 0] respectively, and the address is 0x1f2082000080, which are recorded in two MAPs respectively.
[0079] After recording the index key value and address of the deleted data, the index key value and address of the deleted data in the physically migrated original fare data can be deleted based on the previous record after completing the physical migration of the original fare data. Specifically, according to the index key value and address of the deleted data recorded above, that is, [1E HAK BPL ] and [1E HAK BPL RT 0], and the address is 0x1f2082000080, the indexes KEY_m_cxrCode_m_oriCode_m_destCode and KEY_m_cxrCode_m_oriCode_m_destCode_m_journeyType_m_fareCategor of the domestic fare data table are deleted. Here, the indexes are all MAP containers (associative containers) of STL (standard database), the corresponding index key values are found respectively, and the addresses under the index key values are traversed to delete the values with the same recorded address.
[0080] In some embodiments, the above method further comprises modifying the index key value and address of the physically migrated effective data block.
[0081] Specifically, the index key value and address of the migrated data recorded above are modified. The key values of the indexes KEY_m_cxrCode_m_oriCode_m_destCode and KEY_m_cxrCode_m_oriCode_m_destCode_m_journeyType_m_fareCategor of the domestic fare data table are queried, and the address to be modified is found by traversal, which is modified so that the pointer points to the new moved data.
[0082] Another aspect of the present application discloses a fare data pruning system, which refers to Figure 5, is a relationship diagram of the cutting system and the publishing system, specifically, the cutting system includes:
[0083] a data soft deletion module configured to determine deletion data and effective data in the original freight rate data;
[0084] a data analysis module configured to traverse the original freight rate data, to continuously delete data as a deletion data block to construct a deletion data block linked list, and to continuously effective data as an effective data block to construct an effective data block linked list;
[0085] determine a moving boundary, take the moving boundary as a cutoff point, traverse the deletion data block linked list in a forward direction to construct a new deletion data block linked list, and traverse the effective data block linked list in a reverse direction to construct a new effective data block linked list;
[0086] determine a matching relationship between the deletion data block in the new deletion data block linked list and the effective data block in the new effective data block linked list;
[0087] a data physical deletion module configured to, based on the matching relationship between the deletion data block in the new deletion data block linked list and the effective data block in the new effective data block linked list, perform a physical migration processing on the original freight rate data, and cut the original freight rate data after the physical migration processing.
[0088] Although the embodiments of the present application have been shown and described, it can be understood by those skilled in the art that various changes, modifications, replacements and variations can be made to the embodiments without departing from the principles and spirits of the present application, the scope of the present application is defined by the appended claims and their equivalents.
Claims
1. A method for cutting freight rate data, characterized in that: include: Determine the deleted data and effective data in the original freight rate data; Traverse the original freight rate data, build a deleted data block linked list with the continuous deleted data as the deleted data block, and build a valid data block linked list with the continuous valid data as the valid data block; Determine the moving boundary, use the moving boundary as the cutoff point, traverse the deleted data block linked list forward to construct a new deleted data block linked list, and traverse the effective data block linked list backward to construct a new effective data block linked list; Determine the matching relationship between the deleted data block in the new linked list of deleted data blocks and the valid data block in the new linked list of valid data blocks; Based on the matching relationship between the deleted data blocks in the new linked list of deleted data blocks and the valid data blocks in the new linked list of valid data blocks, the original freight rate data is physically migrated and the original freight rate data after the physical migration is trimmed.
2. A method for cutting freight rate data according to claim 1, characterized in that: Determine the deleted data in the original freight rate data including: Get the filter conditions for deleting data; The original freight rate data is traversed, and deletion marks are set for the freight rate data that meet the screening conditions to determine the deleted data in the original freight rate data.
3. The method for cutting freight rate data according to claim 1, wherein: Determining the moving boundary involves: The deleted data block linked list is traversed forward while the valid data block linked list is traversed backward. When the cumulative size of the traversed valid data blocks is equal to the overall size of the deleted data block linked list, the traversal is stopped and the traversal end address is used as the moving boundary.
4. The method for cutting freight rate data according to claim 1, wherein: Determining the matching relationship between the deleted data in the new linked list of the deleted data block and the effective data in the new linked list of the effective data block includes: S401: Traverse the new linked list of deleted data blocks and the new linked list of effective data blocks in parallel, find deleted data blocks and effective data blocks of the same size, remove them from the linked lists to obtain an updated linked list of deleted data blocks and an updated linked list of effective data blocks, and record the matching relationship between deleted data blocks and effective data blocks of the same size; S402: Parallel traversal of the deleted data block update linked list and the effective data block update linked list, filling the deleted data blocks with the effective data blocks one by one, including: When the effective data block is larger than the deleted data block, split the effective data block into sub-block A that matches the deleted data block and the remaining sub-block B. Record the matching relationship between sub-block A and the deleted data block and insert sub-block B into the effective data block update linked list. When the deleted data block is larger than the effective data block, the deleted data block is split into a sub-block C that matches the effective data block and the remaining sub-block D. The matching relationship between sub-block C and the effective data block is recorded, and sub-block D is inserted into the deleted data update linked list. S401 and S402 are repeatedly executed until the new linked list of deleted data blocks is empty.
5. The method for cutting freight rate data according to claim 1, wherein: Forward traversing the deleted data block linked list to construct a new linked list of deleted data blocks and reverse traversing the effective data block linked list to construct a new linked list of effective data blocks include: Traverse the original linked list of deleted data blocks forward to the moving boundary, and sort the deleted data blocks in descending order to build a new linked list of deleted data blocks; Traverse the original linked list of effective data blocks in reverse order to the moving boundary, and sort them in descending order according to the size of the effective data blocks to construct a new linked list of effective data blocks.
6. A method for cutting freight rate data according to claim 1, characterized in that: Based on the matching relationship between the deleted data blocks in the new linked list of the deleted data blocks and the valid data blocks in the new linked list of the valid data blocks, the physical migration of the original freight rate data includes: The starting and ending addresses of the matching effective data blocks and deleted data blocks are obtained, and the effective data blocks are physically migrated so that the effective data blocks cover the deleted data blocks.
7. A method for cutting freight rate data according to claim 6, characterized in that: The raw freight rate data after physical migration is trimmed, including: Adjust the tail pointer of the original freight rate data after physical migration to delete invalid data.
8. A method for cutting freight rate data according to claim 6, characterized in that: The method further includes: modifying the index key value and address of the physically migrated valid data block.
9. The method for cutting freight rate data according to claim 1, wherein: The method further comprises: Determining the deleted data in the original freight rate data and recording the index key value and address of the deleted data; and Delete the index key values and addresses of the deleted data in the original freight rate data after physical migration.
10. A freight rate data cutting system, characterized in that: include: A data soft deletion module configured to determine deleted data and valid data in the original freight rate data; A data analysis module is configured to traverse the original freight rate data, construct a deleted data block linked list using consecutive deleted data as deleted data blocks, and construct a valid data block linked list using consecutive valid data as valid data blocks; Determine the moving boundary, use the moving boundary as the cutoff point, traverse the deleted data block linked list forward to construct a new deleted data block linked list, and traverse the effective data block linked list backward to construct a new effective data block linked list; Determine the matching relationship between the deleted data block in the new linked list of deleted data blocks and the valid data block in the new linked list of valid data blocks; The data physical deletion module is configured to perform physical migration processing on the original freight rate data based on the matching relationship between the deleted data block in the new linked list of the deleted data block and the effective data block in the new linked list of the effective data block, and to trim the original freight rate data after the physical migration processing.