Freight rate calculation cloud service system, freight rate calculation method and electronic equipment
By introducing a cloud service system for freight calculation in the freight calculation system, the distributed concurrent preprocessing and data preprocessing are adopted to deploy the same machine as the engine, the existing system has high hardware resources requirements and slow data preprocessing speed. The problem of this achieves more efficient data processing and lower operation and maintenance costs.
Patent Information
- Application Number
- CN202411523416.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-29
- Publication Date
- 2025-02-07
AI Technical Summary
The existing freight rate computing system has high requirements for hardware resources, slow data preprocessing speed, poor real-time update effect, high failure rate for transmitting large amounts of data, and cannot solve the data growth problem through horizontal scaling.
A cloud service system for freight calculation adopts the architecture of the general control module, data module and interface module, reduces processing delay through distributed concurrent preprocessing, reduces hardware resource requirements, and deploys it in the same machine as the engine through data preprocessing Reduce data transmission requirements.
It has achieved the reduction of processing delay, reduced hardware resource requirements, improved data loading performance, reduced operation and maintenance costs, and achieved elastic capacity expansion through cloud deployment.
Smart Images
Figure CN119398856A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical field of civil aviation data processing, and in particular to a freight rate calculation cloud service system, a freight rate calculation method and an electronic device. Background Art
[0002] The current domestic freight calculation software adopts the ALL IN ONE system architecture model, which is a high-performance architecture solution designed for freight calculation-intensive, data-intensive, and IO-intensive scenarios. It adopts a "high cohesion and low coupling" model that mixes data and programs "All-in-one", based on a shared memory-based database MMDB and cache, combined with large-capacity SSD (6-20TB), large-capacity memory (2-12TB) and a private 10G multicast network, and is widely used in domestic and international aviation freight software.
[0003] The existing open high-performance mass computing system adopts a single-machine terabyte-level full-data mode, which places high demands on hardware resources. As the amount of system data increases, it can no longer meet the development of such business. Figure 1 This is a schematic diagram of the interaction between the domestic freight rate calculation system (DPRIC) and users. DPRIC includes important business scenarios such as freight rate calculation, query, refund and change, and has very high requirements for the performance, security, scalability, maintainability and other aspects of the system. In particular, the security and efficiency of data processing are of great importance. The current freight rate calculation system adopts a multicast mode. After external pre-processing, the data is sent to each data node in a multicast manner, and the single data transmission volume is up to several terabits. The transmission of huge amounts of data requires an independent network, that is, an independent network card. This processing mode makes the hardware requirements of this system very high, the speed of data pre-processing is relatively slow, the effect of real-time update is relatively poor, and the failure rate of transmitting large amounts of data is high. The cost of data compression and decompression is high, and the data growth problem cannot be solved by horizontal expansion. Summary of the invention
[0004] The present disclosure aims to solve at least one of the technical problems in the above-mentioned technologies to a certain extent, and proposes a cloud-based service system for freight rate calculation, including:
[0005] Master control module, data module and interface module; among them,
[0006] A data mapping table is stored in the master control module;
[0007] The data module is configured as follows:
[0008] Responding to the freight rate calculation request through the interface module; and
[0009] After receiving the data change notification sent by the master control module, new data is loaded and the master control module is notified, so that the master control module updates the data mapping table.
[0010] Furthermore, the data module includes: a data management module, a data update module, a freight rate calculation module and a database interface module; wherein,
[0011] The data update module is configured as follows:
[0012] In response to the data change notification, generate an interface data file;
[0013] The general control module is notified to control the freight rate calculation module to call the interface data file and access the database interface module through the data management module.
[0014] Furthermore, the freight rate calculation cloud service system also includes: a data preprocessing module; the data preprocessing module, the data module and the interface module are distributedly deployed in a number of computing nodes.
[0015] Furthermore, the data preprocessing module and the data module are deployed on the same machine.
[0016] Furthermore, the preprocessing module is configured to collect, organize, publish and preprocess the source data; wherein,
[0017] The source data includes: freight rate data and flight data;
[0018] The data preprocessing includes: cleaning unrelated data, storing unrelated data and inserting new related data.
[0019] Furthermore, at least two computing services are deployed in the same computing node, and each computing service includes: at least one data preprocessing module, at least one data module and at least one interface module.
[0020] Furthermore, data is shared between different computing services in the same computing node.
[0021] Furthermore, the data module is also configured as follows:
[0022] After receiving the data version switching notification sent by the master control module, the subsequent database interface module access request or freight rate calculation module access request is redirected to other database interface modules or other freight rate calculation modules in the current computing node.
[0023] The present disclosure also proposes a freight rate calculation method based on the above-mentioned freight rate calculation cloud service system, including: using the data module to perform the following method:
[0024] Responding to the freight rate calculation request through the interface module; and
[0025] After receiving the data change notification sent by the master control module, new data is loaded and the master control module is notified, so that the master control module updates the data mapping table.
[0026] The present disclosure also proposes an electronic device, including a memory and a processor, wherein the memory stores a computer program or instruction, and when the computer program or instruction is executed by the processor, it is at least used to implement the above-mentioned freight rate calculation method.
[0027] Compared with the prior art, the beneficial effects of the present disclosure are as follows: the present disclosure transforms the existing ALL IN ONE multicast data processing mode into a cloud deployment. The pre-processed data with a single transmission volume of up to hundreds of G or even several TB is transformed into distributed concurrent pre-processing, which not only reduces the processing delay, but also greatly reduces the requirements for hardware resources such as memory and SSD. In addition, the pre-processing and the engine are deployed on the same machine, and there is no need to transmit large-volume data files after pre-processing, and no special dual network environment is required, and elastic expansion is more convenient. By realizing the cloudification of high-performance massive computing systems, on the one hand, such businesses can fully enjoy the technical advantages of cloudification, and on the other hand, the basic technical architecture of Air China PSS business can be unified to reduce operation and maintenance costs.
[0028] Other features and advantages of the present disclosure will be described in the subsequent description, and partly become apparent from the description, or be understood by practicing the present disclosure. The purpose and other advantages of the present disclosure can be realized and obtained by the structures particularly pointed out in the written description and the drawings. The technical solution of the present disclosure is further described below through drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. In the accompanying drawings:
[0030] Figure 1 A schematic diagram of the interaction between the domestic freight rate calculation system and the user provided in the embodiment;
[0031] Figure 2 A schematic diagram of a freight rate calculation cloud service system provided in the present disclosure;
[0032] Figure 3 A schematic diagram of a freight rate calculation cloud service system provided in an embodiment;
[0033] Figure 4 A flow chart of system data changes provided in an embodiment;
[0034] Figure 5 This is a schematic diagram of the data layer of the freight rate calculation cloud service system provided in the embodiment;
[0035] Figure 6 A schematic diagram of a computing microservice call provided in an embodiment;
[0036] Figure 7 The architecture diagram of the freight rate calculation cloud service system provided in the embodiment;
[0037] Figure 8 A schematic diagram of different processes sharing data given in an embodiment;
[0038] Fig. 9 A flowchart of system data version switching given in an embodiment;
[0039] Fig.10 A schematic diagram of an electronic device provided for an embodiment;
[0040] Fig.11 A schematic diagram of a computer-readable storage medium is provided for an embodiment. DETAILED DESCRIPTION
[0041] The present disclosure is described below in conjunction with the accompanying drawings. The preferred embodiments described herein are only used to illustrate and explain the present disclosure and are not used to limit the present disclosure.
[0042] The shortcomings of the domestic freight rate calculation system in the current local data access, multicast database file transmission, freight rate data update and switching, disaster recovery processing, etc. are the problems that this cloud solution needs to solve. Finding a solution that can efficiently read data, expand and solve data growth, improve data loading performance, reduce the number of single data transmissions, and reduce network hardware requirements is the focus of this public discussion.
[0043] Freight data is updated regularly, and a certain delay is allowed. If the data is updated in real time, the domestic freight calculation system cannot use complex data preprocessing technology, and the existing computing engine cannot support it. The distribution of airline data that needs to be accessed by the calculation request is relatively stable, but as the data and market change, the number of cloud service nodes / processes for each data partition still needs to be expanded or reduced. These rely on monitoring and early warning mechanisms to automatically or semi-automatically trigger automated operation and maintenance tools for data migration and parameter adjustment. Based on such premise constraints, the construction method and working principle of this system are discussed.
[0044] Figure 2 The freight rate calculation cloud service system provided in this disclosure includes:
[0045] Master control module, data module and interface module; among them,
[0046] A data mapping table is stored in the master control module;
[0047] The data module is configured as follows:
[0048] Responding to the freight rate calculation request through the interface module; and
[0049] After receiving the data change notification sent by the master control module, new data is loaded and the master control module is notified, so that the master control module updates the data mapping table.
[0050] According to some embodiments of the present disclosure, Figure 3 As shown, the freight rate calculation cloud service system includes: interface service, computing service interface, master control, and local data. The interface service is used for computing system entry service. The computing service interface is used to access the data partition service of each business scenario. The master control is used to control data update, data loading, and data version switching; in addition, this embodiment also includes business services, and the data business service is used to split the ALL IN ONE data into fine-grained data according to the freight rate business.
[0051] Furthermore, the data module includes: a data management module, a data update module, a freight rate calculation module and a database interface module; wherein,
[0052] The data update module is configured as follows:
[0053] In response to the data change notification, generate an interface data file;
[0054] The general control module is notified to control the freight rate calculation module to call the interface data file and access the database interface module through the data management module.
[0055] According to some embodiments of the present disclosure, Figure 4 As shown in the figure, the process of loading new data in the data module is as follows: (1) The master control module sends a data change notification to the data module; (2) The data update module generates an interface data file based on the data change notification; (3) The data update module sends a notification of completion of interface data file generation to the master control module and writes the status to redis; (4) The master control module sends a control command to the data management module based on redis, and the command includes the mapping information contained in the interface data file; (5) The data management module sends a forwarding control command to the freight rate calculation engine interface; (6) The freight rate calculation engine interface calls the interface data file based on the mapping information and accesses the database interface based on the file.
[0056] According to other embodiments of the present disclosure, after the freight calculation system service data is loaded, the consistency of the entire link call must be guaranteed, and TAP (Air Information Cloud Computing Application Platform) provides the ability to specify groups. Service groups include: computing services and business scenario partition data services, that is, deploying interface (interface), SFN (rule display), FD (freight query), FBR (rule freight) and other services; when data changes, the data loading master control notifies the computing service and the airline partition data service; after the data is successfully loaded, the master control is notified; the master control writes the current effective group information to redis; the interface service must access redis every time it requests and set the request group information; the computing service sets the request group information based on the group information at the time of deployment. In this way, the consistency of data effectiveness is guaranteed.
[0057] Furthermore, the freight rate calculation cloud service system also includes: a data preprocessing module; the data preprocessing module, the data module and the interface module are distributedly deployed in a number of computing nodes.
[0058] Furthermore, the data preprocessing module and the data module are deployed on the same machine.
[0059] According to some embodiments of the present disclosure, the data layer implementation principle of the freight rate calculation cloud service system is as follows: cloud deployment, using distributed concurrent preprocessing, not only reduces processing latency, but also greatly reduces the requirements for hardware resources such as memory and SSD. In addition, data preprocessing and computing engines are deployed on the same machine, without the need to transmit large-volume data files after preprocessing, no longer requiring a special dual network environment, and elastic expansion is more convenient.
[0060] Furthermore, the preprocessing module is configured to collect, organize, publish and preprocess the source data; wherein the source data includes: freight rate data and flight data; the data preprocessing includes: cleaning unrelated data, storing unrelated data and inserting new related data.
[0061] According to some embodiments of the present disclosure, the external data that the freight rate calculation cloud service system relies on is divided into three categories: basic data, freight rate data, and flight data. The data sources include DFP, ASFP, etc., and the file format, release cycle, etc. are not unified. The collection module packages the data into a series of data versions according to the time point of receiving the data; source data organization includes: decompressing and parsing the source data, and reorganizing it according to the data business relationship: ① Cleaning unrelated data: unrelated data refers to data that is not referenced by any airline, and is invalid data in the current data version. After cleaning, the data scale can be effectively reduced. ② Storing unrelated data: unrelated data means that the current version is not referenced, but may be referenced in future data versions, so it is stored on the local disk. ③ Inserting new related data: New related data refers to the previous batch of data (that is, the unrelated data of the past data version) referenced in the current data version, which is taken out from the local disk and inserted into the data version; source data publishing includes: publishing data in HTTP mode; data preprocessing (DPRIC Update) generates preprocessing results that can be used directly by the engine in advance, and maps them into the process virtual memory space in a memory-mapped manner. The whole process is like Figure 5 shown.
[0062] Furthermore, at least two sets of computing services are deployed in the same computing node, and each set of computing services includes: at least one of the data preprocessing modules, at least one of the data modules, and at least one of the interface modules.
[0063] Furthermore, data is shared between different computing services in the same computing node.
[0064] According to some embodiments of the present disclosure, Figure 6 As shown in the figure, in order to make full use of physical resources and improve the utilization rate of computing cloud microservices, the data accessed by the fare calculation service is divided into two data groups, airline group 1 and airline group 2, according to the size of the data and the proportion of data access. The specific grouping is adjusted according to the access volume and data proportion; the computing microservice distinguishes the business processes of the two airline groups through the service names with suffixes (group 1 and group 2). When making external calls, different microservice names should be called according to the airline grouping requests. If there are connecting requests or FBR shared code fares from different groups, mutual access between the computing microservice groups (LE group 1 and LE group 2) is required.
[0065] According to some embodiments of the present disclosure, Figure 7As shown in the figure, the DPRIC freight calculation system deployment model can be divided into two categories, namely interface services (stateless), computing services, and business scenario partition data services (stateful). The interface service can label a group of machines as LFEP (Freight Front-end Processing Module), and the pric-LFEP service can be deployed on any machine. The computing service & business scenario partition data service is a single group of machines labeled as Engine (engine, including the host and backup machine), and the same service and corresponding data loading components are deployed on each physical machine.
[0066] According to other embodiments of the present disclosure, the freight rate calculation system business data model requires multiple processes to share the same data, which essentially requires data sharing between containers. Figure 8 As shown in the figure, all PODs of the same physical machine can access the same file by hanging them in the same directory of the host machine; DB file: The DB file is mapped by the data update module DPRIC Update POD using MMap+MAP_PRIVATE to pre-process the source data file and save it to the shared directory; Data loading: When a pod service process uses MMap+MAP_SHARE to map a file, the operating system maps the data to the shared memory RAM. When other POD service processes map files in the same way, the operating system will ensure access to the same shared content RAM. Freight rate data is not real-time data, but is updated regularly. In order to meet the needs of continuous data updates and changes, two groups of TAP services, DPRIC Update and DB Manager, are designed: When DPRIC Update obtains updated data, it will generate a new data file and record the data batch status to redis. When DPRIC-Engine receives a freight rate calculation request, it goes to redis to obtain a new data batch. If the data is updated, the new batch data will be reloaded.
[0067] Furthermore, the data module is also configured to redirect subsequent database interface module access requests or freight rate calculation module access requests to other database interface modules or other freight rate calculation modules in the current computing node after receiving the data version switching notification sent by the master control module.
[0068] According to some embodiments of the present disclosure, Fig. 9 As shown, after the master control module sends the data version switching notification, the data update module generates the corresponding data file (refer to the data change process above) and writes the status to redis. After the data module accepts the freight calculation request through the interface module, the data management module obtains the data file in redis and redirects the access request of the freight calculation engine to the database where the new version of the data is located based on the data file.
[0069] The present disclosure also proposes a freight rate calculation method based on the above-mentioned freight rate calculation cloud service system, including: using the data module to perform the following method:
[0070] Responding to the freight rate calculation request through the interface module; and after receiving the data change notification sent by the master control module, loading new data and notifying the master control module, so that the master control module updates the data mapping table.
[0071] like Fig.10 As shown, the present disclosure provides an electronic device 1000, which includes a memory 1002 and a processor 1001. The memory 1002 stores a computer program or instruction. When the computer program or instruction is executed by the processor 1001, it is at least used to implement the above-mentioned freight rate calculation method.
[0072] like Fig.11 As shown, the present disclosure provides a computer-readable storage medium 1100, in which a computer program or instruction is stored. When the computer program or instruction is executed by a processor, it is at least used to implement the above-mentioned freight rate calculation method.
[0073] The present disclosure also provides a computer program product, which is stored in a computer-readable storage medium. When the computer program product is executed by a processor, it is at least used to implement the above-mentioned freight rate calculation method.
[0074] According to some embodiments of the present disclosure, the freight rate calculation cloud service system provided by the present disclosure adopts an X86 hardware architecture, is configured with 16 Cores, 1024G memory, 6.4T SSD hard disk, TOSFC container (Travelsky Open ServiceFrameword Cloud, a framework developed with C++ technology), OCP4 container image, and TOSFC as the middleware, combined with the Travelsky cloud computing application platform TAP, to design a cloud DPRIC. In addition to considering important indicators such as performance, scalability, and maintainability, the software. If the DPRIC is unavailable due to an overall failure of the TAP cluster or other reasons, and cannot be restored in the short term, emergency recovery measures can be used to cut the traffic back to the cloud system. The recovery time of this switch is in seconds, and the diversion configuration of the LFEP module of the DPRIC software can be changed. After completing the migration of full-scene and full-market business traffic, the old system can still be retained for half a year.
[0075] In summary, this system currently uses ALL IN ONE's self-developed data access solution based on memory mapping. The data is completely localized, the reading efficiency is high, and it supports complex data structures. However, the data growth problem cannot be solved by horizontal expansion. This problem can be well solved by splitting the freight data. The advantage of freight data splitting is that some data is still local, the reading efficiency is high, the data growth problem can be solved by horizontal expansion, and the system transformation cost is low. However, after the data is split, the additional overhead of obtaining remote services and data is increased, and the instruction processing delay is increased. The prototype system verification shows that the increased processing delay is about 0.1 seconds, and the user is basically unaware. The freight data splitting solution not only ensures the reading efficiency, but also solves the data growth problem.
[0076] The DPRIC system uses multicast technology to transmit database files, with a transmission volume of up to T level. Its advantage is that data preprocessing is performed on a single server, and each computing node server can use the data directly after receiving it, without repeated data preprocessing. However, the current cloud environment does not support multicast technology, and the failure rate of transmitting large amounts of data is high, and the cost of data compression and decompression is also high. In order to solve this problem, we adopted distributed preprocessing, so that the data preprocessing will be performed independently on each computing node server, without the need for large-scale network transmission of large amounts of data, and only a small amount of source data transmission is required (TCP protocol can be used). Judging from the verification results of the prototype system, the delay of data preprocessing has been alleviated.
[0077] The freight rate data of the DPRIC system is updated regularly, and the update process cannot affect the computing service. After the new data is preprocessed, all computing requests must be responded to and processed using the new data version. The current system uses an AVE-like method to determine whether to switch data before processing each request. Its advantages are that the solution is simple and effective, and TOSFC natively supports it. However, when there are many data files and the loading process is time-consuming, the request processing delay will increase. In order to eliminate the impact of the loading process, we use an offline method to complete the data loading process and use an incremental update method to complete the data update, which can greatly improve the data loading performance.
[0078] It is obvious that a person skilled in the art can make various changes and modifications to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is also intended to include these modifications and variations.
Claims
1. A freight rate calculation cloud service system, characterized in that: include: Master control module, data module and interface module; among them, A data mapping table is stored in the master control module; The data module is configured as follows: Responding to the freight rate calculation request through the interface module; and After receiving the data change notification sent by the master control module, new data is loaded and the master control module is notified, so that the master control module updates the data mapping table.
2. The freight rate calculation cloud service system according to claim 1, characterized in that: The data module includes: a data management module, a data update module, a freight rate calculation module and a database interface module; wherein, The data update module is configured as follows: In response to the data change notification, generate an interface data file; The general control module is notified to control the freight rate calculation module to call the interface data file and access the database interface module through the data management module.
3. The freight rate calculation cloud service system according to claim 2, characterized in that: Also includes: Data preprocessing module; the data preprocessing module, the data module and the interface module are distributedly deployed in a number of computing nodes.
4. The freight rate calculation cloud service system according to claim 3, characterized in that: The data preprocessing module and the data module are deployed on the same machine.
5. The freight rate calculation cloud service system according to claim 4, characterized in that: The preprocessing module is configured to collect, organize, publish and preprocess source data; wherein, The source data includes: freight rate data and flight data; The data preprocessing includes: cleaning unrelated data, storing unrelated data and inserting new related data.
6. The freight rate calculation cloud service system according to claim 4, characterized in that: At least two computing services are deployed in the same computing node, and each computing service includes: at least one data preprocessing module, at least one data module and at least one interface module.
7. The freight rate calculation cloud service system according to claim 6, characterized in that: Data is shared between different computing services in the same computing node.
8. The freight rate calculation cloud service system according to claim 6, characterized in that: The data module is further configured as follows: After receiving the data version switching notification sent by the master control module, the subsequent database interface module access request or freight rate calculation module access request is redirected to other database interface modules or other freight rate calculation modules in the current computing node.
9. A freight rate calculation method based on the freight rate calculation cloud service system according to any one of claims 1 to 8, characterized in that: include: The data module is used to perform the following method: Responding to the freight rate calculation request through the interface module; and After receiving the data change notification sent by the master control module, new data is loaded and the master control module is notified, so that the master control module updates the data mapping table.
10. An electronic device, comprising a memory and a processor, characterized in that: The memory stores a computer program or instruction, which, when executed by the processor, is at least used to implement the freight rate calculation method according to claim 9.