Airline map display method, system and equipment based on distributed log system and medium

Abstract

The invention belongs to the technical field of aviation maps, and provides a route map display method and system based on a distributed log system, electronic equipment and a storage medium. The method comprises the following steps: acquiring an air ticket search service log, preprocessing, and converting the air ticket search service log into a general format log; designing an index strategy, and indexing the route search record of the general format log into an independent document; carrying out multi-dimensional statistics on the independent documents by utilizing a search engine; and displaying the statistical result in a graphical manner by using a data visualization library. A distributed log system is adopted, a large amount of flight data can be collected and analyzed in real time, compared with a centralized log system, the processing capacity is higher, the processing requirement of large-scale data can be effectively met, and the problem of processing delay or collapse is reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of aviation maps, and in particular relates to a route map display method, system, electronic equipment and storage medium based on a distributed log system. Background Art

[0002] With the increasing prosperity of the aviation transportation industry, the number of flights and passengers has shown a rapid growth trend, and the search volume of the ISHOP international freight rate search system has also increased. In this context, how to effectively analyze and display a large amount of flight search data and mine valuable information is crucial for the operation and management of airlines and airports. Existing data collection and statistical methods are often inefficient and difficult to achieve real-time monitoring and analysis. In addition, the display method for popular routes is relatively simple, and the online map service cannot be used offline, and cannot intuitively reflect the popularity distribution of routes.

[0003] In terms of log collection, existing technologies usually use a centralized log system to collect and process flight data. This system uses a server to collect and store all log data, and then uses data analysis tools to perform data analysis and visualization. In terms of route map display, existing technologies usually use front-end visualization frameworks such as ECharts for display, and mostly use online map services.

[0004] First, since all log data is concentrated on one server, once the server fails, data may be lost, affecting the results of data analysis. Secondly, when processing large-scale data, centralized log systems may encounter performance bottlenecks, affecting the efficiency of data processing and insufficient data real-time performance. Thirdly, existing log systems can usually only provide simple data analysis functions, cannot meet complex data analysis requirements, and cannot use more intuitive visualization methods for display. Finally, existing map display solutions are mostly online map services, which cannot cope with situations where the internal network environment does not allow connection to the external network. Summary of the invention

[0005] To solve the above problems, the present invention provides a route map display method, system, electronic device and storage medium based on a distributed log system. The method can solve the problem of data reliability of a centralized log system. The distributed log system based on ELK can ensure that a single-node system failure will not affect the system. Secondly, it can solve the problem of insufficient real-time performance in the prior art and realize low-latency quasi-real-time data query. Thirdly, it can solve the problem that the prior art solutions cannot meet the needs of complex data analysis. This solution supports Lucence Query and Elasticsearch DSL syntax, and can realize business queries for complex scenarios. Finally, it can solve the problem that the online map display solution cannot be used in the intranet. This solution uses GeoJSON spatial geographic data to realize the display of popular routes in an offline state.

[0006] A route map display method based on a distributed log system of the present invention comprises:

[0007] Collecting air ticket search service logs, performing preprocessing, and converting the air ticket search service logs into logs in a common format;

[0008] Design an indexing strategy to index route search records in the common format log into separate documents;

[0009] Using a search engine to perform multi-dimensional statistics on the independent documents;

[0010] Use the data visualization library to display the statistical results in a graphical way.

[0011] Furthermore, the collecting of air ticket search service logs includes:

[0012] The air ticket search business logs distributed on different servers are monitored and collected in real time.

[0013] Further, the pre-processing includes:

[0014] One or more of filtering irrelevant information, parsing log formats, and converting data types.

[0015] Furthermore, the key fields for designing the index strategy include:

[0016] One or more of departure time, origin, destination and OD route.

[0017] Furthermore, the design index strategy includes:

[0018] Use reasonable word segmentation strategies and data type selection to optimize retrieval performance.

[0019] Furthermore, the method of displaying the statistical results in a graphical manner using a data visualization library includes:

[0020] Build a world map component to display the spatial distribution of routes and traffic volume through dynamic lines and color changes.

[0021] Furthermore, the method of displaying the statistical results in a graphical manner using a data visualization library includes:

[0022] Use the Sankey diagram component to display the traffic direction and scale between different routes, helping to understand passengers' travel preferences and market trends.

[0023] Furthermore, different visualization components are combined together to create a comprehensive and interactive monitoring interface to monitor the business indicators of the search service at the same time.

[0024] The present invention also provides a route map display system based on a distributed log system, the system comprising: a data acquisition module, a data index module, a data analysis module and a data visualization module;

[0025] A data collection module, used to collect air ticket search business logs, perform preprocessing, and convert the air ticket search business logs into a common format log;

[0026] The data indexing module is used to design indexing strategies and index route search records in the common format log into independent documents;

[0027] A data analysis module, used for performing multi-dimensional statistics on the independent documents using a search engine;

[0028] The data visualization module is used to display the statistical results in a graphical way using the data visualization library.

[0029] The present invention also provides an electronic device, comprising:

[0030] processor;

[0031] and a memory storing computer executable instructions, wherein the computer executable instructions, when executed, cause the processor to perform the method described in any one of the above embodiments.

[0032] The present invention also provides a computer storage medium, wherein the computer storage medium stores one or more programs, and when the one or more programs are executed by a processor, the method described in any one of the above embodiments is implemented.

[0033] Compared with the prior art, the present invention has the following advantages:

[0034] 1. Efficiently process large amounts of data: The present invention adopts a distributed log system, which can collect and analyze a large amount of flight data in real time. Compared with a centralized log system, it has stronger processing capabilities, can effectively cope with the processing needs of large-scale data, and reduce processing delays or crashes.

[0035] 2. Reduce costs: The present invention uses Elasticsearch as a data storage and processing platform, which does not require special hardware equipment and has a simple configuration, greatly reducing costs and making it affordable for more small and medium-sized enterprises.

[0036] 3. Support queries for complex business scenarios: After configuring the Elasticsearch data source in Grafana, we can take advantage of its powerful aggregation function and use the Lucence Query and Elasticsearch DSL syntax it provides to perform multi-dimensional statistics on the number of route searches and support queries for more complex business scenarios.

[0037] 4. Visually display data changes and improve user experience: The present invention uses the Echarts plug-in to display the route map in Granafa, which can intuitively display the changing trends and laws of the data, improve the visualization of the data, facilitate data analysis and decision-making, and improve the user experience.

[0038] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 Shows an overall architecture diagram according to an embodiment of the present invention;

[0041] Figure 2 A schematic diagram showing a flow chart of a route map display method based on a distributed log system of the present invention is shown;

[0042] Figure 3 The Elasticsearch index data format according to an embodiment of the present invention is shown;

[0043] Figure 4The structure diagram of the route map display system based on the distributed log system of the present invention is shown;

[0044] Figure 5 A schematic structural diagram of an electronic device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] The same reference numerals in the drawings represent the same or similar elements, components or parts, and thus the repeated description of the same or similar elements, components or parts may be omitted below. It should also be understood that although the first, second, third and other attributives representing the numbers may be used herein to describe various devices, elements, components or parts, these devices, elements, components or parts should not be limited by these attributives. In other words, these attributives are only used to distinguish one from another. For example, the first device may also be called the second device, but it does not deviate from the essential technical solution of the present invention. In addition, the terms "and / or" and "and / or" refer to all combinations including any one or more of the listed items.

[0047] In order to have a deeper understanding of the present invention, the relevant background and some of the terms therein are explained as follows:

[0048]

[0049] Table 1 Component deployment planning table required for this solution

[0050]

[0051] Table 1 mainly lists the component versions, IP and port planning, and the number of reference nodes deployed in this solution. The implementation process does not describe the installation process of each component in detail, but only describes in detail the configuration of the configuration files of the interrelated parts of each component and the main data structures used in the intermediate process. The components used in this solution are all open source components. The detailed deployment process of each component is documented in the official documentation, which can be used as a reference for installation. The specific implementation process of this solution is described in detail below.

[0052] Figure 1 The overall architecture diagram provided for the embodiment of the present invention specifically includes:

[0053] Build a device based on a distributed log system, which is mainly composed of multiple log servers to form a distributed log collection system. Deploy the Filebeat service on each business log server in the cluster, and the Filebeat service will push data to the Logstash cluster in real time. Secondly, build a Logstash cluster composed of multiple service nodes, and at the same time build an Elasticsearch cluster composed of multiple data nodes. After the data is collected by the Filebeat service, it is aggregated to the Logstash cluster, filtered and sorted by Logstash, and finally stored in the Elasticsearch cluster. Thirdly, build the Kibana service and Grafana service. The Kibana service can read the data in Elasticsearch and facilitate complex business queries on the data. The Grafanna service can read the data in Elasticsearch and finally realize the visualization of business data. Finally, in Grafana, introduce the Echarts plug-in to analyze and sort the flight query data in Elasticsearch, and finally realize the popular route map and traffic map.

[0054] Figure 2 A schematic flow chart of a route map display method based on a distributed log system provided in an embodiment of the present invention specifically includes:

[0055] S1: collecting air ticket search service logs, performing preprocessing, and converting the air ticket search service logs into a common format.

[0056] The specific collection and preprocessing details are as follows:

[0057] Data collection is the foundation of the entire system. In this solution, we use Filebeat as the data collector, which is lightweight and easy to deploy, and can monitor and collect flight search business logs distributed on different servers in real time. Filebeat supports multiple log formats and can flexibly configure collection rules to ensure the comprehensiveness and accuracy of data.

[0058] The collected raw data is preprocessed by Logstash, including but not limited to filtering irrelevant information, parsing log formats, converting data types, etc., for subsequent processing. The processed data is sent in batches to the Elasticsearch cluster, using its distributed storage capabilities to ensure high availability and scalability of the data.

[0059] The specific steps are as follows:

[0060] S11: Install Filebeat on all 10 client machines that store business logs.

[0061] S12: Configure filebeat.yml and specify the log collection path and Logstash target address.

[0062] S13: Start the Filebeat service and verify its connection to Logstash.

[0063] S14: Each record file in the business log is in JOSN format, and what is recorded is an ISHOP request message.

[0064] The definitions of some of the main fields in the ISHOP request message are as follows:

[0065] Table 2 Description of main fields of ISHOP request message

[0066]

[0067] S2: Design an indexing strategy to index route search records in a common format into independent documents.

[0068] The specific index and search details are as follows:

[0069] In Elasticsearch, data indexing is a key step in organizing and storing data. Based on business needs, this solution designs an indexing strategy that includes key fields such as departure time, origin, destination, and OD route. Each flight search record is indexed as an independent document, in which each field is carefully designed to support fast retrieval and aggregation operations.

[0070] During the indexing process, we adopted a reasonable word segmentation strategy and data type selection to optimize query performance. At the same time, by setting an appropriate index lifecycle strategy, we automatically manage the creation and deletion of indexes and reduce the maintenance burden.

[0071] The specific steps are as follows:

[0072] S21: Build a Logstash cluster.

[0073] S211: Install Logstash on the specified three nodes.

[0074] S212: Configure Logstash's logstash.conf, set up Beats input, filtering, and Elasticsearch output.

[0075] S213: Start the Logstash service.

[0076] Configure the conversion logic of business logs to ElasicSearch index mapping in Logstash's filter. The main relationships are listed as follows:

[0077] Table 3 Correspondence between ISHOP request message fields and Elasticsearch index fields

[0078]

[0079] S22: Build ElasicSearch.

[0080] S221: Configure Elasticsearch.yml and set the cluster name, node name, and network options.

[0081] S222: Start the Elasticsearch service. You can then check the cluster health status in Kibana.

[0082] S3: Utilize a search engine to perform multi-dimensional statistics on the independent documents.

[0083] The specific analysis details are as follows:

[0084] The core of data analysis is to extract valuable information from massive data. After configuring the Elasticsearch data source in Grafana, we can use its powerful aggregation function to perform multi-dimensional statistics on the number of route searches.

[0085] In this embodiment, we can analyze the search popularity in different time periods of the day through the time field; through the origin and destination fields, we can identify popular OD routes. In addition, using Grafana's visual query editor, we can easily build complex data analysis models and realize real-time data analysis.

[0086] The specific steps are as follows:

[0087] S31: Build Kibana.

[0088] S311: Install Kibana on the specified node.

[0089] S312: Configure kibana.yml to ensure that Kibana can connect to the Elasticsearch cluster.

[0090] S313: Start the Kibana service and access the web interface to verify the connection.

[0091] In Kibana, the format of the Elasticsearch data queried is as follows Figure 3 .

[0092] S32: Build Grafana.

[0093] S321: Install Grafana on the specified node.

[0094] S322: Configure grafana.ini and add Elasticsearch as the data source.

[0095] S323: Start the Grafana service and access the web interface to configure the dashboard.

[0096] Create a dashboard of popular routes in Grafana and create a panel. In the panel edit mode, configure Elasticsearch as the data source of Grafana and write a Lucence Query in the Grafana panel for preliminary data analysis.

[0097] S4: Use the data visualization library to display the statistical results in a graphical way.

[0098] The specific visualization details are as follows:

[0099] Data visualization is to display the analysis results in a graphical way so that users can understand the data intuitively. In Grafana, we use the Echarts plug-in as the main visualization component. First, we built a world map component to show the spatial distribution and traffic size of popular routes through dynamic lines and color changes. Secondly, the Sankey diagram component is used to show the flow direction and scale of traffic between different routes, helping users understand passengers' travel preferences and market trends.

[0100] In addition, Grafana's dashboard feature allows us to combine different visualization components to create a comprehensive, interactive monitoring interface that can monitor both the server's hardware metrics and the search service's business metrics.

[0101] The specific steps are as follows:

[0102] S41: Install the Echarts plug-in to Grafana offline and restart Grafana.

[0103] S42: In the DashBoard of "Popular Route Map", create two panels, "Arrival Route Map" and "Departure Route Map", and edit the two panels separately.

[0104] Take the "Arrival-Route Chart" panel as an example to explain: first select Elasticsearch as the data source, enter the query statement in the figure below in Query, then select the plug-in that introduces the Echarts icon in the Visualization option of the Panel tab on the right, and finally enter the JavaScript code in the code input box in the Display selection to implement the Echarts route chart with code.

[0105] In the DashBoard of "Popular Route Map", create two panels, "Arrival-Flow Chart" and "Departure-Flow Chart", and edit the two panels respectively. Take the "Arrival-Flow Chart" panel as an example to explain: first select Elasticsearch as the data source, enter the query statement in the Query as shown below, then select the plugin to introduce the Echarts icon in the Visualization option of the Panel tab on the right, and finally enter the JavaScript code in the code input box in the Display selection to implement the Echarts flow chart with code.

[0106] The embodiment of the present invention also provides a route map display system based on a distributed log system, such as Figure 4 As shown, including:

[0107] The data collection module 11 is used to collect the air ticket search business log, perform pre-processing, and convert the air ticket search business log into a common format;

[0108] A data indexing module 12, used to design an indexing strategy to index route search records in a common format into independent documents;

[0109] A data analysis module 13 is used to perform multi-dimensional statistics on the independent documents using a search engine;

[0110] The data visualization module 14 is used to display the statistical results in a graphical manner using a data visualization library.

[0111] like Figure 5 As shown, an embodiment of the present invention provides an electronic device, including a processor 1110, a communication interface 1120, a memory 1130 and a communication bus 1140, wherein the processor 1110, the communication interface 1120, and the memory 1130 communicate with each other through the communication bus 1140;

[0112] Memory 1130, used for storing computer programs;

[0113] The processor 1110 is configured to implement any of the above determination methods when executing a program stored in the memory 1130 .

[0114] In the electronic device provided by the embodiment of the present invention, the processor 1110 obtains the fluid flow rate of each branch in different switching states by executing the program stored in the memory 1130, and determines the initial volume flow rate of each branch; corrects the initial volume flow rate according to the pipeline parameters and fluid parameters of each branch when it is in working condition and standard condition, and obtains the standard condition volume flow rate of each branch; obtains multiple total standard condition volume flow rates according to the standard condition volume flow rates of each branch in different switching states, and determines the optimal switching state of each branch through the switching state of each branch when the total standard condition volume flow rate reaches a preset target.

[0115] The communication bus 1140 mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus 1140 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0116] The communication interface 1120 is used for communication between the above electronic device and other devices.

[0117] The memory 1130 may include a random access memory 1130 (RAM), or may include a non-volatile memory 1130 (non-volatile memory), such as at least one disk memory 1130. Optionally, the memory 1130 may also be at least one storage device located away from the processor 1110.

[0118] The above-mentioned processor 1110 can be a general-purpose processor 1110, including a central processing unit 1110 (CPU), a network processor 1110 (NP), etc.; it can also be a digital signal processor 1110 (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0119] An embodiment of the present invention provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors 1110 to implement the determination method of any of the above embodiments.

[0120] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk SolidStateDisk (SSD)), etc.

[0121] Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent substitutions for some of the technical features therein; and these modifications or substitutions do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A route map display method based on a distributed log system, characterized in that: The method comprises: Collecting air ticket search service logs, performing preprocessing, and converting the air ticket search service logs into logs in a common format; Design an indexing strategy to index route search records in the common format log into separate documents; Using a search engine to perform multi-dimensional statistics on the independent documents; Use the data visualization library to display the statistical results in a graphical way.

2. The display method according to claim 1, characterized in that: The collection of air ticket search business logs includes: The air ticket search business logs distributed on different servers are monitored and collected in real time.

3. The display method according to claim 1, characterized in that: The pre-processing comprises: One or more of filtering irrelevant information, parsing log formats, and converting data types.

4. The display method according to claim 1, characterized in that: The key fields for designing the index strategy include: One or more of departure time, origin, destination and OD route.

5. The display method according to claim 1, characterized in that: The design index strategy includes: Use reasonable word segmentation strategies and data type selection to optimize retrieval performance.

6. The display method according to claim 1, characterized in that: The method of using the data visualization library to display the statistical results in a graphical manner includes: Build a world map component to display the spatial distribution of routes and traffic volume through dynamic lines and color changes.

7. The display method according to claim 1, characterized in that: The method of using the data visualization library to display the statistical results in a graphical manner includes: Use the Sankey diagram component to display the traffic direction and scale between different routes, helping to understand passengers' travel preferences and market trends.

8. The display method according to claim 6 or 7, characterized in that: Combine different visualization components together to create a comprehensive and interactive monitoring interface that monitors business metrics of search services simultaneously.

9. A route map display system based on a distributed log system, characterized in that: The system includes: a data acquisition module, a data index module, a data analysis module and a data visualization module; A data collection module, used to collect air ticket search business logs, perform preprocessing, and convert the air ticket search business logs into a common format log; The data indexing module is used to design indexing strategies and index route search records in the common format log into independent documents; A data analysis module, used for performing multi-dimensional statistics on the independent documents using a search engine; The data visualization module is used to display the statistical results in a graphical way using the data visualization library.

10. An electronic device, characterized in that: include: processor; and a memory storing computer executable instructions which, when executed, cause the processor to perform a method according to any one of claims 1-8.

11. A computer storage medium, characterized in that: in, The computer storage medium stores one or more programs, and when the one or more programs are executed by a processor, the method of any one of claims 1 to 8 is implemented.

Images