Traffic planning statistics data linkage retrieval system and method based on multiple servers

Abstract

The present invention discloses a multi-server-based traffic planning statistics data linkage retrieval system and method, which relates to the technical field of traffic planning management. The system includes a collection client, a collection service cluster, a linkage distribution server, a storage service cluster, a search service cluster and a retrieval terminal. By separating the storage and retrieval of big data and constructing a composite word segmentation of traffic planning statistics data, the performance and speed of big data range search and accurate search are improved; by linking the distribution server dimension extraction, spatiotemporal convergence and control forwarding, data linkage is achieved between the two, and the characteristics of the two types of databases are fully combined, which greatly improves the system retrieval performance and enhances the user experience. The present invention constructs a connection relationship between multiple hardware servers, that is, constructs a hardware carrier of a statistical data linkage retrieval system, so that the hardware carrier assists the relevant software function to achieve the purpose of rapid retrieval, providing convenience for relevant staff.

Description

Technical Field

[0001] The present invention relates to the technical field of traffic planning management, and more specifically to a multi-server-based traffic planning statistical data linkage retrieval system and method. Background Art

[0002] At present, traffic planning business urgently needs economic and social, transportation, urban and rural construction, and industrial development statistics, such as yearbook statistics, traffic survey statistics, mobile phone signaling statistics, ETC statistics, and green freight GPS statistics. These statistics appear directly or are indirectly cited in prediction and evaluation traffic planning research reports with high frequency. Different research reports require different indicators or indicators of different time and space ranges to quantify, display, and mine to provide support for research projects. In many cases, researchers do not know the exact information of the indicators, so an accurate and high-performance retrieval system is very important for traffic planning research. A high-performance and accurate retrieval system needs to be configured with corresponding hardware carriers to assist the software in realizing the above functions.

[0003] Therefore, how to build a linkage retrieval system for traffic planning statistical data to achieve fast and accurate retrieval is an urgent problem that technical personnel in this field need to solve. Summary of the invention

[0004] In view of this, the present invention provides a multi-server-based traffic planning statistical data linkage retrieval system and method, through which the function of rapid retrieval of traffic planning statistical data is facilitated.

[0005] In order to achieve the above object, the present invention adopts the following technical solution:

[0006] A multi-server-based traffic planning statistics data linkage retrieval system, comprising a collection client, a collection service cluster, a linkage distribution server, a storage service cluster, a search service cluster and a retrieval terminal;

[0007] The collection client obtains traffic data, processes it into traffic planning statistics data according to a set format, and sends it to the collection service cluster in the form of Kafka messages;

[0008] The collection service cluster stores the traffic planning statistical data in the form of a Kafka service cluster;

[0009] The linkage distribution server reads the traffic planning statistics data from the collection service cluster reading module; extracts the indicator dimension, time dimension and space dimension in the read data, and encodes them respectively, which are referred to as indicator code, time code and space code respectively; performs spatiotemporal aggregation on the time dimension, space dimension and their encoding according to the indicator dimension, obtains a time range encoding list, a space range name and a corresponding space encoding list, and can parse the time name according to the time code, and can aggregate the time range name list according to the parsed time name; distributes the indicator dimension, time dimension, space dimension, the encoding and data values ​​corresponding to each dimension to the storage service cluster; distributes the time range encoding list, space range name and the corresponding space encoding list obtained by spatiotemporal aggregation to the search service cluster;

[0010] The retrieval terminal obtains the search conditions and sends them to the search service cluster, and receives the returned result data set for display; obtains the user selected data, performs a range query or a precise query according to the query method of the user selected data and the query data, and receives the returned query results for display;

[0011] The search service cluster inserts or updates the results of spatiotemporal aggregation according to the indicator dimension, performs a complete match according to the search conditions, implements a fuzzy search, returns the matched result data set to the search terminal, and displays the matched indicator and the spatiotemporal range corresponding to the indicator; performs a complete match on the business type and data source according to the search conditions, and performs a fuzzy search on the table name and indicator name using a custom compound participle;

[0012] The storage service cluster incrementally stores indicator dimensions, time dimensions and space dimensions and corresponding codes, stores data values, corresponding indicator codes, time codes and space codes, and a primary key formed by splicing the codes corresponding to the three dimensions; supports matching the primary key start and end range with the stored primary key, range querying the corresponding data set, or matching the complete primary key with the stored primary key, accurately querying the corresponding data value, and obtaining the query result and returning it to the retrieval terminal.

[0013] Preferably, the client collects traffic data from a database or business system, and constructs traffic planning statistical data according to a set format. The set format fields include business type, data source, table name, indicator name, indicator unit, indicator value, data description, time type, data time, space type, space name, and location description, a total of 12 fields. For example, if the business type is statistical yearbook, the data source is the national statistical yearbook data and the business type is inter-departmental data, the data source is inter-departmental results statistical data, which are set to two different partitions and sent to the collection service cluster in the form of kafka message middleware.

[0014] Preferably, the collection service cluster saves the traffic planning statistics data in a storage medium in partitioned batch order.

[0015] Preferably, the linkage distribution server includes a data reading module, a feature extraction module and a data distribution module;

[0016] The data reading module reads from different partitions in the collection service cluster in parallel, and reads traffic planning statistics data in the same partition in sequence. It is implemented using multiple clients in the same group, one client corresponds to one partition, and each partition has only one client reading sequentially, realizing concurrency between partitions and orderly reading within a partition;

[0017] The feature extraction module extracts the data values ​​of the indicator dimension, time dimension and space dimension in the traffic planning statistical data, wherein the business type, data source, table name and indicator name are the indicator dimension; the time type and data time are the time dimension; the space type, space name and location description are the space dimension; the three dimensions are encoded respectively to obtain the indicator code, time code and space code, and at the same time, the time range code list, space range name and corresponding space code list corresponding to the data are aggregated according to the indicator dimension, wherein the space range name and the space code list correspond one to one;

[0018] The data distribution module sends the indicator dimension, time dimension and space dimension, indicator code, time code and space code, and the data values ​​corresponding to the three dimensions to the storage service cluster; and sends the time range code list, space range name and corresponding space code list to the search service cluster.

[0019] Preferably, the retrieval terminal includes a search client and a query client;

[0020] The search client obtains the business type, data source, and keywords set by the user as search conditions, sends them to the search service cluster, obtains search results, and displays them;

[0021] The query client provides users with query methods, and users can choose to query data by primary key range or query data precisely by primary key. The query results are obtained and displayed according to the query data. The query methods include querying all indicator data and querying indicator data at specified time and space. When querying all indicator data, the range query method is used to query, and the code of the indicator dimension in the query data is used as the primary key start value in the primary key start and end range, and the code of the indicator dimension is used in hexadecimal plus 1 as the primary key end value in the primary key start and end range. According to the primary key start and end range, the data values ​​of all corresponding time dimensions and space dimensions are queried from the storage service cluster according to the primary key dictionary sequence range as the query result. When querying indicator data at specified time and space, the codes corresponding to the indicator, time, and space in the query data are spliced ​​into a complete primary key, and sent to the storage service cluster for precise query of the data values ​​of the corresponding indicator at specified time and space to obtain the query result.

[0022] Preferably, the storage service cluster uses the concatenated string of codes in the indicator code list, time code list and space code list as the primary key; stores the three dimensions in the traffic planning statistics data and other information of the traffic planning statistics data except the extracted dimension, the other information includes the indicator unit, indicator value, data description and the associated indicator code, time code and space code; queries the corresponding data set according to the start and end range of the primary key, or accurately queries the data value according to the complete primary key, fills in the data values ​​corresponding to the indicator dimension, time dimension and space dimension according to the corresponding indicator dimension code, time dimension code and space dimension code, and forms the search results and returns them to the search terminal;

[0023] The storage service cluster stores data in a distributed file cluster according to the primary key dictionary sequence, and provides a Hash index based on the primary key, a Hash fast query based on the primary key, and a range query based on the dictionary sequence.

[0024] The technical effect of the above technical solution is to obtain all traffic planning statistical data within the range of "A2G080108" to "A2G080109" from closed to open according to the primary key dictionary order range, that is, all cumulative passenger volume data details in the monthly statistical data of Inner Mongolia Autonomous Region and return them to the retrieval terminal for analysis and display; by constructing A2G080108202201014000, the detailed information on the cumulative passenger volume in Baotou in January 2022 can be accurately queried.

[0025] Preferably, the data fields stored in the search service cluster include business type, data source, table name, indicator name, indicator unit, indicator dimension code, as well as time range code list, space range name and corresponding space code list;

[0026] The search service cluster includes a compound word segmentation module and a search module;

[0027] The compound word segmentation module calculates the compound words contained in all table names and indicator names; constructs a feature vector matrix after basic word segmentation of the search conditions, compares the basic word segmentations in a circular order to obtain the vector similarity between different basic word segmentations, and forms compound word segmentations with basic word segmentations whose vector similarity among adjacent basic word segmentations is greater than the set threshold, and establishes an ElasticSearch index based on the compound word segmentations;

[0028] The search module supports the retrieval terminal to use the search method provided by ElasticSearch to completely match the business type and data source set by the user, and to perform fuzzy search on the index constructed by compound participles from the table name and indicator name fields using the keywords entered by the user in the search conditions, and to form a result data set that realizes keyword matching in reverse order according to the matching scores and send it to the retrieval terminal.

[0029] Preferably, the threshold value of vector similarity is set to 0.7.

[0030] The technical effect of the above technical solution is to improve the search accuracy through the compound word segmentation module. For example, for the phrase "highway mileage", ik_smart word segmentation is divided into two words "highway" and "mileage", but what the user really needs is the overall word segmentation of highway mileage. The doc2vec tool is used to establish a feature vector matrix of all table names and indicator names after using ik_smart word segmentation. The probability of the previous and next words appearing together is used to determine whether it is a compound word. If the similarity of adjacent basic word segmentation vectors reaches 0.7 or more, it is considered to be a compound word. According to the precise matching of business type and data source and the fuzzy matching of table name, indicator name, time and space, the matching results are returned to the search terminal;

[0031] The user obtains search results from the retrieval terminal, selects corresponding indicator data, and obtains the indicator dimension code, such as "A2G080108", where A represents the statistical yearbook, 2G represents the "monthly statistical data of the Inner Mongolia Autonomous Region", 0801 represents the transportation industry and the postal and telecommunications industry tables, and 08 code indicator "cumulative passenger volume (10,000 people)". The starting value of the storage server query is constructed as A2G080108, and the ending value is the indicator number in hexadecimal plus 1, that is, A2G080109, to query the data of all years and all spatial ranges of the indicator "cumulative passenger volume (10,000 people)".

[0032] Preferably, the process of constructing compound participles by the compound participle module is:

[0033] Step 1: Summarize all indicator names and table names and remove duplicates;

[0034] Step 2: Use ik_smart word segmentation to split all the index names and table names after deduplication into a word segmentation array list, where each word segmentation array in the word segmentation number list corresponds to a table name or index name;

[0035] Step 3: Use doc2vec to construct a word segmentation feature matrix based on the word segmentation array list, and create a compound word segmentation construction list and a compound word segmentation set based on the word segmentation feature matrix; the vector dimension of the word segmentation feature matrix is ​​300, the window size is 3, and the minimum ignored total frequency is 1; the compound word segmentation construction list is used to construct compound word segmentations, and the compound word segmentation set is used to store all compound word segmentations;

[0036] Step 4: Loop through the word segmentation array list and each word in each word segmentation array;

[0037] Step 5: If the compound participle construction list is empty, add the participle of the current cycle to the compound participle construction list; otherwise, determine whether the vector similarity between the current cycle participle and the last element in the compound participle construction list is greater than the set threshold. If so, add the participle of the current cycle to the compound participle construction list, otherwise proceed to step 6;

[0038] Step 6: Determine whether the number of elements in the compound participle construction list is greater than 1. If so, concatenate all the participles in the compound participle construction list into a compound participle and add the compound participle to the compound participle set. Clear the compound participle construction list and return to step 4 to continue the loop until all participles are traversed;

[0039] Step 7: Set all compound participles in the compound participle set as the ElasticSearch extended word library and build the ElasticSearch index. All compound participles in the compound participle set are the final calculation results. Set the compound participle set as the ElasticSearch extended word library and build the ElasticSearch index on this basis.

[0040] A multi-server-based traffic planning statistical data linkage retrieval method comprises the following steps:

[0041] Step 1: Obtain traffic data, process it into traffic planning statistics data according to the set format, and send it to the collection service cluster for storage;

[0042] Step 2: Read the traffic planning statistics data from the collection service cluster, extract the indicator dimension, time dimension and space dimension in the traffic planning statistics data and encode them, aggregate the time range and space list of the indicator dimension in real time, and send them to the storage service cluster and the search service cluster for storage respectively;

[0043] Step 3: The retrieval terminal obtains the business type, data source and input keywords selected by the user as search conditions and sends them to the search service cluster; the retrieval terminal obtains the user selected data and sends it to the storage service cluster

[0044] Step 4: The search service cluster generates a result data set based on the search conditions and feeds it back to the retrieval terminal;

[0045] Generate the primary key start and end range or complete primary key according to the query data and query method selected by the user, perform range query or precise query, and obtain the query results and feed them back to the search terminal;

[0046] Obtain the code of the indicator dimension, time dimension and space dimension of the corresponding data from the query data; if the query method is to obtain all the indicator data, the start and end values ​​of the query primary key are constructed according to the indicator dimension code, and the system searches for the data values ​​of all time and space of the indicator in the storage service cluster according to the primary key by range; if the query method is to obtain the indicator data of specified time and space, the complete primary key is spliced ​​to accurately search for the data value of the specified time and space of the indicator from the storage server according to the hash.

[0047] Through the above technical solutions, it can be known that compared with the prior art, the present invention discloses a multi-server-based traffic planning statistical data linkage retrieval system and method, by building a connection relationship between multiple hardware servers, that is, building a hardware carrier of a statistical data linkage retrieval system, so that the hardware carrier assists the relevant software functions to achieve the purpose of rapid retrieval, providing convenience for relevant staff. The present invention constructs a complete set of processing flows of collection, linkage distribution, storage, search, and retrieval, and proposes a compound segmentation construction algorithm based on traffic planning statistical data, which greatly improves the performance and accuracy of the search. The storage and retrieval of big data are separated, and the performance of accurate search and range search of big data is high and the speed is fast; the search server is flexible, can construct compound segmentation, and has a strong search function. The data linkage between forwarding and range aggregation is controlled by the linkage distribution server, which fully combines the characteristics of the two types of databases, greatly improves the system retrieval performance, and enhances the user experience. According to experiments, the system using this method is more than 10 times faster than directly querying from a traditional structured database. At the same time, the system of the present invention supports personalized search functions of index spatiotemporal aggregation and automatic construction of compound segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0049] Figure 1 The accompanying drawing is a schematic diagram of the structure of a multi-server-based traffic planning statistics data linkage retrieval system provided by the present invention;

[0050] Figure 2 The accompanying drawing is a schematic diagram of the server network deployment structure provided by the present invention;

[0051] Figure 3 The accompanying drawing is a flow chart of the algorithm for calculating compound word segmentation by the search service cluster provided by the present invention. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] Example 1

[0054] The embodiment of the present invention discloses a multi-server-based traffic planning statistics data linkage retrieval system and method, including a collection client, a collection service cluster, a linkage distribution server, a storage service cluster, a search service cluster and a retrieval terminal;

[0055] Collection client: users collect traffic data from the database or business system, and construct messages in the format of business type, data source, table name, indicator name, indicator unit, indicator value, data description, time type, data time, space type, space name, and location description. Then, messages are sent to the collection service cluster in the form of Kafka message middleware. Messages are partitioned by business type and data source.

[0056] The collection service cluster stores the data in partitions and batches into the storage medium.

[0057] The linkage distribution server is connected to the collection service cluster, including data reading, feature extraction, and data distribution modules; the feature extraction module extracts the business type, data source, table name, and indicator name in the data, and encodes them in 36-decimal in turn. The same value is encoded in the same way, and the 36-decimal system is based on 10 numbers and 26 letters. The string of business type code, data source code, table name code, and indicator name code is the indicator dimension code; extract the time type and data time to extract the time dimension and encode it, where the monthly type time dimension is encoded as YYYYMM, the quarterly type time dimension is encoded as YYYYQN, and the N value range is 1-4, representing 4 seasons, and the year type time dimension is encoded as YYYY; and the time range of the data is aggregated in real time, and different times are separated by ",". The spatial type, spatial name, and location description are extracted as spatial dimensions and encoded. The spatial name of the administrative division type uses the administrative division code of the Ministry of Civil Affairs, the spatial names of roads, railways, water transport, and civil aviation use industry codes, and the linear spatial data is encoded as the start and end point pile numbers in the location description of the industry code splicing. The format is as follows CODE_KXXX.XXX-KXXX.XXX; the spatial name list and code list of the read data are aggregated in real time. The positions of the name list and the code list correspond one to one, and different names or codes are separated by ",".

[0058] As the time dimension coding is highly readable, the time name list is no longer stored separately, where the monthly range is such as "201001, 201912, ..., 202101", the quarterly range is such as "2010Q1, 2019Q4, ..., 2020Q2", and the annual range is such as "2010, 2019, ..., 2022"; the spatial dimension is extracted and encoded according to the spatial type, spatial name, and location description, and the spatial name list and code list are aggregated in real time. The name list and the code list correspond one to one, and different names or codes are not stored separately. The codes are separated by ",", such as the administrative division space name list "Beijing, Shanghai, ..., Hebei", and the corresponding space dimension code list "110000, 310000 ..., 130000", the space name list of linear data and the corresponding space dimension code list are such as "Beijing-Harbin Expressway Sifang Bridge to Beijing-Hebei Border of Beijing-Harbin Expressway, Beijing-Hebei Border of Beijing-Harbin Expressway to Beijing-Hebei Border of Beijing-Harbin Expressway", and the corresponding space dimension code list is "G1K000.000-G1K41.091, G1K41.091-G1K63.303". Then the original data and the extracted dimension data are distributed to the storage service cluster and the search service cluster for storage.

[0059] Furthermore, the data reading module is used to read different categories in the collection service cluster in parallel, and read traffic planning statistics data of the same category sequentially. It is implemented using multiple clients of the same group. One client corresponds to one partition, and each partition has only one client reading sequentially, achieving concurrency between partitions and order within partitions.

[0060] Furthermore, a feature extraction module is provided, including extracting indicators, and encoding the three dimensions of time and space; aggregating the time dimension and the space dimension according to the indicators; and automatically obtaining the subject classification labels of the indicators.

[0061] Furthermore, the method for extracting the indicator dimension in the feature extraction module is to encode the business type, data source, chapter name, table name, indicator name, and so on in the data in sequence. The fields with the same value use the same 36-decimal encoding. The length and range of each field encoding are shown in Table 1. Since the granularity of traffic planning statistics is relatively coarse, generally only the month is required. The method for extracting the time dimension is to obtain the information of time, year, season or year from the data. The time dimension encoding composed of year and month is in the format of YYYYMM, the time dimension encoding composed of year and season is in the format of YYYYQN, and the time dimension encoding composed of year is in the format of YYYY; the method for extracting and encoding the spatial dimension is to encode the three fields of spatial type, spatial name, and location description and the spatial dimension encoding composed of the three fields of information encoded step by step, as shown in Table 2:

[0062] Table 1 Indicator dimension coding range

[0063]

[0064] Table 2 Spatial dimension encoding range table

[0065]

[0066] Furthermore, aggregating the time dimension and the space dimension by indicator refers to aggregating the time and space range of each indicator. The method of aggregating by time is to separate different times with "," , such as the monthly time range "201001, 201912, ..., 202101", the quarterly time range is "2010Q1, 2019Q4, ..., 2020Q2", and the annual range is "2010, 2019, ..., 2022". The method of aggregating the spatial dimension code is to aggregate the code of the spatial dimension corresponding to the indicator into one field In the code, they are separated by ",", such as the administrative division space name list "Beijing, Shanghai, ..., Hebei", and the corresponding space dimension code list "110000, 310000..., 130000", the space name list of linear data and the corresponding space dimension code list are such as: "Beijing-Harbin Expressway Sifang Bridge to Beijing-Harbin Expressway Beijing-Hebei border, Beijing-Harbin Expressway Beijing-Hebei border to Beijing-Harbin Expressway Hebei-Tianjin border", and the corresponding space dimension code list is "G1K000.000-G1K41.091, G1K41.091-G1K63.303".

[0067] Furthermore, the linkage distribution server includes a data distribution module, which is to send the extracted indicators, time, space dimension information and encoding, the indicator information and the original statistical data to the storage service cluster first, and then send the indicator dimension data and the time and space aggregation information of the indicator to the search service cluster. After both service clusters are sent successfully, the read position of each category is updated.

[0068] The storage service cluster is connected to the linkage distribution server, and is used to store traffic planning statistical data in a distributed file cluster in a dictionary sequence according to the indicator dimension coding, time dimension coding, and space dimension coding as the primary key, and provide a hash index based on the primary key to facilitate quick search based on the primary key.

[0069] The search service cluster is connected to the linkage server to provide accurate search of business type and data source, as well as fuzzy search of table name, indicator name, time and space;

[0070] Further, the search service cluster includes a compound word segmentation module and a search module;

[0071] The so-called compound participle refers to the basic phrases after the ik_smart word segmentation of the field to be fuzzy searched, the word2vec tool is used to construct the word segmentation feature matrix of these phrases, and then the basic phrases whose adjacent basic phrase vector similarity is greater than 0.7 are combined into compound participles.

[0072] The retrieval terminal is connected to the search service cluster and is used to retrieve statistical indicators from the search service cluster and accurately or range-search for detailed information of corresponding indicators from the storage service cluster.

[0073] Furthermore, the process of constructing compound participles by the compound participle module is as follows:

[0074] S1: Summarize all indicator names and table names and remove duplicates;

[0075] S2: Use ik_smart word segmentation to split all the index names and table names after deduplication into a word segmentation array list, where each word segmentation array corresponds to a table name or index name;

[0076] S3: Use doc2vec to construct a word segmentation feature matrix based on the word segmentation array list, and create a compound word segmentation construction list and a load word segmentation set based on the word segmentation feature matrix; the vector dimension of the word segmentation feature matrix is ​​300, the window size is 3, and the minimum ignored total frequency is 1;

[0077] S4: loop through the word segmentation array list and each word segmentation in each word segmentation array;

[0078] S5: If the word segmentation construction list is empty, the word segmentation of the current cycle is added to the compound word segmentation construction list; otherwise, it is determined whether the vector similarity between the current cycle word segmentation and the last element in the compound word segmentation construction list is greater than the set threshold 0.7. If so, the word segmentation of the current cycle is added to the compound word segmentation construction list, otherwise, it enters S6;

[0079] S6: Determine whether the number of elements in the compound participle construction list is greater than 1. If so, concatenate the compound participle construction list into a compound participle and add the compound participle to the compound participle set; clear the compound participle construction list, return to S4 and continue the loop until all participles are traversed;

[0080] S7: All compound participles in the compound participle set are the final calculation results. The compound participle set is set as the ElasticSearch extended vocabulary, and the ElasticSearch index is built on this basis.

[0081] Further, the retrieval terminal includes a retrieval client and a query client;

[0082] The retrieval client is used to accurately match the index business type and data source from the search service cluster; fuzzy search results from the table name, index name, time range, and space name range based on keyword information and display them;

[0083] The query client is used by the user to select the specified indicator and time and space information, construct the primary key start and end value range according to the indicator dimension code to search for all time and space detailed information of the indicator, or splice the complete primary key according to the indicator dimension code, time dimension code, and space dimension code to accurately find the indicator data details from the storage server according to the Hash.

[0084] Example 2

[0085] like Figure 1 As shown, a multi-server-based linkage retrieval system and method for traffic planning statistics data of this embodiment is applied in the comprehensive traffic space planning analysis and data service platform (TranSPAD). The system includes a collection client 100, a collection service cluster 200, a linkage distribution server 300, a storage service cluster 400, a search service cluster 500 and a retrieval terminal 600; the collection client 100, the collection service cluster 200, the linkage distribution server 300, the search service cluster 500 and the retrieval terminal 600 are connected in sequence; and the linkage distribution server 300 and the retrieval terminal 600 are connected to the storage service cluster 400. The server configuration in this example is shown in Table 3, and the network connection is as shown in Table 3. Figure 2 shown.

[0086] Table 3 Server configuration table

[0087]

[0088]

[0089] The acquisition client 100 and the acquisition service cluster 200 of this embodiment are implemented using the kafka message middleware method. The acquisition client 100 collects data in the format described in Example 1, and partitions it according to the business type and data source. The acquisition service cluster 200 accumulates the collected data into the memory by partition according to the business type and data source, and then synchronizes it to the permanent storage medium in batch order; the permanent storage medium includes a solid-state hard disk, a mechanical disk, a disk array, and a cloud hard disk.

[0090] The linkage distribution server 300 of this embodiment reads traffic planning statistical data from the collection service cluster 200 through the kafka grouping client via the data reading module. Each client in the group corresponds to a partition. According to the description in Example 1, the indicator dimension, time dimension, and space dimension are extracted from the data, and are encoded in sequence according to the data source, table name, and indicator name level, and then spliced ​​into the indicator dimension code. The time dimension and space dimension of each indicator are aggregated according to the indicator in real time, and distributed to the storage service cluster 400, and the distribution sequence number is recorded. If it fails, it is retransmitted; after successful distribution to the storage service cluster 400, it is distributed to the search service cluster 500, and if it fails, it is retransmitted; this can avoid the situation where the detailed data can be retrieved but cannot be queried.

[0091] The storage service cluster 400 of this instance mainly stores and searches for comprehensive transportation planning statistical data details by precise primary key and by primary key dictionary sequence range. Precise search includes finding all data details of a certain indicator in a certain area in a certain year and month by primary key; and searching for data details of a certain indicator in a certain area in a certain year and month by the start and end range of the indicator dimension code.

[0092] The search service cluster 500 of this embodiment is used to construct compound participles and store indicator retrieval information synchronized from the linkage distribution server 300, such as table name, indicator name, time convergence range and space convergence range; specifically, the search service cluster includes a compound participle module and a search module; the search server uses ElasticSearch to implement participle-based retrieval, and supports customized matching participles to improve the accuracy of the search, such as ik_smart participles decompose highway mileage into highway and mileage, ElasticSearch's basic participles will search separately by highway and mileage, and the customized compound participle can intelligently identify highway mileage as a compound participle according to the closeness of the participle connection, and can search according to the overall word of highway mileage; the search module is used to use the ElasticSearch capability to perform precise matching according to business type and data source, and to use compound participles to fuzzy match table names, indicator names, time ranges and space ranges according to user-input keywords, and return to the search terminal after sorting according to the degree of matching.

[0093] The retrieval terminal 600 of this embodiment is used to fuzzily search the corresponding indicator list from the search service cluster 500 according to the business type and data source selected by the user, and the input keywords from the table name, indicator name time range, and space name range; construct the storage server primary key range or complete primary key according to the indicator and data range selected by the user, and range query or accurately search the indicator data details from the storage server 300.

[0094] The technical solution of this embodiment helps to separate the collection, storage and retrieval of big data by constructing a hardware server of the statistical data linkage retrieval system, thereby helping to make the accurate search and range search of big data high in performance and speed; the search server is flexible and can segment words, especially custom compound segment words, and has strong search functions. This solution helps to achieve data linkage between the two by constructing a hardware carrier for the distribution server to control forwarding and linkage servers, and fully combines the characteristics of the two types of databases, greatly improving the system retrieval performance, thereby improving the user experience.

[0095] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0096] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multi-server-based traffic planning statistics data linkage retrieval system, characterized in that: It includes collection client, collection service cluster, linkage distribution server, storage service cluster, search service cluster and retrieval terminal; The collection client obtains traffic data, processes it into traffic planning statistics data according to a set format, and sends it to the collection service cluster in the form of Kafka messages; The collection service cluster stores the traffic planning statistics data in the form of a Kafka service cluster; The linkage distribution server reads the traffic planning statistics data from the collection service cluster; extracts the indicator dimension, time dimension and space dimension from the read data, and encodes them respectively; performs spatiotemporal aggregation of the time dimension, space dimension and their encoding according to the indicator dimension, and obtains a time range encoding list, a space range name and a corresponding space encoding list; distributes the indicator dimension, time dimension, space dimension, the encoding and data values ​​corresponding to each dimension to the storage service cluster; distributes the time range encoding list, space range name and the corresponding space encoding list obtained by the spatiotemporal aggregation of the indicator dimension to the search service cluster; The retrieval terminal obtains the search conditions and sends them to the search service cluster, and receives the returned result data set for display; obtains the user selection data, performs range query or precise query, and receives the returned query results for display; The search service cluster inserts or updates the results of spatiotemporal aggregation according to the indicator dimension, performs complete matching according to the search conditions, implements fuzzy search, returns the matched result data set to the search terminal, and displays the matched indicator and the spatiotemporal range corresponding to the indicator; The storage service cluster incrementally stores the indicator dimension, time dimension and space dimension and the corresponding codes, stores the data value, the corresponding indicator code, time code and space code, and the primary key formed by splicing the codes corresponding to the three dimensions; supports matching the primary key start and end range with the stored primary key, range querying the corresponding data set, obtaining the query result and returning it to the retrieval terminal, or matching the complete primary key with the stored primary key, accurately querying the corresponding data value, obtaining the query result and returning it to the retrieval terminal; The linkage distribution server includes a data reading module, a feature extraction module and a data distribution module; The data reading module reads from different partitions in the collection service cluster in parallel, and reads traffic planning statistics data in the same partition in sequence. It is implemented using multiple clients in the same group, one client corresponds to one partition, and each partition has only one client reading sequentially, realizing concurrency between partitions and orderly reading within a partition; The feature extraction module extracts the data values ​​of the indicator dimension, time dimension and space dimension in the traffic planning statistical data, wherein the business type, data source, table name and indicator name are the indicator dimension; the time type and data time are the time dimension; the space type, space name and location description are the space dimension; the three dimensions are encoded respectively to obtain the indicator code, time code and space code, and at the same time, the time range code list, space range name and corresponding space code list corresponding to the data are aggregated according to the indicator dimension; The data distribution module sends the indicator dimension, time dimension and space dimension, indicator code, time code and space code, and the data values ​​corresponding to the three dimensions to the storage service cluster; and sends the time range code list, space range name and corresponding space code list to the search service cluster.

2. The multi-server-based traffic planning statistics data linkage retrieval system according to claim 1 is characterized in that: The client collects traffic data from a database or business system, constructs a Kafka message in the format of 12 fields including business type, data source, table name, indicator name, indicator unit, indicator value, data description, time type, data time, space type, space name, and location description, sets partitions according to business type and data source, and sends the message to the collection service cluster.

3. The multi-server-based traffic planning statistics data linkage retrieval system according to claim 1 is characterized in that: The collection service cluster saves the traffic planning statistics data in a storage medium in a partitioned batch order.

4. The multi-server-based traffic planning statistics data linkage retrieval system according to claim 1 is characterized in that: The retrieval terminal includes a search client and a query client; The search client obtains the business type, data source, and keywords set by the user as search conditions and sends them to the search service cluster; The query client performs range query or precise query according to the query data and query method selected by the user. The query methods include querying all indicator data and querying the specified time and space data of the indicator. For querying all indicator data corresponding to the range query, the current indicator code and the next indicator code corresponding to the indicator in the query data are used as the primary key start and end range, and sent to the storage service cluster for range query. The data values ​​of all time dimensions and space dimensions corresponding to the primary key dictionary sequence range are queried from the storage service cluster to obtain the query results. The query indicator specifies the time and space data for precise query. The indicators, time, and space in the query data are encoded and spliced ​​into a complete primary key, which is sent to the storage service cluster for precise query. The data value of the specified time and the data value of the specified space of the corresponding indicator are accurately found to obtain the query results.

5. The multi-server-based traffic planning statistics data linkage retrieval system according to claim 4 is characterized in that: The storage service cluster uses the string of codes in the indicator code list, time code list and space code list as the primary key; Store the three dimensions of traffic planning statistics data and other information in the traffic planning statistics data except the extracted dimensions, including indicator units, indicator values, data descriptions, and associated indicator codes, time codes, and space codes; query the corresponding data set according to the start and end range of the primary key, form the search results and return them to the search terminal, or accurately query the data value according to the complete primary key, fill in the data values ​​corresponding to the indicator dimension, time dimension, and space dimension according to the corresponding indicator dimension code, time dimension code, and space dimension code, form the search results and return them to the search terminal; The storage service cluster stores data in a distributed file cluster according to the primary key dictionary sequence, and provides a Hash index based on the primary key, a Hash fast query based on the primary key, and a range query based on the dictionary sequence.

6. The multi-server-based traffic planning statistics data linkage retrieval system according to claim 4 is characterized in that: The data fields stored in the search service cluster include business type, data source, table name, indicator name, indicator unit, indicator dimension code, as well as time range code list, space range name and corresponding space code list; The search service cluster includes a compound word segmentation module and a search module; The compound word segmentation module calculates the compound words contained in all table names and indicator names; constructs a feature vector matrix after basic word segmentation of the search conditions, compares the basic word segmentations in a circular order to obtain the vector similarity between different basic word segmentations, and forms compound word segmentations with basic word segmentations whose vector similarity among adjacent basic word segmentations is greater than the set threshold, and establishes an ElasticSearch index based on the compound word segmentations; The search module supports the retrieval terminal to use the search method provided by ElasticSearch to completely match the business type and data source set by the user, and to perform fuzzy search on the index constructed by compound participles from the table name and indicator name fields using the keywords entered by the user, and to form a result data set in reverse order according to the matching score and send it to the retrieval terminal.

7. The multi-server-based traffic planning statistics data linkage retrieval system according to claim 6 is characterized in that: The process of constructing compound participles by the compound participle module is as follows: Step 1: Summarize all indicator names and table names and remove duplicates; Step 2: Use ik_smart word segmentation to split all the index names and table names after deduplication into a word segmentation array list, where each word segmentation array in the word segmentation array list corresponds to a table name or index name; Step 3: Use doc2vec to construct a word segmentation feature matrix based on the word segmentation array list, and create a compound word segmentation construction list and a compound word segmentation set based on the word segmentation feature matrix; Step 4: Loop through the word segmentation array list and each word in each word segmentation array; Step 5: If the compound participle construction list is empty, add the participle of the current cycle to the compound participle construction list; otherwise, determine whether the vector similarity between the current cycle participle and the last element in the compound participle construction list is greater than the set threshold. If so, add the participle of the current cycle to the compound participle construction list, otherwise proceed to step 6; Step 6: Determine whether the number of elements in the compound participle construction list is greater than 1. If so, concatenate all the participles in the compound participle construction list into a compound participle, add the compound participle to the compound participle set, clear the compound participle construction list, and return to step 4 to continue the loop until all participles are traversed; Step 7: Set all compound participles in the compound participle set to the ElasticSearch extended vocabulary and build an ElasticSearch index.

8. A multi-server-based traffic planning statistical data linkage retrieval method, characterized in that: The method is used for the multi-server-based traffic planning statistics data linkage retrieval system according to any one of claims 1 to 7, comprising the following steps: Step 1: Obtain traffic data, process it into traffic planning statistics data according to the set format, and send it to the collection service cluster for storage; Step 2: Read the traffic planning statistics data from the collection service cluster, extract the indicator dimension, time dimension and space dimension in the traffic planning statistics data and encode them, aggregate the time range and space list of the indicator dimension in real time, and send them to the storage service cluster and the search service cluster for storage respectively; Step 3: The search conditions or user selection data are obtained on the retrieval terminal and sent to the search service cluster and the storage service cluster respectively; Step 4: The search service cluster generates a result data set based on the search conditions and feeds it back to the retrieval terminal; the storage service cluster generates a primary key start and end range or a complete primary key based on the query data and query method, performs a range query or a precise query, obtains the query results and feeds them back to the retrieval terminal.

Images