Massive data flexible query method and system

Abstract

The application provides a mass data flexible query method and system, the method comprises the following steps: based on the data query request of a user, obtaining the index configuration information corresponding to the data query request; inputting the user data query request into a split data node and pushing through an interactive thread; based on the service instance in a healthy state corresponding to the split data node and the data query request of the user pushed by the interactive thread, querying a pre-created Lucene index file to obtain the Lucene index file corresponding to the split data node and the index configuration information; based on the Lucene index file, obtaining a data file and a column storage file, integrating the index configuration information corresponding to the split data node and the data file and the column storage file to obtain a query result. The application takes the Lucene index file as the underlying data storage, has high query efficiency, small resource consumption, and realizes flexible, multi-dimensional and fast query of mass data.

Description

Technical Field

[0001] This invention relates to the field of data query technology, and in particular to a method and system for flexible querying of massive amounts of data. Background Technology

[0002] Currently, massive data queries in the industry are usually based on database queries, which is slow for querying billions of data points and slow for writing the results of billions of data points to files.

[0003] In the insurance industry, data on policy underwriting, claims, receivables, and reinsurance are distributed across various systems. To check policy information, relevant data needs to be retrieved from multiple systems.

[0004] In addition, data generally belongs to company assets, and permissions need to be strictly defined when using the data. Permissions also need to be defined when querying the database. Due to the influence of permissions, the query results will be incomplete or incomplete.

[0005] Because the data in each system is an independent data source, it cannot be directly used by business personnel who are not in R&D positions. Furthermore, the mapping relationships between the systems are complex, and even professional technicians need to spend a lot of time sorting out the relationships and writing query statements. Summary of the Invention

[0006] This invention provides a flexible query method and system for massive data, which solves the problems of incomplete and incomplete query results and long time consumption caused by the influence of permissions and data sources when querying massive data through databases in the existing technology. It improves the applicability of massive data query and enhances the accuracy of query results.

[0007] This invention provides a method for flexible querying of massive amounts of data, including:

[0008] Based on the user's data query request, obtain the index configuration information corresponding to the data query request; the index configuration information includes the number of data splits, the data splitting rules, and the names of the split data nodes.

[0009] User data query requests are input into the split data nodes and pushed through the interaction thread; the interaction thread is generated by starting threads corresponding to the number of data splits, and establishing the same number of connections corresponding to the split data nodes in each thread;

[0010] Based on the user's data query request pushed by the service instance and interaction thread corresponding to the split data node that is in a healthy state, the pre-created Lucene index file is queried to obtain the Lucene index file and index configuration information corresponding to the split data node; based on the Lucene index file corresponding to the split data node, the data file and columnar storage file corresponding to the Lucene index file corresponding to the split data node are obtained.

[0011] The query results are obtained by integrating the index configuration information corresponding to the split data nodes, the data files, and the columnar storage files.

[0012] According to the flexible query method for massive data provided by the present invention, before obtaining the index configuration information corresponding to the user's data query request, the method further includes:

[0013] The system verifies whether a user's data query request is valid by using security authentication services such as user authentication and permission authentication.

[0014] According to the flexible query method for massive data provided by the present invention, the index configuration is established through the following steps:

[0015] S31. Configure the host IP, port number, database name, user, password, and other information required to access the database where the data tables generated by the background processing are located, and complete the data source configuration;

[0016] S32. Select the data table generated by the backend processing from the data source configured in S31, and configure the physical name and Chinese name of the table fields to complete the dataset configuration;

[0017] S33. Based on S32, configure the dataset initialization index file name and field list, configure the basic information related to data storage, display and query of each field in Lucene, and complete the basic index information configuration.

[0018] S34. Split the index data to obtain the relevant configuration for data distribution to each data node, and complete the index distribution configuration.

[0019] According to the flexible query method for massive data provided by the present invention, the data table generated by the background processing includes personalized dimension data and public dimension data.

[0020] According to the flexible query method for massive data provided by the present invention, after step S34 is completed, it further includes:

[0021] The index configuration results are stored in data containers, including: GBase and HBase, used for offline analysis, batch processing, and other scenarios; real-time Kafka, used for real-time analysis, online processing, and other scenarios; and near real-time ElasticSearch, with minute-level latency, used for regulatory reporting, operational monitoring, and other scenarios.

[0022] According to the flexible query method for massive data provided by this invention, an index file is created, including:

[0023] S61. Obtain index configuration information, which includes index name, field list, splitting rules, splitting data storage nodes, and data dictionary configuration.

[0024] S62. Based on the data node names obtained in S61, retrieve the service instances in a healthy state corresponding to each data node from the registry center;

[0025] S63. Start the corresponding number of threads according to the number of data splits obtained in S61. Each thread initializes the storage space for temporarily storing the data allocated to the node, establishes a connection with the corresponding data node, and pushes the temporarily stored data to its corresponding data node in an orderly manner.

[0026] S64. Based on the configuration information obtained in S61, determine whether the dictionary is enabled for the current index. If the dictionary is enabled, proceed to S65; otherwise, proceed directly to S66.

[0027] S65. Query dictionary data from the data source configured in the current index and save it as a Lucene index file to generate a dictionary index;

[0028] S66. Generate query SQL statements based on the configured index information, obtain relevant data from the configured data source, distribute the data to the storage space created by S63 for data interaction threads with each data node according to the configured data allocation rules, and push the data to its corresponding data node through the interaction thread.

[0029] S67. Start each data node and obtain the relevant configurations for writing Lucene index files from the configuration center.

[0030] According to the flexible query method for massive data provided by the present invention, the Lucene index file related configuration includes the index file storage path, data compression method, cache size, and merging strategy.

[0031] According to the flexible query method for massive data provided by the present invention, step S67 further includes:

[0032] After receiving data pushed by the interactive thread, write it into the corresponding index file according to the field types pre-configured in the configuration phase;

[0033] Data used as query conditions is written to Lucene's inverted index file, while data used as query results is written to Lucene's data file. Numeric data is saved to a columnar storage file.

[0034] According to the flexible query method for massive data provided by the present invention, the column storage file adopts Lucene's SortedNumericDoc Values.

[0035] According to the flexible query method for massive data provided by the present invention, step S67 further includes:

[0036] If any data is written incorrectly during the process of correctly writing all data retrieved from the data source into the Lucene index file, the written data will be deleted and the index creation process will be restarted.

[0037] According to the flexible query method for massive data provided by the present invention, the data dictionary configuration includes:

[0038] The data dictionary configuration is completed by initializing dictionary content through the dictionary management page or by querying external data sources using SQL statements.

[0039] According to the flexible query method for massive data provided by the present invention, after obtaining the query results from the data file and columnar storage file corresponding to the Lucene index file based on the Lucene index file, the method further includes:

[0040] The query results returned by each split data node are integrated, and the integrated data is returned to the front-end query page.

[0041] This invention also provides a flexible query system for massive amounts of data, including:

[0042] The acquisition module is used to acquire the index configuration information corresponding to the user's data query request; the index configuration information includes the number of data splits, the data splitting rules, and the names of the split data nodes;

[0043] The request sending module is used to input user data query requests into the split data nodes and push them through the interaction thread; the interaction thread is generated by starting threads corresponding to the number of data splits and establishing the same number of connections corresponding to the split data nodes in each thread.

[0044] The query module is used to query the pre-created Lucene index file based on the user's data query request pushed by the service instance and interaction thread corresponding to the split data node, and obtain the Lucene index file and index configuration information corresponding to the split data node; based on the Lucene index file corresponding to the split data node, obtain the data file and columnar storage file corresponding to the Lucene index file corresponding to the split data node.

[0045] The output module is used to integrate the index configuration information corresponding to the split data nodes, the data files, and the columnar storage files to obtain the query results.

[0046] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the flexible query method for massive data as described above.

[0047] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method for flexibly querying massive amounts of data as described above.

[0048] The proposed method and system for flexible querying of massive data adopts the Lucene service group, uses the Lucene index file as the underlying data storage, and wraps the data query and write processing on it as the platform's basic data service. It has high query efficiency and low resource consumption, and realizes flexible, multi-dimensional and fast querying of massive data. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0050] Figure 1 This is a flowchart illustrating the flexible query method for massive data provided in an embodiment of the present invention;

[0051] Figure 2 This is a functional structure diagram of the flexible query system for massive data provided in the embodiments of the present invention;

[0052] Figure 3 This is a schematic diagram of the configuration index process provided in an embodiment of the present invention;

[0053] Figure 4 This is a schematic diagram of the index creation process provided in an embodiment of the present invention;

[0054] Figure 5 This is a schematic diagram of a user query page provided in an embodiment of the present invention;

[0055] Figure 6 This is a schematic diagram of the user query process provided in an embodiment of the present invention;

[0056] Figure 7 This is a schematic diagram of the structure of the flexible query system for massive data provided in an embodiment of the present invention;

[0057] Figure 8 This is an architecture diagram of the flexible query system for massive data provided in this embodiment of the invention;

[0058] Figure 9 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0060] In the industry, data on policy underwriting, claims, receivables, and reinsurance are distributed across various systems. To query policy information, it's necessary to retrieve relevant data from multiple systems. This technology uses an ETL process to integrate data from multiple systems, allowing for the querying of multi-dimensional policy data from a single system.

[0061] In the industry, most massive data queries are based on database queries, which are slow for querying billions of records. This technology leverages the high query efficiency and low resource consumption of Lucene, creating an inverted index for fields that can be used as query conditions in Lucene to achieve sub-second queries for billions of records.

[0062] In the industry, massive data queries are based on database queries, and writing the results of queries involving billions of data points to files is slow. This technology fully leverages the advantages of distributed systems, where each data node independently writes the query results to a file, which is then handed over to the query node to merge the files. This results in a file download speed improvement that is almost equivalent to the increase in the number of data nodes.

[0063] Data is a company asset, and access control needs to be strictly defined when using data and when querying the database. In this technology, access control issues can be addressed through several aspects, including user role configuration, user's province (city / county) company, user's department (customer group), and user's searchable indexes. The next step will be to control data access control down to the field level.

[0064] Because the data in each system is an independent data source, it cannot be directly used by business personnel (people who do not understand technology). Furthermore, the mapping relationships between the systems are complex, and even professional technicians need to spend a lot of time sorting out the relationships and writing query statements.

[0065] To address the aforementioned technical problems, this invention provides a method, system, and electronic device for flexible querying of massive amounts of data. By integrating data from various systems through ETL processing, a simple categorized selection interface is provided on the front-end page. Users only need to enter the query fields to obtain the corresponding results, thereby solving the problem that ordinary business personnel find it difficult to query data across platforms.

[0066] Figure 1 This is a flowchart illustrating the flexible query method for massive data provided by the present invention, as shown below. Figure 1 As shown, the method includes the following steps:

[0067] Step 110: Based on the user's data query request, obtain the index configuration information corresponding to the data query request; the index configuration information includes the number of data splits, the data splitting rules, and the names of the split data nodes;

[0068] Step 120: Input the user data query request into the split data node and push it through the interaction thread; the interaction thread is generated by starting the thread corresponding to the number of data splits, and establishing the same number of connections corresponding to the split data node in each thread;

[0069] Step 130: Based on the user's data query request pushed by the service instance and interaction thread corresponding to the split data node that is in a healthy state, query the pre-created Lucene index file to obtain the Lucene index file and index configuration information corresponding to the split data node; based on the Lucene index file corresponding to the split data node, obtain the data file and columnar storage file corresponding to the Lucene index file corresponding to the split data node.

[0070] Step 140: Integrate the index configuration information corresponding to the split data nodes, the data files, and the columnar storage files to obtain the query results.

[0071] Figure 2 This is a functional structure diagram of a flexible query platform for massive data provided in an embodiment of the present invention, such as... Figure 2 As shown, the flexible query system is designed for querying massive amounts of detailed data in the insurance industry. The data scope covers 2 billion data points across the entire insurance product line and supports multiple data sources. This system provides a simple and easy-to-use detailed query page, allowing business personnel to accurately and quickly obtain detailed data through simple clicks and drag-and-drop. Its innovation lies in the use of the Lucene service suite. It uses Lucene index files (an index in Lucene can be understood as a table in a database, not an index created for a data table) as the underlying data storage, and wraps data query and write processing on top of this as the platform's basic data services. Leveraging Lucene's high query efficiency and low resource consumption, it achieves flexible, multi-dimensional, and rapid querying of massive amounts of data, visualization analysis, and file download functions.

[0072] The above steps will be explained in detail below with reference to specific embodiments.

[0073] Step 110: Based on the user's data query request, obtain the index configuration information corresponding to the data query request; the index configuration information includes the number of data splits, the data splitting rules, and the names of the split data nodes;

[0074] Specifically, Figure 3 This is a schematic diagram of the configuration index process provided in an embodiment of the present invention, such as... Figure 3 As shown, configuring index information mainly involves the following steps:

[0075] Step 1: Configure the data source: Using the data source management page provided by the data source management module, configure the host IP, port number, database name, user, password, and other information required to access the database where the data tables generated by the backend are located.

[0076] Step 2: Configure the dataset: Using the dataset management page provided by the dataset management module, select the data table generated by the backend processing from the data source configured in Step 1 (or write an SQL query) and configure the physical name and Chinese name of the table fields, etc.

[0077] Step 3: Configure the index: Configuring the index involves configuring the information needed to generate the Lucene index file. Using the index management page provided by the index management module, select the dataset configured in Step 2 to initialize the index file name and field list. Then, configure the storage type of each field's data in Lucene, whether it can be used for query conditions, whether it is displayed as a result, whether it is a dimension or a metric, the data alignment method and display format, as well as the corresponding data dictionary and basic information related to data storage, display, and querying.

[0078] Step 4, Index Distribution Configuration: After configuring the basic index information, the index data can be split according to the actual situation, i.e., the data distribution to various data nodes is configured. Each data node registers itself with the Nacos registry center upon startup (Step 4.1 in the diagram). When entering the index distribution configuration, it retrieves all registered data nodes from the registry center (Step 4.2 in the diagram), and then obtains information such as hardware configuration and remaining disk space of each node by accessing the service interfaces provided by each data node (Step 4.3 in the diagram). Users can configure the number of data splits as needed, distributing the data evenly to specified data nodes, or distributing it according to a hash method or a custom method. During subsequent index creation, the corresponding Lucene index files will be generated based on the splitting rules configured here and the data nodes corresponding to the split data.

[0079] Step 5: Save the index: After completing the above configuration, you can save it. You can start the index creation process below by configuring a scheduled task or manually clicking the "Create Index" button.

[0080] Optionally, the index configuration information data originates from the target user's internal database. For example, the data source for an insurance industry business system includes third-party underwriting, third-party claims, accounts receivable, reinsurance, payment and collection, and sales management systems, which are the sources of Lucene's query data. This data is processed by the data warehouse, which periodically extracts business system data to the Operation Data Store (ODS): the data preparation area. In the ODS data layer, data cleaning and normalization operations are performed, such as removing empty data, dirty data, and outliers. A General Semantic Layer (GSL) is built using a dimensional model. Based on the GSL, the basic data is integrated and summarized into a data service layer for the analytical subject domain, typically a wide table, used to provide subsequent business queries, such as non-motor insurance lists and group accident and health insurance lists.

[0081] Optionally, taking the insurance industry as an example, in a specific embodiment, the back-end data processing includes the following steps:

[0082] Step 1: Policy Scope: The policy scope includes all policies that have been insured in the three-generation underwriting system. Currently, incremental policies are extracted from the logs of the underwriting, claims, payment, and receivable systems on a T+1 basis.

[0083] Step 2: Premiums Receivable: Take the premiums receivable for the policies from Step 1 in the latest base date in the accounts receivable system. This indicator is a point-in-time value.

[0084] Step 3: Underwriting Indicators: Take the policy premium, policy sum assured, revised premium, revised sum assured, earned premium, and other underwriting-related indicators of the policy in Step 1 during the three-generation underwriting process.

[0085] Step 4: Retention Indicators: Take the reinsurance-related indicators from Step 1, such as the ceded premium, ceded ratio, retention amount, and retention ratio, in the reinsurance system.

[0086] Step 5 Claims Indicators: Take the claims-related indicators from Step 1, such as settled claims, outstanding claims, number of settled claims, and number of outstanding claims, in the third-party claims process.

[0087] Step 6: Collection and payment indicators: Take the relevant collection and payment indicators of the policy in Step 1, such as premium income, handling fees, and claims payouts in the collection and payment system.

[0088] Step 7 Common Dimensions: Take the underwriting dimensions of the policy from Step 1, such as: institution, type of insurance, start date, end date, channel, service manager, and other common dimensions.

[0089] Step 8: Wide Table: Take the indicators under the policy from Steps 2-6 and associate them with the common dimensions from Step 7 to finally generate a wide table for Lucene to use as a data source.

[0090] Figure 4 This is a schematic diagram of the index creation process provided in an embodiment of the present invention, such as... Figure 4 As shown, creating an index mainly involves the following steps:

[0091] Step 1: Obtain Index Configuration: When the index creation process is started, the index management service first obtains the index configuration information from the internal database of the system, including the index name, field list, splitting rules, split data storage nodes, and data dictionary configuration, as well as all the configurations required in the index creation process.

[0092] Step 2: Obtain data node services: Based on the data node names obtained in Step 1, retrieve the service instances in a healthy state corresponding to each node from the registry center.

[0093] Step 3: Start the index creation process: Start the corresponding number of threads according to the number of data splits obtained in Step 1. Each thread initializes the storage space for temporarily storing the data allocated to that node, establishes a connection with the corresponding data node, and prepares to push the temporarily stored data to its corresponding data node in an orderly manner.

[0094] Step 4: Enable dictionary? Based on the configuration information obtained in Step 1, determine whether the dictionary is enabled for the current index. If the dictionary is enabled, proceed to Step 5: Generate dictionary index processing. If the dictionary is not enabled, skip Step 5 and proceed directly to Step 6: Query data processing.

[0095] Step 5: Dictionary Index Generation. A dictionary is essentially a two-way key-value pair of numbers and text. For example, a provincial-level organization dictionary might contain "1: Beijing, 2: Tianjin, 3: Hebei... 36: Xinjiang". This system provides two types of dictionaries: one with fixed content, similar to provincial-level organizations, and another with variable content generated by querying different text content using SQL statements. Dictionary data is retrieved from the data source configured in the current index and saved as a Lucene index file for later use.

[0096] Step 6: Distribute query data to each data node: Generate query SQL statements based on the configured index information, obtain relevant data from the configured data source, and distribute the data to the storage space created in Step 3 for data interaction threads with each data node according to the configured data distribution rules. The data is then pushed to the corresponding data node through the interaction threads.

[0097] Step 7: Data Writing Processing: Upon startup, each data node retrieves its Lucene index file writing configuration from the configuration center, including the index file storage path, data compression method, cache size, and merging strategy. After receiving data pushed by the interaction thread, it writes the data to the corresponding index file according to the pre-configured field types. Data used as query conditions is written to Lucene's inverted index file, data used to display query results is written to Lucene's data file, and numeric data such as metric values ​​and dictionary ordinal numbers are saved to a columnar storage file (this system uses Lucene's SortedNumericDocValues). Index creation is complete only after all data retrieved from the data source in Step 6 has been correctly written to the Lucene index file. If any data writing fails during this process, the written data will be deleted, and the index creation process will be restarted to ensure that the data in the index file remains consistent with the data in the data source.

[0098] Understandably, a dictionary is a bidirectional key-value pair of numbers and text. Taking provincial-level organizations as an example, the dictionary content is "1: Beijing, 2: Tianjin, 3: Hebei... 36: Xinjiang". When writing to a Lucene file, the corresponding key-value pairs are obtained from the text and written to the Lucene index. During a query, the corresponding text is retrieved based on the key-value pairs and returned to the user. The dictionary key-value pairs are stored using Lucene's columnar storage method, SortedNumericDocValues, because this storage method has higher read efficiency compared to other data storage methods, which can significantly improve the speed of data aggregation and query. The following uses three dimensions commonly used in the insurance industry—provincial companies, insurance categories (hereinafter referred to as insurance categories), and primary channels (hereinafter referred to as channels)—to introduce how dictionaries are used in data aggregation and query. Table 1 shows some data for these three dimensions:

[0099] Table 1. Partial data from three dimensions

[0100]

[0101] After adopting the dictionary above, the data table changed from Table 2 to Table 3.

[0102] Table 2: Dimensional data before using the data dictionary

[0103] Beijing Corporate Property Insurance Traditional direct selling business Tianjin Home insurance Personal agency business Xinjiang Personal non-motor insurance Professional agency business …… …… ……

[0104] Table 3 shows the dimensional data after using the data dictionary.

[0105] 1 1 1 2 2 2 36 29 7 …… …… ……

[0106] As shown in Tables 2 and 3, the original three fields, each occupying over 20 bytes of string data, can be completely replaced by each field using only 1 byte after transformation. This improves efficiency for both data writing and reading. The only additional cost is the conversion from text to key-value pairs during Lucene index generation and the conversion from key-value pairs to text when displayed to the user; these two processes are negligible for querying and aggregation. The aggregation process based on these three dimensions has changed from comparing three strings to comparing three numbers. Although it's only comparing three numbers, it still increases processing time when dealing with large datasets. Therefore, this system further merges the data from these three dimensions during aggregation, as detailed below:

[0107] Provincial Company: There are 36 data entries in total, with a maximum value of 36. Converted to binary, this is "100100", occupying a maximum of 6 bits.

[0108] Risk category: 29 data entries in total, with a maximum value of 29, which is "11101" in binary, occupying a maximum of 5 bits;

[0109] Channel: There are 7 data entries in total, with a maximum value of 7, which is "111" in binary and occupies a maximum of 3 bits.

[0110] This shows that the three fields occupy only 14 bits in total, while an integer has 32 bits. Therefore, we can use bits 1-6 of an integer to store the provincial company ID, bits 7-11 to store the insurance type ID, and bits 12-14 to store the channel ID. Thus, the original 20+ bytes of three fields are reduced to a single integer. During aggregation, the calculation of the hash values ​​of three strings is eliminated, and the comparison of the three fields becomes a comparison of a single integer. The original 20+ bytes of data transfer between the data nodes is reduced to transferring only a single integer. Since the dimension is now a single integer, sorting and pagination processes become sorting and deduplication of integers, currently implemented using Bitmap, meaning that sorting and deduplication are already completed when the aggregation field data is extracted. Therefore, the aggregation efficiency is significantly improved after introducing the data dictionary.

[0111] It should be noted that the current flexible query system supports two types of data dictionaries: one is fixed or infrequently changed, which can be initialized through the dictionary management page or by querying external data sources using SQL statements; the other is changed with the data and is automatically generated based on the source data content before the index is generated.

[0112] Step 120: Input the user data query request into the split data node and push it through the interaction thread; the interaction thread is generated by starting the thread corresponding to the number of data splits, and establishing the same number of connections corresponding to the split data node in each thread;

[0113] In this step, Nacos is used to configure the number of nodes, service names, port numbers, data storage paths, data compression methods, and other settings for each node, as well as runtime configurations for each node's caching, logging, and timeout. It also enables automatic registration and discovery of services on each data node.

[0114] Step 130: Based on the user's data query request pushed by the service instance and interaction thread corresponding to the split data node that is in a healthy state, query the pre-created Lucene index file to obtain the Lucene index file and index configuration information corresponding to the split data node; based on the Lucene index file corresponding to the split data node, obtain the data file and columnar storage file corresponding to the Lucene index file corresponding to the split data node.

[0115] In this step, data from various systems is first aggregated into a wide table using ETL processing. Next, the connection method, table structure, and index file information for accessing this table are configured. Finally, the index creation process is initiated to generate the index file based on the above configuration. The data from various data sources is integrated to generate the wide table and written to the Lucene index file, allowing users to query the data they need through the front-end page.

[0116] Figure 5 This is a schematic diagram of a user query page provided in an embodiment of the present invention. Figure 6 This is a schematic diagram of the user query process provided in an embodiment of the present invention, such as... Figure 5 , 6 As shown, the process of a user querying data through the flexible query system includes:

[0117] Step 1: Customize query conditions and display fields: After logging into the system, users can open the customized query page (see the above user query page image for details). After entering the page, users can click the "Add" and "Edit" buttons in the upper right corner of the "Query Conditions" area to set the fields used as query conditions as needed, and click the "Edit" button in the upper right corner of the "Query Results" area to select the fields to be displayed in the query results.

[0118] Step 2: Send a query request: After completing Step 1 to customize the query, enter the conditions for this query and click the "Query" button above. The entered query information will be sent to the background for query processing. Wait for the query results to be returned.

[0119] Step 3: Verify the user request. Here, the security authentication information in the request header and the access data information in the request body are extracted from the front-end page request header. The request is verified to be valid through security authentication services such as user authentication and permission authentication. If the verification fails, an exception message is returned to the querying user. If the verification is successful, proceed to Step 4 to continue the query processing.

[0120] Step 4: Obtain Index Configuration: When the query service receives a valid data query request, it first obtains the index configuration information from the internal database of the system through the index management service. This includes all the configurations required during the index creation process, such as the index name, field list, splitting rules, split data storage nodes, and data dictionary configuration.

[0121] Step 5: Obtain data node services: Based on the data node names obtained in Step 4, retrieve the service instances in a healthy state corresponding to each node from the registry center.

[0122] Step 6: Start query data processing: Start the corresponding number of threads based on the number of data splits obtained in Step 4. Each thread establishes a connection with the corresponding data node and submits the user's query request to the corresponding data node.

[0123] Step 7: Data Query and Processing: When each data node starts up, it will obtain the relevant configurations of its Lucene index file from the configuration center, including the index file storage path, data compression method, cache size, merging strategy, etc.

[0124] S140. Integrate the index configuration information corresponding to the split data nodes, the data files, and the columnar storage files to obtain the query results.

[0125] In this step, after receiving the query request data pushed by the interaction thread, the Lucene API is called to query the Lucene index file, and the result data is obtained from the data file and column storage file respectively according to the field type, and the query result is returned to the caller;

[0126] Furthermore, the query results returned by each data node are integrated, and the integrated data is returned to the front-end query page.

[0127] Preferably, the user-customized query data is displayed to the user in the form of a table or chart, and the subscription can be completed by sending the results via email through scheduled queries; each user can only access the data that the administrator has granted permission for, and fields containing sensitive information will be displayed according to a pre-set masking method.

[0128] The following describes the flexible query system for massive data provided by the present invention. The flexible query system for massive data based on multimodality described below can be referred to in correspondence with the flexible query method for massive data described above.

[0129] Figure 7 This is a schematic diagram of the structure of the flexible query system for massive data provided in an embodiment of the present invention; see reference. Figure 7 This is a schematic diagram of the structure of the flexible query system for massive data provided in an embodiment of the present invention, as shown below. Figure 7 As shown, the system 700 includes:

[0130] The acquisition module 710 is used to acquire the index configuration information corresponding to the user's data query request based on the user's data query request; the index configuration information includes the number of data splits, the data splitting rules, and the names of the split data nodes.

[0131] The request sending module 720 is used to input user data query requests into the split data nodes and push them through the interaction thread; the interaction thread is generated by starting threads corresponding to the number of data splits and establishing the same number of connections corresponding to the split data nodes in each thread.

[0132] The query module 730 is used to query a pre-created Lucene index file based on the user's data query request pushed by the service instance and interaction thread corresponding to the split data node that is in a healthy state, to obtain the Lucene index file and index configuration information corresponding to the split data node; and to obtain the data file and columnar storage file corresponding to the Lucene index file corresponding to the split data node based on the Lucene index file corresponding to the split data node.

[0133] The output module 740 is used to integrate the index configuration information corresponding to the split data nodes, the data files, and the columnar storage files to obtain query results.

[0134] Figure 8 This is an architecture diagram of the flexible query system for massive data provided in the embodiments of the present invention, such as... Figure 8 As shown in one specific embodiment, the system is a distributed query system built on the Spring Cloud Alibaba microservice architecture. The basic services primarily provide management of user, department, and role information, data source and dataset information, and log information. The query service provides Lucene index file management, user-customized query management, and responding to query requests, aggregating data from each data node, and returning query results. The configuration center and registry center use Nacos, which not only provides a visual interface for managing configuration information and services but also includes both registry and configuration center functions. It can also integrate with SpringAdmin for status monitoring, simplifying the system architecture to some extent. Each data node in the system is an independent service, providing query, aggregation, and download functions. When a node starts, it obtains runtime information such as file storage and cache configuration from the configuration center and registers itself with the registry center for discovery and invocation by the query service. The relationships between the services will be described in detail later through aspects such as index generation and data querying.

[0135] The system implements the following functions:

[0136] System Management: This module firstly configures basic information such as departments, users, and permissions; secondly, it configures the data source connection for the database containing the wide tables generated by data modeling, i.e., data source management; and thirdly, it configures the table names or query SQL and table fields (including physical names, Chinese names, etc.) for each wide table, i.e., dataset management.

[0137] Index Management: The main function of this module is to generate Lucene index files for querying using the configured datasets corresponding to each wide table. It includes basic information configuration such as the display name, display format, dimensions, metrics, and query conditions for each field; configuration of the data splitting method, such as the number of data splits, or sequential or hash distribution; configuration of master-detail table relationships, such as the relationship between the policy table and the terms table; and configuration of the data update timing and viewing of update results.

[0138] Data Query: This module provides users with the ability to customize query conditions and results to perform data queries or summaries as needed. Users can also perform batch queries by uploading Excel files.

[0139] Configuration Management and Registration Center: This module mainly uses Nacos to configure the number of nodes, service names of each node, port numbers, data storage paths, data compression methods, etc., as well as runtime configurations such as cache, logs, and timeout for each node; and automatic registration and discovery of services for each data node.

[0140] Figure 9 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 9 As shown, the electronic device 900 may include: a processor 910, a communication interface 920, a memory 930, and a communication bus 940. The processor 910, communication interface 920, and memory 930 communicate with each other via the communication bus 940. The processor 910 can call logical instructions from the memory 930 to execute a method for flexible querying of massive amounts of data. This method includes:

[0141] Based on the user's data query request, the index configuration information corresponding to the data query request is obtained. The index configuration information includes the number of data splits, the data splitting rules, and the names of the split data nodes. The user's data query request is input into the split data node and pushed through an interaction thread. The interaction thread is generated by starting threads corresponding to the number of data splits and establishing the same number of connections to the split data node in each thread. Based on the service instance corresponding to the split data node that is in a healthy state and the user's data query request pushed by the interaction thread, the pre-created Lucene index file is queried to obtain the Lucene index file and index configuration information corresponding to the split data node. Based on the Lucene index file corresponding to the split data node, the data file and columnar storage file corresponding to the Lucene index file corresponding to the split data node are obtained. The index configuration information corresponding to the split data node, the data file, and the columnar storage file are integrated to obtain the query result.

[0142] Furthermore, the logical instructions in the aforementioned memory 930 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0143] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute a method for flexibly querying massive amounts of data, the method comprising:

[0144] Based on the user's data query request, the index configuration information corresponding to the data query request is obtained. The index configuration information includes the number of data splits, the data splitting rules, and the names of the split data nodes. The user's data query request is input into the split data node and pushed through an interaction thread. The interaction thread is generated by starting threads corresponding to the number of data splits and establishing the same number of connections to the split data node in each thread. Based on the service instance corresponding to the split data node that is in a healthy state and the user's data query request pushed by the interaction thread, the pre-created Lucene index file is queried to obtain the Lucene index file and index configuration information corresponding to the split data node. Based on the Lucene index file corresponding to the split data node, the data file and columnar storage file corresponding to the Lucene index file corresponding to the split data node are obtained. The index configuration information corresponding to the split data node, the data file, and the columnar storage file are integrated to obtain the query result.

[0145] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform a method for flexibly querying massive amounts of data, the method comprising:

[0146] Based on the user's data query request, the index configuration information corresponding to the data query request is obtained. The index configuration information includes the number of data splits, the data splitting rules, and the names of the split data nodes. The user's data query request is input into the split data node and pushed through an interaction thread. The interaction thread is generated by starting threads corresponding to the number of data splits and establishing the same number of connections to the split data node in each thread. Based on the service instance corresponding to the split data node that is in a healthy state and the user's data query request pushed by the interaction thread, the pre-created Lucene index file is queried to obtain the Lucene index file and index configuration information corresponding to the split data node. Based on the Lucene index file corresponding to the split data node, the data file and columnar storage file corresponding to the Lucene index file corresponding to the split data node are obtained. The index configuration information corresponding to the split data node, the data file, and the columnar storage file are integrated to obtain the query result.

[0147] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0148] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0149] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for flexible querying of massive amounts of data, characterized in that, include: Based on the user's data query request, obtain the index configuration information corresponding to the data query request; the index configuration information includes the number of data splits, the data splitting rules, and the names of the split data nodes. User data query requests are input into the split data nodes and pushed through the interaction thread; the interaction thread is generated by starting threads corresponding to the number of data splits, and establishing the same number of connections corresponding to the split data nodes in each thread; Based on the user's data query request pushed by the service instance and interaction thread corresponding to the split data node that is in a healthy state, the pre-created Lucene index file is queried to obtain the Lucene index file and index configuration information corresponding to the split data node; wherein, the Lucene index file is created based on the data table generated by the background processing, and the data table contains dimensional data; based on the Lucene index file corresponding to the split data node, the data file and columnar storage file corresponding to the Lucene index file corresponding to the split data node are obtained; The index configuration information corresponding to the split data nodes, the data files, and the columnar storage files are integrated to obtain the query results. Specifically, for multiple dimensions in the dimension data, the integration includes: mapping the dimension data of the multiple dimensions to corresponding dictionary IDs using a pre-generated numeric dictionary; merging and compressing the dictionary IDs of the multiple dimensions into different bits of the same integer using a bit-off method based on the number of binary bits occupied by the maximum value of each dimension data in the dictionary, to obtain a compressed dimension integer; and sorting and deduplicating the compressed dimension integer using a BitMap algorithm.

2. The method for flexible querying of massive data according to claim 1, characterized in that, Before obtaining the index configuration information corresponding to the user's data query request, the process further includes: The system verifies whether a user's data query request is valid by using security authentication services such as user authentication and permission authentication.

3. The method for flexible querying of massive data according to claim 1, characterized in that, The index configuration is established through the following steps: S31. Configure the host IP, port number, database name, user, password, and other information required to access the database where the data tables generated by the background processing are located, and complete the data source configuration; S32. Select the data table generated by the backend processing from the data source configured in S31, and configure the physical name and Chinese name of the table fields to complete the dataset configuration; S33. Based on S32, configure the dataset initialization index file name and field list, configure the basic information related to data storage, display and query of each field in Lucene, and complete the basic index information configuration. S34. Split the index data to obtain the relevant configuration for the data distribution to each split data node, and complete the index distribution configuration.

4. The method for flexible querying of massive data according to claim 3, characterized in that, The data tables generated by the backend processing include personalized dimension data and public dimension data.

5. The method for flexible querying of massive data according to claim 3, characterized in that, After S34 is completed, the following is also included: The index configuration results are stored in data containers, including: GBase and HBase, used for offline analysis, batch processing, and other scenarios; real-time Kafka, used for real-time analysis, online processing, and other scenarios; and near real-time ElasticSearch, with minute-level latency, used for regulatory reporting, operational monitoring, and other scenarios.

6. The method for flexible querying of massive data according to claim 1, characterized in that, Create an index file, including: S61. Obtain index configuration information, which includes index name, field list, number of data splits, data splitting rules, split data node names, and data dictionary configuration. S62. Based on the split data node names obtained in S61, retrieve the service instances in a healthy state corresponding to each split data node from the registry center. S63. Start the corresponding number of threads according to the number of data splits obtained in S61. Each thread initializes the storage space for temporarily storing the data allocated to the split data node, establishes a connection with the corresponding split data node, and pushes the temporarily stored data to its corresponding split data node in an orderly manner. S64. Based on the configuration information obtained in S61, determine whether the dictionary is enabled for the current index. If the dictionary is enabled, proceed to S65; otherwise, proceed directly to S66. S65. Query dictionary data from the data source configured in the current index and save it as a Lucene index file to generate a dictionary index; S66. Generate query SQL statements based on the configured index information, obtain relevant data from the configured data source, distribute the data to the storage space created by S63 for interacting with the data threads of each split data node according to the configured data allocation rules, and push the data to its corresponding split data node through the interaction thread. S67. Start each split data node and obtain the relevant configurations for writing the Lucene index file from the configuration center.

7. The method for flexible querying of massive data according to claim 6, characterized in that, The Lucene index file configuration includes the index file storage path, data compression method, cache size, and merging strategy.

8. The method for flexible querying of massive data according to claim 6, characterized in that, The S67 further includes: After receiving data pushed by the interactive thread, write it into the corresponding index file according to the field types pre-configured in the configuration phase; Data used as query conditions is written to Lucene's inverted index file, while data used as query results is written to Lucene's data file. Numeric data is saved to a columnar storage file.

9. The method for flexible querying of massive data according to claim 8, characterized in that, The columnar storage file uses Lucene's Sorted NumericDoc Values.

10. The method for flexible querying of massive data according to claim 6, characterized in that, The S67 further includes: If any data is written incorrectly during the process of correctly writing all data retrieved from the data source into the Lucene index file, the written data will be deleted and the index creation process will be restarted.

11. The method for flexible querying of massive data according to claim 6, characterized in that, The data dictionary configuration includes: The data dictionary configuration is completed by initializing dictionary content through the dictionary management page or by querying external data sources using SQL statements.

12. The method for flexible querying of massive data according to claim 1, characterized in that, After integrating the index configuration information corresponding to the split data nodes, the data files, and the columnar storage files to obtain the query results, the method further includes: The integrated data is returned to the front-end query page.

13. A flexible query system for massive amounts of data, characterized in that, include: The acquisition module is used to acquire the index configuration information corresponding to the user's data query request; the index configuration information includes the number of data splits, the data splitting rules, and the names of the split data nodes; The request sending module is used to input user data query requests into the split data nodes and push them through the interaction thread; the interaction thread is generated by starting threads corresponding to the number of data splits and establishing the same number of connections corresponding to the split data nodes in each thread. The query module is used to query a pre-created Lucene index file based on the user's data query request pushed by the service instance and interaction thread corresponding to the split data node, which are in a healthy state, to obtain the Lucene index file and index configuration information corresponding to the split data node; wherein, the Lucene index file is created based on a data table generated by the background processing, and the data table contains dimensional data; based on the Lucene index file corresponding to the split data node, the data file and columnar storage file corresponding to the Lucene index file corresponding to the split data node are obtained; The output module integrates the index configuration information corresponding to the split data nodes, the data file, and the columnar storage file to obtain query results. Specifically, for multiple dimensions in the dimension data, the output module is used to: map the dimension data of the multiple dimensions to corresponding dictionary IDs using a pre-generated numeric dictionary; merging and compressing the dictionary IDs of the multiple dimensions into different bits of the same integer by bit offset according to the number of binary bits occupied by the maximum value of each dimension data in the dictionary, to obtain a compressed dimension integer; and sorting and deduplicating the compressed dimension integer using a BitMap algorithm.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements any one of the methods for flexible querying of massive data as described in claims 1 to 12.

15. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method for flexible querying of massive data as described in any one of claims 1 to 12.

Images