Data query system and method and electronic equipment

Abstract

The invention provides a data query system and method and electronic equipment, and the system comprises a shunting gateway which is used for carrying out shunting processing on query requests of different complexity categories; the first cluster is used for processing a first category query request with a complexity value lower than a first preset threshold value; the second cluster is isolated from the first cluster and is used for processing a second category query request of which the complexity value is higher than a first preset threshold value; wherein the first cluster or the second cluster is composed of node servers, the number of the node servers is smaller than a second preset threshold value, distributed search and storage engines run on the node servers, data indexes are synchronized between the first cluster and the second cluster in real time, and word segmentation engines are arranged between the first cluster and the second cluster. The word segmentation engine at least comprises a mixed word segmentation mode for combining continuous multi-character word segmentation and single-character word segmentation. According to the method and the device, the technical problems that the hardware cost is high, the query request is easy to block and a single word segmentation strategy cannot give consideration to double index requirements of recall ratio and query speed in the prior art are solved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data query system, method, and electronic device. Background Technology

[0002] With the number of global patent documents exceeding 200 million, the storage capacity of patent data often reaches 30-40TB. Professional queries need to support complex logical operators (such as AND / OR / NOT) and multiple nesting, and the length of query statements can reach tens of thousands of characters.

[0003] Distributed search and storage engines are often used to build such big data retrieval platforms, but traditional optimization solutions such as hot and cold data separation or increasing the number of nodes have significant limitations: On the one hand, to support tens of terabytes of data and provide low-latency responses, traditional architectures require the deployment of a large number of node servers, resulting in huge hardware investment and maintenance costs. On the other hand, complex long query statements consume a large amount of central processing unit (CPU) and memory resources, and running such queries within a single cluster can easily block the cluster, affecting the response speed of regular queries. For example, the single cluster architecture with hot and cold data separation used in patent publication CN120929012A is prone to blocking the cluster, thus affecting the response speed of regular queries. In addition, the word segmentation strategy of existing clusters, similar to that in patent publication CN107247780A, using a single word segmentation mode, cannot balance query speed and recall.

[0004] No effective solutions have yet been proposed for the aforementioned technical problems. Summary of the Invention

[0005] This application provides a data query system, method, and electronic device to solve one or more of the above-mentioned technical problems.

[0006] In a first aspect, embodiments of this application provide a data query system, comprising: a traffic distribution gateway for processing query requests of different complexity categories; a first cluster for processing first-category query requests with complexity values ​​lower than a first preset threshold; and a second cluster deployed in isolation from the first cluster for processing second-category query requests with complexity values ​​higher than the first preset threshold; wherein the first cluster or the second cluster consists of fewer than a second preset threshold of node servers running distributed search and storage engines, the first cluster and the second cluster synchronize data indexes in real time and are respectively equipped with word segmentation engines, the word segmentation engines including at least a hybrid word segmentation mode combining continuous multi-character word segmentation and single-character word segmentation.

[0007] Secondly, this application provides a data query method applied to the aforementioned data query system. The data query method includes: receiving a query request submitted by a user; routing the query request to a first cluster or a second cluster based on the complexity value of the query request, thereby triggering the first cluster or the second cluster to perform word segmentation processing on the query request, and using the result of the word segmentation processing to retrieve the query result in the data index.

[0008] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method described in any of the above-mentioned embodiments.

[0009] The data query system provided in this application includes: a traffic distribution gateway for distributing query requests of different complexity categories; a first cluster for processing first-category query requests with complexity values ​​below a first preset threshold; and a second cluster deployed in isolation from the first cluster for processing second-category query requests with complexity values ​​above the first preset threshold. The first or second cluster consists of fewer than the second preset threshold of node servers running distributed search and storage engines. The first and second clusters synchronize data indexes in real time and are each equipped with a word segmentation engine, which includes at least a hybrid word segmentation mode combining continuous multi-character segmentation and single-character segmentation. In other words, the data query system provided in this application adopts a small cluster model, reducing hardware costs compared to traditional large-node-server architectures. Furthermore, the deployment of mutually isolated first and second clusters, along with the traffic distribution gateway, routes high-complexity query requests to the second cluster and low-complexity query requests to the first cluster, avoiding interference from complex queries on regular queries within the first cluster and meeting the performance requirements of professional users. Furthermore, the embodiments of this application simultaneously meet the dual requirements of recall and search speed through a hybrid word segmentation mode.

[0010] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0011] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments according to this application and should not be construed as limiting the scope of this application.

[0012] Figure 1 This illustration shows a schematic diagram of a data query system architecture in a related art provided in an embodiment of this application;

[0013] Figure 2 This paper shows a structural block diagram of a data query system provided in an embodiment of this application;

[0014] Figure 3 This paper shows a structural block diagram of another data query system provided in an embodiment of this application;

[0015] Figure 4 A flowchart illustrating a hybrid word segmentation method provided in an embodiment of this application is shown;

[0016] Figure 5 This illustration shows a schematic diagram of a patent search result provided in an embodiment of this application;

[0017] Figure 6 This illustration shows a schematic diagram of a syntax tree structure provided in an embodiment of this application;

[0018] Figure 7 This paper shows a structural block diagram of another data query system provided in an embodiment of this application;

[0019] Figure 8 A flowchart of a data query method provided in an embodiment of this application is shown;

[0020] Figure 9 A flowchart of another data query method provided in an embodiment of this application is shown;

[0021] Figure 10 A block diagram of an electronic device used to implement embodiments of this application is shown. Detailed Implementation

[0022] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the concept or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0023] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and all of them fall within the protection scope of the embodiments of this application.

[0024] Terminology Explanation:

[0025] ES, short for Elasticsearch, is a distributed, open-source real-time search and analytics engine built on Apache Lucene and designed to provide fast, scalable, and high-performance search solutions.

[0026] Data indexing is a structured organization method used to speed up data retrieval. It enables rapid location and querying by pre-establishing a mapping between data items and their storage locations.

[0027] Shards are the physical units of index splitting used for distributed storage and parallel search. They include primary shards and replica shards. Primary shards are the original data storage units of the index, while replica shards are copies of the primary shards used for fault tolerance and load balancing.

[0028] Index segments are the underlying collection of storage files within a shard, managed by Lucene.

[0029] Logstash is an open-source data processing pipeline tool that supports collecting data from various input sources, filtering and processing it, and then outputting it to specified storage targets (such as Elasticsearch, files, message queues, etc.).

[0030] Cross-cluster replication (CCR) is a feature of Elasticsearch that automatically replicates data from an index in one cluster (Follower) to another cluster (Leader), achieving near real-time synchronization.

[0031] N-Gram segmentation divides the original text string into multiple fixed-length segments (Gram) by using a sliding window to divide the original text string into N consecutive characters. These segments are then stored as the smallest retrieval unit for indexing, supporting functions such as fuzzy matching, fault-tolerant search, and auto-completion.

[0032] SSDs are hard drives that use solid-state electronic storage chips as their storage medium, replacing the rotating disks and read / write heads of traditional mechanical hard drives for reading and writing data. Because they have no moving mechanical parts, SSDs are superior to mechanical hard drives in terms of read / write speed, shock resistance, and power consumption.

[0033] Queries Per Minute (QPM) is a core performance metric for measuring the query processing capacity and throughput of a computer system, database, or search engine. It quantifies the total number of query requests that a system can successfully process per unit of time (1 minute).

[0034] The main architecture of current data query systems is as follows: Figure 1As shown, a three-tiered distributed architecture is primarily adopted, consisting of an application service layer, a gateway, and a data processing layer from top to bottom. The application service layer faces end users and external systems, typically including a user search interface, a standardized application programming interface (API), and permission management services. It receives user data (e.g., patent data) query requests, verifies user access permissions, and returns formatted search results. The gateway connects the application service layer and the data processing layer, routing query requests forwarded by the application service layer. The data processing layer is deployed as a single cluster, consisting of numerous node servers running distributed search and storage engines. Each node is configured with a primary shard and replica shards for the data index. The primary shard carries the retrieval and storage tasks, while the replica shards provide data redundancy backup, ensuring data reliability and service continuity in the event of a single point of failure in the cluster.

[0035] However, the above architecture has significant limitations: firstly, the hardware cost is high (typically requiring more than 20 server nodes); secondly, complex queries are prone to resource contention, leading to delays in the response time of regular queries. For example, in a single cluster, long queries can block the CPU, causing concurrent query response times to exceed the industry requirement of 150 milliseconds. Furthermore, the existing cluster's word segmentation strategy cannot balance query speed and recall.

[0036] In view of this, embodiments of this application provide a data query system to solve all or part of the above-mentioned technical problems. This data query system is primarily adapted to scenarios requiring efficient and accurate retrieval of massive amounts of data under limited hardware resource constraints. Specific application scenarios include, but are not limited to: patent data retrieval scenarios, specifically targeting massive global patent documents (including semi-structured data such as specifications, claims, and accompanying drawings), enabling efficient querying needs such as accurate matching of patent technical features, rapid verification of legal status, and related retrieval of patent families under scenarios with limited server hardware configuration (small-scale cluster deployment); academic literature retrieval scenarios, specifically targeting massive academic resources such as journal articles, conference papers, and dissertations, supporting multi-dimensional accurate retrieval based on keywords, authors, research fields, and citation relationships, adapting to the lightweight deployment needs of university libraries, research institutions, and other institutions with limited hardware resources; and government archives retrieval scenarios, specifically targeting massive structured government data such as government approval archives, real estate registration archives, and social security archives, enabling rapid cross-departmental archive access and historical data tracing while ensuring data security and compliance. It is understood that the core purpose of the data query system provided in this application embodiment is to resolve the "performance-cost" contradiction in massive data query. Therefore, the data query system provided in this application embodiment can be used in all scenarios with the core requirements of "massive data storage + limited hardware resources + efficient and accurate query".

[0037] Figure 2 The diagram shows a structural block diagram of a data query system provided in an embodiment of this application, such as... Figure 2 As shown, the above data query system includes:

[0038] The traffic distribution gateway 12 is used to distribute query requests of different complexity levels.

[0039] Understandably, the core function of the aforementioned traffic distribution gateway 12 is to receive all query requests initiated by clients, classify the query requests according to preset complexity classification rules, and then route the requests to the corresponding backend query clusters based on the categories, thereby realizing a layered query mechanism of "fast response for lightweight requests and in-depth processing for complex requests".

[0040] Optionally, in this embodiment, the aforementioned traffic distribution gateway 12 is also used to verify the validity of the query statement in the user's input query request. If the verification fails, a syntax error message is returned. For example, suppose the user inputs patent number = ZL202510000001 AND technical field = G06F AND. During the verification, the traffic distribution gateway 12 will recognize that the logical operator AND at the end of the statement is not associated with any query conditions, which is a syntax error due to incomplete logical operator matching. It will then return a syntax error: "The logical operator at the end of the query statement is not associated with a valid query condition. Please complete it and try again." As another example, if the user inputs application date = 2025 / 13 / 01, the traffic distribution gateway 12 will verify that the month "13" of the application date does not conform to the standard range of "1-12" and return a syntax error: "Application date format is invalid. The month must be an integer between 1 and 12. Please enter it in YYYY-MM-DD format."

[0041] The first cluster 14 is used to process first-category query requests with a complexity value lower than a first preset threshold.

[0042] It is understandable that the above-mentioned first category of query requests are low-complexity query requests, such as a client querying the legal status of a patent (valid authorization, authorization date 2022-04-12), applicant name (XX Co., Ltd.), basic patent information (application date, publication date, International Patent Classification (IPC), list of inventors, etc.), legal status change records (application submitted on 2019-06-10 → entered substantive examination on 2020-09-05 → authorized on 2021-11-01), and other basic information.

[0043] The second cluster 16, which is deployed in isolation from the first cluster 14, is used to process second-category query requests with a complexity value higher than the first preset threshold.

[0044] It is understandable that the above-mentioned second category of query requests are high-complexity query requests, such as the client performing trend analysis on patents of a certain technical topic (e.g., the application trend of patents in the field of "artificial intelligence chips" over the past ten years), or searching for competitors' patent layout (e.g., the patent layout of XX company in the field of new energy vehicle battery management system), etc.

[0045] It should be noted that, in the embodiments of this application, the first cluster 14 or the second cluster 16 consists of fewer than a second preset threshold number of node servers running a distributed search and storage engine. The first cluster 14 and the second cluster 16 synchronize data indexes in real time and are respectively equipped with word segmentation engines. The word segmentation engine includes at least a hybrid word segmentation mode that combines continuous multi-character word segmentation and single-character word segmentation.

[0046] Optionally, in this embodiment, the aforementioned distributed search and storage engine typically adopts a distributed architecture, supporting sharded storage of massive amounts of data, full-text search, and high-concurrency queries, including but not limited to: ES engine, Solr engine, OpenSearch engine, or Milvus engine. Preferably, in patent data retrieval scenarios, this embodiment selects the ES engine.

[0047] The aforementioned second preset threshold was determined through extensive experimental results with the goal of reducing costs while meeting the query needs of massive amounts of data (especially patent data), resulting in a significant reduction in the number of node servers included in the first cluster 14 and the second cluster 16 compared to the 20-node server cluster in the prior art.

[0048] For example, either cluster 14 or cluster 16 comprises three node servers running the Elasticsearch engine. In this case, the query system architecture is described in [reference needed]. Figure 3 As shown in the diagram, each node server is configured with at least 512GB of memory and a solid-state drive (SSD) with a read / write speed of at least 2000MB / s. Only 32GB of memory is allocated to the Elasticsearch JVM heap for runtime object storage, while the remaining approximately 480GB is reserved for the operating system as a file system cache. This ensures that the vast majority of index data (greater than or equal to 90%) is read directly from memory rather than from the disk, thereby significantly improving query performance.

[0049] It can be understood that the above single-character word segmentation is usually a text preprocessing method that splits the text data to be processed into the smallest units according to single characters. The splitting process does not consider the semantic association between characters, but only uses the natural boundaries of characters as the basis for word segmentation to ensure rare technical terms. In multi-character word segmentation (N-gram), N is the preset word segmentation length, which can be configured as positive integers such as 2, 3, 4, etc. according to actual needs. In this application, N = 2 or N = 3 is preferred. The splitting process can cover all continuous character combinations in the text and can retain the local semantic association between characters.

[0050] Exemplarily, taking the query statement "quantum communication method" in the query request and the N-gram length of 2-3 characters as an example to illustrate the hybrid word segmentation mode. Specifically, for the single-character word segmentation results are "量", "子", "通", "信", "方", "法", for the 2-gram word segmentation results are "量子", "子通", "通信", "信方", "方法", for the 3-gram word segmentation results are "量子通", "子通信", "通信方", "信方法", and then the index matching fusion module automatically de-duplicates and sorts the word segmentation results to return the patent query result set, as shown in Figure 4 the flowchart of a hybrid word segmentation method provided.

[0051] It can be understood that the hybrid word segmentation mode combining continuous multi-character word segmentation and single-character word segmentation in the embodiments of this application meets the dual index requirements of recall rate and query speed at the same time. Exemplarily, taking the keywords "watch, intelligent, charging" and the related words corresponding to the keywords respectively to form a retrieval formula and inputting it into the data query system provided by the embodiments of this application, the following can be obtained Figure 5 a schematic diagram of a patent query result as shown. Compared with the related art that only uses the single-character word segmentation mode, due to the hybrid word segmentation mode proposed in the embodiments of this application, there is no need to splice single characters together and other operation steps when the position information remains unchanged, but directly takes multi-characters as a whole for parallel retrieval, which significantly improves the recall speed of the query result.

[0052] Optionally, in the embodiments of this application, the Byte Pair Encoding (BPE) algorithm can also be integrated in the multi-character word segmentation (N-gram) to establish short word indexes for high-frequency technical vocabulary, such as "blockchain", "artificial intelligence", etc., thereby reducing the query scanning range.

[0053] The above through Figure 1The data query system shown includes: a traffic distribution gateway 12 for distributing query requests of different complexity categories; a first cluster 14 for processing first-category query requests with complexity values ​​below a first preset threshold; and a second cluster 16, deployed in isolation from the first cluster 14, for processing second-category query requests with complexity values ​​above the first preset threshold. The first cluster 14 or the second cluster 16 consists of fewer than the second preset threshold of node servers running distributed search and storage engines. The first cluster 14 and the second cluster 16 synchronize data indexes in real time and are each equipped with a word segmentation engine, which includes at least a hybrid word segmentation mode combining continuous multi-character segmentation and single-character segmentation. In other words, the data query system of this application adopts a small cluster model, which reduces hardware costs compared to the traditional large-node-server architecture. Furthermore, compared to the single-cluster architecture in patent CN114741467A, this application deploys a first cluster 14 and a second cluster 16 that are isolated from each other. Through the traffic distribution processing of the traffic distribution gateway 12, high-complexity query requests are routed to the second cluster 16, while low-complexity query requests are routed to the first cluster 14. This avoids interference from complex queries on regular queries within the first cluster 14 and meets the performance requirements of professional users. In addition, compared to the single word segmentation method in patent CN119377258A, the embodiment of this application simultaneously meets the dual requirements of recall and query speed through a hybrid word segmentation mode.

[0054] In one possible implementation, the aforementioned traffic splitting gateway 12 includes:

[0055] The complexity calculation module is used to parse the query request to generate a syntax tree, and calculate the above complexity value based on the depth of the syntax tree, the number of logical operators, and the length of the query statement in the query request.

[0056] Optionally, the above-mentioned parsing query request to generate a syntax tree can first perform word segmentation on the query string input by the user according to the predefined patent search syntax rules, decompose the fields, logical operators, parentheses and values ​​into syntax units (Tokens), and then use a syntax analyzer to recursively parse these Tokens according to the syntax rules to construct an abstract syntax tree with logical operators as internal nodes and field conditions as leaves, so that the above-mentioned abstract syntax tree can completely represent the hierarchy and logical structure of the query.

[0057] For example, assuming the query request includes the query statement "Patent No.=ZL202510000001AND Technical Field=G06F", the complexity calculation module performs parsing according to predefined patent search syntax rules: First, the query string is broken down into seven semantically clear syntactic units through word segmentation: Patent No. (field name Token), = (relational operator Token), ZL202510000001 (value Token), AND (logical operator Token), Technical Field (field name Token), = (relational operator Token), and G06F (value Token); then, the parser recursively parses the Tokens according to the rule that fields + operators + values ​​constitute basic query conditions, and logical operators associate multiple conditions. It first combines Patent No. += + ZL202510000001 and Technical Field += + G06F into two basic query conditions as leaf nodes, and then uses the logical operator AND as the root node (internal node), attaching the two leaf nodes to form an abstract syntax tree, such as... Figure 6 As shown. The final generated syntax tree fully expresses the query logic that simultaneously satisfies the exact matching of the patent number ZL202510000001 and the exact matching of the technical field G06F in a hierarchical structure.

[0058] The above-mentioned complexity calculation process and traffic splitting process form a closed-loop logic, which solves the problems of cluster resource waste and high-priority request response delay caused by the indiscriminate forwarding of traditional gateways.

[0059] Optionally, in this embodiment of the application, the complexity calculation module is further configured to multiply the depth of the syntax tree and the number of logical operators, and then add the multiplication result to the length of the query statement adjusted according to a preset ratio to obtain the complexity value.

[0060] Optionally, the preset ratio can be 0.1%. Specifically, the complexity value is determined using the following formula 1:

[0061] C = D × N + L / 1000 (Formula 1)

[0062] Where C is the complexity value, D is the depth of the syntax tree, N is the number of logical operators, and L is the length of the query statement.

[0063] The routing decision module is used to route the query request to either the first cluster 14 or the second cluster 16 based on the complexity category corresponding to the complexity value.

[0064] It is understandable that the aforementioned first preset threshold can be set by combining the distribution of time and resource consumption in historical query logs, query syntax structure analysis, and system performance stress test results. This application embodiment, after extensive data verification, sets the aforementioned first preset threshold to 50.

[0065] Optionally, in this embodiment, query requests with a complexity value less than 50 are typically identified as first-category query requests (i.e., regular query requests), and query requests with a complexity value greater than or equal to 50 are identified as second-category query requests (i.e., complex query requests). Then, first-category query requests are routed to the first cluster 14, thereby achieving a concurrency limit of 3000 QPM and a response time of less than or equal to 150ms. Second-category query requests are routed to the second cluster 16, thereby achieving a response time of less than or equal to 1s and a single-cluster concurrency limit of 500 QPM.

[0066] The load monitoring module is used to monitor the load status of the first cluster 14 and the second cluster 16 in real time, and to buffer the query request and the query request queue after the query request when the load status does not meet the preset conditions.

[0067] Understandably, the aforementioned load status is typically a comprehensive indicator reflecting the cluster's hardware resources and software operating status, including but not limited to: CPU utilization, memory utilization, disk I / O load, network bandwidth load, as well as service operation dimensions such as task queue length, index operation load, and connection load. The load monitoring module collects these multi-dimensional indicators in real time, uses a weighted scoring mechanism to calculate the overall load index and classify load levels, providing a basis for gateway request routing and task scheduling, realizing refined resource management of lightweight clusters, and ensuring the stability and efficiency of massive patent data queries while significantly reducing hardware costs.

[0068] For example, taking the load status as CPU utilization and the corresponding threshold of 80% as an example, the load monitoring module monitors the cluster CPU utilization in real time. When it is determined that the CPU utilization exceeds 80%, it triggers the query queue buffer to ensure that the resources of the first cluster 14 or the second cluster 16 do not exceed the limit.

[0069] Optionally, in this embodiment, primary shards are configured according to the principle of 30-50GB per shard. Then, 30-40TB of data corresponds to 600-1333 primary shards, and the number of replica shards is set to 1 to meet high availability and control storage costs.

[0070] After configuring the shards, this application embodiment also proposes a segmentation strategy within each shard. Optionally, the first cluster 14 or the second cluster 16 further includes a merging module, used to perform a merging operation within each main shard after configuring the main shards for the above-mentioned data index, to merge a first number of index segments before merging into a second number of index segments, wherein the second number is less than the first number. Optionally, in this application embodiment, the second number is less than or equal to 5.

[0071] In one possible implementation, the merging strategy of the aforementioned merging module could be to set "index.merge.policy.max_merged_segment=50gb" to avoid an excessive number of small segments. Specifically, this limits the maximum size of a single target segment after index segment merging to 50GB. That is, when the cluster performs segment merging, it will gradually merge multiple small index segments to ultimately form a large index segment with a capacity not exceeding 50GB, and will not further merge index segments that have reached or exceeded 50GB.

[0072] Taking the above data index as an example of patent data index, during the writing process of patent data index, a large number of small temporary index segments will be generated first (such as N-gram word segmentation index segments for newly added patent documents and incremental index segments for patent status updates). These small index segments will cause the retrieval to require traversing more segment files and increase disk I / O overhead. However, the segment merging driven by the above merging strategy will merge multiple small index segments into a large index segment of nearly 50GB according to the logic of "from small to large and gradually merged" (for example, 10 small index segments of 5GB can be merged into a large index segment of 50GB, or 8 small index segments of 6GB can be merged into a large index segment of 48GB). At the same time, the capacity of a single segment after merging is strictly controlled to not exceed the 50GB limit.

[0073] It should be noted that, in this embodiment, the 50GB threshold was not chosen arbitrarily, but rather is the optimal value obtained through extensive experimental verification, taking into account the hardware configuration of the node server cluster (single node disk capacity, memory size) and the characteristics of patent data (a single patent data entry contains large amounts of information such as text and figures, and the full patent index capacity is adapted to the data). On the one hand, a single segment capacity of 50GB can significantly reduce the total number of index segments, reduce the number of segment traversals during retrieval, thereby reducing disk I / O waiting time and improving the retrieval response speed of patent data. On the other hand, this threshold does not exceed the storage and processing capacity of a single node, avoiding excessive memory consumption and index loading failures caused by merging excessively large index segments, while also reserving sufficient space for redundant backup of index shards. In terms of execution logic, when the cluster detects that the cumulative capacity of small index segments in the patent data index is close to or can be merged into segments of about 50GB, it will automatically trigger a background merging task. During the merging process, it will prioritize using low-load periods of the cluster (such as the low period of patent query requests at night) and reasonably control the number of merging threads to avoid consuming too much CPU and memory resources and affecting normal query services. For large index segments that have reached 50GB, the cluster will no longer merge them to prevent the generation of ultra-large segments from reducing the flexibility of index maintenance (such as index deletion and update) and to avoid the risk of a large amount of patent data being unsearchable due to a single segment failure.

[0074] The segment merging strategy in the above-mentioned merging module works in synergy with the main sharding configuration and word segmentation strategy of the 3-node server cluster. Under the lightweight hardware architecture, it not only ensures the storage stability of the data index but also improves the retrieval efficiency, achieving a balance between cost and performance.

[0075] Optionally, in addition to the hybrid segmentation mode, the above-mentioned word segmentation engine also includes at least a single-character segmentation mode. In some embodiments, the word segmentation engine also includes a dynamic switching module, which is used to identify whether the query request includes phrase-level exact matching identification information. When the query request includes the identification information, it switches to the single-character segmentation mode; when the query request does not include the identification information, it switches to the hybrid segmentation mode.

[0076] It is understandable that the above phrase-level precise matching identifiers are used to limit the matching of complete phrases or specific content without splitting, so as to avoid interference from word segmentation. They may include, but are not limited to: double quotation mark identifiers (such as chemical formula "C6H12O6", etc.), square bracket identifiers, patent data fields (such as application number, patent number, applicant name, etc.), and special prefix identifiers.

[0077] The aforementioned word segmentation engine prioritizes single-character word segmentation mode for precise queries and enables hybrid word segmentation mode for fuzzy queries, further improving query speed and accuracy.

[0078] Optionally, in this embodiment of the application, the word segmentation engine can not only process text, but also perform feature processing on the images in the patent document (extract visual feature vectors), and store these vectors in ES. With the help of ES's vector retrieval capabilities, it can realize the retrieval of similar patents, documents, etc. based on image content, such as finding the patents with the most similar appearance design and mechanical structure to the query image.

[0079] To further improve query response speed, the data query system also includes a third cluster deployed in isolation from the first cluster 14 and the second cluster 16, used to handle third category query requests with a complexity value higher than a third preset threshold. The third preset threshold is greater than the first preset threshold. The third cluster consists of fewer node servers running distributed search and storage engines than the second preset threshold. Data indexes are synchronized in real time among the first cluster 14, the second cluster 16, and the third cluster, and the third cluster is equipped with the word segmentation engine.

[0080] Optionally, in the embodiments of this application, the first cluster 14, the second cluster 16, and the third cluster can use Docker to create a unified image for each cluster node, and Kubernetes is responsible for deployment, scheduling, and management. Based on this, combined with Kubernetes' automatic scaling capability, the number of nodes can be dynamically increased or decreased according to the real-time load, thereby achieving flexible resource supply, cost optimization, and high availability.

[0081] It is understandable that the aforementioned third preset threshold can be set by combining the time and resource consumption distribution of historical query logs, query syntax structure analysis, and system performance stress test results. This application embodiment, after extensive data verification, sets the aforementioned third preset threshold to 200. Complex query requests with a complexity value greater than 50 and less than 200 are routed to the second cluster 16, while ultra-complex query requests with a complexity value greater than or equal to 200 are routed to the third cluster.

[0082] After verification by a large amount of experimental data, the above deployment method achieved the technical effects shown in Table 1 below.

[0083] Table 1

[0084] Query Request Category C value range The target cluster of the route Concurrency limit Response threshold Regular query 0-49 Cluster 1 3000 QPM 150ms Complex Queries 50-199 Cluster 2 500QPM 500ms Extremely complex queries ≥200 Cluster 3 200QPM 1000ms

[0085] It is understood that the overall time taken for the above-mentioned complexity value calculation and traffic splitting judgment method in the embodiments of this application is less than or equal to 10ms, ensuring that the overall response meets the standard.

[0086] In one possible implementation, the data query system described above further includes a data synchronization module, used to achieve real-time synchronization of data indexes of the first cluster 14, the second cluster 16, and the third cluster through Logstash synchronization or cross-cluster replication CCR synchronization.

[0087] Understandably, when using Logstash synchronization, the input plugins of Logstash retrieve newly added or updated documents from the source cluster in real time. After optional filtering, these documents are simultaneously written to the corresponding indexes of the second and third clusters via multiple output plugins, thus achieving data replication across multiple clusters. In contrast, when using Elasticsearch Cross-Cluster Replication (CCR), the original indexes are maintained on the leader cluster, while a follower index is pre-built on the target cluster. Elasticsearch's built-in replication mechanism continuously pulls and replays the index changes from the leader cluster in the order of operations, ensuring near real-time consistency of data indexes across multiple clusters.

[0088] The aforementioned data synchronization module allows multiple physically or logically separate Elasticsearch clusters to share the same latest data copy, thereby improving the stability, performance, and scalability of the data query system.

[0089] In this embodiment of the application, the above-mentioned traffic splitting gateway further includes: a machine learning model, used to dynamically optimize the first preset threshold and the third preset threshold based on historical query data.

[0090] Optionally, the aforementioned machine learning model can be a Long Short-Term Memory (LSTM) model, which uses historical query prediction complexity thresholds to achieve adaptive routing.

[0091] The embodiments of this application will be illustrated below with specific examples.

[0092] In this example, we use an Elasticsearch cluster and patent data as the data type. (Reference) Figure 7 Another data query system structure diagram is provided, including: user terminal (for inputting patent query requests), load balancing node, traffic distribution gateway (for distributing patent query requests of different complexity categories), complexity calculation module (for parsing patent query requests to generate a syntax tree, and calculating the complexity value based on the depth of the syntax tree, the number of logical operators, and the length of the query statement in the patent query request), cluster status monitoring module (for real-time monitoring of the load status of cluster 1, cluster 2, and cluster 3), cluster 1 (processing regular patent query requests), cluster 2 (processing complex patent query requests), cluster 3 (processing ultra-complex patent query requests), hybrid word segmentation engine (combining continuous multi-character word segmentation and single-character word segmentation), ES index cluster (3 nodes / cluster), and SSD storage layer (30-40TB / replica).

[0093] pass Figure 7 The system shown achieves fast and accurate querying of massive amounts of patented data under the hardware condition of deploying a small number (e.g., 3) of high-configuration node servers in each cluster. By multi-cluster traffic distribution and word segmentation optimization, it balances query speed and recall, avoiding performance bottlenecks caused by complex queries.

[0094] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0095] Corresponding to the application scenarios and system of the data query system provided in the embodiments of this application, the embodiments of this application also provide a data query method. For example... Figure 8The diagram shown is a flowchart of a data query method according to an embodiment of this application, including:

[0096] S802, receives query requests submitted by users;

[0097] S804, based on the complexity value of the query request, the query request is routed to the first cluster or the second cluster to trigger the first cluster or the second cluster to perform word segmentation processing on the query request, and the result of the word segmentation processing is used to retrieve the query result in the data index.

[0098] It is understandable that step S804 above receives all query requests initiated by the client, classifies the query requests according to the preset complexity classification rules, and then routes the requests to the corresponding backend query clusters according to the categories, thereby realizing a layered query mechanism of "fast response for lightweight requests and in-depth processing for complex requests".

[0099] Before executing step S804, the data query method further includes: S11, validating the validity of the query statement in the user-input query request, and returning a syntax error message if it fails. For example, suppose the user inputs Patent Number = ZL202510000001 AND Technical Field = G06F AND. During the verification, the distribution gateway 12 will recognize that the logical operator AND at the end of the statement is not associated with any query conditions, which is a syntax error due to incomplete logical operator matching, and will return a syntax error: "The logical operator at the end of the query statement is not associated with a valid query condition. Please complete it and try again." As another example, if the user inputs Application Date = 2025 / 13 / 01, the distribution gateway 12 will verify that the month "13" of the application date does not conform to the standard range of "1-12", and return a syntax error: "Application date format is invalid. The month must be an integer between 1 and 12. Please enter it in YYYY-MM-DD format."

[0100] Optionally, in this embodiment, step S804 may include: S21, routing a first category query request with a complexity value lower than a first preset threshold to a first cluster; S22, routing a second category query request with a processing complexity value higher than the first preset threshold to a second cluster.

[0101] Understandably, the first category of query requests mentioned above are low-complexity queries, such as a client querying the legal status of a patent (valid authorization, authorization date 2022-04-12), applicant name (XX Co., Ltd.), basic patent information (application date, publication date, International Patent Classification (IPC), list of inventors, etc.), and legal status change records (application submitted on 2019-06-10 → entered substantive examination on 2020-09-05 → authorized on 2021-11-01), etc. The second category of query requests mentioned above are high-complexity queries, such as a client performing trend analysis on patents related to a specific technical topic (e.g., the application trend of patents in the field of "artificial intelligence chips" over the past ten years) or searching competitor patent layouts (e.g., XX Company's patent layout in the field of new energy vehicle battery management systems), etc.

[0102] Furthermore, it is understood that the aforementioned first preset threshold can be set by combining the time and resource consumption distribution of historical query logs, query syntax structure analysis, and system performance stress test results. This application embodiment, after extensive data verification, sets the aforementioned first preset threshold to 50. Optionally, in this application embodiment, query requests with a complexity value less than 50 are typically identified as first-category query requests (i.e., regular query requests), and query requests with a complexity value greater than or equal to 50 are identified as second-category query requests (i.e., complex query requests). First-category query requests are then routed to the first cluster, thereby achieving a concurrency limit of 3000 QPM and a response time less than or equal to 150ms. Second-category query requests are routed to the second cluster, thereby achieving a response time less than or equal to 1s and a single-cluster concurrency limit of 500 QPM.

[0103] It should be noted that, in the embodiments of this application, the first cluster or the second cluster deployed in isolation from the first cluster consists of fewer than a second preset threshold number of node servers running distributed search and storage engines. The first cluster and the second cluster synchronize data indexes in real time and are respectively equipped with word segmentation engines. The word segmentation engines include at least a hybrid word segmentation mode that combines continuous multi-character word segmentation and single-character word segmentation.

[0104] Optionally, in this embodiment, the aforementioned distributed search and storage engine typically adopts a distributed architecture, supporting sharded storage of massive amounts of data, full-text search, and high-concurrency queries, including but not limited to: ES engine, Solr engine, OpenSearch engine, or Milvus engine. Preferably, in patent data retrieval scenarios, this embodiment selects the ES engine.

[0105] The above-mentioned second preset threshold is determined through a large number of experimental results with the goal of reducing costs while meeting the query requirements of massive data (especially patent data), such that the number of node servers included in the first cluster and the second cluster is significantly reduced compared to the 20-node server cluster in the prior art.

[0106] Exemplarily, it is set that the first cluster or the second cluster includes 3 node servers running the ES engine. In this case, the architecture of the query system is shown in Figure 3 the figure. Each node server is configured with a memory of not less than 512GB and a solid-state drive SSD with a read / write speed of not less than 2000MB / s. Only 32GB of the memory is allocated to the ES JVM heap for runtime object storage, and the remaining approximately 480GB is all reserved for the operating system as a file system cache, so that the vast majority of index data (greater than or equal to 90%) is directly read from the memory instead of the disk, thereby significantly improving the query performance.

[0107] It can be understood that the above single-character word segmentation is usually a text preprocessing method that splits the text data to be processed into the smallest units according to single characters. The splitting process does not consider the semantic association between characters, and only takes the natural boundary of characters as the basis for word segmentation to ensure rare technical terms. In multi-character word segmentation (N-gram), N is the preset word segmentation length, which can be configured as positive integers such as 2, 3, 4, etc. according to actual needs. In this application, N = 2 or N = 3 is preferred. The splitting process can cover all continuous character combinations in the text and can retain the local semantic association between characters.

[0108] Exemplarily, taking the query statement "quantum communication method" in the query request and the N-gram length of 2-3 characters as an example to illustrate the hybrid word segmentation mode. Specifically, for the single-character word segmentation results are "quantity", "sub", "communication", "letter", "method", "law", for the 2-gram word segmentation results are "quantum", "sub-communication", "communication", "letter-method", "method", for the 3-gram word segmentation results are "quantum communication", "sub-communication method", "communication method", "letter-method", and then the word segmentation results are automatically de-duplicated and sorted by the index matching and fusion module, and the patent query result set is returned, as shown in Figure 4 the figure.

[0109] It can be understood that the hybrid word segmentation mode combining continuous multi-character word segmentation and single-character word segmentation in the embodiments of this application meets the dual index requirements of recall rate and query speed. Exemplarily, taking the keywords "watch, intelligent, charging" and the related words corresponding to the keywords respectively to form a retrieval formula and inputting it into the data query system provided by the embodiments of this application, the result can be obtained as shown in Figure 5The query results shown show that, compared to the single-character segmentation mode used in related technologies, the hybrid segmentation mode proposed in this application does not require the operation steps of concatenating single characters together without changing the position information. Instead, it directly performs parallel retrieval of multiple characters as a whole, which significantly improves the recall speed of query results.

[0110] Optionally, in this embodiment of the application, a Byte Pair Encoding (BPE) algorithm can also be integrated into the multi-character segmentation (N-gram) to establish a short word index for high-frequency technical terms, such as "blockchain" and "artificial intelligence", thereby reducing the scope of query scanning.

[0111] Optionally, in this embodiment, step S804 further includes: S31, parsing the query request to generate a syntax tree; S32, calculating the complexity value based on the depth of the syntax tree, the number of logical operators, and the length of the query statement in the query request.

[0112] Optionally, the above-mentioned parsing query request to generate a syntax tree can first perform word segmentation on the query string input by the user according to the predefined patent search syntax rules, decompose the fields, logical operators, parentheses and values ​​into syntax units (Tokens), and then use a syntax analyzer to recursively parse these Tokens according to the syntax rules to construct an abstract syntax tree with logical operators as internal nodes and field conditions as leaves, so that the above-mentioned abstract syntax tree can completely represent the hierarchy and logical structure of the query.

[0113] For example, assuming the query request includes the query statement "Patent No.=ZL202510000001AND Technical Field=G06F", the complexity calculation module performs parsing according to predefined patent search syntax rules: First, the query string is broken down into seven semantically clear syntactic units through word segmentation: Patent No. (field name Token), = (relational operator Token), ZL202510000001 (value Token), AND (logical operator Token), Technical Field (field name Token), = (relational operator Token), and G06F (value Token); then, the parser recursively parses the Tokens according to the rule that fields + operators + values ​​constitute basic query conditions, and logical operators associate multiple conditions. It first combines Patent No. += + ZL202510000001 and Technical Field += + G06F into two basic query conditions as leaf nodes, and then uses the logical operator AND as the root node (internal node), attaching the two leaf nodes to form an abstract syntax tree, such as... Figure 6As shown. The final generated syntax tree fully expresses the query logic that simultaneously satisfies the exact matching of the patent number ZL202510000001 and the exact matching of the technical field G06F in a hierarchical structure.

[0114] The above-mentioned complexity calculation process and traffic splitting process form a closed-loop logic, which solves the problems of cluster resource waste and high-priority request response delay caused by the indiscriminate forwarding of traditional gateways.

[0115] Step S32 above may also include: S321, multiplying the depth of the syntax tree by the number of logical operators, and adding the result of the multiplication to the length of the query statement adjusted according to a preset ratio to obtain the above complexity value.

[0116] Optionally, the preset ratio can be 0.1%. Specifically, the complexity value is determined using the following formula 1:

[0117] C = D × N + L / 1000 (Formula 1)

[0118] Where C is the complexity value, D is the depth of the syntax tree, N is the number of logical operators, and L is the length of the query statement.

[0119] Optionally, in this embodiment of the application, the above data query method may further include: S41, real-time monitoring of the load status of the first cluster and the second cluster, and buffering the query request and the query request queue following the query request when the load status does not meet the preset conditions.

[0120] Understandably, the aforementioned load status is typically a comprehensive indicator reflecting the cluster's hardware resources and software operating status, including but not limited to: CPU utilization, memory utilization, disk I / O load, network bandwidth load, as well as service operation dimensions such as task queue length, index operation load, and connection load. By collecting these multi-dimensional indicators in real time, a weighted scoring mechanism is used to calculate the overall load index and classify load levels, providing a basis for decision-making on gateway request routing and task scheduling. This enables refined resource management of lightweight clusters, significantly reducing hardware costs while ensuring the stability and efficiency of massive patent data queries.

[0121] For example, taking the load status as CPU utilization and the corresponding threshold of 80% as an example, the cluster CPU utilization is monitored in real time. When it is determined that the CPU utilization exceeds 80%, the query queue buffer is triggered to ensure that the resources of the first cluster or the second cluster do not exceed the limit.

[0122] Optionally, in this embodiment, primary shards are configured according to the principle of 30-50GB per shard. Then, 30-40TB of data corresponds to 600-1333 primary shards, and the number of replica shards is set to 1 to meet high availability and control storage costs.

[0123] After configuring the shards, this application embodiment also proposes a segmentation strategy within each shard. Optionally, the above data query method further includes: S51, after configuring the primary shards for the data index of the first cluster or the second cluster, performing a merge operation within each primary shard to merge a first number of index segments into a second number of index segments, wherein the second number is less than the first number. Optionally, in this application embodiment, the second number is less than or equal to 5.

[0124] In one possible implementation, the above merging strategy could be to set "index.merge.policy.max_merged_segment=50gb" to avoid an excessive number of small segments. Specifically, this limits the maximum size of a single target segment after index segment merging to 50GB. This means that when the cluster performs segment merging, it will gradually merge multiple small index segments to ultimately form a large index segment with a capacity not exceeding 50GB, and will not further merge index segments that have reached or exceeded 50GB.

[0125] Taking the above data index as an example of patent data index, during the writing process of patent data index, a large number of small temporary index segments will be generated first (such as N-gram word segmentation index segments for newly added patent documents and incremental index segments for patent status updates). These small index segments will cause the retrieval to require traversing more segment files and increase disk I / O overhead. However, the segment merging driven by the above merging strategy will merge multiple small index segments into a large index segment of nearly 50GB according to the logic of "from small to large and gradually merged" (for example, 10 small index segments of 5GB can be merged into a large index segment of 50GB, or 8 small index segments of 6GB can be merged into a large index segment of 48GB). At the same time, the capacity of a single segment after merging is strictly controlled to not exceed the 50GB limit.

[0126] It should be noted that, in this embodiment, the 50GB threshold was not chosen arbitrarily, but rather is the optimal value obtained through extensive experimental verification, taking into account the hardware configuration of the node server cluster (single node disk capacity, memory size) and the characteristics of patent data (a single patent data entry contains large amounts of information such as text and figures, and the full patent index capacity is adapted to the data). On the one hand, a single segment capacity of 50GB can significantly reduce the total number of index segments, reduce the number of segment traversals during retrieval, thereby reducing disk I / O waiting time and improving the retrieval response speed of patent data. On the other hand, this threshold does not exceed the storage and processing capacity of a single node, avoiding excessive memory consumption and index loading failures caused by merging excessively large index segments, while also reserving sufficient space for redundant backup of index shards. In terms of execution logic, when the cluster detects that the cumulative capacity of small index segments in the patent data index is close to or can be merged into segments of about 50GB, it will automatically trigger a background merging task. During the merging process, it will prioritize using low-load periods of the cluster (such as the low period of patent query requests at night) and reasonably control the number of merging threads to avoid consuming too much CPU and memory resources and affecting normal query services. For large index segments that have reached 50GB, the cluster will no longer merge them to prevent the generation of ultra-large segments from reducing the flexibility of index maintenance (such as index deletion and update) and to avoid the risk of a large amount of patent data being unsearchable due to a single segment failure.

[0127] The above segment merging strategy, together with the main sharding configuration and word segmentation strategy of the 3-node server cluster, forms a synergistic optimization. Under the lightweight hardware architecture, it not only ensures the storage stability of the data index but also improves the retrieval efficiency, achieving a balance between cost and performance.

[0128] Optionally, in addition to the hybrid segmentation mode, the above-mentioned word segmentation engine also includes at least a single-character segmentation mode. In some embodiments, the word segmentation processing of the query request by the first cluster or the second cluster may include: S61, identifying whether the query request includes phrase-level exact matching identification information; S62, when the query request includes the identification information, switching to the single-character segmentation mode; S63, when the query request does not include the identification information, switching to the hybrid segmentation mode.

[0129] It is understandable that the above phrase-level precise matching identifiers are used to limit the matching of complete phrases or specific content without splitting, so as to avoid interference from word segmentation. They may include, but are not limited to: double quotation mark identifiers (such as chemical formula "C6H12O6", etc.), square bracket identifiers, patent data fields (such as application number, patent number, applicant name, etc.), and special prefix identifiers.

[0130] Through the above steps S61~S63, single-character word segmentation mode is preferentially called for precise queries, and mixed word segmentation mode is enabled for fuzzy queries, which further improves the query speed and accuracy.

[0131] Optionally, in this embodiment of the application, the above data query method can not only process text, but also perform feature processing (extract visual feature vectors) on the images in the patent document, and store these vectors in ES. Then, with the help of ES's vector retrieval capabilities, it can realize the retrieval of similar patents, documents, etc. based on image content, such as finding the patents with the most similar appearance design and mechanical structure to the query image.

[0132] To further improve query response speed, the above data query method may also include: S71, setting up a third cluster deployed in isolation from the first cluster and the second cluster, used to handle third category query requests with a complexity value higher than a third preset threshold, wherein the third preset threshold is greater than the first preset threshold, the third cluster consists of fewer node servers running distributed search and storage engines than the second preset threshold, the first cluster, the second cluster and the third cluster synchronize data indexes in real time, and the third cluster is equipped with the word segmentation engine.

[0133] Optionally, in the embodiments of this application, the first cluster, the second cluster, and the third cluster can use Docker to create a unified image for each cluster node, and Kubernetes is responsible for deployment, scheduling, and management. Based on this, combined with Kubernetes' automatic scaling capability, the number of nodes can be dynamically increased or decreased according to the real-time load, thereby achieving flexible resource supply, cost optimization, and high availability.

[0134] It is understandable that the aforementioned third preset threshold can be set by combining the time and resource consumption distribution of historical query logs, query syntax structure analysis, and system performance stress test results. This application embodiment, after extensive data verification, sets the aforementioned third preset threshold to 200. Complex query requests with a complexity value greater than 50 and less than 200 are routed to the second cluster 16, while ultra-complex query requests with a complexity value greater than or equal to 200 are routed to the third cluster.

[0135] After verification by a large amount of experimental data, the above deployment method achieved the technical effects shown in Table 1 below.

[0136] Table 1

[0137] Query Request Category C value range The target cluster of the route Concurrency limit Response threshold Regular query 0-49 Cluster 1 3000 QPM 150ms Complex Queries 50-199 Cluster 2 500QPM 500ms Extremely complex queries ≥200 Cluster 3 200QPM 1000ms

[0138] Optionally, in this embodiment of the application, the above data query method further includes: S81, realizing real-time synchronization of data indexes of the first cluster, the second cluster and the third cluster through Logstash synchronization or cross-cluster replication CCR synchronization.

[0139] Understandably, when using Logstash synchronization, the input plugins of Logstash retrieve newly added or updated documents from the source cluster in real time. After optional filtering, these documents are simultaneously written to the corresponding indexes of the second and third clusters through multiple output plugins, thus achieving data replication across multiple clusters. In contrast, when using Elasticsearch Cross-Cluster Replication (CCR), the original indexes are maintained on the leader cluster, while a follower index is pre-built on the target cluster. Elasticsearch's built-in replication mechanism continuously pulls and replays the index changes from the leader cluster in the order of operations, ensuring near real-time consistency of data indexes across multiple clusters.

[0140] Optionally, in this embodiment of the application, the above data query method further includes: S91, using a machine learning model to dynamically optimize the first preset threshold and the third preset threshold.

[0141] The aforementioned machine learning model can be a Long Short-Term Memory (LSTM) model, which uses historical query prediction complexity thresholds to achieve adaptive routing.

[0142] Through the above steps, with the hardware configuration of the small cluster, query requests are routed to either the first or second cluster based on their complexity, avoiding query latency caused by resource contention and meeting the performance requirements of professional users. Furthermore, the hybrid word segmentation mode set up through the first and second clusters simultaneously satisfies both recall and query speed requirements.

[0143] The embodiments of this application will be illustrated below with specific examples.

[0144] This example provides a method for querying patent data. (Refer to...) Figure 9 A flowchart of another data query method is provided, which includes:

[0145] S901, User enters query statement;

[0146] S902, Query statement validity check. If it fails, proceed to step S903; if it succeeds, proceed to step S904.

[0147] S903 returns a syntax error message;

[0148] S904, generates an abstract syntax tree;

[0149] S905, the computational complexity is C, where C = D × N + L / 1000, where C is the complexity value, D is the depth of the syntax tree, N is the number of logical operators, and L is the length of the query statement;

[0150] S906, Flow splitting judgment: if the complexity value is less than 50, execute step S907; if the complexity value is greater than or equal to 50, execute step S908.

[0151] S907, routed to ES cluster 1;

[0152] S908, routed to ES cluster 2 or cluster 3;

[0153] S909, ES cluster 1 or ES cluster 2 or cluster 3 respectively call the hybrid word segmentation engine;

[0154] S910, execute the query in ES cluster 1, ES cluster 2, or cluster 3;

[0155] S911, returns the patent search results.

[0156] pass Figure 9 The steps shown enable fast and accurate querying of massive amounts of patented data by deploying a small number (e.g., 3) of high-configuration node servers in each cluster. By optimizing the multi-cluster traffic and word segmentation, the query speed and recall rate are balanced, avoiding performance bottlenecks caused by complex queries.

[0157] Figure 10 This is a block diagram of an electronic device used to implement embodiments of this application. For example... Figure 10 As shown, the electronic device includes a memory 1001 and a processor 1002. The memory 1001 stores a computer program that can run on the processor 1002. When the processor 1002 executes the computer program, it implements the method described in the above embodiments. The number of memories 1001 and processors 1002 can be one or more.

[0158] The electronic device also includes:

[0159] Communication interface 1003 is used to communicate with external devices and perform data exchange and transmission.

[0160] If the memory 1001, processor 1002, and communication interface 1003 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 10The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0161] Optionally, in a specific implementation, if the memory 1001, processor 1002, and communication interface 1003 are integrated on a single chip, then the memory 1001, processor 1002, and communication interface 1003 can communicate with each other through an internal interface.

[0162] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in this application.

[0163] This application also provides a chip including a processor for calling and executing instructions stored in a memory, causing a communication device with the chip installed to perform the method provided in this application.

[0164] This application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in the application embodiment.

[0165] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0166] Further, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate Synchronous DRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0167] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0168] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0169] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0170] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0171] The logic and / or steps described in the flowchart or otherwise herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0172] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0173] Furthermore, the functional units in the various embodiments of this application can be integrated into a single processing module, or each unit can exist physically separately, or two or more units can be integrated into a single module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0174] The above description is merely an exemplary embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data query system, characterized in that, include: A traffic distribution gateway is used to distribute query requests of different complexity levels. The first cluster is used to process first-category query requests with a complexity value lower than a first preset threshold. A second cluster, deployed in isolation from the first cluster, is used to process second-category query requests with a complexity value higher than the first preset threshold. The first cluster or the second cluster consists of fewer than a second preset threshold number of node servers running a distributed search and storage engine. The first cluster and the second cluster synchronize data indexes in real time and are each equipped with a word segmentation engine. The word segmentation engine includes at least a hybrid word segmentation mode that combines continuous multi-character word segmentation and single-character word segmentation.

2. The data query system according to claim 1, characterized in that, The traffic splitting gateway includes: The complexity calculation module is used to parse the query request to generate a syntax tree, and calculate the complexity value based on the depth of the syntax tree, the number of logical operators, and the length of the query statement in the query request. The routing decision module is used to route the query request to the first cluster or the second cluster according to the complexity category corresponding to the complexity value; The load monitoring module is used to monitor the load status of the first cluster and the second cluster in real time, and to buffer the query request and the query request queue following the query request when the load status does not meet the preset conditions.

3. The data query system according to claim 2, characterized in that, The complexity calculation module is further configured to multiply the depth of the syntax tree and the number of logical operators, and then add the multiplication result to the length of the query statement adjusted according to a preset ratio to obtain the complexity value.

4. The data query system according to claim 1, characterized in that, The first cluster or the second cluster includes three node servers running a distributed search and storage engine, each node server being configured with no less than 512GB of memory and a solid-state drive (SSD) with a read / write speed of no less than 2000MB / s.

5. The data query system according to claim 1, characterized in that, The first cluster or the second cluster further includes: The merging module is used to perform a merging operation within each of the primary shards after configuring primary shards for the data index, so as to merge a first number of index segments into a second number of index segments, wherein the second number is less than the first number.

6. The data query system according to claim 1, characterized in that, The word segmentation engine also includes a single-character word segmentation mode, and the word segmentation engine also includes: The dynamic switching module is used to identify whether the query request includes phrase-level exact matching identifier information. When the query request includes the identifier information, it switches to the single-character word segmentation mode; when the query request does not include the identifier information, it switches to the mixed word segmentation mode.

7. The data query system according to claim 1, characterized in that, The system also includes: A third cluster, deployed in isolation from the first cluster and the second cluster, is used to process third category query requests with a complexity value higher than a third preset threshold. The third preset threshold is greater than the first preset threshold. The third cluster consists of fewer node servers running distributed search and storage engines than the second preset threshold. Data indexes are synchronized in real time between the first cluster, the second cluster, and the third cluster, and the third cluster is equipped with the word segmentation engine. The data synchronization module is used to achieve real-time synchronization of data indexes among the first cluster, the second cluster, and the third cluster via Logstash synchronization or cross-cluster replication CCR synchronization.

8. The data query system according to claim 7, characterized in that, The traffic splitting gateway also includes: A machine learning model is used to dynamically optimize the first preset threshold and the third preset threshold based on historical query data.

9. A data query method, characterized in that, The data query method, applied to any one of claims 1 to 8, comprises: Receive query requests submitted by users; Based on the complexity value of the query request, the query request is routed to the first cluster or the second cluster to trigger the first cluster or the second cluster to perform word segmentation processing on the query request, and the result of the word segmentation processing is used to retrieve the query result in the data index.

10. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method of claim 9.

Images