A data retrieval method and system based on ES large model scoring

Abstract

The embodiment of the specification provides a data retrieval method and system based on ES large model scoring, comprising: performing vectorization processing on field content in business data that needs to support text query, and storing generated vector data and business data in an Elasticsearch index document; receiving a user query condition, generating a query vector based on the query condition, and constructing an Elasticsearch composite query in combination with a keyword query to obtain a candidate business data set; converting the user query condition into a scoring standard and inputting it into a large model, and scoring data in the candidate business data set according to the scoring standard by the large model; and sorting the candidate business data set according to the scoring result and returning the sorted data. The application can improve the accuracy of the retrieval result and significantly optimize the retrieval experience of the user.

Description

Technical Field

[0001] This document relates to the field of computer software development technology, and in particular to a data retrieval method and system based on Elasticsearch large model scoring. Background Technology

[0002] When developing SaaS software, as business data is continuously generated within the system, the amount of data increases significantly. The software system needs to provide data query functionality, retrieving the first n records that meet the user's specified criteria. To achieve this, a common practice is to synchronize the data to Elasticsearch and then query the data from Elasticsearch using keywords.

[0003] In some cases, keyword searches alone are insufficient to meet user query needs. For example, when querying engineering materials business data, if a user searches for data containing the material name "commercial concrete," since "commercial concrete" and "concrete" are the same building material, data containing the material name "concrete" will not be found. In such situations, text vectorization of the business data's text attributes, combined with a keyword-based query, and then scoring and ranking the retrieved data using a large-scale model, can better satisfy users' data query needs. Summary of the Invention

[0004] This specification provides one or more embodiments of a data retrieval method based on Elasticsearch large model scoring, including: S1. Vectorize the content of fields in the business data that need to support text queries, and store the generated vector data in the corresponding Elasticsearch index documents along with the business data; S2. Receive user query conditions, generate query vectors based on the query conditions, and construct Elasticsearch composite queries by combining keyword queries to obtain a candidate business data set; S3. The user query conditions are converted into scoring criteria and input into the large model, which then scores the data in the candidate business data set according to the scoring criteria. S4. Sort the candidate business data set according to the scoring results and return the sorted data.

[0005] Furthermore, the vectorization processing of the field content in the business data that needs to support text queries specifically includes: Determine the list of fields that need to support text queries, concatenate the field name and field value of each field, and then combine the text of each field in the field list to obtain the combined text; The combined text is vectorized to obtain vector data.

[0006] Furthermore, the specific steps of concatenating the field names and values ​​of each field are as follows: The system concatenates the field name, field value, and connector of each field according to a preset format, and combines the concatenated results of all fields into a coherent text.

[0007] Furthermore, generating a query vector based on the query conditions specifically includes: Extract the fields and query parameters related to text queries from the user's query conditions, concatenate the text, and obtain the combined query text; The query combination text is vectorized to obtain the query vector.

[0008] Furthermore, the construction of Elasticsearch composite queries by combining keyword queries includes: In composite queries, weight values ​​are configured for vector query conditions and keyword query conditions respectively, in order to adjust the degree of influence of the two types of conditions on the search results.

[0009] Furthermore, constructing a composite Elasticsearch query also includes: The number of records returned by the query is increased according to the preset magnification factor in order to expand the coverage of the candidate data set.

[0010] Furthermore, the step of converting the user query conditions into scoring criteria and inputting them into the large model, and then having the large model score the data in the candidate business data set according to the scoring criteria, specifically involves: The parameters of each field in the user's query conditions are processed into prompt words, and combined with the business attributes of the queried fields, a scoring standard text is generated for the large model to understand. Based on the scoring criteria, a large model is used to evaluate the relevance of each candidate data to the user's query intent and generate a corresponding score.

[0011] This specification provides one or more embodiments of a data retrieval system based on Elasticsearch large model scoring, including: Data processing module: This module is used to vectorize the content of fields in business data that need to support text queries, and to store the generated vector data in the Elasticsearch index document, corresponding to the business data. Data retrieval module: Used to receive user query conditions, generate query vectors based on the query conditions, and construct Elasticsearch composite queries by combining keyword queries to obtain a set of candidate business data; Scoring module: Used to convert the user query conditions into scoring criteria and input them into the large model, which then scores the data in the candidate business data set according to the scoring criteria. Sorting module: Used to sort the candidate business data set according to the scoring results and return the sorted data.

[0012] This specification provides one or more embodiments of an electronic device, including: Processor; and, The memory is configured to store computer-executable instructions, which, when executed, cause the processor to implement the steps of the data retrieval method based on ES large model scoring described above.

[0013] This specification provides one or more embodiments of a storage medium for storing computer-executable instructions, which, when executed, implement the steps of the data retrieval method based on ES large model scoring described above.

[0014] This invention combines Elasticsearch's fast retrieval capabilities with the deep semantic understanding of a large model. The field names and values ​​to be queried in business data are concatenated into combined text and then vectorized for storage, effectively preserving cross-field semantic relationships and laying the foundation for hybrid retrieval. User query conditions are similarly concatenated and vectorized to generate query vectors, which are then used with keyword queries to construct composite queries. By configuring weights, a flexible balance between semantic search and keyword matching is achieved. Simultaneously, an amplification factor is introduced to expand the candidate data set, ensuring retrieval accuracy and avoiding the omission of potentially high-quality data. User query condition prompts are processed into scoring criteria and then fed into a large model for refined scoring and re-ranking of candidate data, making the ranking results more aligned with users' implicit needs and complex business logic. In summary, this application utilizes Elasticsearch for initial screening of massive amounts of data to ensure retrieval efficiency, and then a large model performs deep semantic evaluation on the narrowed candidate set to improve result accuracy, significantly optimizing the user's retrieval experience.

[0015] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in one or more embodiments of this specification or in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating a data retrieval method based on ES large model scoring, provided for one or more embodiments of this specification; Figure 2 A schematic diagram illustrating the composition of a data retrieval system based on ES large model scoring, provided for one or more embodiments of this specification; Figure 3 This is a schematic diagram of the structure of an electronic device provided for one or more embodiments of this specification. Detailed Implementation

[0018] To enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this document.

[0019] Method Implementation Examples According to embodiments of the present invention, a data retrieval method based on Elasticsearch large model scoring is provided. Figure 1 A flowchart illustrating a data retrieval method based on ES large model scoring, provided for one or more embodiments of this specification, is shown below. Figure 1 As shown, the data retrieval method based on ES large model scoring according to an embodiment of the present invention specifically includes: S1. Vectorize the content of fields in the business data that need to support text queries, and store the generated vector data in the Elasticsearch index document, corresponding to the business data.

[0020] Business data refers to records stored in traditional relational databases or similar data sources. Based on the business data, we first determine which fields need to be included in the subsequent text query scope, select the fields to be queried, and ignore fields that are not suitable or unnecessary for semantic search, thus obtaining a list of text query fields.

[0021] After determining the list of fields to be queried, the field names and values ​​of each field are concatenated. Then, the text from each field in the field list is combined to obtain composite text. Composite text specifically refers to a string with complete semantics, formed by concatenating the field names and values ​​of multiple selected fields according to specific rules. The specific concatenation process is as follows: First, for each field in the list, the text is organized according to a preset format. This preset format can be "field name is field value" or "field name: field value," and a separator is added after each pair of information to ensure the readability of the generated text. All these text fragments corresponding to all fields are combined in a certain order to form a coherent and complete paragraph. This final paragraph is the composite text corresponding to the business data.

[0022] The generated combined text is then vectorized, meaning it's converted into numerical vectors that computers can compute and compare using a specific embedding model. These vectors are fixed-length floating-point numbers; the more semantically similar the text, the closer their corresponding vectors are in space. Finally, the processed data is stored in Elasticsearch.

[0023] S2. Receive user query conditions, generate query vectors based on the query conditions, and construct Elasticsearch composite queries by combining keyword queries to obtain a set of candidate business data.

[0024] When a user initiates a search, the system first receives the user's query conditions and identifies the fields involved in the text query and their corresponding query parameter values. These fields typically correspond to a defined list of text query fields. For each identified text query field, a text concatenation operation is performed to generate a query composite text. This query composite text specifically refers to a string formed by combining the user-input query field names and query parameter values ​​according to a preset format. After concatenation, a vectorization model is invoked to process the query composite text and generate the corresponding query vector.

[0025] After obtaining the query vector, a composite Elasticsearch query is constructed by combining it with keyword queries. A composite query is a query mechanism in Elasticsearch that allows combining multiple different types of subqueries and comprehensively calculating the final relevance score for each document. In this scheme, the composite query mainly consists of two core parts: first, a "vector query" clause based on the query vector, used to perform a similarity search on Elasticsearch's vector field to find business data that is semantically closest to the user's query; second, a traditional keyword query clause, which uses Elasticsearch's inverted index to perform exact or fuzzy matching on the user's input keywords. To enable these two query methods to work together, weight values ​​are configured for both the vector query condition and the keyword query condition in the composite query. Each weight value is a numerical value used to adjust the influence of different subqueries on the final document score, thereby adjusting the degree of influence of the two types of conditions on the search results, controlling the bias of the search results, and achieving a balance between semantic search and keyword search.

[0026] Furthermore, to improve the recall rate of subsequent large-scale model scoring and avoid missing potentially relevant data due to threshold limitations in vector or keyword queries, the solution introduces an amplification factor when constructing composite queries. This factor multiplies the number of records the user expects to return by a coefficient greater than 1, resulting in the actual number of records retrieved from Elasticsearch. For example, if a user requests the top 10 most relevant results, the query count can be amplified to 15 or 20, obtaining a broader set of candidate business data than the final requirement, providing richer material for subsequent refined scoring and re-ranking of large-scale models.

[0027] S3. The user query conditions are converted into scoring criteria and input into the large model, which then scores the data in the candidate business data set according to the scoring criteria.

[0028] After obtaining the candidate business data set, the parameters of each field in the user's original query conditions are processed into suggestive terms. These terms are then transformed by combining the business attributes of the queried fields to generate scoring standard text that can be understood by the large model. Suggestive term processing is the process of converting structured query parameters, combined with the business attributes of the queried fields, into a natural language description. After generating the scoring standard text, it is input into the large model along with the candidate business data set. For each data point in the candidate set, its key fields are organized into a descriptive text, which then interacts with the large model. The large model comprehensively considers the various dimensions mentioned in the scoring criteria and outputs a corresponding score.

[0029] S4. Sort the candidate business data set according to the scoring results and return the sorted data.

[0030] After obtaining the score assigned to each candidate business data point by the large model, based on the relevance score given by the large model after comprehensive evaluation from semantic and business logic perspectives, a sorting algorithm is invoked to arrange the data in descending order of score, forming a sorted list. After sorting, the first n data points are extracted from the top of the sorted list as the final result, where n is the number of data points expected by the user when initiating the query. When the data is returned, the complete information of this data is encapsulated into a unified response format and returned to the caller through an interface. This can be displayed to the user through a front-end interface or processed further by other business systems. The returned data content typically includes all fields of the original business data.

[0031] The beneficial effects of this invention are as follows: By combining Elasticsearch's fast retrieval capabilities with the deep semantic understanding of a large model, the field names and values ​​to be queried in business data are concatenated into combined text and then vectorized for storage. This effectively preserves the semantic relationships across fields, laying the foundation for hybrid retrieval. User query conditions are similarly concatenated and vectorized to generate query vectors, which are then used with keyword queries to construct composite queries. Flexible balance between semantic search and keyword matching is achieved through weight configuration. Simultaneously, an amplification factor is introduced to expand the candidate data set, ensuring retrieval accuracy and avoiding the omission of potentially high-quality data. User query condition prompts are processed into scoring criteria and then fed into a large model for refined scoring and re-ranking of candidate data, making the ranking results more aligned with users' implicit needs and complex business logic. In summary, this application utilizes Elasticsearch for initial screening of massive amounts of data to ensure retrieval efficiency, and then a large model performs deep semantic evaluation on the narrowed candidate set to improve result accuracy, significantly optimizing the user's retrieval experience.

[0032] System Implementation Examples According to embodiments of the present invention, a data retrieval system based on Elasticsearch large model scoring is provided. Figure 2 A schematic diagram illustrating the composition of a data retrieval system based on ES large model scoring, provided for one or more embodiments of this specification, is shown below. Figure 2 As shown, the data retrieval system based on ES large model scoring according to an embodiment of the present invention specifically includes: Data processing module 20: This module is used to vectorize the content of fields in business data that need to support text queries, and to store the generated vector data in the Elasticsearch index document, corresponding to the business data. Data retrieval module 22: used to receive user query conditions, generate query vectors based on the query conditions, and construct Elasticsearch composite queries by combining keyword queries to obtain a candidate business data set; Scoring module 24: Used to convert the user query conditions into scoring criteria and input them into the large model, so that the large model can score the data in the candidate business data set according to the scoring criteria; Sorting module 26: Used to sort the candidate business data set according to the scoring results and return the sorted data.

[0033] The embodiments of the present invention are system embodiments corresponding to the above method embodiments. The specific operation of each module can be understood by referring to the description of the method embodiments, and will not be repeated here.

[0034] Device Example 1 This invention provides an electronic device, such as... Figure 3 As shown, it includes: a memory 30, a processor 32, and a computer program stored in the memory 30 and executable on the processor 32. When the computer program is executed by the processor 32, it performs the following method steps: S1. Vectorize the content of fields in the business data that need to support text queries, and store the generated vector data in the corresponding Elasticsearch index documents along with the business data; S2. Receive user query conditions, generate query vectors based on the query conditions, and construct Elasticsearch composite queries by combining keyword queries to obtain a candidate business data set; S3. The user query conditions are converted into scoring criteria and input into the large model, which then scores the data in the candidate business data set according to the scoring criteria. S4. Sort the candidate business data set according to the scoring results and return the sorted data.

[0035] Device Example 2 This invention provides a computer-readable storage medium storing an information transmission implementation program. When executed by a processor 32, the program performs the following method steps: S1. Vectorize the content of fields in the business data that need to support text queries, and store the generated vector data in the corresponding Elasticsearch index documents along with the business data; S2. Receive user query conditions, generate query vectors based on the query conditions, and construct Elasticsearch composite queries by combining keyword queries to obtain a candidate business data set; S3. The user query conditions are converted into scoring criteria and input into the large model, which then scores the data in the candidate business data set according to the scoring criteria. S4. Sort the candidate business data set according to the scoring results and return the sorted data.

[0036] The computer-readable storage media described in this embodiment include, but are not limited to, ROM, RAM, disk, or optical disk.

[0037] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data retrieval method based on Elasticsearch large model scoring, characterized in that, include: S1. Vectorize the content of fields in the business data that need to support text queries, and store the generated vector data in the corresponding Elasticsearch index documents along with the business data; S2. Receive user query conditions, generate query vectors based on the query conditions, and construct Elasticsearch composite queries by combining keyword queries to obtain a candidate business data set; S3. The user query conditions are converted into scoring criteria and input into the large model, which then scores the data in the candidate business data set according to the scoring criteria. S4. Sort the candidate business data set according to the scoring results and return the sorted data.

2. The method according to claim 1, characterized in that, The vectorization processing of the field content in the business data that needs to support text queries specifically includes: Determine the list of fields that need to support text queries, concatenate the field name and field value of each field, and then combine the text of each field in the field list to obtain the combined text; The combined text is vectorized to obtain vector data.

3. The method according to claim 2, characterized in that, The specific steps of concatenating the field names and values ​​of each field are as follows: The system concatenates the field name, field value, and connector of each field according to a preset format, and combines the concatenated results of all fields into a coherent text.

4. The method according to claim 1, characterized in that, The specific steps of generating a query vector based on the query conditions include: Extract the fields and query parameters related to text queries from the user's query conditions, concatenate the text, and obtain the combined query text; The query combination text is vectorized to obtain the query vector.

5. The method according to claim 1, characterized in that, The method of constructing Elasticsearch composite queries by combining keyword queries includes: In composite queries, weight values ​​are configured for vector query conditions and keyword query conditions respectively, in order to adjust the degree of influence of the two types of conditions on the search results.

6. The method according to claim 1, characterized in that, The construction of Elasticsearch composite queries also includes: The number of records returned by the query is increased according to the preset magnification factor in order to expand the coverage of the candidate data set.

7. The method according to claim 1, characterized in that, The specific steps involve converting the user query conditions into scoring criteria and inputting them into a large model, whereby the large model scores the data in the candidate business data set based on the scoring criteria: The parameters of each field in the user's query conditions are processed into prompt words, and combined with the business attributes of the queried fields, a scoring standard text is generated for the large model to understand. Based on the scoring criteria, a large model is used to evaluate the relevance of each candidate data to the user's query intent and generate a corresponding score.

8. A data retrieval system based on ES large model scoring, characterized in that, include: Data processing module: This module is used to vectorize the content of fields in business data that need to support text queries, and to store the generated vector data in the Elasticsearch index document, corresponding to the business data. Data retrieval module: Used to receive user query conditions, generate query vectors based on the query conditions, and construct Elasticsearch composite queries by combining keyword queries to obtain a set of candidate business data; Scoring module: Used to convert the user query conditions into scoring criteria and input them into the large model, which then scores the data in the candidate business data set according to the scoring criteria. Sorting module: Used to sort the candidate business data set according to the scoring results and return the sorted data.

9. An electronic device, characterized in that, include: processor; as well as, A memory configured to store computer-executable instructions, which, when executed, cause the processor to implement the steps of the data retrieval method based on ES large model scoring as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, Used to store computer-executable instructions, which, when executed, implement the steps of the data retrieval method based on ES large model scoring as described in any one of claims 1 to 7.

Images