Hybrid retrieval method and device for lifecycle stream data based on large language model

Abstract

The present invention relates to the technical field of electronic digital data processing and data stream retrieval, and in particular to a hybrid retrieval method and device for lifecycle stream data based on a large language model, wherein the method comprises: using a large language model to convert query information into semantic retrieval English information, full-text retrieval Chinese information and full-text retrieval English information; using a semantic retrieval string, a full-text retrieval string and a filter condition to perform semantic retrieval and full-text retrieval in a preset database; obtaining a preliminary ranking score according to the weighted sum of the semantic retrieval score and the full-text retrieval score; calculating a comprehensive ranking index according to the consistency index of the semantic retrieval and the full-text retrieval and the ranking confidence attenuation factor; re-sorting the preliminary ranking results based on a deep learning model to generate a final result list containing a unique identifier, ranking and content. Thus, the problem that the relevant technical database has incomplete data, inconsistent standards, non-unique stream names, and difficulty in returning accurate streams is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of electronic digital data processing and data stream retrieval, and in particular to a method and device for hybrid retrieval of life cycle stream data based on a large language model. Background Art

[0002] With the advent of the big data era, various types of data are generated and accumulated at extremely high speeds, including a large amount of stream data. In the lifecycle analysis stream database, accurately retrieving stream information according to needs is an important part of lifecycle analysis, and how to effectively retrieve, analyze and utilize these stream data, especially in combination with large language models for hybrid retrieval, has become a hot topic and difficulty in current research.

[0003] In related technologies, the ability of large language models can be introduced to analyze the user's search intention, output unified search key information, and achieve efficient and accurate retrieval of text-multimodal data by combining structured retrieval and vector retrieval.

[0004] However, in the related technologies, existing databases including GaBi, Encoinvent, ELCD, etc. generally have problems such as incomplete data, inconsistent standards, and non-unique stream names, which makes it difficult for traditional full-text retrieval and semantic retrieval to return accurate streams, which urgently needs to be improved. Summary of the invention

[0005] The present invention provides a hybrid retrieval method and device for lifecycle stream data based on a large language model to solve the problems of incomplete data, inconsistent standards, non-unique stream names, etc. in databases in related technologies, which easily lead to traditional full-text retrieval and semantic retrieval being difficult to return accurate streams.

[0006] The first aspect of the present invention provides a hybrid retrieval method for lifecycle stream data based on a large language model, comprising the following steps: receiving a retrieval request input by a user containing query information and filtering conditions for a stream; in response to the retrieval request, using a large language model to convert the query information into semantic retrieval English information, full-text retrieval Chinese information and full-text retrieval English information, and converting them into semantic retrieval strings and full-text retrieval strings respectively; using the semantic retrieval strings, the full-text retrieval strings and the filtering conditions to perform semantic retrieval and full-text retrieval in a preset database, and obtaining a semantic retrieval result list and a full-text retrieval result list sorted by similarity respectively; and converting the semantic retrieval result list and the full-text retrieval result list into the search results. The full-text search result lists are merged, and the corresponding semantic search score and full-text search score are calculated according to each result in the merged result list, so as to obtain a preliminary ranking score according to the weighted sum of the semantic search score and the full-text search score; based on the preliminary ranking score, a comprehensive ranking index is calculated according to the consistency index of the semantic search and the full-text search and the ranking confidence attenuation factor, and based on the comprehensive ranking index, the merged result list is sorted in descending order, a comprehensive sorting result list is returned, and the results that meet the preset number condition are retained to generate a preliminary sorting result; the preliminary sorting results are re-sorted based on the deep learning model to generate a final result list containing a unique identifier, ranking and content.

[0007] Optionally, in one embodiment of the present invention, the converting into a semantic search string and a full-text search string respectively includes: converting the full-text search Chinese information and the full-text search English information into string types, and connecting them with a preset string to obtain the full-text search string.

[0008] Optionally, in one embodiment of the present invention, the converting into a semantic search string and a full-text search string respectively further includes: converting the semantic search English information into an embedding vector using a preset model; and converting the embedding vector into the semantic search string.

[0009] Optionally, in one embodiment of the present invention, the calculation formula for the similarity score between the query statement and the document in the full-text search result list is:

[0010] ,

[0011] in, For statement Included Keywords, For documentation, is the number of documents, is the inverse document frequency, is the document word frequency, For each result in the merged result list, is the query word frequency;

[0012] The vector in the semantic search result list and vector The calculation formula for the similarity between them is:

[0013] ,

[0014] in, For each result in the merged result list, is the number of documents, Respectively represent the vector and the vector No. A quantity.

[0015] Optionally, in one embodiment of the present invention, the calculation formulas for the semantic search score and the full-text search score are:

[0016] ,

[0017] in, For each result in the merged result list, To avoid calculating constants that are too large, and The ranking of each result in the merged result list in the full-text search and semantic search respectively.

[0018] Optionally, in one embodiment of the present invention, the calculation formula for the initial score in the merged result list is:

[0019] ,

[0020] in, and are the weights of full-text retrieval and semantic retrieval, respectively. is the full-text search score, is the semantic retrieval score, For each result in the merged result list, Represents full-text search, stands for semantic retrieval;

[0021] The calculation formula for the consistency index of the semantic retrieval and full-text retrieval is:

[0022] ,

[0023] in, is the total number of search results, and The ranking of each result in the combined result list in full-text search and semantic search respectively;

[0024] The calculation formula for the consistency score of the semantic search and full-text search is:

[0025] ,

[0026] in, For a control Adjustment parameters that affect the final score. is the initial score, is a consistency index of the semantic retrieval and full-text retrieval;

[0027] The calculation formula of the ranking confidence attenuation factor is:

[0028] ,

[0029] in, is the attenuation rate, is the initial score;

[0030] The calculation formula of the comprehensive ranking index is:

[0031] ,

[0032] in, is the consistency score of the semantic retrieval and full-text retrieval, is the ranking confidence decay factor.

[0033] The second aspect of the present invention provides a hybrid retrieval device for lifecycle stream data based on a large language model, including: a receiving module for receiving a search request input by a user containing query information and filtering conditions for a stream; a conversion module for responding to the search request, using a large language model to convert the query information into semantic retrieval English information, full-text retrieval Chinese information and full-text retrieval English information, and converting them into semantic retrieval strings and full-text retrieval strings respectively; an acquisition module for performing semantic retrieval and full-text retrieval in a preset database using the semantic retrieval strings, the full-text retrieval strings and the filtering conditions, and obtaining a semantic retrieval result list and a full-text retrieval result list sorted according to similarity respectively; a calculation module for converting the semantic retrieval result list into a full-text retrieval result list; and a calculation module for converting the semantic retrieval result list into a full-text retrieval result list. The table is merged with the full-text search result list, and the corresponding semantic search score and full-text search score are calculated according to each result in the merged result list, so as to obtain a preliminary ranking score according to the weighted sum of the semantic search score and the full-text search score; a descending sorting module is used to calculate a comprehensive ranking index based on the preliminary ranking score, according to the consistency index of the semantic search and the full-text search and the ranking confidence attenuation factor, and based on the comprehensive ranking index, the merged result list is sorted in descending order, the comprehensive sorting result list is returned, and the results that meet the preset number condition are retained to generate a preliminary sorting result; a re-sorting module is used to re-sort the preliminary sorting results based on the deep learning model, and generate a final result list containing a unique identifier, ranking and content.

[0034] Optionally, in an embodiment of the present invention, the conversion module includes: a first conversion unit, configured to convert the full-text search Chinese information and the full-text search English information into a string type, and connect them with a preset string to obtain the full-text search string.

[0035] Optionally, in one embodiment of the present invention, the conversion module further includes: a second conversion unit, used to convert the semantic retrieval English information into an embedding vector using a preset model; and a conversion unit, used to convert the embedding vector into the semantic retrieval character string.

[0036] Optionally, in one embodiment of the present invention, the calculation formula for the similarity score between the query statement and the document in the full-text search result list is:

[0037] ,

[0038] in, For statement Included Keywords, For documentation, is the number of documents, is the inverse document frequency, is the document word frequency, For each result in the merged result list, is the query word frequency;

[0039] The vector in the semantic search result list and vector The calculation formula for the similarity between them is:

[0040] ,

[0041] in, For each result in the merged result list, is the number of documents, Respectively represent the vector and the vector No. A quantity.

[0042] Optionally, in one embodiment of the present invention, the calculation formulas for the semantic search score and the full-text search score are:

[0043] ,

[0044] in, For each result in the merged result list, To avoid calculating constants that are too large, and The ranking of each result in the merged result list in the full-text search and semantic search respectively.

[0045] Optionally, in one embodiment of the present invention, the calculation formula for the initial score in the merged result list is:

[0046] ,

[0047] in, and are the weights of full-text retrieval and semantic retrieval, respectively. is the full-text search score, is the semantic retrieval score, For each result in the merged result list, Represents full-text search, stands for semantic retrieval;

[0048] The calculation formula for the consistency index of the semantic retrieval and full-text retrieval is:

[0049] ,

[0050] in, is the total number of search results, and The ranking of each result in the combined result list in full-text search and semantic search respectively;

[0051] The calculation formula for the consistency score of the semantic search and full-text search is:

[0052] ,

[0053] in, For a control Adjustment parameters that affect the final score. is the initial score, is a consistency index of the semantic retrieval and full-text retrieval;

[0054] The calculation formula of the ranking confidence attenuation factor is:

[0055] ,

[0056] in, is the attenuation rate, is the initial score;

[0057] The calculation formula of the comprehensive ranking index is:

[0058] ,

[0059] in, is the consistency score of the semantic retrieval and full-text retrieval, is the ranking confidence decay factor.

[0060] An embodiment of the third aspect of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the hybrid retrieval method for lifecycle stream data based on a large language model as described in the above embodiment.

[0061] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-mentioned hybrid retrieval method for lifecycle stream data based on a large language model.

[0062] A fifth aspect of the present invention provides a computer program product, which stores a computer program that, when executed by a processor, implements the above-mentioned hybrid retrieval method for lifecycle stream data based on a large language model.

[0063] The embodiment of the present invention can use a large language model to summarize and extract key information of the user input content, avoid the influence of invalid information on semantic retrieval, use the inverse sorting fusion algorithm to comprehensively sort the full-text retrieval and semantic retrieval results, and accurately retrieve all qualified flows in the database according to the relevant information of the flow input by the user, improve the quality of the query content, improve the accuracy of the retrieval results, and support multi-language queries. Thus, the database in the related technology is solved, and the problems such as incomplete data, inconsistent standards, and non-unique flow names are common, which easily lead to the difficulty of traditional full-text retrieval and semantic retrieval to return accurate flows.

[0064] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0066] Figure 1 A flowchart of a hybrid retrieval method for lifecycle stream data based on a large language model provided according to an embodiment of the present invention;

[0067] Figure 2 A flowchart of a hybrid retrieval method for lifecycle stream data based on a large language model according to an embodiment of the present invention;

[0068] Figure 3 A schematic diagram of the structure of a hybrid retrieval device for lifecycle stream data based on a large language model according to an embodiment of the present invention;

[0069] Figure 4 It is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0070] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0071] The following describes the hybrid retrieval method and device of lifecycle stream data based on a large language model according to an embodiment of the present invention with reference to the accompanying drawings. With respect to the databases in the related technologies mentioned in the above background technology, there are generally problems such as incomplete data, inconsistent standards, and non-unique stream names, which easily lead to the difficulty of traditional full-text search and semantic search to return accurate streams. The present invention provides a hybrid retrieval method for lifecycle stream data based on a large language model. In this method, the large language model can be used to summarize and extract key information of the user input content, avoid the influence of invalid information on semantic search, and use the inverse sorting fusion algorithm to comprehensively sort the full-text search and semantic search results, so as to accurately retrieve all the streams that meet the requirements in the database according to the relevant information of the stream input by the user, improve the query content quality, improve the accuracy of the search results, and support multi-language queries. Thus, the problems such as incomplete data, inconsistent standards, and non-unique stream names, which easily lead to the difficulty of traditional full-text search and semantic search to return accurate streams in the database in the related technologies are solved.

[0072] Specifically, Figure 1 A schematic flow chart of a hybrid retrieval method for lifecycle stream data based on a large language model provided in an embodiment of the present invention.

[0073] like Figure 1 As shown, the hybrid retrieval method for lifecycle stream data based on a large language model includes the following steps:

[0074] In step S101, a search request including flow query information and filter conditions input by a user is received.

[0075] It can be understood that the query information of the flow in the embodiment of the present invention includes but is not limited to: the Chinese and English names of the flow, the chemical and physical properties of the flow, the molecular formula of the flow and the environmental impact of the flow.

[0076] In the actual execution process, the user in the embodiment of the present invention can input the "query information" of the flow according to their own needs, such as the Chinese and English names of the flow, the chemical and physical properties of the flow, the molecular formula of the flow and the environmental impact of the flow. Furthermore, the user in the embodiment of the present invention can also choose whether to input the "filter conditions" of the flow. If not, the filter conditions are empty by default. The embodiment of the present invention improves the quality of the query content by receiving the search request containing query information and filter conditions input by the user, ensures that the key information of the user input content can be summarized and refined by the large language model in the future, avoids the influence of invalid information on semantic retrieval, and expands synonyms according to the name of the flow, so as to generate accurate and comprehensive full-text query content.

[0077] For example, in the embodiment of the present invention, the user can input the English name of the stream as hexafluoropropane, the Chinese name as hexafluoropropylene, the chemical and physical properties as halogenated olefins, the molecular formula as C3H2F6, and the impact on the environment as possible toxicity and environmental impact according to their own needs.

[0078] In step S102, in response to the search request, the query information is converted into semantic search English information, full-text search Chinese information and full-text search English information using a large language model, and into semantic search strings and full-text search strings respectively.

[0079] It can be understood that the large language model in the embodiment of the present invention has powerful natural language processing capabilities and broad application prospects.

[0080] As a possible implementation method, the embodiment of the present invention can respond to the search request, use the large language model to pre-process the query information, summarize and condense the content of the "query information", and convert the query information into three parts: semantic search English information, full-text search Chinese information and full-text search English information. Among them, the semantic search information contains English keywords for semantic search, and the full-text search information contains the original name of the stream used for full-text search and its synonyms, which are converted into semantic search strings and full-text search strings respectively, so as to support multi-language queries.

[0081] The embodiment of the present invention can use a large language model to convert the query information input by the user into Chinese and English query strings, thereby solving the retrieval difficulty problem caused by inconsistent database languages. At the same time, other languages ​​can be expanded to achieve cross-language stream information retrieval.

[0082] Optionally, in one embodiment of the present invention, converting into a semantic search string and a full-text search string respectively includes: converting the full-text search Chinese information and the full-text search English information into string types, and connecting them with a preset string to obtain the full-text search string.

[0083] It can be understood that the preset string connection in the embodiment of the present invention may be a string “OR” connection.

[0084] Specifically, the embodiment of the present invention can convert "full text search Chinese information" and "full text search English information" into a string type, and connect them with the string "OR" to obtain a full text search string. , thereby connecting search strings in different languages, simplifying user operations, improving search efficiency, providing support for meeting the needs of users in different languages, and further improving the accuracy of search results.

[0085] It should be noted that the preset string connection can be set by those skilled in the art according to actual conditions and is not specifically limited here.

[0086] Optionally, in one embodiment of the present invention, the conversion into a semantic search string and a full-text search string respectively includes: using a preset model to convert the semantic search English information into an embedding vector; and converting the embedding vector into a semantic search string.

[0087] It can be understood that the preset model in the embodiment of the present invention can be an embedded model, such as an OpenAIEmbeddings model.

[0088] In the actual implementation process, the embodiment of the present invention can use the OpenAI Embeddings model to convert the "semantic retrieval English information" into an embedding vector, and then convert the embedding vector into a semantic retrieval string. , thereby improving the accuracy of retrieval results and supporting complex queries in multiple languages.

[0089] It should be noted that the preset model can be set by those skilled in the art according to actual conditions and is not specifically limited here.

[0090] In step S103, semantic search and full-text search are performed in a preset database using the semantic search string, the full-text search string and the filter conditions to obtain a semantic search result list and a full-text search result list sorted by similarity, respectively.

[0091] It is understandable that the embodiment of the present invention can use the search string and the filter condition to carry out a mixed search in the database.

[0092] As a possible implementation method, an embodiment of the present invention can use a full-text search string and filter conditions to perform a full-text search in a database, and return a full-text search result list sorted by similarity, and use a semantic search string and filter conditions to perform a semantic search in a database, and return a semantic search result list sorted by similarity. Thus, a semantic search result list sorted by similarity and a full-text search result list are obtained.

[0093] In one embodiment of the present invention, the calculation formula for the similarity score between the query statement and the document in the full-text search result list is:

[0094] ,

[0095] in, For the query statement and Documentation The similarity score between For statement Included Keywords, For documentation, is the number of documents, is the inverse document frequency, is the document word frequency, For each result in the merged results list, is the query word frequency.

[0096] is the inverse document frequency, which can measure The rarity of a document among all documents indirectly reflects the amount of information it contains, and is calculated as follows:

[0097] ,

[0098] in, Representatives include The number of documents, Indicates the total number of documents. For statement Included Keywords;

[0099] is the document word frequency, representative word In the documentation The relative frequency in is calculated as follows;

[0100] ,

[0101] in, represent In the documentation The number of times it appears in Representative Documents The number of words, Represents the average number of words in an article; parameter is an adjustment factor. The larger the value, the greater the influence of document word frequency. Indicates ignoring the influence of document word frequency, generally set to 1.2; parameter is the normalization factor, represents no normalization at all. Represents complete normalization, usually 0.75.

[0102] Query word frequency, representative words In the statement The relative frequency in is calculated as follows:

[0103] ,

[0104] in, represent In the query statement The number of times it appears in is an adjustment factor. The larger the value, the greater the influence of query frequency. Indicates ignoring the influence of query word frequency, usually set to 0.

[0105] Furthermore, in the semantic search result list in the embodiment of the present invention, the vector and vector The calculation formula for the similarity between them is:

[0106] ,

[0107] in, For vector and vector The similarity between them is in [0,1], and the closer to 1, the higher the similarity. is the number of documents, Represents vectors and vector No. A quantity.

[0108] The embodiment of the present invention can use semantic search strings, full-text search strings and filtering conditions to perform semantic search and full-text search in a preset database. The full-text search can widely capture flows with similar names, and the semantic search can accurately locate flows with similar contents, solving the problems of non-unique flow names and incomplete data. The embodiment of the present invention can set the weights of the two search methods by itself, thereby further improving the accuracy of the search results.

[0109] It should be noted that the preset database can be set by those skilled in the art according to actual conditions and is not specifically limited here.

[0110] In step S104, the semantic search result list and the full-text search result list are merged, and the corresponding semantic search score and full-text search score are calculated according to each result in the merged result list, so as to obtain a preliminary ranking score according to the weighted sum of the semantic search score and the full-text search score.

[0111] It is understandable that the embodiments of the present invention can comprehensively rank the results returned by the full-text search and the semantic search based on the Reciprocal Rank Fusion (RRF) algorithm.

[0112] During the actual execution process, an embodiment of the present invention can merge the result list returned by semantic retrieval and the result list returned by full-text retrieval. For example, the full-text retrieval returns {result 1, result 2}, and the semantic retrieval returns {result 2, result 3}, then the merged list is {result 1, result 2, result 3}, and the corresponding semantic retrieval score and full-text retrieval score are calculated based on each result in the merged result list, so as to obtain a preliminary ranking score based on the weighted sum of the semantic retrieval score and the full-text retrieval score, which can further improve the accuracy of the retrieval results.

[0113] In one embodiment of the present invention, the calculation formulas for the semantic search score and the full-text search score are:

[0114] ,

[0115] in, For each result in the merged results list, To avoid calculating constants that are too large, and The ranking of each result in the merged result list in full-text search and semantic search respectively.

[0116] Specifically, for each result in the merge list , its full-text retrieval score can be calculated according to the above formula and semantic retrieval score , To avoid calculating constants with too large values, you can set them yourself. and The ranking of each result in the combined result list in full-text search and semantic search respectively. If a result comes from only one search method, then the score of the other search method is 0. For example, if a result is only retrieved by full-text search, its semantic search score is 0.

[0117] In one embodiment of the present invention, the calculation formula for the initial score in the merged result list is:

[0118] ,

[0119] in, and They are the weights for full-text search and semantic search, which can be set by yourself. is the full-text search score, is the semantic retrieval score, For each result in the merged results list, Represents full-text search, Stands for semantic retrieval.

[0120] For the results , calculate the consistency index (CI) of the results of the two retrieval methods:

[0121] ,

[0122] in, is the total number of search results. If the document does not appear in the full-text search, =total_results + 1; if the document does not appear in the semantic search, = total_results + 1; and are the rankings of the result in full-text retrieval and semantic retrieval respectively;

[0123] Will and Combined, the calculation formula for the consistency score of semantic retrieval and full-text retrieval is:

[0124] ,

[0125] in, is an adjustment parameter used to control The degree of influence on the final score can be set in the range of [0, 1]. is the initial score, It is the consistency index of the results of the two retrieval methods;

[0126] Furthermore, considering that the reliability of the retrieval results may decrease as the ranking decreases, the confidence decay factor (CDF) is introduced. The calculation formula of the ranking confidence decay factor is:

[0127] ,

[0128] in, is the attenuation rate, is the initial score;

[0129] Furthermore, the final calculation formula for the comprehensive ranking index is:

[0130] ,

[0131] in, is the consistency score between semantic retrieval and full-text retrieval, is the ranking confidence decay factor.

[0132] Specifically, for each result in the combined list, its comprehensive ranking can be calculated according to the above formula, thereby providing support for the subsequent generation of a final result list containing a unique identifier, ranking and content, thereby further improving the accuracy of the retrieval results.

[0133] In step S105, based on the preliminary ranking score, the comprehensive ranking index is calculated according to the consistency index of semantic retrieval and full-text retrieval and the ranking confidence attenuation factor, and based on the comprehensive ranking index, the merged result list is sorted in descending order, the comprehensive ranking result list is returned, and the results that meet the preset number conditions are retained to generate a preliminary ranking result.

[0134] It can be understood that the results satisfying the preset number condition in the embodiment of the present invention may be the top-k results.

[0135] As a possible implementation method, an embodiment of the present invention can calculate a comprehensive ranking index based on the preliminary ranking score, according to the consistency indicators of the two retrieval methods, semantic retrieval and full-text retrieval, and the ranking confidence attenuation factor, and based on the comprehensive ranking index, sort the merged result list in descending order, return the comprehensive ranking result list and the corresponding scores, retain the top-k results, provide preliminary retrieval ranking results, and filter out obviously irrelevant results.

[0136] The embodiment of the present invention improves the accuracy of the search results and integrates the advantages of two search methods: full-text search can widely capture flows with similar names, while semantic search can accurately locate flows with similar contents, thereby solving problems such as non-unique flow names and incomplete data.

[0137] It should be noted that the preset number condition can be set by those skilled in the art according to actual conditions and is not specifically limited here.

[0138] In step S106, the preliminary sorting result list is re-sorted based on the deep learning model to generate a final result list including a unique identifier, ranking and content.

[0139] In the actual implementation process, the embodiment of the present invention can reorder the preliminary search results based on the deep learning model, and return a final result list containing a unique identifier (uuid), ranking and content. The reordering method based on the deep learning model can further improve the accuracy of the search results. The deep learning network structure includes an input layer, an encoding layer, a relevance scoring layer, and a sorting layer. The specific operation method is as follows:

[0140] (1) Input layer: based on sorted document list , for each document , and the query statement , , Make an input pair , and concatenate using a special delimiter:

[0141] ,

[0142] in, A marker indicating the start of a task. It is a separator between tasks and converts the text into a token sequence that the model can understand.

[0143] (2) Encoding layer: Use the pre-trained Transformer model to encode each input pair Encode the full text and semantic features respectively to obtain their vector representations:

[0144] ,

[0145] ,

[0146] Due to the presence of special separators, the Transformer model acts on the input pair Multiple vectors can be obtained from the above. In order to represent the encoding of the entire input, keep The vector corresponding to the mark is the first vector. The full-text features and semantic features are weighted fused:

[0147] ,

[0148] in, and They are full-text vector representation and semantic vector representation respectively. and They are the weights of full-text retrieval and semantic retrieval respectively.

[0149] (3) Relevance scoring layer: using the linear layer And the activation function (e.g. Softmax) to compute the relevance score for each input pair:

[0150] ,

[0151] in, is the weighted fusion result of full-text features and semantic features. is a linear layer.

[0152] (4) Sorting layer: Sort by similarity score from high to low to get a rearranged document list:

[0153] ,

[0154] in, To sort documents, Score the relevance of each input pair. Also returns the document's unique identifier (uuid), rank, and content to get the final result list.

[0155] Specifically, it can be combined Figure 2 As shown, the working principle of the hybrid retrieval method for lifecycle stream data based on a large language model in the embodiment of the present invention is described in detail with a specific embodiment.

[0156] like Figure 2 As shown, the embodiment of the present invention may include the following steps:

[0157] Step S201: input query content and filtering conditions.

[0158] Among them, the user in the embodiment of the present invention can input the query information of the flow according to his own needs, and can also choose whether to input the filtering condition of the flow.

[0159] Step S202: extract, embed and convert into a query string using a large language model.

[0160] The embodiment of the present invention can utilize a large language model to convert the query information and the filter conditions into a search string, that is, into a query string.

[0161] Step S203: Perform full-text search.

[0162] Among them, the embodiment of the present invention can perform full-text search in the database.

[0163] Step S204: perform semantic retrieval.

[0164] Among them, the embodiment of the present invention can perform semantic retrieval in the database.

[0165] Step S205: Return the full-text search results.

[0166] Among them, the embodiment of the present invention can return full-text search results sorted by similarity.

[0167] Step S206: Return the semantic search result.

[0168] Among them, the embodiment of the present invention can return semantic search results sorted by similarity.

[0169] Step S207: Integrate the search results.

[0170] Among them, the embodiment of the present invention can merge the result lists returned by the full-text search and the semantic search to integrate the search results.

[0171] Step S208: Calculate the weighted score of the search results.

[0172] Among them, the embodiment of the present invention can calculate the weighted sum of the semantic search score and the full-text search score as the preliminary ranking score.

[0173] Step S209: Return the final sorting result list.

[0174] Among them, the embodiment of the present invention can re-sort the preliminary sorting results based on the deep learning model and return a final result list including a unique identifier, ranking and content.

[0175] According to the hybrid retrieval method for lifecycle stream data based on a large language model proposed in an embodiment of the present invention, the large language model can be used to summarize and extract key information of the user input content, avoid the influence of invalid information on semantic retrieval, and use the inverse sorting fusion algorithm to comprehensively sort the full-text retrieval and semantic retrieval results, so as to accurately retrieve all qualified streams in the database according to the relevant information of the stream input by the user, improve the quality of the query content, improve the accuracy of the retrieval results, and support multi-language queries. Thus, the problem that the database in the related technology generally has incomplete data, inconsistent standards, and non-unique stream names, which easily leads to the difficulty of traditional full-text retrieval and semantic retrieval to return accurate streams is solved.

[0176] Next, a hybrid retrieval device for lifecycle stream data based on a large language model according to an embodiment of the present invention will be described with reference to the accompanying drawings.

[0177] Figure 3 It is a structural diagram of a hybrid retrieval device for lifecycle stream data based on a large language model according to an embodiment of the present invention.

[0178] like Figure 3 As shown, the lifecycle stream data hybrid retrieval device 10 based on a large language model includes: a receiving module 100, a conversion module 200, an acquisition module 300, a calculation module 400, a descending sorting module 500 and a reordering module 600.

[0179] Specifically, the receiving module 100 is used to receive a search request input by a user that includes query information of a flow and a filter condition.

[0180] The conversion module 200 is used to respond to the search request and use the large language model to convert the query information into semantic search English information, full-text search Chinese information and full-text search English information, and convert them into semantic search strings and full-text search strings respectively.

[0181] The acquisition module 300 is used to perform semantic search and full-text search in a preset database using the semantic search string, the full-text search string and the filter condition, and obtain a semantic search result list and a full-text search result list sorted by similarity, respectively.

[0182] The calculation module 400 is used to merge the semantic search result list and the full-text search result list, and calculate the corresponding semantic search score and full-text search score according to each result of the merged result list, so as to obtain a preliminary ranking score according to the weighted sum of the semantic search score and the full-text search score.

[0183] The descending sorting module 500 is used to calculate the comprehensive ranking index based on the preliminary ranking score, according to the consistency index of semantic retrieval and full-text retrieval and the ranking confidence attenuation factor, and based on the comprehensive ranking index, sort the merged result list in descending order, return the comprehensive sorting result list, and retain the results that meet the preset number conditions to generate a preliminary sorting result.

[0184] The re-ranking module 600 is used to re-rank the preliminary ranking results based on the deep learning model to generate a final result list including a unique identifier, ranking and content.

[0185] Optionally, in one embodiment of the present invention, the conversion module 200 includes: a first conversion unit.

[0186] The first conversion unit is used to convert the full-text search Chinese information and the full-text search English information into a character string type, and connect them with a preset character string to obtain a full-text search character string.

[0187] Optionally, in one embodiment of the present invention, the conversion module 200 further includes: a second conversion unit and a conversion unit.

[0188] The second conversion unit is used to convert the semantically retrieved English information into an embedding vector using a preset model.

[0189] The conversion unit is used to convert the embedding vector into a semantic retrieval string.

[0190] Optionally, in one embodiment of the present invention, the calculation formula for the similarity score between the query statement and the document in the full-text search result list is:

[0191] ,

[0192] in, For statement Included Keywords, For documentation, is the number of documents, is the inverse document frequency, is the document word frequency, For each result in the merged results list, is the query word frequency;

[0193] Vectors in the semantic search result list and vector The calculation formula for the similarity between them is:

[0194] ,

[0195] in, For each result in the merged results list, is the number of documents, Represents vectors and vector No. A quantity.

[0196] Optionally, in one embodiment of the present invention, the calculation formulas for the semantic search score and the full-text search score are:

[0197] ,

[0198] in, For each result in the merged results list, To avoid calculating constants that are too large, and The ranking of each result in the merged result list in full-text search and semantic search respectively.

[0199] Optionally, in one embodiment of the present invention, the calculation formula for the initial score in the merged result list is:

[0200] ,

[0201] in, and are the weights of full-text retrieval and semantic retrieval, respectively. is the full-text search score, is the semantic retrieval score, For each result in the merged results list, Represents full-text search, stands for semantic retrieval;

[0202] The calculation formula for the consistency index of semantic retrieval and full-text retrieval is:

[0203] ,

[0204] in, is the total number of search results, and The ranking of each result in the merged result list in full-text search and semantic search respectively;

[0205] The calculation formula for the consistency score of semantic retrieval and full-text retrieval is:

[0206] ,

[0207] in, For a control Adjustment parameters that affect the final score. is the initial score, It is the consistency index of semantic retrieval and full-text retrieval;

[0208] The calculation formula of ranking confidence decay factor is:

[0209] ,

[0210] in, is the attenuation rate, is the initial score;

[0211] The calculation formula of the comprehensive ranking index is:

[0212] ,

[0213] in, is the consistency score between semantic retrieval and full-text retrieval, is the ranking confidence decay factor.

[0214] It should be noted that the above explanation of the embodiment of the method for hybrid retrieval of lifecycle stream data based on a large language model is also applicable to the device for hybrid retrieval of lifecycle stream data based on a large language model in this embodiment, and will not be repeated here.

[0215] According to the hybrid retrieval device for lifecycle stream data based on a large language model proposed in an embodiment of the present invention, the large language model can be used to summarize and extract key information of the user input content, avoid the influence of invalid information on semantic retrieval, and use the inverse sorting fusion algorithm to comprehensively sort the full-text retrieval and semantic retrieval results, so as to accurately retrieve all qualified streams in the database according to the relevant information of the stream input by the user, improve the quality of the query content, improve the accuracy of the retrieval results, and support multi-language queries. Thus, the problem that the database in the related technology generally has incomplete data, inconsistent standards, and non-unique stream names, which easily leads to the difficulty of traditional full-text retrieval and semantic retrieval to return accurate streams, is solved.

[0216] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device may include:

[0217] Memory 401 , processor 402 , and a computer program stored in the memory 401 and executable on the processor 402 .

[0218] When the processor 402 executes the program, the hybrid retrieval method for lifecycle stream data based on a large language model provided in the above embodiment is implemented.

[0219] Furthermore, the electronic device further comprises:

[0220] The communication interface 403 is used for communication between the memory 401 and the processor 402 .

[0221] The memory 401 is used to store computer programs that can be executed on the processor 402 .

[0222] The memory 401 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0223] If the memory 401, the processor 402 and the communication interface 403 are implemented independently, the communication interface 403, the memory 401 and the processor 402 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0224] Optionally, in a specific implementation, if the memory 401, the processor 402 and the communication interface 403 are integrated on a chip, the memory 401, the processor 402 and the communication interface 403 can communicate with each other through an internal interface.

[0225] The processor 402 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention.

[0226] This embodiment also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method for hybrid retrieval of lifecycle stream data based on a large language model is implemented as described above.

[0227] An embodiment of the present invention further provides a computer program product on which a computer program is stored, and when the program is executed by a processor, the method for hybrid retrieval of lifecycle stream data based on a large language model as described above is implemented.

[0228] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0229] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "N" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0230] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present invention belong.

[0231] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection with one or N wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary and then storing it in a computer memory.

[0232] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0233] A person skilled in the art may understand that all or part of the steps in the above-mentioned embodiment method may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0234] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0235] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A hybrid retrieval method for lifecycle stream data based on a large language model, characterized in that: The following steps are involved: Receiving a search request input by a user including query information and filter conditions of a flow; In response to the search request, the query information is converted into semantic search English information, full-text search Chinese information and full-text search English information by using a large language model, and converted into semantic search character strings and full-text search character strings respectively; Using the semantic search string, the full-text search string and the filtering condition to perform semantic search and full-text search in a preset database, respectively obtaining a semantic search result list and a full-text search result list sorted according to similarity; Merging the semantic search result list and the full-text search result list, and calculating a corresponding semantic search score and a full-text search score according to each result in the merged result list, so as to obtain a preliminary ranking score according to a weighted sum of the semantic search score and the full-text search score; Based on the preliminary ranking score, a comprehensive ranking index is calculated according to the consistency index of semantic retrieval and full-text retrieval and the ranking confidence attenuation factor, and based on the comprehensive ranking index, the merged result list is sorted in descending order, a comprehensive sorting result list is returned, and results that meet a preset number condition are retained to generate a preliminary sorting result; Re-ranking the preliminary ranking results based on the deep learning model to generate a final result list including a unique identifier, a ranking, and content; The calculation formula for the similarity score between the query statement and the document in the full-text search result list is: ， in, For statement Included Keywords, For documentation, is the number of documents, is the inverse document frequency, is the document word frequency, For each result in the merged result list, is the query word frequency; The vector in the semantic search result list and vector The formula for calculating the similarity between them is: ， in, For each result in the merged result list, is the number of documents, Respectively represent the vector and the vector No. Quantity; The calculation formulas for the semantic search score and the full-text search score are: ， in, For each result in the merged result list, To avoid calculating constants that are too large, and The ranking of each result in the combined result list in full-text search and semantic search respectively; The calculation formula for the initial score in the merged result list is: ， in, and are the weights of full-text retrieval and semantic retrieval, respectively. is the full-text search score, is the semantic retrieval score, For each result in the merged result list, Represents full-text search, stands for semantic retrieval; The calculation formula for the consistency index of the semantic retrieval and full-text retrieval is: ， in, is the total number of search results, and The ranking of each result in the combined result list in full-text search and semantic search respectively; The calculation formula for the consistency score of the semantic search and full-text search is: ， in, For a control Adjustment parameters that affect the final score. is the initial score, is a consistency index of the semantic retrieval and full-text retrieval; The calculation formula of the ranking confidence attenuation factor is: ， in, is the attenuation rate, is the initial score; The calculation formula of the comprehensive ranking index is: ， in, is the consistency score of the semantic retrieval and full-text retrieval, is the ranking confidence decay factor.

2. The hybrid retrieval method for lifecycle stream data based on a large language model according to claim 1 is characterized in that: The conversion into a semantic search string and a full-text search string respectively includes: The full-text search Chinese information and the full-text search English information are converted into character string types, and connected with a preset character string to obtain the full-text search character string.

3. The hybrid retrieval method for lifecycle stream data based on a large language model according to claim 1 is characterized in that: The conversion into a semantic search string and a full-text search string respectively also includes: Using a preset model to convert the semantically retrieved English information into an embedding vector; The embedding vector is converted into the semantic search string.

4. A hybrid retrieval device for lifecycle stream data based on a large language model, characterized in that: include: A receiving module, used for receiving a search request input by a user containing query information and filter conditions of a flow; A conversion module, for responding to the search request, using a large language model to convert the query information into semantic search English information, full-text search Chinese information and full-text search English information, and converting them into semantic search strings and full-text search strings respectively; An acquisition module, used to perform semantic search and full-text search in a preset database using the semantic search string, the full-text search string and the filtering condition, and obtain a semantic search result list and a full-text search result list sorted by similarity, respectively; A calculation module, used for merging the semantic search result list and the full-text search result list, and calculating the corresponding semantic search score and full-text search score according to each result of the merged result list, so as to obtain a preliminary ranking score according to the weighted sum of the semantic search score and the full-text search score; A descending sorting module is used to calculate a comprehensive ranking index based on the preliminary ranking score, according to the consistency index of semantic retrieval and full-text retrieval and the ranking confidence attenuation factor, and to sort the merged result list in descending order based on the comprehensive ranking index, return the comprehensive sorting result list, and retain the results that meet the preset number condition to generate a preliminary sorting result; A re-ranking module, used to re-rank the preliminary ranking results based on a deep learning model to generate a final result list including a unique identifier, a ranking and content; The calculation formula for the similarity score between the query statement and the document in the full-text search result list is: ， in, For statement Included Keywords, For documentation, is the number of documents, is the inverse document frequency, is the document word frequency, For each result in the merged result list, is the query word frequency; The vector in the semantic search result list and vector The calculation formula for the similarity between them is: ， in, For each result in the merged result list, is the number of documents, Respectively represent the vector and the vector No. Quantity; The calculation formulas for the semantic search score and the full-text search score are: ， in, For each result in the merged result list, To avoid calculating constants that are too large, and The ranking of each result in the combined result list in full-text search and semantic search respectively; The calculation formula for the initial score in the merged result list is: ， in, and are the weights of full-text retrieval and semantic retrieval, respectively. is the full-text search score, is the semantic retrieval score, For each result in the merged result list, Represents full-text search, stands for semantic retrieval; The calculation formula for the consistency index of the semantic retrieval and full-text retrieval is: ， in, is the total number of search results, and The ranking of each result in the combined result list in full-text search and semantic search respectively; The calculation formula for the consistency score of the semantic search and full-text search is: ， in, For a control Adjustment parameters that affect the final score. is the initial score, is a consistency index of the semantic retrieval and full-text retrieval; The calculation formula of the ranking confidence attenuation factor is: ， in, is the attenuation rate, is the initial score; The calculation formula of the comprehensive ranking index is: ， in, is the consistency score of the semantic retrieval and full-text retrieval, is the ranking confidence decay factor.

5. The hybrid retrieval device for lifecycle stream data based on a large language model according to claim 4 is characterized in that: The conversion module comprises: The first conversion unit is used to convert the full-text search Chinese information and the full-text search English information into a character string type, and connect them with a preset character string to obtain the full-text search character string.

6. The hybrid retrieval device for lifecycle stream data based on a large language model according to claim 4 is characterized in that: The conversion module also includes: A second conversion unit, used to convert the semantically retrieved English information into an embedding vector using a preset model; A conversion unit is used to convert the embedding vector into the semantic search string.

7. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the hybrid retrieval method for lifecycle stream data based on a large language model as described in any one of claims 1 to 3.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program is executed by a processor to implement the hybrid retrieval method for lifecycle stream data based on a large language model as described in any one of claims 1 to 3.

9. A computer program product, comprising a computer program, characterized in that The computer program is executed to implement the hybrid retrieval method for lifecycle stream data based on a large language model as described in any one of claims 1 to 3.

Images