Method and computer device for searching standard text of confidential scene
Patent Information
- Application Number
- CN202511717431.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-21
- Publication Date
- 2026-10-09
AI Technical Summary
[0003]目前的文本检索技术大多建立在关键词匹配，或基本语义嵌入的基础上，因此这种检索方式存在检索结果准确性低，检索过程易脱密等问题
[0039]本申请实施例使用不同的大语言嵌入模型对输入的查询文本进行不同的向量表示，从而根据两种向量表进行检索以得到两种检索结果，相比直接对原始文本进行检索的方式，向量化文本将原始文本内容隐藏，具有提高文本保密性，起到了防止文本内容泄露的作用，将这两种检索进行处理得到第一结果集的方式，实现了将不同的大语言嵌入模型的检索结果融合，以提高检索结果的可靠性的目的；构建第三向量和第四向量再检索以得到检索结果，实现了将查询向量检索与文本向量检索的结合，一方面能够提高检索结果的可信度，另一方面具有防止通过攻击者通过试探性检索攻击推断涉密文本事件发生的概率。
Smart Images
Figure CN122885153A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of text retrieval technology, and in particular to a method and computer device for retrieving standard text in classified scenarios. Background Technology
[0002] Classified documents refer to documents that contain sensitive information related to government, military, scientific research, finance, and other fields. With the increasing number of classified documents, how to conduct secure retrieval of these documents has become a key research focus.
[0003] Most current text retrieval technologies are based on keyword matching or basic semantic embedding. Therefore, this retrieval method has problems such as low accuracy of retrieval results and easy decryption during the retrieval process. Summary of the Invention
[0004] At least one aspect and advantage of this application will be set forth in part in the description which follows, or may be apparent from the description, or may be obtained by practicing the subject matter of this disclosure.
[0005] According to the first aspect of this application, a method for retrieving standard text in classified scenarios is provided, the method comprising:
[0006] Generate a first vector representation and a second vector representation based on the input query text;
[0007] The first relevant result is determined based on the first vector representation;
[0008] The second correlation result is determined based on the second vector representation;
[0009] The first result set is determined based on the first and second correlation results;
[0010] The third and fourth vectors are constructed based on the first result set, and the search results are obtained based on the third and fourth vectors.
[0011] The first and second vector representations are generated based on different large language embedding models.
[0012] According to one embodiment of this application, the first correlation result is the document set corresponding to the first m vectors whose approximation with the first vector is greater than a first threshold.
[0013] The second correlation result is the set of documents corresponding to the first m vectors whose approximation is greater than the second threshold.
[0014] The first result set consists of results that are consistent with the first and second correlation results;
[0015] The m is a natural number not less than 10.
[0016] According to one embodiment of this application, the process of determining the first result set includes:
[0017] Identify the common documents that are included in both the first result set and the second result set;
[0018] A first sequence is determined based on a first result set and shared documents, and a second sequence is determined based on a second result set and shared documents;
[0019] The first result set is determined based on documents that have a consistent order within the first and second sequences;
[0020] The first sequence contains shared documents sorted by the approximation of the first vector representation, and the second sequence contains shared documents sorted by the approximation of the second vector representation.
[0021] According to one embodiment of this application, the first result set also includes documents with greater approximation than documents with a consistent order, but with inconsistent order.
[0022] According to one embodiment of this application, after documents with a consistent order within the first and second sequences, the documents with the lowest approximation are removed;
[0023] Calculate the first vector or the second document based on the document with the lowest approximation.
[0024] According to one embodiment of this application, the third vector is the vector corresponding to the center of the first vector set, and the first vector set is the first vector representation of the document in the first result set;
[0025] The fourth vector is the vector corresponding to the center of the second vector set, and the second vector set is the second vector representation of the document in the first result set.
[0026] According to one embodiment of this application, the third vector is obtained based on vectors in the first vector set and their approximation to the first vector; the fourth vector is obtained based on vectors in the second vector set and their approximation to the second vector.
[0027] According to one embodiment of this application, the process of obtaining the search results includes:
[0028] The third correlation result is obtained based on the third vector, and the fourth correlation result is obtained based on the fourth vector.
[0029] Based on the approximation of the third or fourth vector, the third and fourth correlation results are deduplicated and merged to obtain the second result set.
[0030] According to one embodiment of this application, in response to the first result set being empty or below a third threshold, the first relevant result and the second relevant result are merged to obtain the search result.
[0031] According to a second aspect of this application, a retrieval device for standard texts in classified scenarios is provided, comprising:
[0032] The text vectorization module is used to generate a first vector representation and a second vector representation based on the input query text, wherein the first vector representation and the second vector representation are generated based on different large language embedding models;
[0033] The first result determination module is used to determine the first relevant result based on the first vector representation;
[0034] The second result determination module is used to determine the second relevant result based on the second vector representation;
[0035] The result set determination module is used to determine a first result set based on the first relevant result and the second relevant result.
[0036] The retrieval result determination module is used to construct a third vector and a fourth vector based on the first result set, and to obtain retrieval results based on the third vector and the fourth vector.
[0037] According to a third aspect of this application, a computer device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.
[0038] According to a fourth aspect of this application, a computer-readable storage medium is provided that stores a computer program, which, when executed by a processor, implements the steps of the above-described method.
[0039] This application uses different large language embedding models to represent the input query text with different vectors, thereby obtaining two types of search results based on two vector tables. Compared to directly searching the original text, vectorized text hides the original text content, which improves text confidentiality and prevents text content leakage. Processing these two searches to obtain the first result set achieves the fusion of search results from different large language embedding models, thereby improving the reliability of the search results. Constructing a third and fourth vector for further searching to obtain search results combines query vector retrieval with text vector retrieval. This not only improves the credibility of the search results but also prevents attackers from inferring the probability of confidential text incidents through probing search attacks. Attached Figure Description
[0040] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below.
[0041] Figure 1 A flowchart illustrating a method for retrieving standard text in classified scenarios, provided as an embodiment of this application;
[0042] Figure 2 This is a block diagram of a device for retrieving standard text in classified scenarios, provided as an embodiment of this application. Detailed Implementation
[0043] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0044] It should be noted that although the functional modules are divided in the device diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than the module division in the device or the order in the flowchart.
[0045] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.
[0046] According to one embodiment of this application, a method for retrieving standard text in classified scenarios is provided, such as... Figure 1 As shown, the method includes steps S101 to S105.
[0047] Step S101: Generate a first vector representation and a second vector representation based on the input query text, wherein the first vector representation and the second vector representation are generated based on different large language embedding models.
[0048] Specifically, the large language embedding model used can be either a homologous large language embedding model or a heterogeneous large language embedding model. Homologous large language embedding models refer to different versions of the model provided by the same service provider, such as using the large language embedding modules Qwen3-Embedding-0.6B and Qwen-2.5 0.5B from Tongyi Qianwen. Heterogeneous large language embedding models refer to models with different technical architectures provided by different service providers, such as Qwen3-Embedding-0.6B and ritrieve_zh_v1.
[0049] In practice, the choice of large language embedding model can be based on the nature of the classified institution. For example, if the classified institution is a medical institution, Qwen3-Embedding-0.6B, BGE-M3, etc., can be used.
[0050] It should be noted that different large language embedding models can be selected based on confidential scenarios, text retrieval targets, etc., and this application embodiment does not limit this.
[0051] Step S102: Determine the first correlation result based on the first vector representation.
[0052] It should be noted that the first relevance result is a set of text identifiers. Each text identifier can be queried through a mapping relationship (obtained by processing the standard text using a large language embedding model that generates the first vector representation) to find the feature vector of the corresponding vectorized standard text. The first relevance result is then determined based on the vector approximation calculation result. The feature vectors within the mapping relationship include multi-dimensional data representing classified texts, which are obtained by processing the large language embedding model that generates the first vector representation.
[0053] In this embodiment of the application, the standard text exists in the form of vector representation, so the text content is hidden during the text retrieval process. Therefore, this method of pre-representing classified text as vector has the effect of preventing classified documents from being declassified, and at the same time, it is faster than text content retrieval.
[0054] In practice, the primary relevant result can be determined based on document similarity targets. Text similarity targets are determined using similarity thresholds. For example, a similarity threshold greater than 90% is considered high similarity, and the text corresponding to highly similar vectors is used to determine the retrieval target.
[0055] Step S103: Determine the second correlation result based on the second vector representation.
[0056] It should be noted that the second correlation result is a set of text identifiers. Each text identifier can be queried through a mapping relationship (obtained by processing the standard text using a large language embedding model that generates the second vector representation) to find the feature vector of the corresponding standard text. The second correlation result is then determined based on the vector approximation calculation result. The feature vectors included in the mapping relationship include multi-dimensional data of classified texts, which are obtained by processing the large language embedding model that generates the second vector representation.
[0057] It should be noted that, because the query text uses different large language embedding models, the results retrieved by the second vector representation in a standard text library that has already been vectorized may differ.
[0058] Step S104: Determine the first result set based on the first correlation result and the second correlation result.
[0059] In this embodiment of the application, the first result set is used to represent the intersection of the first relevant result and the second relevant result, that is, the standard text that appears in both search results is used as the search result.
[0060] Step S105: Construct a third vector and a fourth vector based on the first result set, and obtain the search results based on the third vector and the fourth vector.
[0061] It should be noted that each standard text in the first result set includes two representations, generated by different large language embedding models. In practice, the vectors generated by the corresponding large language embedding models can be determined based on the text identifier in the first result set.
[0062] It should be noted that the third or fourth vector is used to represent the common features of all standard texts included in the first set of results. These common features are constructed from vectors generated by the same large language embedding model. By constructing vectors from the first result set, the intent of the input query text is understood as common text features, thereby retrieving results with a higher degree of matching to the user's intent using these common text features.
[0063] This application uses different large language embedding models to represent the input query text with different vectors, thereby obtaining two types of search results based on two vector tables. Compared to directly searching the original text, vectorized text hides the original text content, which improves text confidentiality and prevents text content leakage. Processing these two searches to obtain the first result set achieves the fusion of search results from different large language embedding models, thereby improving the reliability of the search results. Constructing a third and fourth vector for further searching to obtain search results combines query vector retrieval with text vector retrieval. This not only improves the credibility of the search results but also prevents attackers from inferring the probability of confidential text incidents through probing search attacks.
[0064] In some embodiments, the first correlation result is the set of documents corresponding to the first m vectors whose approximation is greater than the first threshold.
[0065] The second relevant result is the second vector, which represents the set of documents corresponding to the first m vectors whose approximation is greater than the second threshold.
[0066] The first result set consists of results that are consistent with the first and second relevant results.
[0067] m is a natural number not less than 10.
[0068] In practice, the corresponding document set can be determined based on the number of first-relevant results. For example, when the document retrieval volume is less than a predetermined threshold, the first-relevant results are used as all retrieval results; when the document retrieval volume is less than the predetermined threshold, the first m vectors with an approximation greater than the first threshold are used as the first-relevant results.
[0069] In some embodiments, the process of determining the first result set includes:
[0070] Identify the common documents that are simultaneously included in both the first and second relevant result sets;
[0071] The first sequence is determined based on the first relevant results and shared documents, and the second sequence is determined based on the second relevant results and shared documents;
[0072] The first result set is determined based on documents that have a consistent order within the first and second sequences;
[0073] The first sequence contains shared documents sorted by the approximation of the first vector representation, and the second sequence contains shared documents sorted by the approximation of the second vector representation.
[0074] It should be noted that each text in the first result set of this embodiment exhibits high textual features across the retrieval results of different large language embedding models, indicating a high degree of similarity in the textual features generated by different large language embedding models for the same text. Performing secondary retrieval based on these highly similar textual features can minimize the discrepancies in retrieval results from different large language embedding models and improve the reliability of the retrieval results.
[0075] Specifically, the first relevant results may include multiple documents that are sorted from highest to lowest similarity, or they may be unordered.
[0076] For example, in some embodiments, when the number of texts with an approximation greater than a first threshold is insufficient (less than 10), the first correlation result may be unordered. In this case, the approximation of each text can be recorded, so that when the shared documents are determined, the first sequence can be obtained by sorting them according to the approximation.
[0077] For example, in some embodiments, when the number of texts whose approximation to the first vector is greater than the first threshold is too large (not less than 10), they are sorted in descending order of approximation, and the texts ranked first are taken as the first relevant results. At this time, the texts included in the first relevant results are ordered.
[0078] Based on this, the same situation exists for the second related result.
[0079] Since the order of the same text in the first and second relevant results may differ, the order of each shared text in the first and second relevant results is re-determined, resulting in a first sequence and a second sequence. Consequently, the order of shared documents in the first and second relevant results may be inconsistent, meaning their similarity may differ across different sets of relevant results. For example, suppose the first relevant results include (doc1, doc2, doc3, doc4, doc5, doc6, doc7, doc9, doc10), the second relevant results include (doc1, doc2, doc7, doc6, doc10, doc11, doc12, doc13, doc3, doc9), and the corresponding shared documents include (doc1, doc2, doc3, doc6, doc7, doc9, doc10). In this embodiment, the number of texts in the first and second related results is sufficient. The texts included in each of the two related result sets are sorted in descending order of similarity. Thus, the text order included in the first sequence is (doc1, doc2, doc3, doc6, doc7, doc9, doc10), and the text order included in the second sequence is (doc1, doc2, doc7, doc6, doc10, doc9, doc3). Therefore, the first result set is (doc1, doc2).
[0080] In one embodiment, the first result set also includes documents with greater approximation than those with a consistent order, but with inconsistent order.
[0081] It should be noted that a higher similarity indicates that the document ranks higher among the two relevant results. Based on documents with a consistent order, the text ranked higher in the first and second relevant results is used as the first result set. This expands the number of texts in the first result set, prevents the omission of highly similar documents with inconsistent rankings, and ensures both the accuracy and recall of subsequent searches.
[0082] In some embodiments, after determining documents with a consistent order within the first and second sequences, documents after the lowest approximation are removed; the first and second sequences are then determined based on the documents with the lowest approximation.
[0083] In practice, when there are a large number of documents with a consistent order, such as three documents with a consistent order, the document with the lowest similarity among these three documents is removed; or the similarity of each of these three documents is compared with the similarity threshold, and the document with a similarity lower than the threshold is removed.
[0084] It should be noted that the approximation threshold can be the average of the approximations of all documents included in the first or second relevant results, or the approximation threshold can be the weighted average of the approximations of all documents included in the first or second relevant results.
[0085] In some embodiments, the third vector is the vector corresponding to the center of the first vector set, and the first vector set is the first vector representation of the document in the first result set; the fourth vector is the vector corresponding to the center of the second vector set, and the second vector set is the second vector representation of the document in the first result set.
[0086] It should be noted that in this embodiment, the third vector refers to the central vector of the first vector set, which is used to characterize the common semantic features of the first vector representation of all texts in the first result set.
[0087] It should be noted that in this embodiment, the fourth vector refers to the central vector of the second vector set, which is used to characterize the common semantic features of the second vector representation of all texts in the first result set.
[0088] It should be noted that the centroid calculation used in this embodiment is for the central vector. In specific implementations, other algorithms can also be used, such as the vector formed by taking the median of each dimension of the vector space, which will not be listed here.
[0089] In some embodiments, the third vector is obtained based on vectors within the first vector set and their approximation to the first vector; the fourth vector is obtained based on vectors within the second vector set and their approximation to the second vector.
[0090] This embodiment first calculates the product of the vector of each document in the first result set and the weight of its corresponding approximation, then sums and normalizes the products to obtain the center vector. The center vector is then obtained by calculating a weighted average of the vectors corresponding to the documents in the first result set.
[0091] In this embodiment, the third or fourth vector is determined by first calculating a weighted sum vector of the vectors corresponding to the documents in the first result set, and then normalizing the weighted sum vector. That is, the weighted centroid algorithm is used to calculate the center vector of the vectors corresponding to the documents in the first result set. The algorithm formula for the center vector is as follows: ;
[0092] in, For weighted sum vectors, Let the magnitude of the weighted sum vector be . , Let be the normalized weight of the i-th text. Let be the vector of the i-th text. Let be the weight of the i-th text.
[0093] For example, suppose the first set of vectors for all documents included in the first result set is: The second vector set is: ,in, Let i be the representation of the i-th first vector in the first vector set. Let represent the weight of the i-th first vector in the first vector set; Let i be the representation of the i-th second vector in the second vector set. Let be the weight represented by the i-th second vector in the second vector set.
[0094] Therefore, the process of determining the third vector in this embodiment is as follows:
[0095] Determine the weight represented by each first vector in the first vector set. In this embodiment, the approximation of the i-th first vector in the first vector set is used as... ;
[0096] The weight represented by each first vector in the first vector set Normalization is performed to obtain the normalized weights represented by each first vector in the first vector set. , ;
[0097] Calculate the sum vector of the first vector set. , ;
[0098] Calculate the Euclidean magnitude of the first vector set. , ;
[0099] Will and The ratio of is used as the third vector.
[0100] The fourth vector is calculated using the same algorithm, which will not be elaborated here.
[0101] In some embodiments, the process of obtaining search results includes:
[0102] The third correlation result is obtained based on the third vector, and the fourth correlation result is obtained based on the fourth vector.
[0103] Based on the approximation of the third or fourth vector, the third and fourth correlation results are deduplicated and merged to obtain the second result set.
[0104] In practice, the third vector is used to represent the first vector of each text included in the text library, and the similarity is calculated to obtain the third relevance result.
[0105] In practice, the fourth vector is used to represent the first vector of each text included in the text library, and the similarity is calculated to obtain the fourth correlation result.
[0106] It should be noted that the process of obtaining the third and fourth correlation results can refer to the method of determining the first or second correlation results, which will not be elaborated here.
[0107] In practice, the intersection of the third and fourth correlation results can be taken, and the results can be sorted in descending order of similarity to obtain the second result set.
[0108] In some embodiments, in response to the first result set being empty or below a third threshold, the first relevant result and the second relevant result are merged to obtain the search result.
[0109] In practice, the union of the first and second relevant results is taken as the final target and sorted by similarity to obtain the retrieval results. This can avoid the problem of not being able to retrieve consistent texts or retrieving too few texts due to the large differences in retrieval results from different large language embedding models, thereby improving the robustness of the system.
[0110] In some embodiments, to ensure that users can perform text viewing, copying, downloading, and other operations within their access permissions, the following steps may be performed after obtaining the search results: determining the corresponding text security level based on the metadata of each text included in the search results; matching the user account permissions corresponding to the search text with the text security level of each text included in the search results, and determining the successfully matched text as the target search result for display and output processing.
[0111] Specifically, text security classifications generally include public, internal, secret, confidential, and top secret.
[0112] Specifically, user account permissions limit the range of text security levels that can be viewed. For example, a user account with ordinary user permissions typically corresponds to a text security level of Public or Public and Internal; a user account with general security permissions might correspond to a text security level of Secret; a user account with high security permissions corresponds to a text security level of Confidential; and a user account with top secret permissions corresponds to a text security level of Top Secret.
[0113] It should be noted that the text security level included in the user account permissions and the scope of authorization can be set according to business needs, and this application does not limit this.
[0114] In some embodiments, in order to help users quickly determine the relevance of search results to their search purpose, the text included in the target search results can be preprocessed. The preprocessing operations include at least content exclusivity and summary generation; and the preprocessed text is then displayed and output.
[0115] The text retrieval method provided in this application is applied to a computer device, which may include, but is not limited to, a server, a desktop computer, etc. Specifically, it can perform structured parsing of text to construct a tree index structure, and retrieve query requests through the tree index structure.
[0116] One embodiment of this application provides a retrieval device for standard text in classified scenarios, such as... Figure 2 As shown, the device 20 includes: a text vectorization module 201, a first result determination module 202, a second result determination module 203, a result set determination module 204, and a retrieval result determination module 205.
[0117] The text vectorization module 201 is used to generate a first vector representation and a second vector representation based on the input query text, wherein the first vector representation and the second vector representation are generated based on different large language embedding models;
[0118] The first result determination module 202 is used to determine the first relevant result based on the first vector representation;
[0119] The second result determination module 203 is used to determine the second relevant result based on the second vector representation;
[0120] Result set determination module 204 is used to determine a first result set based on the first correlation result and the second correlation result;
[0121] The retrieval result determination module 205 is used to construct a third vector and a fourth vector based on the first result set, and to obtain retrieval results based on the third vector and the fourth vector.
[0122] This application uses different large language embedding models to represent the input query text with different vectors, thereby obtaining two types of search results based on two vector tables. Compared to directly searching the original text, vectorized text hides the original text content, which improves text confidentiality and prevents text content leakage. Processing these two searches to obtain the first result set achieves the fusion of search results from different large language embedding models, thereby improving the reliability of the search results. Constructing a third and fourth vector for further searching to obtain search results combines query vector retrieval with text vector retrieval. This not only improves the credibility of the search results but also prevents attackers from inferring the probability of confidential text incidents through probing search attacks.
[0123] Furthermore, the first relevance result is the set of documents corresponding to the first m vectors whose approximation is greater than the first threshold;
[0124] The second relevant result is the second vector, which represents the set of documents corresponding to the first m vectors whose approximation is greater than the second threshold.
[0125] The first result set consists of results that are consistent with the first and second relevant results.
[0126] m is a natural number not less than 10.
[0127] Furthermore, the process of determining the first result set includes:
[0128] Identify the common documents that are included in both the first result set and the second result set;
[0129] A first sequence is determined based on a first result set and shared documents, and a second sequence is determined based on a second result set and shared documents;
[0130] The first result set is determined based on documents that have a consistent order within the first and second sequences;
[0131] The first sequence contains shared documents sorted by the approximation of the first vector representation, and the second sequence contains shared documents sorted by the approximation of the second vector representation.
[0132] Furthermore, the first result set also includes documents with a higher approximation than those with a consistent order, but with inconsistent order.
[0133] Furthermore, after identifying documents with a consistent order within the first and second sequences, the documents with the lowest approximation are removed.
[0134] Calculate the first vector or the second document based on the document with the lowest approximation.
[0135] Furthermore, the third vector is the vector corresponding to the center of the first vector set, and the first vector set is the first vector representation of the document within the first result set;
[0136] The fourth vector is the vector corresponding to the center of the second vector set, and the second vector set is the second vector representation of the document in the first result set.
[0137] Furthermore, the third vector is obtained based on the vectors in the first vector set and their approximation with the first vector; the fourth vector is obtained based on the vectors in the second vector set and their approximation with the second vector.
[0138] Furthermore, the process of obtaining search results includes:
[0139] The third correlation result is obtained based on the third vector, and the fourth correlation result is obtained based on the fourth vector.
[0140] Based on the approximation of the third or fourth vector, the third and fourth correlation results are deduplicated and merged to obtain the second result set.
[0141] Furthermore, in response to the first result set being empty or below the third threshold, the first relevant result and the second relevant result are merged to obtain the search result.
[0142] The retrieval device for classified scenario standard texts in this embodiment can execute the retrieval method for classified scenario standard texts shown in the embodiments of this application. The implementation principle is similar and will not be described again here.
[0143] Another embodiment of this application provides a computer device including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method.
[0144] Specifically, the processor can be a CPU, a general-purpose processor, a DSP, an ASIC, an FPGA, or other programmable logic device, transistor logic device, hardware component, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computational functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.
[0145] Specifically, the processor connects to the memory via a bus, which may include a path for transmitting information. The bus can be a PCI bus or an EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc.
[0146] The memory may be ROM or other types of static storage devices that can store static information and instructions, RAM or other types of dynamic storage devices that can store information and instructions, or EEPROM, CD-ROM or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited thereto.
[0147] Optionally, the memory stores the code of a computer program that executes the scheme of this application, and the execution is controlled by a processor. The processor executes the application code stored in the memory to implement the operation of the apparatus provided in the above embodiments.
[0148] Another embodiment of this application provides a computer-readable storage medium storing computer-executable instructions for performing the methods provided in the above embodiments.
[0149] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0150] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.
[0151] The above is a detailed description of the preferred embodiments of this application. However, this application is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.
Claims
1. A method for retrieving standard text in classified scenarios, characterized in that, include: Generate a first vector representation and a second vector representation based on the input query text; The first relevant result is determined based on the first vector representation; The second correlation result is determined based on the second vector representation; The first result set is determined based on the first and second correlation results; The third and fourth vectors are constructed based on the first result set, and the search results are obtained based on the third and fourth vectors. The first and second vector representations are generated based on different large language embedding models.
2. The method for retrieving standard text in classified scenarios as described in claim 1, characterized in that, The first correlation result is the set of documents corresponding to the first m vectors whose approximation is greater than the first threshold. The second correlation result is the set of documents corresponding to the first m vectors whose approximation is greater than the second threshold. The first result set consists of results that are consistent with the first and second correlation results; The m is a natural number not less than 10.
3. The method for retrieving standard text in classified scenarios as described in claim 2, characterized in that, The process of determining the first result set includes: Identify the shared documents that are present in both the first and second relevant results; A first sequence is determined based on a first result set and shared documents, and a second sequence is determined based on a second result set and shared documents; The first result set is determined based on documents that have a consistent order within the first and second sequences; The first sequence contains shared documents sorted by the approximation of the first vector representation, and the second sequence contains shared documents sorted by the approximation of the second vector representation.
4. The method for retrieving standard text in classified scenarios as described in claim 3, characterized in that, The first result set also includes documents with a greater approximation than those with a consistent order, but with inconsistent order.
5. The method for retrieving standard text in classified scenarios as described in claim 3, characterized in that, After identifying documents with a consistent order within the first and second sequences, remove documents with the lowest approximation. The first and second sequences are determined based on the documents with the lowest approximation.
6. The method for retrieving standard text in classified scenarios as described in claim 1, characterized in that, The third vector is the vector corresponding to the center of the first vector set, and the first vector set is the first vector representation of the document in the first result set; The fourth vector is the vector corresponding to the center of the second vector set, and the second vector set is the second vector representation of the document in the first result set.
7. The method for retrieving standard text in classified scenarios as described in claim 6, characterized in that, The third vector is obtained based on the vectors in the first vector set and their approximation with the first vector; the fourth vector is obtained based on the vectors in the second vector set and their approximation with the second vector.
8. The method for retrieving standard text in classified scenarios as described in claim 1, characterized in that, The process of obtaining the search results includes: The third correlation result is obtained based on the third vector, and the fourth correlation result is obtained based on the fourth vector. Based on the approximation of the third or fourth vector, the third and fourth correlation results are deduplicated and merged to obtain the second result set.
9. The method for retrieving standard text in classified scenarios as described in claim 1, characterized in that, If the first result set is empty or below the third threshold, the first relevant result and the second relevant result are merged to obtain the search result.
10. A computer device, characterized in that, It includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of any one of claims 1 to 9.