Text retrieval method and apparatus

Abstract

Embodiments of the present application disclose a text retrieval method and device, the text retrieval method comprising: performing structured processing and semantic segmentation on a to-be-retrieved text to obtain a segment set; wherein the segment set satisfies structural integrity and semantic integrity; performing knowledge vectorization and meta-information vectorization on segments in the segment set to obtain a knowledge vector and a meta-information vector; constructing a clustering function based on a pre-constructed cosine similarity function, and performing inter-cluster aggregation and hierarchical construction according to the clustering function and a preset clustering threshold to obtain a multi-level retrieval structure; wherein the clustering function is generated based on the knowledge vector and the meta-information vector; and performing text retrieval according to the multi-level retrieval structure and a query vector generated based on a large language model. The present application realizes semantic modeling and hierarchical retrieval of documents, overcomes the problem of insufficient semantic understanding and result planarization in traditional keyword matching retrieval, and breaks through the limitation of existing semantic vector retrieval in multi-version knowledge organization.

Description

Technical Field

[0001] This application relates to semantic retrieval technology, and in particular to a text retrieval method and apparatus. Background Technology

[0002] Currently, the field of water conservancy and hydropower engineering has accumulated a vast amount of diverse documents and materials during the design, construction, and operation management processes, including design specifications, monitoring reports, operational experience, and accident case studies. These materials are primarily text-based, highly specialized, and semantically complex. Efficiently and accurately retrieving and utilizing these materials is crucial for ensuring project safety, improving operational efficiency, and supporting scientific research and innovation.

[0003] In existing technologies, there are two main types of methods for retrieving engineering texts: one is keyword-based matching methods, such as inverted indexes and BM25. Keyword-based matching methods are simple to implement and relatively efficient, but they rely on literal matching, making it difficult to understand semantic relationships and cover the diversity of actual semantic expressions, resulting in poor retrieval performance and difficulty in supporting the comparative analysis and comprehensive judgment needs of engineering applications. The other type is semantic vector-based methods, such as BERT and Sentence-BERT, which achieve retrieval through vector similarity calculation, breaking through the limitations of keywords to some extent. However, they still lack the ability to organize and compare knowledge from multiple sources and versions, making it difficult to meet the requirements of engineering scenarios for comparability, completeness, and usability. In recent years, retrieval-enhanced generation methods have gradually emerged. These methods combine semantic retrieval with large language models, which can improve question answering and generation capabilities. However, their input still relies on a single query statement, and the knowledge organization is mainly flat, making it difficult to meet the high requirements of comprehensiveness and robustness in engineering scenarios.

[0004] It should be noted that the above introduction to the technical background is only for the purpose of providing a clear and complete explanation of the technical solutions of this application and facilitating understanding by those skilled in the art. It should not be assumed that these technical solutions are known to those skilled in the art simply because they have been described in the background section of this application. Summary of the Invention

[0005] To address at least one of the aforementioned problems, embodiments of this application provide a text retrieval method and apparatus.

[0006] The specific technical solution of this application embodiment is as follows:

[0007] According to a first aspect of the embodiments of this application, a text retrieval method is provided, the text retrieval method comprising:

[0008] The text to be retrieved is subjected to structured processing and semantic segmentation to obtain a set of fragments; wherein the set of fragments satisfies both structural integrity and semantic integrity.

[0009] The fragments in the fragment set are vectorized into knowledge vectors and meta-information vectors to obtain knowledge vectors and meta-information vectors.

[0010] A clustering function is constructed based on a pre-built cosine similarity function, and inter-cluster aggregation and hierarchical construction are performed according to the clustering function and a preset clustering threshold to obtain a multi-level retrieval structure; wherein, the clustering function is generated based on the knowledge vector and meta-information vector;

[0011] Text retrieval is performed based on the multi-level retrieval structure and query vectors generated based on a large language model.

[0012] Furthermore, the text to be retrieved undergoes structured processing and semantic segmentation to obtain a set of fragments, including:

[0013] The text to be retrieved is segmented into multiple segments based on preset boundary conditions; the boundary conditions include titles, paragraphs, tables, and formulas.

[0014] The segmented fragments are vectorized based on a semantic embedding model to obtain semantic embedding vectors;

[0015] Knowledge merging is performed based on the cosine similarity of adjacent segments and a preset semantic integrity threshold to obtain the segment set, wherein the cosine similarity is obtained based on the semantic embedding vector.

[0016] Furthermore, the text retrieval method also includes:

[0017] The cosine similarity is calculated based on the semantic embedding vector, the type of the boundary condition, and the penalty coefficient.

[0018] Furthermore, the cosine similarity based on adjacent segments The fragment set is obtained by merging knowledge based on a preset semantic integrity threshold, including:

[0019] Based on the cosine similarity of adjacent segments, the semantic integrity threshold, and the semantic embedding vector, a merging operation is performed on adjacent segmented segments to obtain a first segment set;

[0020] Based on the cosine similarity of adjacent segments in the first segment set and the semantic integrity threshold, the segments in the first segment set are merged to obtain the second segment set;

[0021] Perform the above iterative operation until the cosine similarity of adjacent segments is less than the semantic integrity threshold, and obtain the segment set.

[0022] Furthermore, the step of performing knowledge vectorization and meta-information vectorization on the fragments in the fragment set to obtain knowledge vectors and meta-information vectors includes:

[0023] The knowledge vector is obtained by vectorizing the fragments in the fragment set using a semantic embedding model;

[0024] The metadata associated with the fragments in the fragment set is converted into strings and vectorized according to the semantic embedding model to obtain the metadata vector.

[0025] Furthermore, a clustering function is constructed based on a pre-built cosine similarity function, including:

[0026] The cosine similarity function is generated based on the knowledge vector, the meta-information vector, and the similarity weights of the knowledge vectors.

[0027] Based on the cosine similarity function and the fragment set Construct the clustering function.

[0028] Further, the step of performing inter-cluster aggregation and hierarchical construction based on the clustering function and a preset clustering threshold to obtain a multi-level retrieval structure includes:

[0029] If the value of the clustering function is greater than the clustering threshold, inter-cluster aggregation is performed;

[0030] The semantic center vector of the topic is determined based on the aggregation results; the semantic center vector of the topic and the fragment constitute the multi-level retrieval structure.

[0031] Furthermore, determining the semantic center vector of the topic based on the aggregation result includes:

[0032] Identify subject clusters from different texts using multi-level directories;

[0033] The semantic center vector of the topic is determined based on the topic cluster.

[0034] Furthermore, the text retrieval method also includes:

[0035] By utilizing a large language model, a diverse set of queries is obtained based on the user's original query statement;

[0036] The diverse query set is mapped to the query vector using a semantic embedding model.

[0037] Furthermore, the method of utilizing a large language model to obtain a diverse query set based on the user's original query statement includes:

[0038] Using a large language model, the user's original query statement is parsed into standardized key-value pairs;

[0039] The standardized key values ​​are semantically expanded and rewritten to obtain the diverse query set.

[0040] Further, the text retrieval based on the multi-level retrieval structure and the query vector generated based on the large language model includes:

[0041] Based on the multi-level retrieval structure and the query vector generated based on the large language model, subject-level retrieval and fragment-level retrieval are performed to obtain the first retrieval result;

[0042] The first retrieval result is reordered based on a pre-trained reordering model to obtain the second retrieval result.

[0043] According to a second aspect of the embodiments of this application, a text retrieval device is provided, the text retrieval device comprising:

[0044] The preprocessing unit is used to perform structured processing and semantic segmentation on the text to be retrieved to obtain a set of fragments; wherein the set of fragments satisfies structural integrity and semantic integrity.

[0045] The vectorization unit is used to perform knowledge vectorization and meta-information vectorization on the fragments in the fragment set to obtain knowledge vectors and meta-information vectors.

[0046] The retrieval structure generation unit is used to construct a clustering function based on a pre-built cosine similarity function, and to perform inter-cluster aggregation and hierarchical construction according to the clustering function and a preset clustering threshold to obtain a multi-level retrieval structure; wherein, the clustering function is generated based on the knowledge vector and the meta-information vector;

[0047] The retrieval unit is used to perform text retrieval based on the multi-level retrieval structure and the query vector generated based on the large language model.

[0048] According to a third aspect of the embodiments of this application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the text retrieval method described above.

[0049] According to a fourth aspect of the embodiments of this application, a computer storage medium is provided that stores instructions thereon, which, when executed individually or jointly by at least one processor of a computer device, cause the computer device to perform the text retrieval method described above.

[0050] According to a fifth aspect of the embodiments of this application, a computer program product is provided, including instructions that, when executed individually or jointly by at least one processor of a computer device, cause the computer device to perform the text retrieval method described above.

[0051] The beneficial effects of the embodiments of this application are as follows: This application realizes semantic modeling and hierarchical retrieval of documents, overcomes the problems of insufficient semantic understanding and flat results in traditional keyword matching retrieval, and breaks through the limitations of existing semantic vector retrieval in multi-version knowledge organization.

[0052] Specific embodiments of this application are disclosed in detail with reference to the following description and accompanying drawings, indicating how the principles of this application can be adopted. It should be understood that the embodiments of this application are not limited in scope. Within the spirit and scope of the appended claims, embodiments of this application include many changes, modifications, and equivalents. Features described and / or shown for one embodiment may be used in the same or similar manner in one or more other embodiments, combined with features in other embodiments, or substituted for features in other embodiments. Attached Figure Description

[0053] The accompanying drawings described herein are for illustrative purposes only and are not intended to limit the scope of this application in any way. Furthermore, the shapes and scales of the components in the drawings are merely illustrative to aid in understanding this application and do not specifically limit the shapes and scales of the components. Those skilled in the art, guided by the teachings of this application, can select various possible shapes and scales to implement this application according to specific circumstances.

[0054] Figure 1 This is a flowchart of the text retrieval method according to the first embodiment of this application;

[0055] Figure 2 This is a flowchart illustrating the method for structuring and semantically segmenting the text to be retrieved according to an embodiment of this application.

[0056] Figure 3 This is a flowchart of the knowledge merging method according to an embodiment of this application;

[0057] Figure 4 This is a flowchart of the fragment knowledge vectorization and meta-information vectorization method according to an embodiment of this application;

[0058] Figure 5 This is a flowchart of the method for constructing a clustering function based on the cosine similarity function in an embodiment of this application;

[0059] Figure 6 This is a flowchart illustrating the method for generating a multi-level retrieval structure according to an embodiment of this application;

[0060] Figure 7 This is a flowchart of the method for determining the semantic center vector of a topic based on the aggregation results in an embodiment of this application;

[0061] Figure 8This is a flowchart of the text retrieval method according to the second embodiment of this application;

[0062] Figure 9 This is a flowchart illustrating the method for generating a diversity query set according to an embodiment of this application;

[0063] Figure 10 This is a flowchart illustrating the method for text retrieval based on a multi-level retrieval structure and query vectors according to an embodiment of this application.

[0064] Figure 11 This is a structural block diagram of a text retrieval device according to an embodiment of this application;

[0065] Figure 12 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0066] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are only used to illustrate this application and are not intended to limit the scope of the present invention. After reading this application, any modifications of this application by those skilled in the art in various equivalent forms fall within the scope defined by the appended claims.

[0067] In the embodiments of this application, the terms "first," "second," etc., are used to distinguish different elements by name, but do not indicate the spatial arrangement or chronological order of these elements, and these elements should not be limited by these terms. The term "and / or" includes any one or more of the terms listed in association and all combinations thereof. The terms "comprising," "including," "having," etc., refer to the presence of the stated features, elements, components, or assemblies, but do not exclude the presence or addition of one or more other features, elements, components, or assemblies.

[0068] In this application, the terms "upper," "lower," "left," "right," "front," "rear," "top," "bottom," "inner," "outer," "middle," "vertical," "horizontal," "lateral," and "longitudinal" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are primarily for the purpose of better describing this application and its embodiments, and are not intended to limit the indicated device, element, or component to having a specific orientation, or to be constructed and operated in a specific orientation.

[0069] Positional relationships such as "parallel" or "perpendicular" not only include completely "parallel" or "perpendicular" positional relationships, but also positional relationships with angular deviations relative to completely "parallel" or "perpendicular" within a preset deviation range.

[0070] Furthermore, in addition to indicating location or positional relationship, some of the aforementioned terms may also have other meanings. For example, the term "above" may also be used in some cases to indicate a certain dependency or connection relationship. Those skilled in the art can understand the specific meaning of these terms in this application based on the specific circumstances.

[0071] Furthermore, the terms "installation," "setup," "equipped with," "connection," "linking," and "socketing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral structure; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium, or an internal connection between two devices, components, or parts. Those skilled in the art can understand the specific meaning of these terms in this application based on the specific circumstances.

[0072] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0073] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein in the specification of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0074] Based on the aforementioned background technology, existing technologies suffer from the following problems: the knowledge organization is flat, failing to effectively represent the multi-dimensional information of the same knowledge point from different sources or versions, resulting in insufficient completeness and comparability of search results; the query method is singular, making it difficult to cover the diversity of semantic expressions, thus limiting both recall and precision. Therefore, there is an urgent need to combine hierarchical knowledge representation with diverse natural language for querying to overcome the bottlenecks of existing technologies and improve the accuracy, comprehensiveness, and robustness of engineering text retrieval.

[0075] In one embodiment of this application, a text retrieval method is provided. Figure 1 This is a flowchart of a text retrieval method according to the first embodiment of this application, wherein the method is specifically applied to the server side. In specific implementation, as follows... Figure 1 As shown, the text retrieval method includes:

[0076] S101: Text to be retrieved Structured processing and semantic segmentation are performed to obtain a set of fragments. The set of fragments satisfies both structural integrity and semantic integrity.

[0077] S102: Perform knowledge vectorization and meta-information vectorization on the fragments in the fragment set to obtain knowledge vectors. and meta-information vector ;

[0078] S103: Based on a pre-constructed cosine similarity function Constructing clustering functions The clustering function and preset clustering threshold are used to perform inter-cluster aggregation and hierarchical construction to obtain a multi-level retrieval structure; wherein, the clustering function is based on the knowledge vector. and meta-information vector generate;

[0079] S104: Perform text retrieval based on the multi-level retrieval structure and the query vector generated based on the large language model.

[0080] According to the above embodiments, this application first performs structured processing and semantic segmentation on the text to be retrieved to obtain a set of fragments; then, it performs knowledge vectorization and meta-information vectorization on the fragments in the set of fragments to obtain knowledge vectors and meta-information vectors; then, it constructs a clustering function based on a pre-built cosine similarity function, and performs inter-cluster aggregation and hierarchical construction according to the clustering function and a preset clustering threshold to obtain a multi-level retrieval structure; finally, it performs text retrieval based on the multi-level retrieval structure and the query vector generated based on a large language model, realizing semantic modeling and hierarchical retrieval of documents, overcoming the problems of insufficient semantic understanding and flat results in traditional keyword matching retrieval, and breaking through the limitations of existing semantic vector retrieval in multi-version knowledge organization.

[0081] Text retrieval is performed based on the multi-level retrieval structure and query vectors generated based on a large language model.

[0082] It should be noted that the text to be searched can be text from any field, such as engineering-related text, such as text in the field of water conservancy and hydropower engineering, which includes documents and materials in the process of engineering design, construction and operation management.

[0083] There are usually multiple texts to be searched, and these multiple texts can be used as a document set. :

[0084]

[0085] in, This represents the total number of documents.

[0086] For files of different formats, format conversion is required first. Documents of various formats should be converted to Markdown format separately.

[0087]

[0088]

[0089] in, This is a file format conversion function. .

[0090] The original formats include, but are not limited to: PDF, WORD, TXT, PNG, etc. For PDF and PNG format text, Optical Character Recognition (OCR) can be used for processing. For WORD format text, conversion can be performed based on the PYTHON-DOCX library. For TXT format text, the file extension can be directly modified.

[0091] In some embodiments, such as Figure 2 As shown in step S101, the text to be retrieved undergoes structured processing and semantic segmentation to obtain a set of fragments, including:

[0092] S201: The text to be retrieved is segmented into multiple segments according to preset boundary conditions; the boundary conditions include titles, paragraphs, tables, and formulas;

[0093] S202: Vectorize the segmented fragments based on the semantic embedding model to obtain semantic embedding vectors;

[0094] S203: Cosine similarity based on adjacent segments and preset semantic integrity threshold The knowledge is merged to obtain the set of fragments. The cosine similarity Based on the semantic embedding vector get.

[0095] In some embodiments, the preset boundary conditions include headings, paragraphs, tables, and formulas, etc., but this application is not limited thereto.

[0096] By utilizing preset boundary conditions, each text to be retrieved can be... Perform sentence segmentation and divide it into... A segment:

[0097]

[0098] in, For the text to be retrieved The result is a set of fragments.

[0099] Each text to be retrieved is processed based on preset boundary conditions. Segmenting sentences ensures the structural integrity of each segment.

[0100] In S202, for each segment obtained from the segmentation in S201, it is necessary to vectorize it, that is, to calculate the semantic embedding vector. Specifically, the following semantic embedding model needs to be utilized. The segmented fragments are vectorized to obtain semantic embedding vectors. :

[0101]

[0102]

[0103] in, For a d-dimensional vector space, Semantic embedding model For example, it could be a model like BGE or SBERT.

[0104] In some embodiments, after S202, the text retrieval method of this application further includes:

[0105] Cosine similarity is calculated based on the semantic embedding vector, the type of the boundary conditions, and the penalty coefficient. .

[0106] Cosine similarity The calculation formula is as follows:

[0107]

[0108] in, ; This indicates whether strong boundary conditions exist (titles, tables, formulas, etc.). This is the penalty coefficient.

[0109] In the specific implementation of S203, a semantic integrity threshold needs to be set first. .

[0110] In some embodiments, semantic integrity threshold It can be a fixed value as follows:

[0111]

[0112] in, , These represent the mean and standard deviation of the similarity, respectively.

[0113] In some embodiments, semantic integrity threshold The value can be adaptive, for example, using the following quantile threshold:

[0114]

[0115] in, For quantiles, .

[0116] In the specific implementation of S203, it is necessary to calculate the cosine similarity of adjacent segments. With semantic integrity threshold Compare the cosine similarity of adjacent segments. Above the semantic integrity threshold Then, adjacent segments will be merged. The merging operator is defined as follows:

[0117] If adjacent segments satisfy a similarity higher than a threshold, for example and Cosine similarity Above the semantic integrity threshold Then and merge:

[0118]

[0119] When performing a merge operation, it is necessary to concatenate the text content and its Markdown structure.

[0120] It should be noted that merging adjacent segments may not be performed in a single round. For merging new adjacent segments, it is necessary to determine whether further merging is needed. For example, after merging the second and third segments to form a new segment, and this new segment is adjacent to both the original first and fourth segments, it is necessary to determine whether the new segment needs to be merged with the original first and fourth segments.

[0121] In some embodiments, in S203, such as Figure 3 As shown, based on the cosine similarity of adjacent segments and preset semantic integrity threshold The knowledge is merged to obtain the set of fragments, including:

[0122] S301: Based on the cosine similarity of adjacent segments, the semantic integrity threshold, and the semantic embedding vector, a merging operation is performed on adjacent segmented segments to obtain a first segment set;

[0123] S302: Based on the cosine similarity of adjacent segments in the first segment set and the semantic integrity threshold, merge the segments in the first segment set to obtain a second segment set;

[0124] S303: Perform the above iterative operation until the cosine similarity of adjacent segments is less than the semantic integrity threshold, and obtain the segment set.

[0125] The above iterative process requires processing each text. This involves knowledge merging. In practice, the iterative process described above is as follows:

[0126] ① Calculate the set of similarities between all adjacent segments ;

[0127] ② Take ;

[0128] In practice, if the cosine similarity between the current segment and both adjacent segments exceeds the threshold... The algorithm selects adjacent segments with a high cosine similarity to the current segment and merges them.

[0129] ③If Then execute the merge:

[0130]

[0131] And delete ;

[0132] ④ Update local similarity And repeat step ①;

[0133] ⑤ When The process terminates at a certain time, resulting in each text. The final collection of fragments:

[0134]

[0135] For each text All knowledge is merged, and the output is a set of segments that satisfy both structural completeness and semantic completeness. .

[0136] In some embodiments, in S102, the fragments in the fragment set are subjected to knowledge vectorization and meta-information vectorization to obtain knowledge vectors. and meta-information vector ,like Figure 4 As shown, in specific implementation, it may include:

[0137] S401: Use a semantic embedding model to vectorize the fragments in the fragment set to obtain the knowledge vector. ;

[0138] S402: Based on the semantic embedding model, extract the fragments from the fragment set. The accompanying metadata is converted into a string and then vectorized to obtain the metadata vector. .

[0139] In practice, it is necessary to process all text fragments segmented by S101. The semantic embedding model is used to vectorize the fragments in the fragment set, where:

[0140]

[0141] in, , , which represents the total number of all file fragments.

[0142] Using semantic embedding models Mapping all fragments to a vector space yields the knowledge vector. :

[0143]

[0144] in, It is a d-dimensional vector space.

[0145] For each segment Meta-information attached to Convert it to a string and embed it into a vector to obtain the meta-information vector. :

[0146]

[0147] Among them, metadata This includes: headings, paragraph numbers, source document identifiers, table / formula types, and location indexes.

[0148] In some embodiments, in S103, such as Figure 5 As shown, constructing a clustering function based on a pre-built cosine similarity function can include the following:

[0149] S501: Generate the cosine similarity function based on the knowledge vector, meta-information vector, and similarity weights of the knowledge vector;

[0150] S502: Based on the cosine similarity function and the fragment set Construct the clustering function.

[0151] Among them, the cosine similarity function as follows:

[0152]

[0153] in, For the similarity weights of knowledge vectors, .

[0154] To determine the aggregation relationship between fragments, a cosine similarity function can be used. Constructing clustering functions :

[0155]

[0156] in, , They are two sets of fragments respectively ( , Belonging to , This represents the average semantic similarity between clusters.

[0157] In some embodiments, in S103, such as Figure 6 As shown, based on the clustering function and preset clustering threshold, inter-cluster aggregation and hierarchical construction are performed to obtain a multi-level retrieval structure, which may include the following:

[0158] S601: If the value of the clustering function is greater than the clustering threshold, perform inter-cluster aggregation;

[0159] S602: Determine the semantic center vector of the topic based on the aggregation results; the semantic center vector of the topic and the fragment constitute the multi-level retrieval structure.

[0160] Inter-cluster aggregation can be based on preset clustering criteria; in specific implementation, a clustering threshold can be used. Based on this, when the following conditions are met:

[0161]

[0162] Then perform inter-cluster aggregation:

[0163]

[0164] The process of inter-cluster aggregation can be performed iteratively in a bottom-up or top-down manner until the similarity between any two clusters is less than 1. This completes the final inter-cluster aggregation.

[0165] Each topic semantic center vector Defined as all knowledge vectors under a topic The mean, semantic center vector as follows:

[0166]

[0167] Based on the semantic center vector of the topic obtained above It is possible to construct "topic-fragment" The hierarchical knowledge structure of “)” enables semantic connections between texts while preserving the traceability of the original information.

[0168] In some embodiments, in S602, such as Figure 7 As shown, the semantic center vector of a topic is determined based on the aggregation results, including:

[0169] S701: Identify subject clusters from different texts through multi-level directories;

[0170] S702: Determine the semantic center vector of the topic based on the topic cluster.

[0171] In practice, based on the clustering results, knowledge fragments can be organized into a two-level hierarchical structure:

[0172] ① First-level directory Representing different knowledge topics, each topic This corresponds to a set of semantically related fragments;

[0173] ② A second-level directory is a collection of different text fragments under the same topic:

[0174]

[0175] in This is a clustering mapping function, representing the topic number to which the fragment belongs. .

[0176] Based on the aforementioned topic clusters, for each topic This allows us to obtain the semantic center vector of the mean of all knowledge vectors under a given topic. .

[0177] In some embodiments, such as Figure 8 As shown, the text retrieval method of this application also includes:

[0178] S801: Utilizes a large language model to obtain a diverse set of queries based on the user's original query statement. ;

[0179] S802: Utilize a semantic embedding model to integrate the diverse query set Mapped to the query vector .

[0180] In practice, the user inputs the original query statement. :

[0181]

[0182] Among them, the original query statement For a set of statements, The original question entered by the user.

[0183] In some embodiments, such as Figure 9 As shown, by utilizing a large language model, a diverse set of queries can be obtained based on the user's original query statement, which may include the following:

[0184] S901: Utilizes a large language model to parse the user's original query statement into standardized key-value pairs. ;

[0185] S902: Perform semantic expansion and rewriting on the standardized key values ​​to obtain the diverse query set. .

[0186] Specifically, based on the original query statement The query semantics can be parsed as follows:

[0187] Call the Large Language Model (LLM) on the original query statement Semantic element extraction and structured parsing are performed to obtain standardized key values. :

[0188]

[0189] This key value It can be directly used as the basis for subsequent diversified generation and knowledge matching.

[0190] Based on the large language model for key values By performing semantic expansion and rewriting, the diverse query set is obtained. :

[0191]

[0192] in For the number of generated, , The generation methods include, but are not limited to:

[0193] ①Synonym substitution:

[0194] ②Word order adjustment:

[0195] ③ Hierarchical expansion: e.g., "dam deformation" → "structural deformation monitoring"

[0196] ④ Condition completion: such as wait.

[0197] Each generated Each of these can correspond to a structured expression:

[0198]

[0199] Specifically, it is necessary to address the above. In Filter:

[0200] ① Control the number of generators To prevent excessive rewriting;

[0201] ② Calculation and generation Inter-similarity:

[0202]

[0203] like ( for If the similarity between elements is high, it is considered semantically redundant and will be deleted. ;

[0204] ③ For low confidence levels or those that do not conform to grammatical constraints They should be removed.

[0205] For the final filtered query set Using semantic embedding models Mapping to a vector space yields the query vector. :

[0206]

[0207] It should be noted that each can be retained. Corresponding structured information This is for subsequent retrieval mapping.

[0208] This application utilizes a combination of hierarchical knowledge representation and diverse natural language queries to achieve semantic modeling and hierarchical retrieval of engineering documents. It overcomes the problems of insufficient semantic understanding and flat results in traditional keyword matching retrieval, and breaks through the limitations of existing semantic vector retrieval in multi-version knowledge organization.

[0209] In the embodiment, in S104, as Figure 10 As shown, text retrieval is performed based on the multi-level retrieval structure and the query vector generated based on the large language model, including:

[0210] S1001: Perform subject-level retrieval and fragment-level retrieval based on the multi-level retrieval structure and the query vector generated based on the large language model to obtain the first retrieval result;

[0211] S1002: The first retrieval result is reordered based on a pre-trained reordering model to obtain the second retrieval result.

[0212] During the main body layer retrieval, it is necessary to calculate the query vector. semantic center vector of the topic similarity :

[0213]

[0214] You can select similarity higher front There are 5 topics. The number of topics, n, can be set according to specific circumstances, for example, n can be set to 5.

[0215] Based on the selected candidate topics, fragment-level retrieval can be performed, and the similarity of fragments under the candidate topics can be calculated to obtain the retrieval results. :

[0216] ,

[0217] You can select search results High score There are 10 segments. The number of segments, m, can be set according to specific circumstances. For example, m can be set to 10 as the first search result.

[0218] Based on the m search results selected above Through a pre-trained reordering model Reorder the segments to obtain the score for each segment. :

[0219]

[0220] Pre-trained ranking model The BGE-RERANKER-V2-M3 model can be used, but this application is not limited to it. Finally, the results are sorted from high to low according to the comprehensive score, and the second search result is obtained.

[0221] The second search result is returned in the form of topic aggregation. This second search result supports tracing back to the original text paragraphs, author and journal information, and can realize semantic-level retrieval and intelligent comparative analysis of papers on water conservancy and hydropower engineering.

[0222] This application utilizes a combination of hierarchical knowledge representation and diverse natural language queries to achieve semantic modeling and hierarchical retrieval of engineering documents. It overcomes the problems of insufficient semantic understanding and flat results in traditional keyword matching retrieval, and breaks through the limitations of existing semantic vector retrieval in organizing knowledge from multiple sources and versions. It provides a high-precision and robust semantic retrieval method for the intelligent organization and comprehensive utilization of massive data in the field of water conservancy and hydropower engineering.

[0223] This application takes the retrieval of papers on water conservancy and hydropower engineering as an example to illustrate how to achieve intelligent parsing, semantic modeling, and hierarchical retrieval of academic literature based on hierarchical knowledge representation and diversified natural language query technology.

[0224] Step (1): Document preprocessing and semantic segmentation

[0225] 150 Chinese academic papers in the field of water conservancy and hydropower engineering were selected as the data source, and all texts were in PDF format.

[0226] Optical Character Recognition (OCR) technology based on the MinerU 2.5 model is used to recognize and extract structured data from the paper content, obtaining structured elements (boundary conditions) including main text, tables, formulas, and titles. After format conversion, each paper can be standardized into a Markdown document.

[0227]

[0228] After format conversion, semantic segmentation is performed based on boundaries such as headings, paragraphs, tables, and formulas to ensure sentence and semantic integrity, forming a collection of fragments:

[0229]

[0230] Furthermore, for each file It needs to be vectorized based on a semantic embedding model (such as BGE-M3) to obtain semantic embedding vectors. .

[0231] Based on semantic embedding vectors Cosine similarity can be calculated. Cosine similarity semantic integrity threshold of adjacent vectors (For example By comparing these, we can determine whether knowledge merging is necessary to ensure semantic consistency within each segment.

[0232] The knowledge items obtained after document preprocessing and semantic segmentation are as follows:

[0233] The collection of all documents, of which 3941 knowledge entries are derived from 150 documents, is presented here for illustrative purposes only and is not intended to limit the scope of the application.

[0234] In this example, 83.2% of the knowledge length is within 700 characters, and the remaining 16.8% of the knowledge length has an average of about 900 characters, which meets the maximum word length of the word embedding input of BGE-M3.

[0235] Step (2): Knowledge and Meta-information Vectorization

[0236] For each semantic fragment By using a semantic embedding model for semantic vectorization mapping, a 1024-dimensional knowledge vector can be obtained in this embodiment. :

[0237]

[0238] Furthermore, metadata about the paper is extracted, including title, author, journal name, keywords, publication date, etc., and fragments from the fragment set are then extracted using a semantic embedding model. The accompanying metadata is converted into a string and then vectorized to obtain the metadata vector. :

[0239]

[0240] Step (3): Topic Clustering and Hierarchical Knowledge Organization

[0241] Based on the semantic vector obtained in step (2) and The cosine similarity function can be obtained. :

[0242]

[0243] in, .

[0244] In this embodiment, a hierarchical clustering method based on complete linkage is used, with a clustering threshold. As a control parameter, the above cosine similarity function is used. Convert to distance metric :

[0245]

[0246] In topic clustering, the distance between any two clusters is defined as the maximum distance between sample pairs within the cluster. for:

[0247]

[0248] In specific clustering, when When the distance between all clusters is greater than the clustering threshold, cluster merging is performed. Ultimately, the similarity of any sample within a cluster is no less than [a certain value]. The clustering results are obtained. For the center vector of each cluster, the mean of the samples within the cluster needs to be taken and normalized.

[0249] when When the value is 0.8, several semantically consistent topic clusters are obtained:

[0250]

[0251] For example, the knowledge fragments in cluster 10 are as follows, and the similarity between the knowledge fragments is relatively high:

[0252]

[0253]

[0254]

[0255] Furthermore, for each of the above topics We need to calculate the semantic center vector:

[0256]

[0257] Building based on "topic-fragment ( A two-level hierarchical structure This provides multi-level index support for subsequent queries and retrievals.

[0258] Step (4): Query parsing, semantic generation and multi-level retrieval

[0259] When the user enters a natural language query statement (For example: "What methods are used to assess the structural safety of the Three Gorges Dam?"), the system calls a large language model (e.g., DeepSeek) to perform the following processing:

[0260] Step 4.1: Query Parsing

[0261] First, a large language model is used to perform structured parsing of the input sentence, extracting core semantic elements and outputting a standardized dictionary:

[0262]

[0263] Step 4.2: Generating Diverse Queries

[0264] Based on a large language model, diverse queries with semantically equivalent or approximate meanings can be generated:

[0265]

[0266]

[0267]

[0268] Step 4.3: Through Inter-similarity The duplicate filters retain n valid queries, where n can be 2, and this application does not limit it to this.

[0269] Step 4.4: Vectorization and Hierarchical Retrieval

[0270] For each query expression Calculate semantic vectors:

[0271]

[0272] Based on the hierarchical knowledge structure in step (3), the following two-level retrieval can be performed:

[0273] ① Topic-level retrieval: Calculating the query vector Similarity to the topic center vector:

[0274] )

[0275] Select the highest scorer There are several themes, among which... You can choose 5.

[0276] ② Fragment-level retrieval: Calculate the similarity of fragments under candidate topics to obtain retrieval results:

[0277] ,

[0278] Select the highest scorer Among the segments, You can take 10.

[0279] ③ Fusion Ranking: Calculate the scores of 10 segments using a pre-trained re-ranking model.

[0280]

[0281] Pre-trained ranking model The BGE-RERANKER-V2-M3 model can be used, but this application is not limited to it.

[0282] After the above search, multiple search results are obtained. These results are sorted from highest to lowest based on their overall scores, and the sorted result set is output.

[0283] Step 4.5: Result Display

[0284] The sorted results are returned in the form of topic aggregation, allowing users to trace back to view the original paragraphs, authors, and journal information, enabling semantic-level retrieval and intelligent comparative analysis of papers on water conservancy and hydropower engineering.

[0285] This application provides a text retrieval device applied to the aforementioned text retrieval method. This text retrieval device and the text retrieval method in one embodiment of this application are based on the same inventive concept and have similar problem-solving principles. Therefore, the implementation of the text retrieval device is the same as the implementation of the text retrieval method in one embodiment of this application, and repeated details will not be described again. The terms "unit" or "module" used below can refer to a combination of software and / or hardware that performs a predetermined function. Although the system described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0286] like Figure 11 As shown, the text retrieval device includes: a preprocessing unit 1101, a vectorization unit 1102, a retrieval structure generation unit 1103, and a retrieval unit 1104.

[0287] The preprocessing unit 1101 is used to perform structured processing and semantic segmentation on the text to be retrieved to obtain a set of fragments; wherein the set of fragments satisfies structural integrity and semantic integrity.

[0288] The vectorization unit 1102 is used to perform knowledge vectorization and meta-information vectorization on the fragments in the fragment set to obtain knowledge vectors and meta-information vectors.

[0289] The retrieval structure generation unit 1103 is used to construct a clustering function based on a pre-built cosine similarity function, and to perform inter-cluster aggregation and hierarchical construction according to the clustering function and a preset clustering threshold to obtain a multi-level retrieval structure; wherein, the clustering function is generated based on the knowledge vector and the meta-information vector;

[0290] The retrieval unit 1104 is used to perform text retrieval based on the multi-level retrieval structure and the query vector generated based on the large language model.

[0291] In some embodiments, the preprocessing unit 1101 includes:

[0292] The segmentation module is used to segment the text to be retrieved into multiple segments according to preset boundary conditions; the boundary conditions include titles, paragraphs, tables, and formulas.

[0293] The vectorization module is used to vectorize the segmented fragments based on the semantic embedding model to obtain semantic embedding vectors;

[0294] The knowledge merging module is used to merge knowledge based on the cosine similarity of adjacent segments and a preset semantic integrity threshold to obtain the segment set, wherein the cosine similarity is obtained based on the semantic embedding vector.

[0295] In some embodiments, the text retrieval device further includes:

[0296] The cosine similarity calculation module is used to calculate the cosine similarity based on the semantic embedding vector, the type of the boundary condition, and the penalty coefficient.

[0297] In some embodiments, the knowledge merging module includes:

[0298] The first merging submodule is used to perform a merging operation on adjacent segmented segments based on the cosine similarity of adjacent segments, the semantic integrity threshold, and the semantic embedding vector, to obtain a first segment set;

[0299] The second merging submodule is used to merge the segments in the first segment set based on the cosine similarity of adjacent segments in the first segment set and the semantic integrity threshold to obtain a second segment set.

[0300] The first merging submodule and the second merging submodule alternately perform merging operations until the cosine similarity of adjacent segments is less than the semantic integrity threshold, thus obtaining the segment set.

[0301] In some embodiments, the vectorization unit 1102 includes:

[0302] The knowledge vector generation module is used to perform vectorized mapping on the fragments in the fragment set using a semantic embedding model to obtain the knowledge vector;

[0303] The meta-information vector generation module is used to convert the meta-information attached to the fragments in the fragment set into strings and vectorize them according to the semantic embedding model to obtain the meta-information vector.

[0304] In some embodiments, the text retrieval further includes:

[0305] The cosine similarity function generation unit is used to generate the cosine similarity function based on the knowledge vector, the meta-information vector, and the similarity weights of the knowledge vector;

[0306] Clustering function construction unit, used to construct clustering functions based on the cosine similarity function and the fragment set. Construct the clustering function.

[0307] In some embodiments, the retrieval structure generation unit includes:

[0308] The aggregation module is used to perform inter-cluster aggregation when the value of the clustering function is greater than the clustering threshold;

[0309] The center vector determination module is used to determine the semantic center vector of a topic based on the aggregation results; the semantic center vector of the topic and the fragment constitute the multi-level retrieval structure.

[0310] In some embodiments, the retrieval structure generation unit includes:

[0311] The topic cluster determination submodule is used to determine topic clusters from different texts through multi-level directories;

[0312] The center vector determination submodule is used to determine the semantic center vector of the topic based on the topic cluster.

[0313] In some embodiments, the text retrieval device further includes:

[0314] The query set generation unit is used to generate diverse query sets based on the user's original query statement using a large language model.

[0315] A set mapping unit is used to map the diverse query set to the query vector using a semantic embedding model.

[0316] In some embodiments, the query set generation unit includes:

[0317] The parsing module is used to parse the user's original query statement into standardized key-value pairs using a large language model;

[0318] The extended rewriting module is used to semantically extend and rewrite the standardized key values ​​to obtain the diverse query set.

[0319] In some embodiments, the retrieval unit includes:

[0320] The first retrieval module is used to perform subject-level retrieval and fragment-level retrieval based on the multi-level retrieval structure and the query vector generated based on the large language model, and obtain the first retrieval result.

[0321] The second retrieval module is used to reorder the first retrieval results based on a pre-trained reordering model to obtain the second retrieval results.

[0322] Figure 12 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention, such as... Figure 12As shown, the electronic device 1200 includes: a processor 1201, a memory 1202, and a bus 1203.

[0323] The processor 1201 and the memory 1202 communicate with each other via the bus 1203.

[0324] The processor 1201 is used to call program instructions in the memory 1202 to execute the methods provided in the above-described method embodiments.

[0325] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described text retrieval method.

[0326] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described text retrieval method.

[0327] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0328] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0329] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The function specified in one or more boxes.

[0330] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0331] In the description of this specification, the references to terms such as "an embodiment," "a specific embodiment," "some embodiments," "for example," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0332] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A text retrieval method, characterized in that, include: The text to be retrieved is subjected to structured processing and semantic segmentation to obtain a set of fragments; wherein the set of fragments satisfies both structural integrity and semantic integrity. The fragments in the fragment set are vectorized into knowledge vectors and meta-information vectors to obtain knowledge vectors and meta-information vectors. A clustering function is constructed based on a pre-built cosine similarity function, and inter-cluster aggregation and hierarchical construction are performed according to the clustering function and a preset clustering threshold to obtain a multi-level retrieval structure; wherein, the clustering function is generated based on the knowledge vector and meta-information vector; Text retrieval is performed based on the multi-level retrieval structure and the query vector generated based on the large language model. The step of performing inter-cluster aggregation and hierarchical construction based on the clustering function and a preset clustering threshold to obtain a multi-level retrieval structure includes: If the value of the clustering function is greater than the clustering threshold, inter-cluster aggregation is performed; The semantic center vector of the topic is determined based on the aggregation results; the semantic center vector of the topic and the fragment constitute the multi-level retrieval structure.

2. The text retrieval method according to claim 1, characterized in that, The text to be retrieved undergoes structured processing and semantic segmentation to obtain a set of fragments, including: The text to be retrieved is segmented into multiple segments based on preset boundary conditions; the boundary conditions include titles, paragraphs, tables, and formulas. The segmented fragments are vectorized based on a semantic embedding model to obtain semantic embedding vectors; Knowledge merging is performed based on the cosine similarity of adjacent segments and a preset semantic integrity threshold to obtain the segment set, wherein the cosine similarity is obtained based on the semantic embedding vector.

3. The text retrieval method according to claim 2, characterized in that, Also includes: The cosine similarity is calculated based on the semantic embedding vector, the type of the boundary condition, and the penalty coefficient.

4. The text retrieval method according to claim 2, characterized in that, The knowledge merging based on the cosine similarity of adjacent segments and a preset semantic integrity threshold yields the segment set, which includes: Step 1: Based on the cosine similarity of adjacent segments, the semantic integrity threshold, and the semantic embedding vector, perform a merging operation on adjacent segmented segments to obtain a first segment set; Step 2: Merge the segments in the first segment set based on the cosine similarity of adjacent segments in the first segment set and the semantic integrity threshold to obtain the second segment set; Perform steps 1 and 2 until the cosine similarity of adjacent segments is less than the semantic integrity threshold, and obtain the segment set.

5. The text retrieval method according to claim 1, characterized in that, The step of performing knowledge vectorization and meta-information vectorization on the fragments in the fragment set to obtain knowledge vectors and meta-information vectors includes: The knowledge vector is obtained by vectorizing the fragments in the fragment set using a semantic embedding model; The metadata associated with the fragments in the fragment set is converted into strings and vectorized according to the semantic embedding model to obtain the metadata vector.

6. The text retrieval method according to claim 1, characterized in that, Clustering functions are constructed based on a pre-built cosine similarity function, including: The cosine similarity function is generated based on the knowledge vector, the meta-information vector, and the similarity weights of the knowledge vectors. Based on the cosine similarity function and the fragment set Construct the clustering function.

7. The text retrieval method according to claim 1, characterized in that, The determination of the semantic center vector of the topic based on the aggregation result includes: Identify subject clusters from different texts using multi-level directories; The semantic center vector of the topic is determined based on the topic cluster.

8. The text retrieval method according to claim 1, characterized in that, Also includes: By utilizing a large language model, a diverse set of queries is obtained based on the user's original query statement; The diverse query set is mapped to the query vector using a semantic embedding model.

9. The text retrieval method according to claim 8, characterized in that, The method of utilizing a large language model to obtain a diverse set of queries based on the user's original query statement includes: Using a large language model, the user's original query statement is parsed into standardized key-value pairs; The standardized key values ​​are semantically expanded and rewritten to obtain the diverse query set.

10. The text retrieval method according to claim 9, characterized in that, The text retrieval based on the multi-level retrieval structure and the query vector generated based on the large language model includes: Based on the multi-level retrieval structure and the query vector generated based on the large language model, subject-level retrieval and fragment-level retrieval are performed to obtain the first retrieval result; The first retrieval result is reordered based on a pre-trained reordering model to obtain the second retrieval result.

11. A text retrieval device, characterized in that, include: The preprocessing unit is used to perform structured processing and semantic segmentation on the text to be retrieved to obtain a set of fragments; wherein the set of fragments satisfies structural integrity and semantic integrity. The vectorization unit is used to perform knowledge vectorization and meta-information vectorization on the fragments in the fragment set to obtain knowledge vectors and meta-information vectors. The retrieval structure generation unit is used to construct a clustering function based on a pre-built cosine similarity function, and to perform inter-cluster aggregation and hierarchical construction according to the clustering function and a preset clustering threshold to obtain a multi-level retrieval structure; wherein, the clustering function is generated based on the knowledge vector and the meta-information vector; The retrieval unit is used to perform text retrieval based on the multi-level retrieval structure and the query vector generated based on the large language model. The retrieval structure generation unit includes: The aggregation module is used to perform inter-cluster aggregation when the value of the clustering function is greater than the clustering threshold; The center vector determination module is used to determine the semantic center vector of a topic based on the aggregation results; the semantic center vector of the topic and the fragment constitute the multi-level retrieval structure.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the text retrieval method according to any one of claims 1 to 10.

13. A computer storage medium storing instructions thereon, characterized in that, When the instructions are executed individually or jointly by at least one processor of a computer device, the computer device performs the text retrieval method according to any one of claims 1 to 10.

14. A computer program product comprising instructions, characterized in that, When the instructions are executed individually or jointly by at least one processor of a computer device, the computer device performs the text retrieval method according to any one of claims 1 to 10.

Images