Retrieval enhancement generation method based on hybrid strategy and related equipment

Abstract

The invention discloses a retrieval enhancement generation method and related equipment based on a hybrid strategy, and the method comprises the steps: obtaining a target document, and determining whether to segment the target document or not and determining a segment strategy according to the size and paragraph level information of the target document; extracting a keyword of the target document or the fragment and generating a vector, and inserting the keyword, the vector and the target document into a vector database; retrieval content is obtained, the retrieval content is preprocessed, and keywords and retrieval words are extracted; and performing matching retrieval in the vector database according to the keyword and the retrieval word, and returning the fragment content meeting the requirement. According to the embodiment of the invention, the content generation accuracy can be improved. The method can be widely applied to the technical field of artificial intelligence.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a retrieval enhancement generation method and related equipment based on a hybrid strategy. Background Technology

[0002] With the powerful capabilities of Large Language Models (LLMs) in natural language processing tasks such as text generation, question answering systems, and dialogue interaction, ensuring the accuracy, factuality, and reliability of their generated content has become a key research focus. Retrieval-Augmented Generation (RAG) technology has emerged to address this need. By retrieving relevant information from external knowledge bases as context, it effectively guides and constrains the generation process of large language models, compensating for the potential outdated, inaccurate, or illusory nature of their inherent knowledge.

[0003] However, existing retrieval enhancement generation methods still face many challenges in practical applications, especially in the two core stages of document preprocessing (index building) and retrieval recall, where there are problems of single strategies and insufficient adaptability, resulting in the need to improve the accuracy of generated content. Summary of the Invention

[0004] The main objective of this application is to propose a retrieval enhancement generation method and related equipment based on a hybrid strategy, aiming to improve the accuracy of the generated content.

[0005] To achieve the above objectives, one aspect of this application proposes a retrieval enhancement generation method based on a hybrid strategy, the method comprising the following steps:

[0006] Obtain the target document, and determine whether to segment the target document and the segmentation strategy based on the size and paragraph hierarchy information of the target document; Extract the segment keywords of the target document or the segment and generate a vector, then insert the segment keywords, the vector, and the target document into a vector database; Obtain the search content, preprocess the search content, and extract search keywords and terms; The vector database is matched and searched based on the search keywords and terms, and the content that meets the requirements is returned.

[0007] In some embodiments, determining whether to fragment the target document and the fragmentation strategy based on the size and paragraph level information of the target document includes: If the target document is less than or equal to the first threshold, the target document is treated as a fragment.

[0008] In some embodiments, determining whether to fragment the target document and the fragmentation strategy based on the size and paragraph level information of the target document includes: If the target document is larger than the first threshold, the target document is parsed to obtain the paragraph level information of the target document; If there is no paragraph level, the text will be segmented according to the preset number of characters per segment and the preset number of overlapping characters per segment.

[0009] In some embodiments, the method further includes: If there are paragraph levels, calculate the average number of characters in each paragraph level, and segment the paragraphs according to the paragraph level whose average number of characters is closest to the second threshold.

[0010] In some embodiments, the method further includes: If there exists a paragraph with more than the third threshold in number of characters at level n, and there is a subset of paragraphs, then the text is segmented according to paragraphs at level n+1; where n is an integer greater than or equal to 2. If no subset paragraph exists, the text is fragmented according to the preset number of fragment characters and the preset number of overlapping fragment characters.

[0011] In some embodiments, the method further includes: If the number of characters in an n-level paragraph is less than or equal to the third threshold, the paragraphs are segmented according to the n-level paragraphs. If the total length of all n-level paragraphs under the n-1 level is less than or equal to the fourth threshold, an n-1 level paragraph title is added to the content of each n-level paragraph, and the paragraph title is used as the keyword for the corresponding segment.

[0012] In some embodiments, the step of performing a matching search on the vector database based on the search keywords and the search terms, and returning content segments that meet the requirements, includes: Based on the search keywords, perform keyword matching and retrieval in the vector database, and return the content of the successfully matched segments; Based on the search terms, a semantic vector matching search is performed in the vector database, and the content of the segment with a matching score greater than a preset score is returned.

[0013] To achieve the above objectives, another aspect of this application proposes a retrieval enhancement generation apparatus based on a hybrid strategy, the apparatus comprising: The first module is used to obtain the target document and determine whether to segment the target document and the segmentation strategy based on the size and paragraph hierarchy information of the target document. The second module is used to extract the segment keywords of the target document or the segment and generate a vector, and insert the segment keywords, the vector and the target document into the vector database; The third module is used to obtain the search content, preprocess the search content, and extract search keywords and terms; The fourth module is used to perform matching and retrieval in the vector database based on the search keywords and search terms, and return the segmented content that meets the requirements.

[0014] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the methods described above.

[0015] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.

[0016] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer program product, including a computer program that, when executed by a processor, implements the methods described above.

[0017] The embodiments of this application include at least the following beneficial effects: This application provides a retrieval enhancement generation method, apparatus, electronic device, and program product based on a hybrid strategy. This solution obtains a target document, determines whether to segment the target document and the segmentation strategy based on the size and paragraph hierarchy information of the target document, extracts keywords from the target document or segment and generates vectors, inserts the keywords, vectors, and target document into a vector database, and dynamically selects the optimal segmentation strategy based on the actual length and structural characteristics of the target document; obtains retrieval content, preprocesses the retrieval content and extracts keywords and search terms, performs matching retrieval in the vector database based on the keywords and search terms, and returns segmented content that meets the requirements, thereby improving the comprehensive relevance and coverage of the retrieval results and thus improving the accuracy of the generated content. Attached Figure Description

[0018] Figure 1 This is a flowchart of a retrieval enhancement generation method based on a hybrid strategy provided in an embodiment of this application; Figure 2 This is a block diagram of a retrieval enhancement generation system based on a hybrid strategy, provided in an embodiment of this application. Figure 3 This is a flowchart illustrating a method for generating a vector database based on a hybrid strategy, as provided in an embodiment of this application. Figure 4 This is a flowchart illustrating a retrieval based on a vector database, as provided in an embodiment of this application. Figure 5This is a schematic diagram of the structure of a retrieval enhancement generation device based on a hybrid strategy provided in an embodiment of this application; Figure 6 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0020] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0021] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0023] Before providing a detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.

[0024] Retrieval Augmented Generation (RAG) is a technical framework that combines information retrieval systems with the generative capabilities of large language models. Simply put, its core idea is to have the large model "search" for information in a knowledge base before answering a question, and then organize and generate the answer based on the found information.

[0025] Chunking, in the context of retrieval enhancement techniques, refers to the process of dividing lengthy original documents (such as PDFs, Word documents, and web articles) into smaller, more manageable segments or chunks. These chunks are the basic units that are subsequently converted into vectors and stored in a vector database.

[0026] Vector: In the context of retrieval enhancement generation, this specifically refers to text vectors, also known as embeddings. It is a string of numbers: a vector is an array of numbers, such as [0.023, -0.542, 0.118, ..., 0.831]. This array typically has hundreds or even thousands of dimensions (the number of numbers). It is the mathematical "fingerprint" of the text: this sequence of numbers is not random; it is generated through a complex machine learning model (called an embedding model, such as OpenAI's text-embedding-ada-002). This model can capture and condense the semantic information (i.e., meaning) of a piece of text. Semantic similarity, vector similarity: the core magic lies in the fact that texts with similar meanings will have vectors that are close in mathematical space.

[0027] In related technologies, traditional methods in the document preprocessing and index building stages typically employ a uniform text segmentation strategy, the most common being sliding window segmentation based on a fixed number of characters or tokens. While simple to implement, this method ignores the inherent semantic structure and logical units of a document. For example, when processing rigorously structured technical manuals or legal contracts, fixed-length segmentation easily breaks down a complete logical unit (such as a clause or a functional description module), resulting in incomplete semantic segments and creating "semantic gaps." This not only fragments the retrieved information but also severely impacts the downstream generation model's understanding of the overall context, leading to biased answers, omissions of key points, or logical inconsistencies. Although some segmentation methods based on paragraphs or chapters exist, they often lack a universal judgment mechanism for various types and heterogeneous documents, failing to dynamically select the optimal segmentation strategy based on the document's actual length and structural characteristics.

[0028] In the retrieval and recall phase, existing systems typically rely on a single retrieval mode. While keyword-based retrieval (such as BM25) is efficient and accurate when handling queries containing explicit entities and terms, its inherent lexical matching nature makes it ineffective in addressing the semantic diversity and ambiguity of user queries, resulting in a serious lexical gap problem. Conversely, pure vector semantic retrieval, although excellent in semantic similarity matching, may lack recall precision for queries containing specific names, precise codes, or rare terms, easily missing crucial, precisely matched segments. Current hybrid retrieval solutions often simply mechanically merge the two types of retrieval results, lacking intelligent strategy selection and dynamic weight adjustment based on query content characteristics. Consequently, in complex query scenarios, the overall relevance and coverage of the retrieval results remain unsatisfactory.

[0029] In view of this, this application provides a retrieval enhancement generation method, apparatus, electronic device, and program product based on a hybrid strategy. This solution obtains a target document, determines whether to segment the target document and the segmentation strategy based on the document's size and paragraph hierarchy information, extracts keywords from the target document or segmentation, generates vectors, inserts the keywords, vectors, and target document into a vector database, and dynamically selects the optimal segmentation strategy based on the target document's actual length and structural characteristics. It then obtains retrieval content, preprocesses the retrieval content, extracts keywords and search terms, performs matching searches in the vector database based on the keywords and search terms, and returns segmented content that meets the requirements. This improves the overall relevance and coverage of the retrieval results, thereby enhancing the accuracy of the generated content.

[0030] The retrieval enhancement generation method based on a hybrid strategy provided in this application relates to the field of information technology. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited thereto. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the retrieval enhancement generation method based on a hybrid strategy, but is not limited to the above forms.

[0031] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0032] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0033] Figure 1 This is an optional flowchart of a retrieval enhancement generation method based on a hybrid strategy provided in an embodiment of this application. Figure 1 The method may include, but is not limited to, steps S101 to S104.

[0034] Step S101: Obtain the target document and determine whether to segment the target document and the segmentation strategy based on the size of the target document and paragraph level information; Step S102: Extract the segment keywords of the target document or segment and generate vectors, then insert the segment keywords, vectors, and target document into the vector database; Step S103: Obtain the search content, preprocess the search content, and extract search keywords and terms; Step S104: Perform a matching search in the vector database based on the search keywords and search terms, and return the segmented content that meets the requirements.

[0035] It should be noted that this embodiment of the invention supports the processing of various common document types, such as txt, word, pdf, and markdown formats. After a user uploads a document to the system, the system's document parsing module will parse the uploaded document and extract tables, images, and text paragraph structures from it.

[0036] Specifically, the indexing process is as follows: Different target document sizes and paragraph levels correspond to different sharding strategies. The decision to shard the target document and the sharding strategy is determined based on the document size and paragraph level. If the target document is not sharded, keywords are directly extracted. If it is sharded, keywords for each shard are extracted and vectors are generated. The keywords, vectors, and the target document are then inserted into a vector database. The retrieval process is as follows: The user-inputted search content is obtained, preprocessed, and keywords and search terms are extracted. Preprocessing includes, but is not limited to, content cleaning. AI large language models can be used to analyze the user-inputted search content and extract paragraph titles, keywords, and other search terms. Based on the keywords and search terms, different matching searches are performed in the vector database to return sharded content that meets the requirements. It should be noted that different matching search methods have different requirements.

[0037] Reference Figure 2 The retrieval enhancement generation system proposed in this embodiment consists of a document parsing module, a document content indexing module, and a document content retrieval module. The document indexing module involves multiple segmentation strategies, primarily including determining whether to enable segmentation based on document length, segmenting based on a preset number of segment characters plus a preset number of overlapping segment characters, determining the number of paragraph levels for segmentation based on the average number of characters per paragraph level, and an index-retrieval paragraph segmentation mechanism. The document content retrieval module involves multiple retrieval strategies, primarily including user input data cleaning, document paragraph title extraction, keyword extraction, keyword retrieval mechanism, semantic vector retrieval mechanism, and retrieval segmentation deduplication mechanism. In practical applications, these strategies are invoked or referenced according to specific circumstances.

[0038] In some embodiments, determining whether to fragment the target document and the fragmentation strategy based on the size of the target document and paragraph hierarchy information includes: Step S201: If the target document is less than or equal to the first threshold, the target document is treated as a fragment.

[0039] First, the size of the target document uploaded by the user is checked. If the size of the target document is less than or equal to a preset first threshold, the entire target document is treated as a segment. After the system extracts keywords from the segment and generates a vector, the segment is inserted into the vector database.

[0040] In some embodiments, determining whether to fragment the target document and the fragmentation strategy based on the size of the target document and paragraph hierarchy information includes: Step S301: If the target document is larger than the first threshold, parse the target document to obtain the paragraph level information of the target document; Step S302: If there is no paragraph level, perform segmentation according to the preset number of segment characters and the preset number of overlapping segments.

[0041] If the size of the target document exceeds the first threshold, the parsing results of the target document need to be checked to determine whether a paragraph structure exists. If no paragraph structure exists, the document is fragmented according to a preset number of fragment characters and a preset number of overlapping fragment characters. That is, the fragmentation strategy fragments according to a fixed number of characters. At the same time, to ensure the semantic integrity of the fragments during retrieval, a fixed number of repeated characters are maintained between fragments. It should be noted that the first threshold is determined according to the actual situation, and this embodiment does not impose specific restrictions.

[0042] In some embodiments, the retrieval enhancement generation method further includes: Step S303: If there are paragraph levels, calculate the average number of characters in each paragraph level and segment the paragraphs according to the paragraph level whose average number of characters is closest to the second threshold.

[0043] If the parsed result of the target document contains paragraph structure, the hierarchy of the paragraph structure and the average number of characters at each level will be further analyzed to determine which level of the target document paragraph should be segmented. If the average number of characters at a certain level in the target document is closest to the segment character count specified by the system's second threshold, then the document will be segmented according to that paragraph level. After generating the segments using this method, keywords will be extracted from all segments, vectors will be generated, and the segments will be inserted into the vector database. It should be noted that the second threshold is determined based on the actual situation, and this embodiment does not impose specific restrictions.

[0044] In some embodiments, the retrieval enhancement generation method further includes: Step S401: If there is a paragraph with more than 2 characters at level n and there is a subset of paragraphs, then the paragraphs are divided into segments according to level n+1; n is an integer greater than or equal to 2. Step S402: If no subset paragraph exists, perform segmentation according to the preset number of segment characters and the preset number of overlapping segment characters.

[0045] If, after calculation, it is determined that the target document should be segmented according to level n paragraphs, and it is found that the number of characters in a certain level n paragraph exceeds the third threshold set by the system, then, if the paragraph has child paragraphs, it will be segmented according to level n+1 paragraphs; otherwise, it will be segmented according to the preset number of segment characters plus the preset number of overlapping segment characters. It should be noted that the third threshold is determined based on the actual situation, and this embodiment does not impose specific restrictions.

[0046] In some embodiments, the retrieval enhancement generation method further includes: Step S403: If the number of characters in an n-level paragraph is less than or equal to the third threshold, segment the text according to the n-level paragraph; if the total length of all n-level paragraphs under the n-1 level is less than or equal to the fourth threshold, add a title of the n-1 level paragraph to the content of each n-level paragraph, and use the paragraph title as the keyword of the corresponding segment.

[0047] In the strategy of segmenting by paragraph level, if, after calculation, segmenting by level n paragraphs, and the total length of all level n paragraphs under level n-1 does not exceed the fourth threshold set by the system, then the content of each level n paragraph is combined with the title of the level n-1 paragraph. The paragraph title automatically becomes the keyword for that segment, used for keyword matching during retrieval. It should be noted that the fourth threshold is determined based on actual circumstances, and this embodiment does not impose specific limitations.

[0048] The above-mentioned index-retrieval paragraph segmentation strategy means that after the system calculates and generates a segment, this segment is called a retrieval segment. If this segment is matched by a search term, the entire content of this segment will be returned. Simultaneously, this segment can be further segmented according to a preset number of segments and a preset number of overlapping characters. The resulting segments are called index segments. The keywords and vector values ​​of these segments are indexed and stored for matching input content during retrieval.

[0049] In some embodiments, a matching search is performed in a vector database based on search keywords and search terms to return content segments that meet the requirements, including: Step S501: Perform keyword matching retrieval in the vector database based on the search keywords, and return the content of the successfully matched segments; Step S502: Perform semantic vector matching retrieval in the vector database based on the search terms, and return the segmented content with a matching score greater than the preset score.

[0050] For searching article paragraph titles, a keyword matching method will be used to search, matching the segment keywords in the vector database with the paragraph title content, and returning all matching segment content.

[0051] When searching for keywords in an article paragraph, a keyword matching method will be used to retrieve all content in the paragraphs that match the keywords.

[0052] For text containing other search terms, semantic vector matching will be used for retrieval. In this retrieval method, the system will set a scoring threshold, and content segments with semantic vector matching scores higher than the threshold will be returned.

[0053] The retrieval function involved in this embodiment of the invention adopts a multi-strategy parallel operation mode. During the application of different strategies, multiple identical shards may be returned by different strategies. After the multi-strategy retrieval returns, the system of this embodiment of the invention performs deduplication processing on all returned shard IDs to ensure the uniqueness of the returned shards.

[0054] The search process of this application will be described below with a specific embodiment.

[0055] See Figure 3 The process of a document indexing mechanism according to an embodiment of the present invention is as follows: The first step is to check the size of the knowledge base document uploaded by the user, i.e. the target document. If the document size does not exceed the first threshold, the keywords in the document content are extracted directly and the entire document content is inserted into the vector database; otherwise, proceed to the second step.

[0056] The second step involves parsing the document content, extracting elements such as paragraphs, tables, and images, and then checking the extraction results. If the paragraph hierarchy information cannot be extracted, the document is segmented according to a preset number of characters plus a preset number of overlapping characters. In other words, the segmentation strategy uses a fixed number of characters for each segment, while maintaining a fixed number of repeated characters between segments to ensure the semantic integrity of the segmentation during retrieval. Otherwise, proceed to the third step.

[0057] The third step involves the system further analyzing the hierarchical structure of the document paragraphs and the average number of characters at each level. This analysis determines which level of the document paragraphs should be used for segmentation. If the average number of characters at a certain level is closest to the system-specified segment size, then the document will be segmented at that level. If the calculation determines that the document should be segmented according to the nth level of paragraphs, and it is found that the number of characters in a certain nth level paragraph exceeds the system's set segment length threshold, then the process proceeds to the fourth step. After generating the segments using this method, the system will extract keywords from all segments, generate vectors, and insert the segments into the vector database.

[0058] The fourth step is to split a paragraph into segments based on level n if the character length of the segment exceeds the system-specified threshold and there are sub-segments. If the segment does not have sub-segments, it will be split into segments based on the preset number of segments plus the preset number of overlapping segments.

[0059] Fifth, if the system, based on the previous steps, determines to segment according to level n paragraphs, and the total length of all level n paragraphs under level n-1 does not exceed the system-set threshold, then the content of each level n paragraph is combined with the title of the level n-1 paragraph. The paragraph title automatically becomes the keyword for that segment, used for keyword matching during retrieval.

[0060] Step 6: This embodiment of the invention supports setting an index-retrieval shard for each shard. After the system calculates and generates a shard according to the previous steps, this shard is called a retrieval shard. That is, if this shard is matched by a search term, the entire content of this shard will be returned completely. Simultaneously, this shard can be further sharded according to a specified number of shard characters plus a specified number of overlapping shard characters. The resulting shards are called index shards. The keywords and vector values ​​of these shards will be indexed and stored for matching input content during retrieval.

[0061] Step 7: After completing the document indexing, this embodiment of the invention provides a multi-strategy-based content retrieval mechanism, the specific process of which is as follows: Figure 4 As shown, the input search content is first cleaned and search terms are extracted. The user input is analyzed by an AI large language model, and the titles of the article paragraphs, keywords of the article paragraphs, and other texts related to the search are extracted from the user input.

[0062] Step 8: For the article paragraph titles or keywords extracted in step 7, a keyword retrieval strategy mechanism is used to match the indexed keywords and paragraph title content in the segment, and all matching segment content is returned.

[0063] In the ninth step, the remaining search terms extracted in the seventh step will be retrieved using semantic vector matching. In this retrieval method, the system will set a scoring threshold, and the content of the segments whose semantic vector matching scores are higher than the threshold will be returned.

[0064] Step 10: The system has a retrieval fragment deduplication mechanism. Since the same fragment in the vector database may be retrieved and returned by multiple strategies, the deduplication mechanism provided by the system will deduplicate duplicate fragments.

[0065] Please see Figure 5 This application also provides a retrieval enhancement generation apparatus based on a hybrid strategy, which can implement the above-described method. The apparatus includes: The first module is used to obtain the target document and determine whether to split the target document and the splitting strategy based on the size of the target document and paragraph hierarchy information. The second module is used to extract the segment keywords of the target document or segment and generate vectors, and insert the segment keywords, vectors and target document into the vector database; The third module is used to obtain the search content, preprocess the search content, and extract search keywords and terms; The fourth module is used to perform matching searches in the vector database based on search keywords and terms, and return the content segments that meet the requirements.

[0066] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0067] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0068] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0069] Please see Figure 6 , Figure 6 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 601 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 602 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 602 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 602 and is called and executed by the processor 601 using the methods described in the embodiments of this application. The input / output interface 603 is used to implement information input and output; The communication interface 604 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 605 transmits information between various components of the device (e.g., processor 601, memory 602, input / output interface 603, and communication interface 604); The processor 601, memory 602, input / output interface 603, and communication interface 604 are connected to each other within the device via bus 605.

[0070] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0071] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0072] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0073] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0074] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0075] The retrieval enhancement generation method, apparatus, electronic device, and program product based on hybrid strategies provided in this application have at least the following beneficial effects: 1. This invention proposes multiple document fragmentation strategies. By analyzing the content of the target document, the paragraph structure of the document is analyzed based on the parsing results of the document content. A certain fragmentation strategy is adopted according to the specific structure of the document, which enhances the flexibility and scalability of document indexing technology in retrieval enhancement generation technology; 2. In the document indexing part, a technical solution is designed to apply multiple strategies to process fragmented content simultaneously. For example, when using a strategy of fragmentation based on level n paragraphs, if a certain level n paragraph in the document exceeds the threshold of fragment length, a fragmentation strategy based on level n+1 paragraphs can be adopted. Fragmentation can also be based on a strategy of specifying the number of fragment characters + specifying the number of overlapping fragment characters, which reflects the diversity of document fragmentation strategies in the retrieval enhancement generation technology solution; 3. The concept of search term extraction + multi-strategy retrieval in the text retrieval part of the retrieval enhancement generation technology solution is proposed. By analyzing the input search content, different types of search terms are extracted, and then keyword retrieval and vector retrieval strategies are adopted according to the type of search terms to improve the overall recall rate.

[0076] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0077] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0078] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0079] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0080] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0081] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0082] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0083] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0084] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0085] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0086] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A retrieval enhancement generation method based on a hybrid strategy, characterized in that, The method includes the following steps: Obtain the target document, and determine whether to segment the target document and the segmentation strategy based on the size and paragraph hierarchy information of the target document; Extract the segment keywords of the target document or the segment and generate a vector, then insert the segment keywords, the vector, and the target document into a vector database; Obtain the search content, preprocess the search content, and extract search keywords and terms; The vector database is matched and searched based on the search keywords and terms, and the content that meets the requirements is returned.

2. The method according to claim 1, characterized in that, The step of determining whether to fragment the target document and the fragmentation strategy based on the size and paragraph hierarchy information of the target document includes: If the target document is less than or equal to the first threshold, the target document is treated as a fragment.

3. The method according to claim 1, characterized in that, The step of determining whether to fragment the target document and the fragmentation strategy based on the size and paragraph hierarchy information of the target document includes: If the target document is larger than the first threshold, the target document is parsed to obtain the paragraph level information of the target document; If there is no paragraph level, the text will be segmented according to the preset number of characters per segment and the preset number of overlapping characters per segment.

4. The method according to claim 3, characterized in that, The method further includes: If there are paragraph levels, calculate the average number of characters in each paragraph level, and segment the paragraphs according to the paragraph level whose average number of characters is closest to the second threshold.

5. The method according to claim 4, characterized in that, The method further includes: If there exists a paragraph with more than the third threshold in number of characters at level n, and there is a subset of paragraphs, then the text is segmented according to paragraphs at level n+1; where n is an integer greater than or equal to 2. If no subset paragraph exists, the text is fragmented according to the preset number of fragment characters and the preset number of overlapping fragment characters.

6. The method according to claim 5, characterized in that, The method further includes: If the number of characters in an n-level paragraph is less than or equal to the third threshold, the paragraphs are segmented according to the n-level paragraphs. If the total length of all n-level paragraphs under the n-1 level is less than or equal to the fourth threshold, an n-1 level paragraph title is added to the content of each n-level paragraph, and the paragraph title is used as the keyword for the corresponding segment.

7. The method according to claim 1, characterized in that, The step of matching and retrieving content in the vector database based on the search keywords and search terms, and returning content that meets the requirements, includes: Based on the search keywords, perform keyword matching and retrieval in the vector database, and return the content of the successfully matched segments; Based on the search terms, a semantic vector matching search is performed in the vector database, and the content of the segment with a matching score greater than a preset score is returned.

8. A retrieval enhancement generation device based on a hybrid strategy, characterized in that, The device includes: The first module is used to obtain the target document and determine whether to segment the target document and the segmentation strategy based on the size and paragraph hierarchy information of the target document. The second module is used to extract the segment keywords of the target document or the segment and generate a vector, and insert the segment keywords, the vector and the target document into the vector database; The third module is used to obtain the search content, preprocess the search content, and extract search keywords and terms; The fourth module is used to perform matching and retrieval in the vector database based on the search keywords and search terms, and return the segmented content that meets the requirements.

9. An electronic device, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method as described in any one of claims 1-7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images