A document retrieval method, device and medium based on a vector database

Abstract

The application relates to the technical field of database indexing, and discloses a document retrieval method and device based on a vector database and a medium, which comprises the following steps: parsing a to-be-processed document into a sequence of semantic units arranged in sequence; dynamically generating a judgment threshold according to the local statistical characteristics of a plurality of similarity change rates before a current processing position, comparing the similarity change rate with the judgment threshold, and marking the current processing position as a semantic boundary when the similarity change rate is greater than the judgment threshold; determining a candidate document cut block set of different granularity levels by setting different sliding step lengths and window sizes; screening a target cut block combination from the candidate document cut block set according to a scoring result and the semantic coincidence degree between the candidate cut blocks; determining a dense feature vector based on the target cut block combination, and storing the dense feature vector into a vector database; and establishing an index structure for the dense feature vector based on an index plug-in of the vector database, so that the similarity of the document can be retrieved.

Description

Technical Field

[0001] This application relates to the field of database indexing technology, and in particular to a document retrieval method, device and medium based on vector database. Background Technology

[0002] Document retrieval technology is a core branch of information retrieval. The goal of document retrieval is to quickly locate text information relevant to a user's query from a large-scale document collection. With the development of deep learning and vector representation technology, document retrieval architecture based on vector databases has gradually become the mainstream form. This architecture typically includes processing steps such as document preprocessing, text segmentation, vector embedding, index building, and similarity calculation.

[0003] Currently, most document retrieval methods are based on document structure tags, utilizing paragraph tags, chapter titles, or punctuation marks inherent in the document as segmentation criteria. This method is effective in processing standardized documents and has been widely adopted in existing retrieval systems. However, these segmentation methods have shortcomings in practical applications. For plain text lacking clear structural tags or documents with heterogeneous formats, the segmentation effect is not ideal, resulting in poor final document retrieval results. Summary of the Invention

[0004] This specification provides one or more embodiments of a document retrieval method, device, and medium based on a vector database to solve the technical problems raised in the background art.

[0005] One or more embodiments of this specification employ the following technical solutions: This specification provides one or more embodiments of a document retrieval method based on a vector database, the method comprising: Obtain the document to be processed, and parse the document to be processed into a sequence of semantic units arranged in order; The semantic similarity between adjacent semantic units in the semantic unit sequence is determined, and the similarity change rate is calculated based on the semantic similarity. According to the local statistical characteristics of the similarity change rates of multiple preceding similarity units at the current processing position, a judgment threshold is dynamically generated. The similarity change rate is compared with the judgment threshold. When the similarity change rate is greater than the judgment threshold, the current processing position is marked as a semantic boundary. Based on the semantic boundary, different sliding step sizes and window sizes are set to determine candidate document chunk sets at different granularity levels; Each candidate segment in the candidate document segment set is scored based on a preset weighting coefficient to obtain a scoring result; Based on the scoring results and the semantic overlap between candidate segments, a target segment combination is selected from the candidate document segment set; Based on the target block combination, a dense feature vector is determined, and the dense feature vector is stored in a vector database; Based on the indexing plugin of the vector database, an index structure is established for the dense feature vectors to achieve document similarity retrieval.

[0006] It should be noted that this application first dynamically generates a judgment threshold by combining the similarity change rate of adjacent semantic units with the local statistical features of multiple preceding change rates, thereby identifying semantic boundaries in the document. This ensures that the segmentation process does not rely on the document's own formatting tags. Simultaneously, the target segment combinations obtained through semantic overlap and scoring are evaluated for semantic consistency and redundancy before being stored in the vector database. This ensures that the segments ultimately stored in the vector database are semantically coherent and have low information redundancy. When similarity retrieval is performed after the index structure is established, the matching process between the query vector and each segment vector in the database is based on semantically complete and density-optimized segments, which can improve the relevance between the retrieval results and the query intent.

[0007] Furthermore, when the similarity change rate is not greater than the determination threshold, the method further includes: The currently processed semantic unit is merged with the semantic units in the current document block buffer, and the sliding window is moved forward by one semantic unit to continue the next round of similarity calculation and comparison.

[0008] It's important to note that when the similarity change rate of the current semantic unit does not exceed the judgment threshold, the current semantic unit is merged into the current document chunk buffer, and the sliding window is controlled to advance one unit forward to continue calculation and comparison. This allows multiple adjacent semantic units in semantically continuous regions to be accumulated sequentially into the same chunk buffer until a true semantic boundary is encountered before being encapsulated and output. Through this progressive merging mechanism, the determination of the cut point no longer depends on a fixed-length window or external rules, but is entirely driven by the degree of semantic change within the document. When the similarity change rate remains low, the buffer can naturally aggregate and maintain complete thematic paragraphs or logical arguments; boundary cutting is only triggered when there is a significant semantic shift.

[0009] Furthermore, the preset weighting coefficients include at least two of the following: the semantic consistency score between the candidate segment and its context, the semantic redundancy between the candidate segment and other candidate segments in the same batch, and the adaptation score between the candidate segment and the user's query intent. The semantic redundancy is configured with a negative weighting coefficient to suppress the contribution of redundancy and repetition to the scoring results.

[0010] It should be noted that this application introduces at least two preset weighting coefficients in the scoring process. When semantic redundancy is configured with a negative weighting coefficient, candidate segments with high content overlap will naturally have their overall score reduced during the scoring process, thereby reducing the possibility of duplicate information resulting from multi-path parallel derivation entering the final output set. Simultaneously, the introduction of a semantic consistency score ensures that the selected segments maintain logical coherence with the original text context, avoiding discontinuities in argumentation caused by segmentation. The query intent adaptation score allows the scoring process to adjust the priority of segments based on different information needs and preferences.

[0011] Furthermore, when scoring candidate segments, the semantic consistency score is configured with a positive weight coefficient, the semantic redundancy score is configured with a negative weight coefficient, and the user query intent adaptation score is configured with a dynamic weight coefficient. The dynamic weight coefficient is adjusted according to the identification result of the query intent type. When the query intent is a fact-checking type, the adaptation score weight of fine-grained segments is increased, and when the query intent is a summary or overview type, the adaptation score weight of coarse-grained segments is increased.

[0012] It should be noted that this application assigns a positive weighting coefficient to the semantic consistency score during the scoring process to enhance the coherence between the segment and the original text, assigns a negative weighting coefficient to the semantic redundancy score to suppress the interference of repeated content, and assigns a dynamic weighting coefficient to the adaptation score for the user's query intent. When the query intent is fact-checking, the adaptation score weight of fine-grained segments is increased, making the scoring process more inclined to output short segments with precise positioning and dense information; when the query intent is summary or overview, the adaptation score weight of coarse-grained segments is increased, making the scoring process more inclined to output long segments with complete context.

[0013] Furthermore, the step of selecting target segment combinations from the candidate document segment set based on the scoring results and the semantic overlap between candidate segments includes: Using a non-maximum suppression algorithm or a greedy selection strategy, candidate segments are sorted based on the score results. Adjacent segments are examined sequentially. When the semantic overlap between two segments is higher than a preset cross threshold, the candidate segment with the lower score is removed, and the candidate segment with the higher score is retained as the target segment combination.

[0014] It should be noted that after obtaining the scoring results of each candidate segment, this application employs a non-maximum suppression algorithm or a greedy selection strategy to sort the candidate segments from highest to lowest score, and then sequentially examines the semantic overlap between adjacent segments. When the overlap between two segments exceeds a preset crossover threshold, the system retains only the segment with the higher score and discards the one with the lower score. Through this mechanism, a large number of semantically overlapping candidate segments originally generated by multi-path parallel derivation are gradually filtered out, and in the final output target segment combination, different segments maintain a low degree of redundancy, while the optimal segment within each local region is retained.

[0015] Furthermore, determining the dense feature vector based on the target block combination includes: The target segments are combined and input into a preset embedding model, which transforms them into dense feature vectors of fixed dimensions.

[0016] It should be noted that this application inputs the selected target segments into a pre-defined embedding model, transforming them into dense feature vectors of fixed dimensions, and then uses an indexing plugin of a vector database to build an index structure for these vectors. Since the segments entering the vectorization and indexing stages have already undergone semantic boundary detection, multi-granularity scoring, and overlap filtering, each segment possesses semantic self-consistency and low redundancy, and its granularity is adapted to downstream retrieval requirements. Therefore, after these segments are transformed into vectors and indexed, subsequent similarity retrieval can be performed in a high-information-density vector space. The retrieval process does not need to handle a large number of semantically overlapping or fragmented redundant entries, thus making the recall results more focused on document fragments that are highly relevant to the query semantics and have complete context.

[0017] Furthermore, the method also includes: Collect feedback signals generated during the retrieval for the target block combination; Based on the feedback signal, the weighting coefficients used to determine the judgment threshold and / or to score candidate segments are periodically adjusted.

[0018] It should be noted that this application collects feedback signals generated during the actual retrieval process of target chunk combinations, and periodically adjusts the generation parameters of the judgment threshold and / or the weight coefficients used for candidate chunk scoring accordingly. Since the feedback signals truly reflect the actual performance of the cutting results in downstream retrieval tasks, such as whether the search content is coherent or contains noise, the system can identify deviations in the current cutting strategy and correct the threshold or weights accordingly by transmitting the feedback signals back to the parameter regulator in the cutting stage. After multiple cycles of adjustment, the sensitivity of the judgment threshold and the ratio of scoring weights gradually match the document content characteristics and user retrieval needs, resulting in continuous improvement in the cutting quality of subsequent documents.

[0019] Furthermore, the step of periodically adjusting the weighting coefficients used in determining the judgment threshold and / or scoring candidate segments based on the feedback signal includes: When the feedback signal indicates that there is semantic incoherence due to information truncation, the judgment threshold is lowered to make the judgment of semantic boundaries more stringent. When the feedback signal indicates a decrease in retrieval accuracy due to redundant noise, the weighting coefficients used when scoring candidate segments are adjusted to increase the penalty weight for semantic redundancy.

[0020] It should be noted that this application performs differentiated parameter adjustments based on the defect type indicated by the feedback signal. When the feedback signal indicates semantic incoherence due to information truncation, the judgment threshold is lowered, making the judgment of semantic boundaries more stringent. This encourages adjacent semantic units to be merged, resulting in a larger output chunk size and more complete contextual information. When the feedback signal indicates a decrease in retrieval accuracy due to redundant noise, the scoring weight is adjusted to increase the penalty coefficient for semantic redundancy. This effectively suppresses highly repetitive candidate chunks during the scoring process, significantly reducing redundant content in the final output chunk combination. Through these targeted parameter adjustments, the system can solve the information breakage problem caused by overly fine chunks and the noise interference problem caused by overly coarse chunks without changing the overall cutting framework, thereby continuously improving the cutting quality over long-term operation.

[0021] This specification provides one or more embodiments of a document retrieval device based on a vector database, comprising: At least one processor and bus; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to: Obtain the document to be processed, and parse the document to be processed into a sequence of semantic units arranged in order; The semantic similarity between adjacent semantic units in the semantic unit sequence is determined, and the similarity change rate is calculated based on the semantic similarity. According to the local statistical characteristics of the similarity change rates of multiple preceding similarity units at the current processing position, a judgment threshold is dynamically generated. The similarity change rate is compared with the judgment threshold. When the similarity change rate is greater than the judgment threshold, the current processing position is marked as a semantic boundary. Based on the semantic boundary, different sliding step sizes and window sizes are set to determine candidate document chunk sets at different granularity levels; Each candidate segment in the candidate document segment set is scored based on a preset weighting coefficient to obtain a scoring result; Based on the scoring results and the semantic overlap between candidate segments, a target segment combination is selected from the candidate document segment set; Based on the target block combination, a dense feature vector is determined, and the dense feature vector is stored in a vector database; Based on the indexing plugin of the vector database, an index structure is established for the dense feature vectors to achieve document similarity retrieval.

[0022] This specification provides one or more embodiments of a non-volatile computer storage medium storing computer-executable instructions, which, when executed by a computer, can perform the following: Obtain the document to be processed, and parse the document to be processed into a sequence of semantic units arranged in order; The semantic similarity between adjacent semantic units in the semantic unit sequence is determined, and the similarity change rate is calculated based on the semantic similarity. According to the local statistical characteristics of the similarity change rates of multiple preceding similarity units at the current processing position, a judgment threshold is dynamically generated. The similarity change rate is compared with the judgment threshold. When the similarity change rate is greater than the judgment threshold, the current processing position is marked as a semantic boundary. Based on the semantic boundary, different sliding step sizes and window sizes are set to determine candidate document chunk sets at different granularity levels; Each candidate segment in the candidate document segment set is scored based on a preset weighting coefficient to obtain a scoring result; Based on the scoring results and the semantic overlap between candidate segments, a target segment combination is selected from the candidate document segment set; Based on the target block combination, a dense feature vector is determined, and the dense feature vector is stored in a vector database; Based on the indexing plugin of the vector database, an index structure is established for the dense feature vectors to achieve document similarity retrieval.

[0023] The above-described at least one technical solution adopted in the embodiments of this specification can achieve the following beneficial effects: This application first dynamically generates a judgment threshold by combining the similarity change rate of adjacent semantic units with the local statistical features of multiple preceding change rates, thereby identifying semantic boundaries in the document and making the segmentation process independent of the document's own format marks. Simultaneously, the target segment combinations obtained through semantic overlap and scoring are evaluated for semantic consistency and redundancy before entering vectorized storage, ensuring that the segments ultimately stored in the vector database are semantically coherent and have low information redundancy. When similarity retrieval is performed after the index structure is established, the matching process between the query vector and each segment vector in the database is based on semantically complete and density-optimized segments, which can improve the relevance between the search results and the query intent. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings: Figure 1 A flowchart illustrating a document retrieval method based on a vector database provided for one or more embodiments of this specification; Figure 2 This is a schematic diagram of the structure of a document retrieval device based on a vector database, provided for one or more embodiments of this specification. Detailed Implementation

[0025] This specification provides a document retrieval method, device, and medium based on a vector database.

[0026] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0027] Figure 1 This diagram illustrates a document retrieval method based on a vector database, provided for one or more embodiments of this specification. This process can be executed by a document retrieval system. Certain input parameters or intermediate results in the process can be manually adjusted to help improve accuracy.

[0028] The method flow steps of the embodiments in this specification are as follows: S101, Obtain the document to be processed, and parse the document to be processed into a sequence of semantic units arranged in order.

[0029] In the embodiments described in this specification, the system can receive multimodal raw documents, including text, PDF, and image formats, through standard application programming interfaces (APIs), web crawlers, or local file uploads. The input document undergoes data cleaning and deep parsing to extract its directory tree, table structure, paragraph styles, and key metadata, while also performing deduplication and noise reduction. The document content can be broken down into the smallest text fragments with independent and complete meaning, i.e., semantic units. Each semantic unit can be a complete sentence, a clause extracted from a complex sentence, a list item, or a table cell. The final output is a sequence of semantic units arranged in the original document order, serving as the smallest operational object for all subsequent semantic calculations.

[0030] S102, determine the semantic similarity of adjacent semantic units in the semantic unit sequence, calculate the similarity change rate based on the semantic similarity, dynamically generate a judgment threshold according to the local statistical characteristics of multiple similarity change rates preceding the current processing position, compare the similarity change rate with the judgment threshold, and mark the current processing position as a semantic boundary when the similarity change rate is greater than the judgment threshold.

[0031] In the embodiments of this specification, the system can input each semantic unit into a preset vector extraction model to obtain its feature vector in a multidimensional semantic space, forming a feature vector sequence that corresponds one-to-one with the semantic unit sequence. Along the sequence progression direction, the cosine similarity between any two adjacent feature vectors is calculated sequentially to obtain a similarity sequence reflecting the continuous changes in text semantics.

[0032] Based on this similarity sequence, the system calculates the rate of change between adjacent similarities. This rate of change can amplify local semantic abrupt change signals, and can show significant changes at logical transitions in the text, while the value is lower and more stable at smooth sections of the argument.

[0033] The system employs a local sliding window mechanism to dynamically analyze the local features of the change rates of multiple historical similarities preceding the current position to be judged, including the central tendency and dispersion of these change rates. Based on these local statistical features, a dynamic judgment threshold is adaptively generated, which can be automatically adjusted according to the differences in writing style in different areas of the document.

[0034] The calculated similarity change rate is compared in real time with the dynamic judgment threshold. If the similarity change rate is greater than the judgment threshold, the current position is determined to be a significant semantic turning point. A hard logical segmentation mark is generated at this point, and the currently accumulated semantic units are encapsulated into an independent document block. At the same time, the buffer is cleared to prepare for the next block. If the similarity change rate is not greater than the judgment threshold, the current context semantics can be determined to be continuous. The currently processed semantic units are merged into the current document block buffer, and the sliding window is controlled to advance one semantic unit to the next, and the next round of similarity calculation and comparison is continued.

[0035] S103, based on the semantic boundary, determine the candidate document segment set at different granularity levels by setting different sliding step sizes and window sizes.

[0036] In the embodiments of this specification, the system can use the hard logic cutting marker generated in step S102 as a physical baseline to initiate the parallel evolution of multiple independent pipelines. By setting different sliding step sizes and window sizes, multiple levels of candidate segments are generated synchronously: coarse granularity corresponds to chapters or large paragraphs, medium granularity corresponds to standard paragraphs, and fine granularity corresponds to key sentence groups or phrases. To address the information overlap between different granularities, different overlap weights are set within each path and between paths to ensure that each candidate segment retains necessary contextual connection information on both sides of the physical boundary.

[0037] S104, score each candidate segment in the candidate document segment set based on a preset weight coefficient to obtain a score result.

[0038] In this embodiment of the specification, the system can perform a full traversal of all candidate segments generated in step S103 and calculate the comprehensive value score of each candidate segment using a preset multi-factor scoring function. The score may include at least two of the following: the semantic consistency score between the segment and its preceding and following context, the semantic redundancy between the segment and other candidate segments in the same batch, and the adaptation score between the segment and the user's query intent. The semantic redundancy score is configured with a negative weighting coefficient to suppress the contribution of redundancy to the scoring result; the semantic consistency score and the query intent adaptation score are configured with positive weighting coefficients. The weighting coefficient of the query intent adaptation score can be dynamically adjusted according to the query intent type identification result.

[0039] S105, Based on the scoring results and the semantic overlap between candidate segments, select the target segment combination from the candidate document segment set.

[0040] In the embodiments of this specification, the system may employ a non-maximum suppression algorithm or a greedy selection strategy to sort candidate segments from high to low based on the scoring results. Adjacent segments are examined sequentially; when the semantic overlap between two segments exceeds a preset cross-cutting threshold, the candidate segment with the lower score is removed, while the candidate segment with the higher score is retained. Through global overlap filtering, the system removes candidate segments with highly overlapping semantics and redundant content, retaining only the optimal segment combination with independent information increments as the target segment combination output, thus completing the segmentation of the document to be processed.

[0041] S106, determine a dense feature vector based on the target block combination, and store the dense feature vector in a vector database.

[0042] In the embodiments described in this specification, the system can take the filtered target segments as input and output a dense feature vector. Since the dense feature vector retains the semantic information within the segments, it can be used for subsequent similarity calculation and retrieval matching. The dense feature vector is then transmitted to a vector database for persistent storage.

[0043] S107, Based on the indexing plugin of the vector database, an index structure is established for the dense feature vectors to achieve document similarity retrieval.

[0044] In the embodiments described in this specification, the vector database can be configured with a dedicated indexing plugin. The system calls the plugin's function to establish an efficient index structure for the stored dense feature vectors. This index structure can take the form of an inverted index or a hierarchical navigable small-world graph, used to accelerate approximate nearest neighbor search in large-scale vector sets. After the index is established, the system can receive external query requests, convert the query content into vectors, and use the established index structure to perform similarity retrieval in the database, quickly recalling the top few target chunk combinations most relevant to the query semantics, thereby achieving efficient retrieval of document content.

[0045] It should be noted that this application first dynamically generates a judgment threshold by combining the similarity change rate of adjacent semantic units with the local statistical features of multiple preceding change rates, thereby identifying semantic boundaries in the document. This ensures that the segmentation process does not rely on the document's own formatting tags. Simultaneously, the target segment combinations obtained through semantic overlap and scoring are evaluated for semantic consistency and redundancy before being stored in the vector database. This ensures that the segments ultimately stored in the vector database are semantically coherent and have low information redundancy. When similarity retrieval is performed after the index structure is established, the matching process between the query vector and each segment vector in the database is based on semantically complete and density-optimized segments, which can improve the relevance between the retrieval results and the query intent.

[0046] Furthermore, when the similarity change rate is not greater than the determination threshold, the method flow steps of this embodiment are as follows: S201, merge the currently processed semantic unit with the semantic units in the current document block buffer, and control the sliding window to advance one semantic unit unit to continue the next round of similarity calculation and comparison.

[0047] In the embodiments of this specification, when the similarity change rate is not greater than the judgment threshold, the system can determine that the current context semantics are continuous and the logical breakpoint condition has not been met. The system can append and merge the currently processed semantic unit into the currently established document block buffer, which already stores multiple semantic units that have been previously determined to be continuous. At the same time, the system controls the sliding window to move backward by one semantic unit, that is, to advance the processing focus to the next semantic unit, and continues to perform the next round of similarity calculation and comparison operations for adjacent semantic units based on the updated window position. Through this cyclical process of gradual merging and window advancement, the system continuously accumulates semantic units within the same semantic topic until it encounters a position where the similarity change rate is greater than the judgment threshold, thus triggering boundary cutting, thereby ensuring that the semantics within each output document block remain coherent and are not prematurely truncated.

[0048] It's important to note that when the similarity change rate of the current semantic unit does not exceed the judgment threshold, the current semantic unit is merged into the current document chunk buffer, and the sliding window is controlled to advance one unit forward to continue calculation and comparison. This allows multiple adjacent semantic units in semantically continuous regions to be accumulated sequentially into the same chunk buffer until a true semantic boundary is encountered before being encapsulated and output. Through this progressive merging mechanism, the determination of the cut point no longer depends on a fixed-length window or external rules, but is entirely driven by the degree of semantic change within the document. When the similarity change rate remains low, the buffer can naturally aggregate and maintain complete thematic paragraphs or logical arguments; boundary cutting is only triggered when there is a significant semantic shift.

[0049] Furthermore, the preset weighting coefficients include at least two of the following: the semantic consistency score between the candidate segment and its context, the semantic redundancy between the candidate segment and other candidate segments in the same batch, and the adaptation score between the candidate segment and the user's query intent. The semantic redundancy is configured with a negative weighting coefficient to suppress the contribution of redundancy and repetition to the scoring results.

[0050] In the embodiments described in this specification, a multi-factor weighted approach can be used to filter out candidate segments that have high information density and match user needs. For each candidate segment in the candidate document segment set, the system calculates three evaluation scores.

[0051] The first item is the semantic consistency score. In practice, the system can extract the adjacent text segments before and after the candidate segment in the original text, calculate the cosine similarity between the feature vector of the segment and the vectors of the preceding and following segments, and take the mean of the two similarities as the contextual semantic consistency score of the segment, which is used to measure the logical coherence of the segment when it is integrated into the original text.

[0052] The second item is semantic redundancy. In practice, the system compares the feature vectors of the candidate segment with those of other candidate segments generated in the same batch, calculates the semantic overlap between each pair, and uses the number of other segments with an overlap higher than a set threshold, or the cumulative overlap value, as the redundancy index of that segment. This score is configured with a negative weighting coefficient. When a segment highly overlaps with the content of other segments, this score will lower its overall score, thereby suppressing duplicate content from entering the final result.

[0053] The third item is the query intent matching score. In practice, the system can pre-configure a query intent classifier. When a user enters a query, the classifier outputs the query intent type, including fact-checking or summary / generalization. For fact-checking intents, the system assigns higher matching scores to finer-grained candidate segments; for summary / generalization intents, the system assigns higher matching scores to coarser-grained candidate segments. The weighting coefficient of this matching score is dynamically adjusted according to the intent type.

[0054] The system can sum the scores of at least two of the above items according to their respective configured weight coefficients to obtain the final score for each candidate segment. Semantic redundancy, due to its negative weight, contributes to reducing the overall score during the summation. The score results are used in subsequent segment selection processes.

[0055] It should be noted that this application introduces at least two preset weighting coefficients in the scoring process. When semantic redundancy is configured with a negative weighting coefficient, candidate segments with high content overlap will naturally have their overall score reduced during the scoring process, thereby reducing the possibility of duplicate information resulting from multi-path parallel derivation entering the final output set. Simultaneously, the introduction of a semantic consistency score ensures that the selected segments maintain logical coherence with the original text context, avoiding discontinuities in argumentation caused by segmentation. The query intent adaptation score allows the scoring process to adjust the priority of segments based on different information needs and preferences.

[0056] Furthermore, when scoring candidate segments, the semantic consistency score is configured with a positive weight coefficient, the semantic redundancy score is configured with a negative weight coefficient, and the user query intent adaptation score is configured with a dynamic weight coefficient. The dynamic weight coefficient is adjusted according to the identification result of the query intent type. When the query intent is a fact-checking type, the adaptation score weight of fine-grained segments is increased, and when the query intent is a summary or overview type, the adaptation score weight of coarse-grained segments is increased.

[0057] It should be noted that this application assigns a positive weighting coefficient to the semantic consistency score during the scoring process to enhance the coherence between the segment and the original text, assigns a negative weighting coefficient to the semantic redundancy score to suppress the interference of repeated content, and assigns a dynamic weighting coefficient to the adaptation score for the user's query intent. When the query intent is fact-checking, the adaptation score weight of fine-grained segments is increased, making the scoring process more inclined to output short segments with precise positioning and dense information; when the query intent is summary or overview, the adaptation score weight of coarse-grained segments is increased, making the scoring process more inclined to output long segments with complete context.

[0058] Furthermore, in the process of selecting the target segment combination from the candidate document segment set based on the scoring results and the semantic overlap between candidate segments, the method flow steps of this embodiment are as follows: S301, using a non-maximum suppression algorithm or a greedy selection strategy, the candidate segments are sorted based on the score results, and adjacent segments are examined in turn. When the semantic overlap between two segments is higher than a preset cross threshold, the candidate segment with the lower score is removed, and the candidate segment with the higher score is retained as the target segment combination.

[0059] In the embodiments of this specification, after obtaining the scoring results of all candidate segments, the system can use a non-maximum suppression algorithm or a greedy selection strategy to extract the globally optimal combination. The system can first sort the candidate segments in descending order of score. Then, it iterates through the sorted candidate segments sequentially. For the currently selected segment, the system compares it with its adjacent segments that have not yet been eliminated, calculating the semantic overlap between the two segments. The system pre-sets a cross-cutting threshold; when the semantic overlap between two segments is higher than this threshold, it indicates that there is a high degree of overlap or information redundancy in their content. In this case, the system eliminates the candidate segments with lower scores, retaining only the segments with higher scores. In this way, the system processes all candidate segments sequentially, and the semantic overlap between the remaining segments is all below the preset threshold. Furthermore, the system retains the highest-scoring representative segment within each local region. These retained segment combinations constitute the final target segment combination.

[0060] It should be noted that after obtaining the scoring results of each candidate segment, this application employs a non-maximum suppression algorithm or a greedy selection strategy to sort the candidate segments from highest to lowest score, and then sequentially examines the semantic overlap between adjacent segments. When the overlap between two segments exceeds a preset crossover threshold, the system retains only the segment with the higher score and discards the one with the lower score. Through this mechanism, a large number of semantically overlapping candidate segments originally generated by multi-path parallel derivation are gradually filtered out, and in the final output target segment combination, different segments maintain a low degree of redundancy, while the optimal segment within each local region is retained.

[0061] Furthermore, in the process of determining the dense feature vector based on the target block combination, the method flow steps of this embodiment are as follows: S401, the target segment combination is input into a preset embedding model and transformed into a dense feature vector of fixed dimensions.

[0062] In the embodiments of this specification, the system can use the filtered target segments as input and transmit them to a preset embedding model. This embedding model is pre-trained and can map text data to a high-dimensional semantic space. For each input target segment, the embedding model converts it into a dense feature vector of fixed dimensions, where the value in each dimension represents the feature strength of the segment in the corresponding semantic dimension. The converted dense feature vector retains the semantic information within the segment and is used for subsequent similarity calculation and retrieval matching.

[0063] It should be noted that this application inputs the selected target segments into a pre-defined embedding model, transforming them into dense feature vectors of fixed dimensions, and then uses an indexing plugin of a vector database to build an index structure for these vectors. Since the segments entering the vectorization and indexing stages have already undergone semantic boundary detection, multi-granularity scoring, and overlap filtering, each segment possesses semantic self-consistency and low redundancy, and its granularity is adapted to downstream retrieval requirements. Therefore, after these segments are transformed into vectors and indexed, subsequent similarity retrieval can be performed in a high-information-density vector space. The retrieval process does not need to handle a large number of semantically overlapping or fragmented redundant entries, thus making the recall results more focused on document fragments that are highly relevant to the query semantics and have complete context.

[0064] Furthermore, the judgment threshold and weight coefficients are adjusted. The method flow steps of this embodiment are as follows: S501, Collect feedback signals generated during the retrieval of the target block combination.

[0065] In the embodiments described in this specification, during the actual operation of the retrieval enhancement generation system, the system continuously monitors and collects feedback signals related to the combination of target segments. These feedback signals include: first, end-user interaction behavior, including user actions such as liking or disliking the generated answer, whether the regeneration function is triggered, the duration of time spent on the answer page, and subsequent follow-up or corrective questions; and second, quality assessment indicators automatically generated by the large language model, including answer fidelity scores and answer relevance scores. Answer fidelity is used to detect whether the generated content can find clear evidence within the retrieved segments, while answer relevance is used to assess whether the output answer matches the user's original query intent.

[0066] S502, based on the feedback signal, periodically adjust the weight coefficients used to determine the judgment threshold and / or score the candidate blocks.

[0067] In the embodiments of this specification, the system can identify the defect type of the collected feedback signals. When the feedback signal indicates that the retrieved content lacks coherence and the contextual information is fragmented, the system determines that the current segmentation strategy has a semantic breakage defect. At this time, it sends an adjustment instruction to the semantic boundary detection stage to lower the generation parameter of the judgment threshold, making it easier for the similarity change rate to exceed the threshold, thereby making the judgment of semantic boundaries more stringent, and adjacent semantic units are more likely to be merged into larger document blocks. When the feedback signal indicates that the generated content contains too much irrelevant information and the retrieval recalls a large number of irrelevant fragments, the system determines that the current segmentation strategy has a redundancy noise overload defect. At this time, it sends an adjustment instruction to the block selection stage to rebalance the weight coefficients in the multi-factor scoring, specifically by increasing the penalty weight of semantic redundancy or dynamically changing the weight preference of the query intent adaptation score. The above adjustments can be executed according to a preset periodic rhythm, so that the system's segmentation parameters can be gradually optimized with the feedback of the retrieval effect, adapting to changes in different document content and user needs.

[0068] It should be noted that this application collects feedback signals generated during the actual retrieval process of target chunk combinations, and periodically adjusts the generation parameters of the judgment threshold and / or the weight coefficients used for candidate chunk scoring accordingly. Since the feedback signals truly reflect the actual performance of the cutting results in downstream retrieval tasks, such as whether the search content is coherent or contains noise, the system can identify deviations in the current cutting strategy and correct the threshold or weights accordingly by transmitting the feedback signals back to the parameter regulator in the cutting stage. After multiple cycles of adjustment, the sensitivity of the judgment threshold and the ratio of scoring weights gradually match the document content characteristics and user retrieval needs, resulting in continuous improvement in the cutting quality of subsequent documents.

[0069] Furthermore, in the process of periodically adjusting the weight coefficients used to determine the judgment threshold and / or score the candidate segments based on the feedback signal, the method flow steps of this embodiment are as follows: S601, when the feedback signal indicates that there is semantic incoherence due to information truncation, the determination threshold is lowered to make the determination of semantic boundaries more stringent.

[0070] In the embodiments of this specification, when the feedback signal collected by the system indicates that the retrieval results lack coherence and the contextual information is fragmented, it is determined that the current cutting process has an information truncation defect, that is, the block volume is too small, resulting in semantic incompleteness. The system can send an adjustment instruction to the semantic boundary detection stage to reduce the generation parameter of the dynamic judgment threshold.

[0071] S602, when the feedback signal indicates that there is a decrease in retrieval accuracy due to redundant noise, the weight coefficient used when scoring candidate segments is adjusted to increase the penalty weight for semantic redundancy.

[0072] In the embodiments of this specification, when the feedback signal collected by the system indicates that the retrieval has retrieved a large number of fragments unrelated to the query intent and the accuracy of the generated content has decreased, it is determined that the current segmentation process has a redundancy noise overload defect, that is, the segment volume is too large, resulting in too much irrelevant information in a single segment. The system sends an adjustment instruction to the segment selection stage to reconfigure the multi-factor scoring weight coefficients used when scoring candidate segments.

[0073] The penalty weight for semantic redundancy refers to the negative weight coefficient assigned by the system to the semantic redundancy evaluation factor when comprehensively scoring candidate segments. The physical meaning of this negative weight coefficient is that when a candidate segment has a high degree of semantic overlap with other candidate segments in the same batch, the redundancy evaluation value of that candidate segment will increase accordingly. Multiplying this increased redundancy evaluation value by the negative weight coefficient yields a negative score contribution. This negative score contribution lowers the comprehensive score of the candidate segment, thereby reducing the probability that the redundant segment will be retained in subsequent screening processes.

[0074] After the candidate document segment set is generated, the system performs a redundancy evaluation on each candidate segment in the set. For the candidate segment currently being evaluated, the system extracts its feature vector and compares it with the feature vectors of every other candidate segment in the candidate document segment set, calculating the semantic overlap between them one by one. After traversal, the system selects the largest semantic overlap value as the redundancy evaluation value for that candidate segment. The larger the redundancy evaluation value, the higher the degree of content overlap between that segment and another segment in the same batch.

[0075] When calculating the overall score of the candidate slice, the system uses the redundancy assessment value as a component of the scoring function. This scoring function pre-configures a weighting coefficient corresponding to the redundancy assessment value, and this weighting coefficient is negative. Multiplying the redundancy assessment value by the negative weighting coefficient produces a negative contribution to the score. The larger the redundancy assessment value, the larger the absolute value of this negative contribution, and the stronger its downward effect on the overall score; conversely, the smaller the redundancy assessment value, the smaller the absolute value of this negative contribution, and the weaker its negative impact on the overall score.

[0076] When the feedback signals collected by the system indicate that the retrieval has retrieved a large amount of duplicate or highly similar content, the system determines that there is a redundancy noise overload defect. During periodic adjustments, the system increases the absolute value of this negative weight coefficient. After multiplying the increased negative weight coefficient by the redundancy assessment value, the resulting negative score contribution is larger than before the adjustment, further suppressing the overall score of highly overlapping candidate segments. After scoring and sorting, the suppressed redundant segments are placed at the end of the descending order and are subsequently prioritized for elimination during the non-maximum suppression screening process. Through this method, the system can effectively suppress highly overlapping segments from entering the final target segment combination when processing new documents.

[0077] It should be noted that this application performs differentiated parameter adjustments based on the defect type indicated by the feedback signal. When the feedback signal indicates semantic incoherence due to information truncation, the judgment threshold is lowered, making the judgment of semantic boundaries more stringent. This encourages adjacent semantic units to be merged, resulting in a larger output chunk size and more complete contextual information. When the feedback signal indicates a decrease in retrieval accuracy due to redundant noise, the scoring weight is adjusted to increase the penalty coefficient for semantic redundancy. This effectively suppresses highly repetitive candidate chunks during the scoring process, significantly reducing redundant content in the final output chunk combination. Through these targeted parameter adjustments, the system can solve the information breakage problem caused by overly fine chunks and the noise interference problem caused by overly coarse chunks without changing the overall cutting framework, thereby continuously improving the cutting quality over long-term operation.

[0078] Figure 2 A schematic diagram of the structure of a document retrieval device based on a vector database, provided for one or more embodiments of this specification, includes: At least one processor and bus; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to: Obtain the document to be processed, and parse the document to be processed into a sequence of semantic units arranged in order; The semantic similarity between adjacent semantic units in the semantic unit sequence is determined, and the similarity change rate is calculated based on the semantic similarity. According to the local statistical characteristics of the similarity change rates of multiple preceding similarity units at the current processing position, a judgment threshold is dynamically generated. The similarity change rate is compared with the judgment threshold. When the similarity change rate is greater than the judgment threshold, the current processing position is marked as a semantic boundary. Based on the semantic boundary, different sliding step sizes and window sizes are set to determine candidate document chunk sets at different granularity levels; Each candidate segment in the candidate document segment set is scored based on a preset weighting coefficient to obtain a scoring result; Based on the scoring results and the semantic overlap between candidate segments, a target segment combination is selected from the candidate document segment set; Based on the target block combination, a dense feature vector is determined, and the dense feature vector is stored in a vector database; Based on the indexing plugin of the vector database, an index structure is established for the dense feature vectors to achieve document similarity retrieval.

[0079] This specification provides one or more embodiments of a non-volatile computer storage medium storing computer-executable instructions, which, when executed by a computer, can perform the following: Obtain the document to be processed, and parse the document to be processed into a sequence of semantic units arranged in order; The semantic similarity between adjacent semantic units in the semantic unit sequence is determined, and the similarity change rate is calculated based on the semantic similarity. According to the local statistical characteristics of the similarity change rates of multiple preceding similarity units at the current processing position, a judgment threshold is dynamically generated. The similarity change rate is compared with the judgment threshold. When the similarity change rate is greater than the judgment threshold, the current processing position is marked as a semantic boundary. Based on the semantic boundary, different sliding step sizes and window sizes are set to determine candidate document chunk sets at different granularity levels; Each candidate segment in the candidate document segment set is scored based on a preset weighting coefficient to obtain a scoring result; Based on the scoring results and the semantic overlap between candidate segments, a target segment combination is selected from the candidate document segment set; Based on the target block combination, a dense feature vector is determined, and the dense feature vector is stored in a vector database; Based on the indexing plugin of the vector database, an index structure is established for the dense feature vectors to achieve document similarity retrieval.

[0080] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, devices, and non-volatile computer storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0081] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0082] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0083] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0084] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0085] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The aforementioned units can be implemented in hardware or software.

[0086] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0087] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A document retrieval method based on a vector database, characterized in that, The method includes: Obtain the document to be processed, and parse the document to be processed into a sequence of semantic units arranged in order; The semantic similarity between adjacent semantic units in the semantic unit sequence is determined, and the similarity change rate is calculated based on the semantic similarity. According to the local statistical characteristics of the similarity change rates of multiple preceding similarity units at the current processing position, a judgment threshold is dynamically generated. The similarity change rate is compared with the judgment threshold. When the similarity change rate is greater than the judgment threshold, the current processing position is marked as a semantic boundary. Based on the semantic boundary, different sliding step sizes and window sizes are set to determine candidate document chunk sets at different granularity levels; Each candidate segment in the candidate document segment set is scored based on a preset weighting coefficient to obtain a scoring result; Based on the scoring results and the semantic overlap between candidate segments, a target segment combination is selected from the candidate document segment set; Based on the target block combination, a dense feature vector is determined, and the dense feature vector is stored in a vector database; Based on the indexing plugin of the vector database, an index structure is established for the dense feature vectors to achieve document similarity retrieval; The preset weighting coefficients include at least two of the following: the semantic consistency score between the candidate segment and its context, the semantic redundancy between the candidate segment and other candidate segments in the same batch, and the adaptation score between the candidate segment and the user's query intent. The semantic redundancy is configured with a negative weighting coefficient to suppress the contribution of redundancy and duplication to the scoring results. When scoring candidate segments, the semantic consistency score is configured with a positive weight coefficient, the semantic redundancy score is configured with a negative weight coefficient, and the user query intent adaptation score is configured with a dynamic weight coefficient. The dynamic weight coefficient is adjusted according to the identification result of the query intent type. When the query intent is a fact-checking type, the adaptation score weight of fine-grained segments is increased, and when the query intent is a summary or overview type, the adaptation score weight of coarse-grained segments is increased.

2. The method according to claim 1, characterized in that, When the similarity change rate is not greater than the determination threshold, the method further includes: The currently processed semantic unit is merged with the semantic units in the current document block buffer, and the sliding window is moved forward by one semantic unit to continue the next round of similarity calculation and comparison.

3. The method according to claim 1, characterized in that, The step of selecting target segment combinations from the candidate document segment set based on the scoring results and the semantic overlap between candidate segments includes: Using a non-maximum suppression algorithm or a greedy selection strategy, candidate segments are sorted based on the score results. Adjacent segments are examined sequentially. When the semantic overlap between two segments is higher than a preset cross threshold, the candidate segment with the lower score is removed, and the candidate segment with the higher score is retained as the target segment combination.

4. The method according to claim 1, characterized in that, The step of determining the dense feature vector based on the target block combination includes: The target segments are combined and input into a preset embedding model, which transforms them into dense feature vectors of fixed dimensions.

5. The method according to claim 1, characterized in that, The method further includes: Collect feedback signals generated during the retrieval for the target block combination; Based on the feedback signal, the weighting coefficients used to determine the judgment threshold and / or score the candidate blocks are periodically adjusted.

6. The method according to claim 5, characterized in that, The step of periodically adjusting the weighting coefficients used to determine the judgment threshold and / or score candidate segments based on the feedback signal includes: When the feedback signal indicates that there is semantic incoherence due to information truncation, the judgment threshold is lowered to make the judgment of semantic boundaries more stringent. When the feedback signal indicates a decrease in retrieval accuracy due to redundant noise, the weighting coefficients used when scoring candidate segments are adjusted to increase the penalty weight for semantic redundancy.

7. A document retrieval device based on a vector database, characterized in that, include: At least one processor and bus; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to: Obtain the document to be processed, and parse the document to be processed into a sequence of semantic units arranged in order; The semantic similarity between adjacent semantic units in the semantic unit sequence is determined, and the similarity change rate is calculated based on the semantic similarity. According to the local statistical characteristics of the similarity change rates of multiple preceding similarity units at the current processing position, a judgment threshold is dynamically generated. The similarity change rate is compared with the judgment threshold. When the similarity change rate is greater than the judgment threshold, the current processing position is marked as a semantic boundary. Based on the semantic boundary, different sliding step sizes and window sizes are set to determine candidate document chunk sets at different granularity levels; Each candidate segment in the candidate document segment set is scored based on a preset weighting coefficient to obtain a scoring result; Based on the scoring results and the semantic overlap between candidate segments, a target segment combination is selected from the candidate document segment set; Based on the target block combination, a dense feature vector is determined, and the dense feature vector is stored in a vector database; Based on the indexing plugin of the vector database, an index structure is established for the dense feature vectors to achieve document similarity retrieval; The preset weighting coefficients include at least two of the following: the semantic consistency score between the candidate segment and its context, the semantic redundancy between the candidate segment and other candidate segments in the same batch, and the adaptation score between the candidate segment and the user's query intent. The semantic redundancy is configured with a negative weighting coefficient to suppress the contribution of redundancy and duplication to the scoring results. When scoring candidate segments, the semantic consistency score is configured with a positive weight coefficient, the semantic redundancy score is configured with a negative weight coefficient, and the user query intent adaptation score is configured with a dynamic weight coefficient. The dynamic weight coefficient is adjusted according to the identification result of the query intent type. When the query intent is a fact-checking type, the adaptation score weight of fine-grained segments is increased, and when the query intent is a summary or overview type, the adaptation score weight of coarse-grained segments is increased.

8. A non-volatile computer storage medium, characterized in that, It stores computer-executable instructions, which, when executed by a computer, can achieve the following: Obtain the document to be processed, and parse the document to be processed into a sequence of semantic units arranged in order; The semantic similarity between adjacent semantic units in the semantic unit sequence is determined, and the similarity change rate is calculated based on the semantic similarity. According to the local statistical characteristics of the similarity change rates of multiple preceding similarity units at the current processing position, a judgment threshold is dynamically generated. The similarity change rate is compared with the judgment threshold. When the similarity change rate is greater than the judgment threshold, the current processing position is marked as a semantic boundary. Based on the semantic boundary, different sliding step sizes and window sizes are set to determine candidate document chunk sets at different granularity levels; Each candidate segment in the candidate document segment set is scored based on a preset weighting coefficient to obtain a scoring result; Based on the scoring results and the semantic overlap between candidate segments, a target segment combination is selected from the candidate document segment set; Based on the target block combination, a dense feature vector is determined, and the dense feature vector is stored in a vector database; Based on the indexing plugin of the vector database, an index structure is established for the dense feature vectors to achieve document similarity retrieval; The preset weighting coefficients include at least two of the following: the semantic consistency score between the candidate segment and its context, the semantic redundancy between the candidate segment and other candidate segments in the same batch, and the adaptation score between the candidate segment and the user's query intent. The semantic redundancy is configured with a negative weighting coefficient to suppress the contribution of redundancy and duplication to the scoring results. When scoring candidate segments, the semantic consistency score is configured with a positive weight coefficient, the semantic redundancy score is configured with a negative weight coefficient, and the user query intent adaptation score is configured with a dynamic weight coefficient. The dynamic weight coefficient is adjusted according to the identification result of the query intent type. When the query intent is a fact-checking type, the adaptation score weight of fine-grained segments is increased, and when the query intent is a summary or overview type, the adaptation score weight of coarse-grained segments is increased.

Images