Knowledge base father-son fragmentation method based on image-text composite vector, medium and equipment

Abstract

The invention discloses a knowledge base father-son fragmentation method based on an image-text composite vector, a medium and equipment. The method comprises the steps of dividing fragments with a father-son relationship based on a hierarchical structure; generating a hierarchical context prefix containing all ancestor titles for each fragment; for the fragment containing the image, generating an image vector through a visual language model by using the context of the fragment; vectorizing the text content of each fragment, and for the image-containing fragment, splicing an image vector and a text vector to form an image-text composite vector; and storing the composite vectors of all the fragments and the metadata containing the hierarchical path identifier into a vector database. During retrieval, user query is vectorized and then similarity retrieval is performed, hierarchical aggregation and sorting are performed on results according to hierarchical path identifiers, and a structured retrieval context is returned. According to the method and the device, the maintenance of a document hierarchical structure, the deep fusion of image-text semantics and the structured organization of retrieval results are realized, and the retrieval accuracy of the multi-modal knowledge base is improved.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and information retrieval technology, specifically to a knowledge base parent-child fragmentation method, medium, and device based on graph-text composite vectors. Background Technology

[0002] Currently, the "Agent + Knowledge Base" architecture is commonly used when building industry-level intelligent agents. This model typically segments original documents (such as technical manuals and reports) into several text fragments, which are then stored in a vector database using vectorization techniques to build a knowledge base. When a user makes a query, the system performs a similarity search in the knowledge base based on the query vectors and provides the retrieved relevant fragments as context to a Large Language Model (LLM) to generate the final answer. The core of this model lies in the quality of the knowledge base construction, especially the document segmentation (chunking) method, which directly determines the accuracy of the retrieval and the completeness of the context.

[0003] To improve retrieval performance, various knowledge base segmentation methods have been proposed in the industry. Common methods include rule-based segmentation with fixed lengths and dynamic segmentation based on sliding windows. These methods primarily focus on the continuity of text content and retrieval efficiency, but they have significant shortcomings when processing structured and multimodal documents.

[0004] (1) Lack of image information processing: Existing mainstream segmentation schemes are mainly designed for plain text. When processing multimodal documents containing images, images are usually treated as attachments or ignored directly, failing to effectively encode the semantic information of images and integrate it into the vector representation of the knowledge base. This results in the system being unable to accurately match relevant information when performing mixed text and image queries or retrieving content that requires understanding images, leading to a decrease in retrieval accuracy.

[0005] (2) Disruption of document hierarchy: Fixed-length or sliding window segmentation methods mechanically cut the document, completely destroying the original hierarchical logical structure of the document, such as titles and chapters. The search results are a series of isolated text fragments, losing the parent-child relationship and contextual logic between chapters, making the generated answers lack overall coherence and making it difficult to understand the specific location and relationship of complex concepts in the document.

[0006] (3) Mismatch between fragment granularity and content: The pre-set fixed fragment length cannot adapt to the actual semantic boundaries of the document content. For documents with complex structures, it may lead to a complete section being forcibly split, resulting in semantic fragments; it may also merge multiple paragraphs with weak correlation, causing information redundancy and noise, affecting retrieval accuracy.

[0007] Some exploratory work has been done to address the issue of structure preservation. For example, the open-source project MDSplit proposes "parent-child splitting" based on the heading hierarchy of Markdown documents, which involves concatenating child heading blocks with their parent heading text to retain some contextual information in the child splits. However, these methods are usually only applied to text and do not systematically solve core problems such as the integration of image semantics, the joint representation of image and text vectors, and structured retrieval output based on complete hierarchical relationships. Summary of the Invention

[0008] In view of the above problems, this application provides a technical solution for parent-child fragmentation of knowledge base based on composite vectors of images and text, which solves the technical problems of loss of image semantic information, fragmentation of document hierarchical structure and difficulty in cross-modal retrieval of images and text in existing knowledge base fragmentation methods when processing mixed image and text documents.

[0009] To achieve the above objectives, in a first aspect, this application provides a knowledge base parent-child segmentation method based on graph-text composite vectors, the method comprising:

[0010] S1: Obtain the multimodal document to be processed, and convert the multimodal document into a structured document that retains the original title hierarchy. The multimodal document contains at least text and images.

[0011] S2: Based on the title hierarchy of the structured document, the document content is divided into multiple segments, and a parent-child hierarchy relationship is established for each segment to generate parent title block segments and child title block segments. Each segment corresponds to a title and its subordinate continuous content.

[0012] S3: Generate hierarchical context information for each segment, specifically including: for each segment, if it has ancestor segments, concatenate the title text of all its ancestor segments to generate the hierarchical context prefix of the segment, wherein the ancestor segments are all the parent segments of the segment up to the document root segment; for the parent title block segment containing the image, use its own hierarchical context prefix and the text content under its title as the context to perform semantic understanding on the image and generate an image vector;

[0013] S4: Vectorize the text content of each segment to generate a text vector; for parent title block segments containing images, concatenate the image vector and text vector corresponding to the parent title block segment to generate a composite vector of the parent title block segment; for subtitle block segments and parent title block segments that do not contain images, use their text vectors directly as the composite vector of the segment.

[0014] S5: Store the composite vector of each fragment and its corresponding fragment metadata in the vector database. The fragment metadata includes at least a hierarchical path identifier for identifying the position of the fragment in the document hierarchy.

[0015] S6: Receive user query information, vectorize the query information into query vectors, use the query vectors to perform similarity retrieval in the vector database, and obtain a preliminary matching composite vector set;

[0016] S7: Based on the hierarchical path identifiers corresponding to each composite vector in the preliminarily matched composite vector set, perform hierarchical aggregation and sorting on the search results to generate a structured search context;

[0017] S8: Return the structured search context to the user.

[0018] Furthermore, its own hierarchical context prefix and the text content under its heading serve as context, specifically including:

[0019] For the current parent title block segment to be processed, extract the ancestor title text of each level in its hierarchical context prefix to form an ancestor title sequence [T1,T2,...,T]. k ], where T1 is the top-level ancestor title, T k Set the title of the subsection for the parent title block;

[0020] Retrieve the text content C under the parent title block title, and compare the text content C with each ancestor title T in the ancestor title sequence. i Each input to a relevance scoring model yields a set of relevance scores [S1, S2, ..., S]. k ], where i takes the value [1, k];

[0021] The correlation scores [S1,S2,...,S] are used to... k Input a weight assignment function to generate a set of normalized dynamic weights [W1, W2, ..., W...]. k ], where the dynamic weight W i With correlation score S i Positive correlation;

[0022] The corresponding ancestor title texts are weighted and fused using the dynamic weights to generate a weighted ancestor title context C. ancestor The calculation formula is: C ancestor =Σ(W i ×Embed(T i ), where Embed(·) is the text embedding function;

[0023] The weighted ancestor title context C ancestor This is concatenated with the text content C to form the final weighted context C. final The calculation formula is as follows: C final=Concat(C ancestor ,C), where Concat(·) represents the concatenation function.

[0024] Furthermore, the following steps are included after step S5:

[0025] After the vector database is stored, all fragments are traversed to identify the set of child fragments with the same direct parent header block fragment identifier;

[0026] For each pair of image vectors V in the sub-segment set i and V j Calculate the semantic similarity or relationship category between the two, where the relationship category includes continuous illustrations, contrast charts, or unrelated illustrations;

[0027] The calculated semantic relationships are stored as cross-shard association metadata in the form of a relationship graph or a relationship list.

[0028] In the index of the vector database, backlinks are established for image vectors with strong semantic associations. This allows the retrieval of an image vector to be directly located through the backlinks when an image vector is found. Then, based on the fragment metadata stored in the vector database that is bound to each image vector, the fragment to which these associated image vectors belong can be determined.

[0029] Furthermore, semantic understanding is performed on the image to generate image vectors, specifically including the following fine-grained alignment steps:

[0030] For a parent title block slice containing at least one image, the text content C under the title of the parent title block slice is divided into several text segments [Seg1, Seg2, ..., Seg...]. m ];

[0031] For each image Img in the parent title block segment n Using a visual-language alignment model, the image is computed to match each text segment Seg. p The correlation score, where p ranges from [1, m];

[0032] Img per image n Assign one or more text fragments with the highest relevance as the unique descriptive text (Desc) for the image. m ;

[0033] Image Img n The specific descriptive text Desc corresponding to this image m A large visual language model is input together to generate a fine-grained aligned image vector V. img-m ;

[0034] Perform mean pooling on all fine-grained aligned image vectors corresponding to the parent title block slice, and use the pooling result as the image vector corresponding to the parent title block slice.

[0035] Furthermore, the method also includes:

[0036] After generating the image vectors for all slices, perform the following steps for each slice:

[0037] Based on the text vector of this segment, the text content topic of this segment is extracted through semantic clustering. text ;

[0038] Based on all image vectors in this segment, semantic topics are extracted from the image vectors of this segment through semantic clustering. img ;

[0039] Calculate Topic text and Topic img The semantic consistency score between shards is used. When the semantic consistency score of a shard is lower than a preset threshold, cross-shard retrieval is performed, specifically including:

[0040] Calculate the semantic topic of the image vector of this segment. img Text content topics in other segments text semantic consistency score;

[0041] When the semantic topic of the image vector of this segment is... img If the similarity between the text topic of another segment and the text content of another segment exceeds a preset similarity threshold, it is determined that the image set in that segment is semantically highly related to the text content of another segment, and a hierarchical structure correction record is generated.

[0042] The hierarchical structure correction record includes: the image set identifier suspected of misattribution, the segment identifier, the identifier of another segment with semantically high relevance, and the Topic in that segment. text and Topic img The semantic consistency score between the segments, and the semantic topic of the image vectors in that segment. img Similarity to the text topic of another segment and the confidence score of attribution;

[0043] The attribution confidence score is based on the Topic in the segment. text and Topic img The semantic consistency score between the segments and the semantic topic of the image vectors of the segment. img The semantic consistency score with the text topic of another segment is calculated.

[0044] Furthermore, semantic understanding is performed on the image to generate an image vector, specifically including:

[0045] Determine whether the text content C under the title of the parent title block segment meets the preset image description sufficiency condition. The image description sufficiency condition is configured as follows: when the length of the text content C is lower than a first length threshold, or when the text content C is judged to be insufficiently described by a pre-trained text sufficiency evaluation model, it is determined that it is not satisfied.

[0046] If the condition is not met, then based on enhanced contextual information, a hypothetical image description text C related to the semantics of the segment is generated using a text generation language model. h The enhanced context information includes at least one of the following: the hierarchical context prefix of the segment, the summary information of the multimodal document, and the topic keywords extracted from the multimodal document;

[0047] The text content C is compared with the hypothetical image description text C. h The text is then concatenated to generate an enhanced text context.

[0048] The enhanced text context and the image are input together into the visual language large model to generate the image vector.

[0049] Furthermore, the composite vector of each slice and its corresponding slice metadata are stored in the vector database, specifically including:

[0050] For each slice, its composite vector is parsed into independent text component vectors and image component vectors;

[0051] The text component vector is associated with the fragment metadata and stored as a record in the text vector sub-library of the vector database; the image component vector is associated with the fragment metadata and stored as a record in the image vector sub-library of the vector database.

[0052] Among them, text component vector records and image component vector records belonging to the same segment are associated with a unique segment identifier corresponding to that segment;

[0053] Using the query vector to perform similarity retrieval in the vector database specifically includes:

[0054] The query vector is used for similarity retrieval in the text vector sub-database to obtain a preliminary matching set of text component vectors and its corresponding first similarity score set;

[0055] The query vector is used for similarity retrieval in the image vector sub-database to obtain a preliminary matching set of image component vectors and its corresponding second similarity score set;

[0056] Based on the unique segment identifier, the set of text component vectors and the set of image component vectors are associated. For a segment that appears in any preliminary matching set, its corresponding component vector and its similarity score are retrieved and associated from another sub-database to ensure that complete text and image component vector information and corresponding scores are recalled for each matching segment.

[0057] For each associated segment, a comprehensive similarity score is calculated based on the first similarity score of the text component vector and the second similarity score of the image component vector of the segment, using a preset score fusion rule. All matching segments are then sorted according to the comprehensive similarity score.

[0058] The text component vectors, image component vectors, and their complete metadata corresponding to the sorted segments are integrated to generate a sorted preliminary matching composite vector set.

[0059] Furthermore, following step S2, the method further includes a step of semantically refining the parent title block fragments, specifically including:

[0060] For the parent title block segment, semantic segmentation point detection is performed on the text content it contains to identify the boundary positions where the topic changes significantly, and thus obtain the semantic boundary.

[0061] Based on the detected semantic boundaries, the parent title block segment is divided into multiple semantically coherent micro-block segments. Each micro-block segment inherits the hierarchical context prefix of the parent title block segment and generates a derived title based on the title of the parent title block segment and the sub-topic content corresponding to the micro-block segment.

[0062] The images in the parent title block are reassigned to the corresponding micro-blocks based on the semantic relationship between the images and the text content within the micro-blocks.

[0063] After the semantic refinement step is completed, the micro-block fragment replaces its original parent title block fragment and serves as the execution unit for subsequent steps S3 to S8.

[0064] In a second aspect, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the knowledge base parent-child segmentation method based on graph-text composite vectors as described in the first aspect of this application.

[0065] In a third aspect, this application provides an electronic device having a computer program stored thereon, including a processor and a storage medium, wherein the computer program is stored on the storage medium, and when executed by the processor, the computer program implements the knowledge base parent-child segmentation method based on graph-text composite vectors as described in the first aspect of this application.

[0066] Unlike existing technologies, the above-disclosed technical solution discloses a knowledge base parent-child segmentation method, medium, and device based on image-text composite vectors, aiming to solve the problems of image semantic loss, document structure fragmentation, and difficulties in cross-modal retrieval in existing technologies. The method includes: acquiring multimodal documents and converting them into structured documents that retain their title hierarchy; dividing the content into segments with parent-child relationships based on this hierarchy; generating a hierarchical context prefix for each segment containing all ancestor titles; for segments containing images, using their hierarchical context prefixes and their own text as context, understanding the image semantics through a visual language model and generating image vectors; vectorizing the text content of each segment into text vectors; for image-containing segments, concatenating their image vectors with text vectors to form image-text composite vectors, while other segments directly use text vectors as composite vectors; and storing the composite vectors of all segments and metadata containing hierarchical path identifiers in a vector database. During retrieval, the user query is vectorized, and a similarity search is performed in the vector database. Hierarchical aggregation and sorting are then performed based on the hierarchical path identifiers corresponding to the segment in the search results, and finally, the structured retrieval context is returned to the user. This application achieves the preservation of document hierarchical structure, deep fusion of text and image semantics, and structured organization of retrieval results, significantly improving the retrieval accuracy and answer interpretability of the multimodal knowledge base.

[0067] The above description of the invention is merely an overview of the technical solution of this application. In order to enable those skilled in the art to better understand the technical solution of this application and to implement it based on the description and drawings, and to make the above-mentioned objectives and other objectives, features and advantages of this application easier to understand, the following description is provided in conjunction with the specific embodiments and drawings of this application. Attached Figure Description

[0068] The accompanying drawings are only used to illustrate the principles, implementation methods, applications, features, and effects of specific embodiments of this application and other related content, and should not be considered as limitations on this application.

[0069] In the accompanying drawings of the instruction manual:

[0070] Figure 1 This is a flowchart of the knowledge base parent-child segmentation method based on graph-text composite vectors as described in the first exemplary embodiment of this application;

[0071] Figure 2This is a flowchart of the knowledge base parent-child segmentation method based on graph-text composite vectors as described in the second exemplary embodiment of this application;

[0072] Figure 3 This is a flowchart of the knowledge base parent-child segmentation method based on graph-text composite vectors as described in the third exemplary embodiment of this application;

[0073] Figure 4 This is a flowchart of the knowledge base parent-child segmentation method based on graph-text composite vectors as described in the fourth exemplary embodiment of this application;

[0074] Figure 5 This is a flowchart of the knowledge base parent-child segmentation method based on graph-text composite vectors as described in the fifth exemplary embodiment of this application;

[0075] Figure 6 This is a schematic diagram of an electronic device according to an exemplary embodiment of this application;

[0076] The reference numerals used in the above figures are explained as follows:

[0077] 10. Electronic equipment; 101. Processor; 102. Storage medium. Detailed Implementation

[0078] To explain in detail the possible application scenarios, technical principles, specific feasible solutions, and the objectives and effects that this application can achieve, the following detailed description is provided in conjunction with the listed specific embodiments and accompanying drawings. The embodiments described herein are only used to more clearly illustrate the technical solutions of this application, and are therefore only examples, and should not be used to limit the scope of protection of this application.

[0079] In this document, the term "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The term "embodiment" appearing in various places throughout the specification does not necessarily refer to the same embodiment, nor does it specifically limit its independence or connection with other embodiments. In principle, in this application, as long as there are no technical contradictions or conflicts, the technical features mentioned in each embodiment can be combined in any way to form corresponding implementable technical solutions.

[0080] Unless otherwise defined, the technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the use of related terms herein is merely for the purpose of describing particular embodiments and is not intended to limit this application.

[0081] In the description of this application, the term "and / or" is used to describe the logical relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A exists, B exists, and A and B exist simultaneously. Additionally, the character " / " in this document generally indicates that the preceding and following objects have an "or" logical relationship.

[0082] In this application, terms such as “first” and “second” are used only to distinguish one entity or operation from another, and do not necessarily require or imply any actual quantity, hierarchy or order relationship between these entities or operations.

[0083] Without further limitations, the use of terms such as “comprising,” “including,” “having,” or other similar open-ended expressions in this application is intended to cover non-exclusive inclusion, which does not exclude the presence of additional elements in a process, method, or product that includes the stated elements, such that a process, method, or product that includes a list of elements may include not only those defined elements but also other elements not expressly listed, or elements inherent to such a process, method, or product.

[0084] In this application, expressions such as "greater than", "less than", and "exceeding" are understood to exclude the stated number; expressions such as "above", "below", and "within" are understood to include the stated number. Furthermore, in the description of the embodiments of this application, "multiple" means two or more (including two), and similar expressions related to "multiple" are also understood in this way, such as "multiple groups" and "multiple times", unless otherwise explicitly specified.

[0085] In the first aspect, such as Figure 1 As shown, this application provides a knowledge base parent-child segmentation method based on graph-text composite vectors, the method comprising:

[0086] S1: Obtain the multimodal document to be processed, and convert the multimodal document into a structured document that retains the original title hierarchy. The multimodal document contains at least text and images.

[0087] S2: Based on the title hierarchy of the structured document, the document content is divided into multiple segments, and a parent-child hierarchy relationship is established for each segment to generate parent title block segments and child title block segments. Each segment corresponds to a title and its subordinate continuous content.

[0088] S3: Generate hierarchical context information for each segment, specifically including: for each segment, if it has ancestor segments, concatenate the title text of all its ancestor segments to generate the hierarchical context prefix of the segment, wherein the ancestor segments are all the parent segments of the segment up to the document root segment; for the parent title block segment containing the image, use its own hierarchical context prefix and the text content under its title as the context to perform semantic understanding on the image and generate an image vector;

[0089] S4: Vectorize the text content of each segment to generate a text vector; for parent title block segments containing images, concatenate the image vector and text vector corresponding to the parent title block segment to generate a composite vector of the parent title block segment; for subtitle block segments and parent title block segments that do not contain images, use their text vectors directly as the composite vector of the segment.

[0090] S5: Store the composite vector of each fragment and its corresponding fragment metadata in the vector database. The fragment metadata includes at least a hierarchical path identifier for identifying the position of the fragment in the document hierarchy.

[0091] S6: Receive user query information, vectorize the query information into query vectors, use the query vectors to perform similarity retrieval in the vector database, and obtain a preliminary matching composite vector set;

[0092] S7: Based on the hierarchical path identifiers corresponding to each composite vector in the preliminarily matched composite vector set, perform hierarchical aggregation and sorting on the search results to generate a structured search context;

[0093] S8: Return the structured search context to the user.

[0094] In this embodiment, a multimodal document refers to a document that simultaneously contains multiple information carriers such as text and images, and the document format includes PDF, Word, HTML, XLS, XLSX, etc.

[0095] A structured document is a document that retains the original title hierarchy (such as chapter, section, subsection, etc.) after format conversion. In this application, it is preferred to use a Markdown format document, which can clearly present the logical organization of the document and provide a foundation for subsequent segmentation processing.

[0096] The parent-child hierarchy refers to the hierarchical relationship formed based on the document title hierarchy. The parent title block is a block that contains higher-level titles and their corresponding content, while the child title block is a block that belongs to the parent title block and contains lower-level titles and their corresponding content. The two form a clear subordinate relationship.

[0097] The hierarchical context prefix refers to the text prefix formed by concatenating the title text of all ancestor fragments (all parent fragments from the direct parent fragment to the document root fragment) for fragments that have ancestor fragments. Its function is to preserve the hierarchical position information of the fragment in the entire document.

[0098] Image vectors are vector data that represent the semantic features of images after semantic understanding is performed on images through visual language models. They are a key data form for realizing the fusion retrieval of image and text information.

[0099] Text vectors are vector data obtained after word segmentation and embedding of text content. They can accurately represent the semantic information of the text and provide data support for text retrieval.

[0100] A composite vector is a vector formed by concatenating the image vector and the text vector of a parent title block segment that contains an image. For subtitle blocks and parent title blocks that do not contain images, the text vector is a composite vector. Composite vectors are the core data carrier for cross-modal retrieval.

[0101] Shard metadata refers to auxiliary information related to shards. It includes at least the hierarchical path identifier used to identify the position of the shard in the document hierarchy, and may also include information such as the original file path, image hash, shard creation time, and business tags, providing necessary auxiliary support for the storage, retrieval, and management of shards.

[0102] Vector databases are databases specifically designed for storing and managing vector data. Open-source vector databases such as Milvus and FAISS can be used, which have efficient vector similarity retrieval capabilities and can meet the storage and retrieval needs of large-scale vector data.

[0103] A query vector is a vector obtained by vectorizing user query information (including plain text queries or mixed queries containing images). It serves as the basis for similarity comparison with composite vectors stored in a vector database.

[0104] Structured retrieval context refers to the set of retrieval results that retains the original hierarchical relationship of documents after the preliminary matching composite vector set is aggregated and sorted according to hierarchical path identifiers. It can provide high-quality input for large models to generate coherent and accurate answers.

[0105] The core objective of this application is to address the shortcomings of existing knowledge base fragmentation methods in terms of multimodal information utilization, hierarchical relationship preservation, and cross-modal retrieval efficiency. It aims to achieve efficient organization and accurate retrieval of multimodal knowledge bases through a complete workflow. The detailed principles of each step are as follows:

[0106] In step S1, the multimodal document to be processed is first obtained. This document contains at least two core types of information: text and images. Using specialized format conversion tools and algorithms, multimodal documents of different formats are uniformly converted into structured Markdown documents. Markdown format is concise and easy to parse, perfectly preserving the original document's heading hierarchy. Whether it's a first-level heading, a second-level heading, or a lower-level heading, it can be accurately presented after conversion, laying a solid foundation for subsequent segmentation processing based on heading hierarchy and establishing parent-child hierarchical relationships.

[0107] In step S2, based on the hierarchical structure of the structured document's headings, the document content is divided into multiple segments. Each segment corresponds to a heading and its subordinate continuous content, ensuring that the content of each segment is logically related. Simultaneously, a parent-child hierarchical relationship is established for each segment according to the heading hierarchy, clearly distinguishing between parent and child heading segments. For example, in the document, "1. Technical Overview" is the parent heading segment corresponding to a first-level heading, and "1.1 Technical Background" is a child heading segment belonging to it. This division method fully preserves the hierarchical logical structure of the document and avoids the context fragmentation problem caused by traditional segmentation methods.

[0108] In step S3, hierarchical context information is first generated for each segment. Specifically, for segments with ancestor segments, the title texts of all ancestor segments are concatenated to generate a hierarchical context prefix. This prefix fully presents the segment's hierarchical position within the entire document, ensuring that the segment's semantic information includes not only its own content but also information from its parent hierarchy. For parent title block segments containing images, their own hierarchical context prefix and the text content under their title are used as the context background. This context is then input into a large-scale visual language model (such as the CLIP model) for image semantic understanding. The large-scale visual language model can accurately capture the semantic features of images by combining contextual information, thereby generating image vectors that accurately represent the semantics of images, ensuring that image information can be effectively converted into vector data that can be used for retrieval.

[0109] In step S4, the text content of each segment is first segmented into words, and then the segmented text is converted into text vectors using an embedding model (such as BERT). These text vectors accurately represent the semantic information of the text. For parent title block segments containing images, the image vector generated in step S3 is concatenated with the text vector generated in this step according to a preset concatenation rule to form a composite vector. This concatenation method achieves deep integration of image semantics and text semantics, enabling the composite vector to simultaneously represent the text and image information of the segment. For subtitle block segments and parent title block segments that do not contain images, since they do not involve image information or the image information has been associated with the parent segment, their text vectors are directly used as composite vectors to ensure that all segments have a unified vector representation form.

[0110] In step S5, the composite vector of each fragment is first stored in a vector database (such as Milvus or FAISS). These vector databases have efficient vector indexing and similarity retrieval capabilities, which can meet the storage and fast retrieval needs of large-scale composite vectors. At the same time, fragment metadata is stored in association with the composite vector. Fragment metadata includes at least a hierarchical path identifier, which clearly indicates the specific location of the fragment in the document hierarchy. In addition, it may also include the original file path (for tracing the original document), image hash (for image uniqueness identification), fragment creation time, business tags, and other information, providing necessary auxiliary support for subsequent retrieval, aggregation, scheduling, and other operations.

[0111] In step S6, user query information is first received. This query information can be either a plain text query or a mixed query including images. For plain text queries, it is converted into query vectors using the same embedding model as the text vector generation process. For mixed queries including images, the text and image parts are converted into corresponding vectors using both the embedding model and the visual language model, respectively, and then fused into a query vector according to preset rules. Subsequently, the generated query vectors are used to perform similarity retrieval in a vector database. By calculating the similarity (such as cosine similarity) between the query vectors and the composite vectors stored in the database, several composite vectors with the highest similarity are selected to form a preliminary matching set of composite vectors, providing a foundation for subsequent result optimization.

[0112] In step S7, the search results are hierarchically aggregated based on the hierarchical path identifiers corresponding to each composite vector in the initially matched composite vector set. Specifically, all sub-heading blocks belonging to the same parent heading block are merged to form complete context blocks, ensuring that each context block retains the hierarchical logical relationship of the original document. Subsequently, the aggregated context blocks are sorted according to the similarity score between the query vector and the composite vector, with context blocks having higher similarity ranking higher. This ultimately generates a structured search context, giving the search results a clear logical structure and priority.

[0113] In step S8, the structured search context generated in step S7 is returned to the user. This structured search context can also be used as input to a larger model to generate more accurate and coherent natural language answers. Compared to traditional search results, the structured search context returned in this step preserves the hierarchical relationships of documents, allowing users to clearly understand the position and logical connections of the search results within the original documents, greatly improving the readability and usability of the search results. For larger models, the structured input helps them better understand the contextual logic, thereby generating answers that better meet user needs.

[0114] This application has the following beneficial effects:

[0115] By converting multimodal documents into structured documents that retain their heading hierarchy, and dividing them into parent-child segments based on this hierarchy, while generating hierarchical context prefixes for each segment, this method overcomes the limitations of traditional segmentation methods that fragment the document's chapter structure. In multi-turn dialogue scenarios, it can quickly recover complete contextual logic based on parent-child hierarchical relationships, effectively improving the contextual coherence of generated answers and significantly reducing the occurrence of missing information.

[0116] This application constructs an image-text composite vector, which unifies the representation of image semantics and text semantics in the vector space. By combining document hierarchy and parent segment text as context for image understanding, the large model can accurately grasp the semantic connotation of an image in a specific chapter, achieving true cross-modal retrieval.

[0117] The structured search context generated through hierarchical aggregation and sorting not only includes information fragments related to the query but also preserves the hierarchical logical relationships of the original document. Users can clearly understand the source and logical connections of the search results, greatly improving the readability and usability of the results. At the same time, the structured presentation also makes the search results more interpretable, making it easier for users to understand the basis for the results, especially suitable for searching highly structured documents such as technical manuals and academic papers.

[0118] like Figure 2As shown, in some embodiments, the context is defined by its own hierarchical context prefix and the text content under its heading, specifically including:

[0119] S31: For the current parent title block segment to be processed, extract the ancestor title text of each level in its hierarchical context prefix to form an ancestor title sequence [T1,T2,...,T...]. k ], where T1 is the top-level ancestor title, T k Set the title of the subsection for the parent title block;

[0120] S32: Obtain the text content C under the parent title block subheading, and compare the text content C with each ancestor title T in the ancestor title sequence. i Each input to a relevance scoring model yields a set of relevance scores [S1, S2, ..., S]. k ], where i takes the value [1, k];

[0121] S33: The correlation scores [S1,S2,...,S] are... k Input a weight assignment function to generate a set of normalized dynamic weights [W1, W2, ..., W...]. k ], where the dynamic weight W i With correlation score S i Positive correlation;

[0122] S34: Use the dynamic weights to perform weighted fusion on the corresponding ancestor title texts to generate a weighted ancestor title context C. ancestor The calculation formula is: C ancestor =Σ(W i ×Embed(T i ), where Embed(·) is the text embedding function;

[0123] S35: Transfer the weighted ancestor title context C ancestor This is concatenated with the text content C to form the final weighted context C. final The calculation formula is as follows: C final =Concat(C ancestor ,C), where Concat(·) represents the concatenation function.

[0124] In this embodiment, the ancestor title sequence refers to the title sequence formed by arranging the titles from the top-level ancestor title to the title of the parent title block fragment itself in hierarchical order for the currently to-be-processed parent title block fragment, denoted as [T1, T2, ..., T...]. k ], where T1 is the top-level ancestor title, T kThis sequence represents the title of the current parent title block segment and fully presents the hierarchical source information of the current parent title block segment.

[0125] A relevance scoring model is a model used to calculate the degree of relevance between text content and its ancestor title. It can output a quantitative relevance score based on indicators such as semantic similarity. In this application, a semantic similarity calculation model based on a pre-trained language model can be used to ensure the accuracy of the scoring results.

[0126] The relevance score is calculated using a relevance scoring model. It is a quantitative value representing the degree of semantic relevance between the text content and the corresponding ancestral title, denoted as [S1, S2, ..., S]. k The higher the value, the stronger the correlation.

[0127] The weighting function is a function used to generate normalized dynamic weights based on the relevance score. Its core feature is that the generated dynamic weights are positively correlated with the relevance score, which can achieve weight bias towards highly relevant ancestor titles. In this application, commonly used normalization functions such as the SoftMax function can be used.

[0128] Dynamic weights refer to the weight values ​​obtained after normalizing the relevance scores using a weighting function, denoted as [W1, W2, ..., W...]. k The sum of these values ​​is 1, which is used to weight and fuse the corresponding ancestor title texts, making the fused context more targeted.

[0129] A text embedding function (Embed(·)) is a function used to convert ancestor title text into text vectors, mapping the semantic information of the text into low-dimensional dense vectors. Preferably, the text embedding function can be implemented using the embedding layer of a pre-trained language model such as BERT or RoBERTa.

[0130] Weighted Ancestor Title Context (C ancestor ( ) refers to the contextual information obtained by weighting and summing the embedding vectors of the corresponding ancestor title texts through dynamic weights, which can highlight the semantic contribution of ancestor titles that are highly related to the content of the current parent title block fragment text.

[0131] The concatenation function (Concat()) is used to concatenate weighted ancestor title contexts (C++, C ... ancestor The text content (C) of the current parent title block is concatenated with the text content (C) of the current title block to form the final weighted context (C). final The function can be used to concatenate text strings or vectors. In this application, it is specifically used to concatenate text strings to ensure the integrity and coherence of the context.

[0132] In practical applications, the relevance between different ancestor titles and the text content of the current parent title block varies. Simply concatenating all ancestor title texts directly as context may introduce irrelevant or weakly related information, interfering with the accuracy of image semantic understanding. Therefore, this application further optimizes the context generation process, with the specific steps and principles as follows:

[0133] First, for the current parent title block segment to be processed, trace all its ancestor segments (all parent segments from the direct parent segment to the document root segment), extract the title text of each ancestor segment and the title text of the current parent title block segment itself, and arrange them in order from the top level to the current level to form the ancestor title sequence [T1, T2, ..., T]. k This sequence fully covers all hierarchical source information of the current parent title block fragment, providing foundational data for subsequent correlation analysis.

[0134] Then, retrieve the text content C under the current parent title block title, and compare the text content C with each ancestor title T in the ancestor title sequence. i (i takes values ​​[1, k]) as input to the relevance scoring model. The relevance scoring model, based on the semantic understanding capabilities of a pre-trained language model, calculates the relationship between the text content C and each ancestor title T. i The semantic relevance between them is evaluated, and the corresponding relevance score S is output. i This ultimately forms a set of correlation scores [S1, S2, ..., S]. k This step allows for precise quantification of the correlation between different ancestral titles and the current text content.

[0135] Then the calculated correlation scores [S1,S2,...,S] are used to... k Input a weighting function (such as the SoftMax function), which normalizes the relevance scores and generates a set of dynamic weights [W1, W2, ..., W...] that sum to 1. k Among them, the dynamic weight W i With correlation score S i Positive correlation, meaning the text content C is related to a certain ancestor title T. i The higher the correlation, the higher the corresponding weight W. i The larger the value, the more weight is allocated to highly relevant ancestor titles.

[0136] Then the text embedding function Embed() is called to embed each ancestor title T in the ancestor title sequence. i The process generates the corresponding text embedding vector, Embed(Ti). Then, the embedding vector of each ancestor title is multiplied by its corresponding dynamic weight according to the weighted summation formula, and the summation is performed to obtain the weighted ancestor title context C.ancestor This context highlights the semantic contribution of highly relevant ancestor titles while mitigating the interference of low-relevance ancestor titles, making the contextual information more targeted.

[0137] Finally, the weighted ancestor title context C is concatenated using the concatenation function Concat(). ancestor Concatenate the text content C of the current parent title block segment to form the final weighted context C. final The final weighted context C final It retains hierarchical source information that is highly relevant to the current text content, and also includes the specific text content of the current parent title block segment, providing accurate and effective contextual support for image semantic understanding.

[0138] The above scheme, through a dynamic weighting mechanism, focuses the final weighted context on information highly relevant to the text content of the current parent title block, avoiding interference from irrelevant or weakly related upper-level title information on image semantic understanding. Based on this precise context, the visual language model performs image semantic analysis, enabling it to more accurately grasp the semantic connotation of an image in a specific scene. The generated image vectors more realistically reflect the actual meaning of the image, providing a strong guarantee for improving cross-modal retrieval accuracy.

[0139] This application automatically evaluates the relevance of text content to its ancestor titles using a relevance scoring model and dynamically assigns weights based on the evaluation results, replacing the traditional method of manually setting fixed rules or simple concatenation. This mechanism can adapt to the structural characteristics and content differences of different documents, and regardless of the hierarchical complexity of the document, it can generate contextual information that meets the needs of the current scenario, making the system more intelligent, robust, and applicable to a wider range of situations.

[0140] The dynamic weighted fusion method proposed in this application can filter out key hierarchical information and assign it higher weights, reducing the proportion of redundant information and making the contextual information more refined and targeted. This not only reduces the processing complexity of the visual language model but also reduces noise interference, further improving the quality and efficiency of image vector generation.

[0141] In some embodiments, such as Figure 3 As shown, the following steps are included after step S5:

[0142] S301: After the vector database is stored, traverse all fragments and identify the set of child fragments with the same direct parent header block fragment identifier;

[0143] S302: For each pair of image vectors V in the sub-segment set i and V jCalculate the semantic similarity or relationship category between the two, where the relationship category includes continuous illustrations, contrast charts, or unrelated illustrations;

[0144] S303: Store the calculated semantic relationships as cross-shard association metadata in the form of a relationship graph or a relationship list;

[0145] S304: In the index of the vector database, backlinks are established for image vectors with strong semantic associations, so that when an image vector is retrieved, other image vectors associated with that image vector can be directly located through the backlinks, and then the slice to which these associated image vectors belong can be determined according to the slice metadata stored in the vector database that is bound to each image vector.

[0146] In this embodiment, the direct parent title block fragment identifier refers to the identification information used to uniquely identify the parent title block fragment. It can be in the form of a string, a number sequence, etc. Its core function is to associate all child fragments belonging to the parent title block fragment, providing a basis for identifying the set of child fragments.

[0147] A sub-fragment set is a collection of all sub-heading block fragments that share the same direct parent heading block fragment identifier. Sub-fragments in this set belong to the same parent heading block fragment in the document hierarchy and have a natural logical relationship.

[0148] Semantic similarity is an index used to quantify the degree of similarity between the semantics represented by two image vectors. In this application, common vector similarity calculation methods such as cosine similarity and Euclidean distance can be used. The higher the value, the more similar the semantics of the two images are.

[0149] Relationship categories refer to categories based on the semantic association between image vectors. Specifically, they include continuous illustrations (images that have a logical sequence and jointly represent a process or step), comparison charts (images used to compare the features and data of different objects), and irrelevant illustrations (images that have no obvious semantic association). This classification provides a clear direction for the utilization of image association.

[0150] Cross-segment association metadata refers to metadata used to record the semantic relationships between image vectors in a sub-segment set. It is stored in the form of a relationship graph or a relationship list. The relationship graph can intuitively present the relationship network between image vectors, while the relationship list facilitates fast querying and traversal, providing data support for cross-segment image association retrieval.

[0151] Backlinks refer to bidirectional links established in a vector database index for image vectors with strong semantic relationships. Their core function is to enable rapid location of related image vectors. When a retrieval hits one of the image vectors, other strongly related image vectors can be found directly through the link.

[0152] Specifically, in step S301, after storing the composite vector and fragment metadata, all fragments stored in the vector database are traversed. Based on the direct parent title block fragment identifier in the fragment metadata, all subtitle block fragments with the same direct parent title block fragment identifier are selected to form a sub-fragment set. The core of this step is to mine sub-fragment groups with potential logical connections based on document hierarchy, thus defining the scope for subsequent image association analysis.

[0153] In step S302, for the identified sub-segment set, the image vector contained in each sub-segment is extracted. For each pair of image vectors V in the sub-segment set... i and V j On the one hand, the semantic similarity between the two images is calculated using vector similarity methods (such as cosine similarity) to quantify their semantic association. On the other hand, based on the semantic features of the images and their application scenarios, the relationship category between the two images is determined, specifically including continuous illustrations, comparative charts, or unrelated insets. Through this step, the semantic association between image vectors in the sub-segment set can be fully understood.

[0154] In step S303, the semantic similarity and relationship categories between the image vectors calculated in step S302 are organized in the form of a relationship graph or a relationship list to form cross-segment association metadata. In the relationship graph, each node represents an image vector, and the edges between nodes represent semantic relationships. The weight of the edges can be determined by semantic similarity, and the relationship category is also labeled. The relationship list records information such as the identifier, semantic similarity, and relationship category of each pair of image vectors in tabular form. This cross-segment association metadata is stored in association with the segment metadata to provide a basis for association queries in subsequent retrieval.

[0155] In step S304, in the index of the vector database, backlinks are established for image vectors with strong semantic relationships (semantic similarity higher than a preset threshold or relationship category of continuous illustration or comparison chart). Backlinks are bidirectional, meaning that image vector V... i Pointing to image vector V j At the same time, image vector V j It also points to the image vector V i By establishing backlinks, the limitations of fragment boundaries can be broken, enabling rapid location of associated image vectors and providing efficient indexing support for cross-fragment image association retrieval.

[0156] When a user's query matches a specific image vector, the system directly locates other image vectors with strong semantic relationships to that vector through the backlinks. Then, based on the fragment metadata stored in the vector database that is bound to each associated image vector, the system determines the fragment to which each associated image vector belongs. This step allows for the retrieval of semantically related images scattered across different sub-fragments, along with their corresponding fragment information, enriching the search results.

[0157] This application overcomes the limitations of shard boundaries by mining the semantic relationships between images in different sub-shards under the same parent title block, establishing backlinks and cross-shard metadata associations. During the retrieval process, it can proactively aggregate semantically related images scattered across different sub-sections (such as multiple steps of an experiment, different perspectives of a product, a set of comparative data charts, etc.) and present them to the user, providing more complete and coherent visual information and helping the user to more comprehensively understand the query content.

[0158] By querying an image, users can not only obtain the segment information of that image, but also access a series of other semantically strongly related images and their corresponding segment information through backlinks. This greatly enriches the content of the search results and expands the depth and breadth of the search. This associative search mode can meet users' needs for in-depth information exploration, avoid the information bias caused by searching only a single image, and significantly improve the search effect.

[0159] By analyzing the semantic relationships between image vectors and storing associated metadata, the system can utilize image information more efficiently. During the retrieval process, related images can be quickly located without fully traversing all image vectors, improving retrieval efficiency. At the same time, clear relationship category annotations help users understand the relationship logic between images, further enhancing the utilization value of image information.

[0160] In some embodiments, such as Figure 4 As shown, semantic understanding is performed on the image to generate an image vector, specifically including the following fine-grained alignment steps:

[0161] S401: For a parent title block slice containing at least one image, divide the text content C under the title of the parent title block slice into several text segments [Seg1, Seg2, ..., Seg...]. m ];

[0162] S402: For each image Img in this parent title block slice n Using a visual-language alignment model, the image is computed to match each text segment Seg. p The correlation score, where p ranges from [1, m];

[0163] S403: Img for each image n Assign one or more text fragments with the highest relevance as the unique descriptive text (Desc) for the image. m ;

[0164] S404: Transfer image Img n The specific descriptive text Desc corresponding to this image m A large visual language model is input together to generate a fine-grained aligned image vector V. img-m ;

[0165] S405: Perform mean pooling on all fine-grained aligned image vectors corresponding to the parent title block slice, and use the pooling result as the image vector corresponding to the parent title block slice.

[0166] In some embodiments, a text fragment is a sub-text unit denoted as [Seg1, Seg2, ..., Seg3], obtained by segmenting the text content C of a parent title block containing at least one image according to rules such as semantic integrity and sentence boundaries. m Each text fragment has relatively independent semantics, providing a foundation for achieving fine-grained alignment between images and text.

[0167] The visual-language alignment model is a model used to calculate the degree of association between an image and a text fragment. This model can simultaneously understand the visual features of the image and the semantic features of the text, and output the association score between the two. In this application, advanced visual-language pre-trained models such as CLIP and ALBEF can be used.

[0168] The relevance score is a quantitative value that represents the degree of semantic relevance between an image and a text fragment, calculated using a visual-language alignment model. It is calculated for each image (Img). n and each text fragment Seg p Output the corresponding correlation score, where p ranges from [1, m]. A higher score indicates a stronger correlation between the image and the text fragment.

[0169] Dedicated description text (Desc) m () refers to Img per image n The assigned one or more text fragments that are most relevant to the image can accurately describe the core semantic features of the image and are key text information for achieving fine-grained alignment between the image and the text.

[0170] Fine-grained aligned image vectors (V img-m () refers to the image Img n Its corresponding exclusive description text Desc mThe image vector generated after inputting into the large visual language model not only represents the visual features of the image, but also integrates textual semantic features that are highly related to the image, thus achieving fine-grained semantic alignment between the image and the text.

[0171] Mean pooling is an operation that averages all finely aligned image vectors corresponding to the parent title block slice. This operation yields a vector that represents the overall semantic features of all images in the parent title block slice, which is then used as the final image vector for that parent title block slice.

[0172] For a parent title block segment containing multiple images, the image vector is generated directly using the entire text content of the parent segment as context. However, in reality, a parent segment may contain multiple images and multiple text segments, with different images often corresponding to different text fragments. Using the entire text as context may lead to inaccurate image semantic understanding and introduce irrelevant noise. This embodiment achieves precise matching between images and text through a fine-grained alignment step, optimizing the image vector generation effect. The specific steps and principles are as follows:

[0173] In step S401, for a parent title block segment containing at least one image, the text content C under its title is obtained. Based on rules such as semantic integrity, sentence boundaries, and punctuation, the text content C is segmented into several text segments [Seg1, Seg2, ..., Seg...]. m During the segmentation process, it is necessary to ensure that the semantics of each text segment are relatively independent and complete, and to avoid semantic breaks due to improper segmentation, so as to provide high-quality text data for subsequent calculation of the correlation between images and text segments.

[0174] In step S402, for each image Img in the parent title block slice... n This is compared with each text segment Seg obtained from the segmentation. p (The value of p ranges from [1, m]) Input the data into the visual-language alignment model. The visual-language alignment model can simultaneously extract the visual features of the image and the semantic features of the text fragment, and calculate the semantic correlation between the two based on these features, outputting the corresponding correlation score. Through this step, the degree of correlation between each image and each text fragment can be accurately quantified.

[0175] In step S403, based on the correlation score, Img is assigned to each image. n Select one or more text fragments with the highest relevance and assign them as the specific descriptive text (Desc) corresponding to the image. m Dedicated descriptive text accurately reflects the core semantics of an image and is the text information most closely associated with the image. This method enables a preliminary fine-grained matching between the image and the text.

[0176] In step S404, each image Img n Its corresponding exclusive description text Desc m Both are used as input to the visual language model. The visual language model combines the visual features of the image with the semantic features of the specific descriptive text to generate an image vector V that accurately represents the semantics of the image and is finely aligned with the text. img-m This vector avoids interference from irrelevant text fragments and can more accurately reflect the true semantics of the image and its specific relationship with the context.

[0177] In step S405, all fine-grained aligned image vectors (V) corresponding to the parent title block slice are processed. img-1 V img-2 ,...,V img-n Perform mean pooling. By calculating the average of the corresponding dimensions of all image vectors, a vector that can represent the overall semantic features of all images in the parent title block segment is obtained. This pooling result is used as the final image vector corresponding to the parent title block segment for the construction of subsequent composite vectors.

[0178] The above scheme uses a fine-grained alignment step to match unique descriptive text to each image, avoiding the noise interference introduced by using the entire text of the parent segment (containing irrelevant text fragments) to describe the image. The visual language model generates image vectors based on the precise matching relationship between the image and the unique descriptive text, ensuring that the generated image vectors accurately reflect the true semantics of the image and its specific relationship with the context, greatly improving the quality and accuracy of the image vectors.

[0179] In complex text-image scenarios such as technical documents and research reports, a parent chapter often contains multiple illustrations, each with different explanatory text. The fine-grained alignment mechanism in this application can accurately identify the unique descriptive text corresponding to each image, establishing a precise "image-text" correspondence. This effectively solves the problem of ambiguous image-text matching in traditional methods and significantly enhances the system's ability to handle complex mixed text-image scenarios.

[0180] The composite vector formed by concatenating finely aligned image vectors and text vectors exhibits higher semantic consistency and accuracy. In cross-modal retrieval, the more precise matching between the query vector and the composite vector effectively improves the relevance of search results, further optimizing cross-modal retrieval accuracy and providing users with search results that better meet their needs.

[0181] In some embodiments, the method further includes:

[0182] After generating the image vectors for all slices, perform the following steps for each slice:

[0183] Based on the text vector of this segment, the text content topic of this segment is extracted through semantic clustering. text ;

[0184] Based on all image vectors in this segment, semantic topics are extracted from the image vectors of this segment through semantic clustering. img ;

[0185] Calculate Topic text and Topic img The semantic consistency score between shards is used. When the semantic consistency score of a shard is lower than a preset threshold, cross-shard retrieval is performed, specifically including:

[0186] Calculate the semantic topic of the image vector of this segment. img Text content topics in other segments text semantic consistency score;

[0187] When the semantic topic of the image vector of this segment is... img If the similarity between the text topic of another segment and the text content of another segment exceeds a preset similarity threshold, it is determined that the image set in that segment is semantically highly related to the text content of another segment, and a hierarchical structure correction record is generated.

[0188] The hierarchical structure correction record includes: the image set identifier suspected of misattribution, the segment identifier, the identifier of another segment with semantically high relevance, and the Topic in that segment. text and Topic img The semantic consistency score between the segments, and the semantic topic of the image vectors in that segment. img Similarity to the text topic of another segment and the confidence score of attribution;

[0189] The attribution confidence score is based on the Topic in the segment. text and Topic img The semantic consistency score between the segments and the semantic topic of the image vectors of the segment. img The semantic consistency score with the text topic of another segment is calculated.

[0190] In this embodiment, semantic clustering refers to an unsupervised learning method based on vector similarity. By calculating the distance or similarity between vectors, semantically similar vectors are grouped into the same category. In this application, it is used to extract core themes from text vector sets and image vector sets respectively.

[0191] Text content topic (Topic) textThis refers to thematic information that represents the core semantics of a text segment after semantic clustering of the text segments. It is a high-level summary of the content of the text segments.

[0192] The semantic topic of an image vector refers to the topic information that can characterize the core semantics of the image in a segment after semantic clustering of all image vectors in the segment, reflecting the overall semantic features of the image within the segment.

[0193] Semantic consistency score refers to the score used to quantify the content topic of a text. text ) and semantic topics of image vectors img The index of semantic matching degree between the two topics is that the higher the value, the more consistent the topics of the text and images are. In this application, it can be obtained by calculating the cosine similarity between the two topic vectors.

[0194] Cross-segment retrieval refers to the retrieval process that compares the semantic theme of the image vector of a segment with the text content theme of all other segments when an inconsistency between the image and text themes is detected, in order to find the correct segment to which the image may belong.

[0195] Hierarchical structure correction records refer to structured data that records image-related information suspected of being misattributed and potentially correctly attributed fragment information, providing a basis for manual review or automatic correction of image attribution.

[0196] The attribution confidence score is a quantitative indicator calculated based on the semantic consistency score of the image and text topics within a segment and the similarity score of the image topic with the text topics of other segments. It is used to characterize the credibility of an image being assigned to another segment, and the higher the score, the stronger the credibility.

[0197] In real-world multimodal documents, there may be instances where images do not match the topic of their respective text segments (e.g., illustrations placed in the wrong chapter). Existing technologies lack mechanisms to detect such misattribution of images and text, leading to a decline in the quality of knowledge base data and affecting retrieval accuracy. This application addresses this issue by adding steps for verifying consistency between image and text topics and providing cross-segment correction suggestions. The specific principles are as follows:

[0198] After generating image vectors for all segments, semantic clustering is first performed on each segment based on its text vectors. The clustering algorithm groups semantically similar text vectors together, and the semantic information corresponding to the cluster centers is extracted as the text content topic for that segment. text Similarly, semantic clustering is performed on all image vectors of this segment, and the semantic information corresponding to the cluster centers is extracted as the semantic topic of the image vectors of this segment. img By extracting themes, the fragmented text and image content is abstracted into core themes, which facilitates subsequent consistency assessment.

[0199] Calculate text content topic text Image Vector Semantic Topic img The semantic consistency score between the two topic vectors is calculated by taking the cosine similarity between them. The calculated semantic consistency score is compared with a preset threshold. If the score is higher than or equal to the preset threshold, it indicates that the text and image topics within the segment are consistent and no further processing is required. If the score is lower than the preset threshold, it is determined that the images within the segment may have been misattributed, and cross-segment retrieval needs to be performed.

[0200] Then, the cross-segment retrieval process is initiated to calculate the semantic topic of the image vector for that segment. img Text content topics in all other segments text The semantic consistency score is calculated. By traversing the text topics of all other segments in the vector database, the degree of matching between the image topic and the text topic of each segment is comprehensively compared to find the possible correct segment to which the image topic belongs.

[0201] When the image vector semantic topic of this segment img When the similarity between the image set and the text topic of another segment exceeds a preset similarity threshold, it is determined that the image set within this segment is semantically highly related to the text content of the other segment, meaning that the image set may belong to the other segment. At this point, a hierarchical structure correction record is generated, including the identifier of the image set suspected of misclassification, the current segment identifier, the identifier of the other segment with semantically high relevance, and the Topic in the current segment. text and Topic img The evaluation criteria include the semantic consistency score between the images, the similarity between the image topic and the text topic of another segment, and the attribution confidence score. The attribution confidence score is calculated by weighting the semantic consistency scores of the image and text topics in the current segment and the semantic consistency scores between the image topic and the text topic of another segment, and comprehensively reflects the confidence of the image attribution correction.

[0202] This application employs a text-image topic consistency verification mechanism to automatically identify potential formatting or attribution errors in the original document (such as illustrations placed in the wrong chapter, images unrelated to the text topic, etc.), providing clear clues and basis for manual review or automatic correction. By promptly correcting these errors, the quality of the underlying data in the knowledge base is effectively improved, avoiding retrieval biases caused by data errors.

[0203] The hierarchical structure correction records contain detailed information on the text and image topics and their attribution confidence scores. This not only guides manual data correction but also provides data support for subsequent knowledge base optimization strategies. For example, by analyzing frequently occurring attribution error types in the correction records, document preprocessing and fragmentation rules can be optimized, reducing text and image attribution errors at the source.

[0204] By correcting image attribution errors, the system ensures accurate matching between images and corresponding text content, guaranteeing that the composite vector of image and text vectors accurately represents the relevant semantics. During the retrieval process, the more precise matching between the composite vector and the query vector indirectly improves the accuracy and reliability of cross-modal retrieval.

[0205] In some embodiments: semantic understanding of the image is performed to generate an image vector, specifically including:

[0206] Determine whether the text content C under the title of the parent title block segment meets the preset image description sufficiency condition. The image description sufficiency condition is configured as follows: when the length of the text content C is lower than a first length threshold, or when the text content C is judged to be insufficiently described by a pre-trained text sufficiency evaluation model, it is determined that it is not satisfied.

[0207] If the condition is not met, then based on enhanced contextual information, a hypothetical image description text C related to the semantics of the segment is generated using a text generation language model. h The enhanced context information includes at least one of the following: the hierarchical context prefix of the segment, the summary information of the multimodal document, and the topic keywords extracted from the multimodal document;

[0208] The text content C is compared with the hypothetical image description text C. h The text is then concatenated to generate an enhanced text context.

[0209] The enhanced text context and the image are input together into the visual language large model to generate the image vector.

[0210] In this embodiment, the image description sufficiency condition refers to the condition used to determine whether the text content of the parent title block segment is sufficient to support image semantic understanding. It includes two dimensions: text length condition and text quality condition. If the condition is met, it indicates that the text content can be effectively used for image semantic understanding; otherwise, context enhancement is required.

[0211] The first length threshold is a critical value used to determine whether the length of the text content meets the requirements of the image description. It is set by the user according to the actual application scenario, such as 50 characters, 100 characters, etc. When the length of the text content is lower than this threshold, it is determined that the text length is insufficient.

[0212] A text adequacy assessment model refers to a pre-trained model used to assess the adequacy of text content in describing an image. It can output the assessment results of text description adequacy based on indicators such as semantic completeness of text and relevance to the image. In this application, a pre-trained language model based on the Transformer architecture can be fine-tuned to achieve this.

[0213] Enhanced contextual information refers to auxiliary information used to supplement the generation of hypothetical image description text, including at least one of the following: the hierarchical context prefix of the segment, the summary information of the multimodal document, and the topic keywords extracted from the multimodal document, which can support the generation of description text that conforms to the segment semantics.

[0214] A text generation language model refers to a pre-trained language model with text generation capabilities. In this application, large language models such as the GPT series and LLaMA can be used to generate hypothetical image description text related to segmented semantics based on enhanced contextual information.

[0215] Hypothetical image description text (C) h Image description text (ACR) refers to image description text generated by a text generation language model based on enhanced contextual information and related to segmented semantics. It is used to supplement the deficiencies of the original text content and provide more adequate contextual support for image semantic understanding.

[0216] Enhancing text context refers to combining the original text content C with the hypothetical image description text C. h The resulting text context, after being spliced ​​together, combines the core information of the original text with supplementary descriptive information, providing richer and more accurate semantic support for image vector generation.

[0217] In practical applications, some multimodal documents may have parent title block text that is too short, vaguely described, or poorly related to the image, resulting in insufficient contextual support for image semantic understanding and inaccurate generated image vectors. This embodiment addresses this issue through a context enhancement mechanism, the specific principle of which is as follows:

[0218] For a parent title block segment containing an image, the first step is to determine whether the text content C under its title meets the preset image description sufficiency condition. This condition has two dimensions: first, text length judgment, comparing the length of text content C with a first length threshold; if the length is lower than the first length threshold, it is determined that the sufficiency condition is not met; second, text quality judgment, inputting text content C into a pre-trained text sufficiency evaluation model, which evaluates the text based on indicators such as semantic completeness and potential relevance to the image. If the evaluation result is insufficient description, it is determined that the sufficiency condition is not met. If either of the above insufficient conditions is met, the text content C is determined as not meeting the image description sufficiency condition.

[0219] If the text content C is determined not to meet the sufficiency requirements for image description, then enhanced contextual information is collected. Enhanced contextual information includes at least one of the following: the hierarchical contextual prefix of the segment (containing all ancestor title texts, reflecting the segment's hierarchical position), the summary information of the multimodal document (reflecting the overall core content of the document), and the topic keywords extracted from the multimodal document (representing the document's core theme). By collecting multi-dimensional contextual information, comprehensive support is provided for generating high-quality hypothetical image description text.

[0220] The collected enhanced contextual information is input into the text generation language model. Based on the enhanced contextual information, combined with its own semantic understanding and text generation capabilities, the model generates a hypothetical image description text C that is semantically related to the segment. h The generated C h It can supplement the deficiencies of the original text content, accurately match the theme of the segment and the potential semantics of the image, and provide effective support for image semantic understanding.

[0221] Compare the original text content C with the hypothetical image description text C. h The images are then concatenated to form an enhanced textual context. This enhanced textual context retains the effective information of the original text while supplementing necessary descriptive content, resulting in richer semantics and a better fit to the image features. The enhanced textual context and the image are then input into a large-scale visual language model. The model combines the enhanced context information to perform semantic understanding of the image and generate image vectors that accurately represent the image's semantics.

[0222] This embodiment effectively addresses situations where illustration captions in documents are insufficient, missing, or of low quality through a context enhancement mechanism. Even if the original text content cannot provide sufficient support for image semantic understanding, hypothetical image description text can be generated to supplement the context, ensuring that the image vector generation process is always guided by high-quality text. This avoids image semantic understanding failure or deviation due to insufficient contextual information, significantly improving the system's adaptability to complex image and text scenarios.

[0223] When generating hypothetical image description text, the system fully utilizes global information such as hierarchical context prefixes of image segments, document summaries, and topic keywords. This information reflects the overall theme and style of the document, making the generated hypothetical description text more consistent with the overall context of the document. This avoids situations where the description is disconnected from the document's theme and enhances the semantic consistency between local image processing and the global document content.

[0224] Image vectors generated based on enhanced text context can more accurately represent the semantic features of images and their association with document content. In cross-modal retrieval, the composite vector formed by accurate image vectors and text vectors can achieve more precise matching with query vectors, thereby further improving the accuracy and relevance of cross-modal retrieval.

[0225] For images with insufficient text descriptions, the system can automatically generate hypothetical descriptive text without the need for manual supplementation, greatly reducing the cost of manual intervention and improving the automation and efficiency of multimodal document processing. It is especially suitable for the construction of large-scale multimodal knowledge bases.

[0226] In some embodiments, the composite vector of each slice and its corresponding slice metadata are stored in a vector database, specifically including:

[0227] For each slice, its composite vector is parsed into independent text component vectors and image component vectors;

[0228] The text component vector is associated with the fragment metadata and stored as a record in the text vector sub-library of the vector database; the image component vector is associated with the fragment metadata and stored as a record in the image vector sub-library of the vector database.

[0229] Among them, text component vector records and image component vector records belonging to the same segment are associated with a unique segment identifier corresponding to that segment;

[0230] Using the query vector to perform similarity retrieval in the vector database specifically includes:

[0231] The query vector is used for similarity retrieval in the text vector sub-database to obtain a preliminary matching set of text component vectors and its corresponding first similarity score set;

[0232] The query vector is used for similarity retrieval in the image vector sub-database to obtain a preliminary matching set of image component vectors and its corresponding second similarity score set;

[0233] Based on the unique segment identifier, the set of text component vectors and the set of image component vectors are associated. For a segment that appears in any preliminary matching set, its corresponding component vector and its similarity score are retrieved and associated from another sub-database to ensure that complete text and image component vector information and corresponding scores are recalled for each matching segment.

[0234] For each associated segment, a comprehensive similarity score is calculated based on the first similarity score of the text component vector and the second similarity score of the image component vector of the segment, using a preset score fusion rule. All matching segments are then sorted according to the comprehensive similarity score.

[0235] The text component vectors, image component vectors, and their complete metadata corresponding to the sorted segments are integrated to generate a sorted preliminary matching composite vector set.

[0236] In some embodiments, a text component vector refers to the vector portion obtained after parsing the fragmented composite vector that only represents the semantic features of the text. It is an important component of the composite vector and is directly related to the text content.

[0237] Image component vectors refer to the vector parts that represent only the semantic features of the image after parsing the composite vectors of the slices. They are another important component of the composite vectors and are directly related to the image content.

[0238] A text vector sub-database refers to a sub-database within a vector database specifically designed to store text component vectors and their corresponding fragment metadata. It possesses efficient text vector similarity retrieval capabilities, providing support for text portion retrieval in both pure text queries and mixed-modal queries.

[0239] The image vector sub-database refers to a sub-database in the vector database specifically used to store image component vectors and corresponding fragment metadata. It has efficient image vector similarity retrieval capabilities and provides support for image partial retrieval in pure image queries or mixed modality queries.

[0240] A unique fragment identifier is a string or number sequence used to uniquely identify each fragment. It is the core identifier that associates text component vector records and image component vector records of the same fragment, ensuring that different component vectors of the same fragment can be accurately associated during the retrieval process.

[0241] The first similarity score set refers to the set of similarity scores obtained after performing similarity retrieval on the text vector sub-database using the query vector, corresponding to the initially matched text component vectors. The higher the score, the higher the matching degree between the text component vector and the query vector.

[0242] The second similarity score set refers to the set of similarity scores obtained after performing similarity retrieval on the image vector sub-database using the query vector, corresponding to the initially matched image component vectors. The higher the score, the higher the matching degree between the image component vector and the query vector.

[0243] The score fusion rule refers to the rule used to merge the first similarity score and the second similarity score of the same segment into a comprehensive similarity score. It can be set according to the actual application scenario, such as the weighted average rule, the maximum value rule, the minimum value rule, etc. In this application, the weighted average rule is preferred.

[0244] The comprehensive similarity score is the score obtained by fusing the first similarity score and the second similarity score of the same segment according to the score fusion rule. It comprehensively reflects the overall matching degree between the segment and the query vector and is the core basis for ranking the search results.

[0245] This embodiment improves the flexibility and accuracy of retrieval through an optimized scheme that combines database sharding and score fusion. The specific principle is as follows:

[0246] For each slice, its composite vector is parsed into independent text component vectors and image component vectors. The text component vectors contain only text semantic features, and the image component vectors contain only image semantic features. The text component vector is associated with the metadata of that slice and stored as a record in the text vector sub-database of the vector database; simultaneously, the image component vector is associated with the metadata of that slice and stored as a record in the image vector sub-database of the vector database. Text component vector records and image component vector records within the same slice are associated using a unique slice identifier, ensuring rapid location of both types of component vectors within the same slice during subsequent retrieval. This sharded storage allows text vectors and image vectors to employ their respective optimal indexing strategies and storage structures, improving storage efficiency and retrieval performance.

[0247] Upon receiving user query information and vectorizing it into a query vector, a multi-sub-database parallel retrieval process is initiated. On one hand, the query vector is input into the text vector sub-database for similarity retrieval. The similarity between the query vector and all text component vectors in the text vector sub-database is calculated, and several text component vectors with high similarity are selected to form a preliminary matching set of text component vectors. The corresponding first similarity score set is recorded. On the other hand, the query vector is input into the image vector sub-database for similarity retrieval. The similarity between the query vector and all image component vectors in the image vector sub-database is calculated, and several image component vectors with high similarity are selected to form a preliminary matching set of image component vectors. The corresponding second similarity score set is recorded. Parallel retrieval fully utilizes the retrieval capabilities of both the text vector sub-database and the image vector sub-database, improving retrieval efficiency.

[0248] Based on unique segment identifiers, the initially matched sets of text component vectors and image component vectors are associated and integrated. For segments appearing in the text component vector set, their corresponding image component vectors and their second similarity scores are retrieved from the image vector sub-database using the unique segment identifier; for segments appearing in the image component vector set, their corresponding text component vectors and their first similarity scores are retrieved from the text vector sub-database using the unique segment identifier. Through this association and integration, it is ensured that the text component vectors, image component vectors, and their corresponding similarity scores for each matched segment are completely retrieved, avoiding information loss.

[0249] For each piece that has completed association and integration, a comprehensive similarity score is calculated based on a preset score fusion rule (such as a weighted average rule). For example, if the weight of the text component vector is set to 0.6 and the weight of the image component vector is set to 0.4, then the comprehensive similarity score = first similarity score × 0.6 + second similarity score × 0.4. All matching pieces are then sorted in descending order based on the calculated comprehensive similarity score, with pieces having higher comprehensive similarity scores appearing earlier in the order. Finally, the text component vectors, image component vectors, and their complete metadata corresponding to the sorted pieces are integrated to generate a sorted preliminary matching composite vector set, providing high-quality input for subsequent hierarchical aggregation.

[0250] The above solution, through database sharding and parallel retrieval mechanisms, can flexibly handle pure text queries, pure image queries, and mixed-modal queries. For pure text queries, relevant segments can be quickly retrieved through the text vector sub-database, and their image component vector information can be retrieved accordingly, providing users with richer results. For pure image queries, relevant segments can be accurately matched through the image vector sub-database, and their text component vector information can be retrieved accordingly, achieving cross-modal retrieval from image to text. For mixed-modal queries, the retrieval capabilities of both text and image sub-databases can be utilized simultaneously to comprehensively match relevant segments, significantly improving the flexibility and comprehensiveness of query response.

[0251] Database sharding allows text vectors and image vectors to employ their own dedicated indexing optimization strategies. For example, text vectors use semantic-based indexes, while image vectors use visual feature-based indexes, improving the retrieval efficiency of single-type vectors. Simultaneously, the parallel retrieval mechanism reduces overall retrieval time and improves system response speed. Furthermore, the comprehensive similarity score integrates matching information from both text and image dimensions, more comprehensively and accurately reflecting the relevance of shards to the query, resulting in more reasonable ranking of search results and improving the accuracy of search results and user experience.

[0252] The text vector sub-database and image vector sub-database are independent yet linked by a unique shard identifier, resulting in a clear and concise storage structure. This structure facilitates the separate maintenance and updating of text and image vectors, such as optimizing text vector indexing strategies individually or updating image vector data in batches, thus reducing system maintenance costs. Furthermore, the sharded storage supports horizontal scaling of both the text and image vector sub-databases, allowing for flexible enhancement of storage and retrieval capabilities for single-type vectors based on business needs, thereby improving system scalability.

[0253] Database sharding allows for the allocation of storage resources based on the different characteristics of text and image vectors, avoiding resource waste caused by mixed storage. For example, for a knowledge base with a high proportion of text vectors, more storage resources can be allocated to the text vector sub-database; for a knowledge base with a high proportion of image vectors, the resource allocation of the image vector sub-database can be optimized to achieve rational utilization of storage resources.

[0254] like Figure 5 As shown. Following step S2, the process further includes a step of semantically refining the parent title block fragments, specifically including:

[0255] S501: For the parent title block segment, perform semantic segmentation point detection on the text content it contains, identify the boundary positions where the topic changes significantly, and obtain the semantic boundary.

[0256] S502: Based on the detected semantic boundaries, the parent title block segment is divided into multiple semantically coherent micro-block segments, wherein each micro-block segment inherits the hierarchical context prefix of the parent title block segment and generates a derived title based on the title of the parent title block segment and the sub-topic content corresponding to the micro-block segment.

[0257] S503: The image in the parent title block segment is reassigned to the corresponding micro-block segment according to the semantic relationship between the image and the text content in the micro-block segment;

[0258] After the semantic refinement step is completed, the micro-block fragment replaces its original parent title block fragment and serves as the execution unit for subsequent steps S3 to S8.

[0259] In this embodiment, semantic segmentation point detection refers to the process of identifying the boundary locations where significant topic shifts occur within the text content of the parent title block segment using natural language processing techniques. Semantic segmentation points are key nodes for topic switching in the text content, and in this application, techniques such as sentence similarity and topic models can be used to achieve this.

[0260] Semantic boundaries refer to the boundary positions that divide text paragraphs of different topics, identified by semantic segmentation point detection. Each semantic boundary corresponds to a topic transition point, dividing the text content of the parent title block into multiple semantically coherent parts.

[0261] Micro-block fragmentation refers to small, highly semantically coherent fragments obtained by segmenting the parent title block fragments based on semantic boundaries. Each micro-block fragment corresponds to a specific subtopic and is the basic execution unit for subsequent processing.

[0262] Derivative titles are unique titles generated for each micro-block segment. They are generated based on the title of the parent title block segment and the corresponding sub-topic content of the micro-block segment. They can accurately reflect the core content of the micro-block segment. For example, if the parent title is "3.1 System Working Principle", the derivative title of a certain micro-block segment can be "3.1-a System Initialization Principle".

[0263] The execution unit refers to the basic unit used in subsequent steps S3 to S8 for operations such as generating hierarchical context information, vectorization, storage, and retrieval. In this application, after semantic refinement is completed, micro-block fragments replace the original parent title block fragments as the execution unit.

[0264] Segmentation based solely on the title level may result in some parent title blocks having excessively long content and containing multiple subtopics, leading to a "mixed topic within a block" problem. This results in inaccurate vector representations and the return of irrelevant content during retrieval. This embodiment achieves adaptive optimization of segmentation granularity by semantically refining the parent title block segments. The specific principle is as follows:

[0265] For each parent title block segment, semantic segmentation point detection is first performed on its contained text content. An algorithm based on inter-sentence similarity and topic models is used to calculate the semantic similarity between adjacent sentences or paragraphs. When the similarity is below a preset threshold, it is determined that a significant topic shift has occurred, and this location is the semantic segmentation point. By traversing the entire text content of the parent title block segment, all semantic segmentation points are identified, and the corresponding semantic boundaries are obtained. The detection of semantic segmentation points ensures the semantic coherence within the subsequently segmented micro-block segments.

[0266] Then, based on the detected semantic boundaries, the parent title block segment is divided into multiple semantically coherent micro-block segments. Each micro-block segment corresponds to an independent sub-topic, with highly consistent semantic content within its text, avoiding the problem of mixed topics within the original parent title block segment. Simultaneously, each micro-block segment inherits the hierarchical context prefix of the parent title block segment, ensuring that its hierarchical position information is not lost; and a derived title is generated based on the title of the parent title block segment and the content of the corresponding sub-topic of the micro-block segment. This derived title accurately summarizes the core content of the micro-block segment, facilitating user understanding and rapid identification during retrieval.

[0267] Then, for the images contained in the original parent title block segment, they are reassigned to the corresponding micro-block segments based on the semantic relationship between the images and the text content in each micro-block segment. By calculating the semantic similarity between the image vector and the text vector of each micro-block segment, the image is assigned to the micro-block segment with the highest semantic similarity, ensuring that each image is bound to the most relevant text content, thus achieving accurate matching of images and text.

[0268] After completing the micro-block partitioning, derived title generation, and image reallocation, the original parent title block partition is replaced by a micro-block partition, serving as the execution unit for subsequent steps S3 to S8. Subsequent operations such as hierarchical context information generation, text and image vectorization, composite vector storage, query retrieval, and aggregation are all based on micro-block partitioning, ensuring that all processing flows can utilize the semantically refined and precisely partitioned data.

[0269] This application uses semantic refinement to divide lengthy and thematically complex parent title blocks into multiple semantically singular micro-blocks, achieving adaptive adjustment of the block granularity. Compared to fixed-length blocks, the granularity of micro-blocks is determined by the semantic structure of the text content, enabling it to adapt to mixed text and image content of varying complexity. This avoids problems of overly fine fragmentation or information redundancy, allowing each processing unit to have highly focused content.

[0270] Micro-block fragments are semantically coherent and thematically focused. The text and image vectors generated from them can more accurately represent the semantic features of the corresponding sub-topics, and the composite vectors have stronger semantic targeting. During the retrieval process, the query vector can achieve more precise matching with the composite vectors of the micro-block fragments, thereby more accurately locating the specific sub-topics that users care about, avoiding the return of large sections of irrelevant content, and significantly improving the accuracy of the retrieval.

[0271] By reassigning images from the original parent title block to corresponding micro-blocks based on semantic association, it is ensured that each image is bound to the most relevant text content. This precise image-text matching relationship makes image vector generation more targeted, and at the same time, it enables accurate association between images and corresponding text subtopics during retrieval, further improving the effectiveness of cross-modal retrieval.

[0272] User queries can be matched with the most semantically relevant micro-blocks, and the returned search results are no longer broad chapter content, but precise information fragments focused on specific sub-topics. These fragments retain derived headings and hierarchical context information, with a clear structure and specific content, greatly improving the granularity and usability of search results, making it convenient for users to quickly obtain the information they need. This is especially suitable for document retrieval scenarios with high degree of structure and rich sub-topics, such as technical manuals and research reports.

[0273] In other embodiments, the method of this application further includes: deploying a monitoring agent in the vector database and retrieval service to periodically collect the aforementioned indicator data, providing a basis for scheduling decisions. Real-time monitored shard load indicators include shard storage space usage, number of retrieval requests per unit time, retrieval response time, CPU and memory utilization, etc.; retrieval hotspot indicators include shard retrieval frequency ranking, continuous retrieval duration, and the matching popularity of user query keywords with shard topics, etc.

[0274] Then, based on the collected monitoring data, scheduling decisions are made using load balancing and hotspot identification algorithms. The load balancing algorithm calculates the load balance of each node, and triggers shard migration or splitting when the load of a node exceeds a preset threshold. The hotspot identification algorithm identifies hotspot shards by statistically analyzing their retrieval frequency and popularity, migrating them to nodes with better performance or splitting them to distribute retrieval pressure. Simultaneously, scheduling strategies are adjusted based on time dimensions (such as peak hours on weekdays and off-peak hours at night) and business tags (such as core business shards and general business shards), prioritizing the allocation of high-quality resources to core business shards.

[0275] After the scheduling decision is generated, the system performs shard migration, splitting, or merging operations. During shard migration, shard data is migrated from the original node to the target node through incremental synchronization, keeping the original shards available during the migration process. After the migration is completed, the routing address is switched to ensure uninterrupted service. During shard splitting, shards with excessive load or excessively long content are split into multiple sub-shards according to semantic relevance and stored on different nodes. During shard merging, multiple shards with extremely low retrieval frequency and strong content relevance are merged into one shard to save storage resources.

[0276] In some embodiments, the system also supports efficient matching for cross-modal retrieval. Specifically, it establishes independent index paths for text and images through bidirectional indexing, enabling text queries to quickly locate relevant segments through the text vector index, and image queries to directly match target segments through the image vector index, without traversing all composite vectors, thus greatly improving the matching efficiency of cross-modal retrieval. Especially in large-scale vector databases, bidirectional indexing can significantly reduce retrieval time and improve system response speed.

[0277] The text vector index and image vector index of the bidirectional index are independent of each other. When a text vector or an image vector needs to be updated, the corresponding index can be maintained separately without rebuilding the entire index, reducing the cost and complexity of index maintenance. For example, when the text content is updated, only the text vector index needs to be updated, while the image vector index remains unchanged, ensuring the efficient operation of the system.

[0278] By combining hierarchical path identifiers, business tags, and original file paths from the shard metadata, bidirectional indexes can support multi-dimensional search filtering. For example, users can specify to search for relevant shards under a certain business tag, or limit the search to a certain document level. Through joint filtering of the index and metadata, the accuracy and relevance of search results can be further improved.

[0279] In a second aspect, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the knowledge base parent-child segmentation method based on graph-text composite vectors as described in the first aspect of this application.

[0280] The computer-readable storage medium may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory.

[0281] The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD ROM); the magnetic surface memory may be a disk storage device or a magnetic tape storage device.

[0282] The volatile memory may be random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synclink dynamic random access memory (SLDRAM), and direct memory bus random access memory (DRRAM). The computer-readable storage media described in the embodiments of this application are intended to include these and any other suitable types of memory.

[0283] like Figure 6 As shown, in a third aspect, this application provides an electronic device 10, including a processor 101 and a storage medium 102, wherein a computer program is stored on the storage medium, and when the computer program is executed by the processor, it implements the knowledge base parent-child sharding method based on graph-text composite vector as described in the first aspect of this application.

[0284] In some embodiments, the processor may be implemented by software, hardware, firmware, or a combination thereof, and may be a circuit, one or more of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field-programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, or a microprocessor, thereby enabling the processor to execute some or all of the steps or any combination thereof in the knowledge base parent-child fragmentation method based on graph-text composite vectors described in the various embodiments of this application.

[0285] Finally, it should be noted that although the above embodiments have been described in the text and drawings of this application, this should not limit the scope of patent protection of this application. Any technical solutions that are based on the essential concept of this application and utilize the content described in the text and drawings of this application, resulting in equivalent structural or procedural substitutions or modifications, as well as the direct or indirect application of the technical solutions of the above embodiments to other related technical fields, are all included within the scope of patent protection of this application.

Claims

1. A knowledge base parent-child fragmentation method based on graph-text composite vectors, characterized in that, The method includes: S1: Obtain the multimodal document to be processed, and convert the multimodal document into a structured document that retains the original title hierarchy. The multimodal document contains at least text and images. S2: Based on the title hierarchy of the structured document, the document content is divided into multiple segments, and a parent-child hierarchy relationship is established for each segment to generate parent title block segments and child title block segments. Each segment corresponds to a title and its subordinate continuous content. S3: Generate hierarchical context information for each segment, specifically including: for each segment, if it has ancestor segments, concatenate the title text of all its ancestor segments to generate the hierarchical context prefix of the segment, wherein the ancestor segments are all the parent segments of the segment up to the document root segment; for the parent title block segment containing the image, use its own hierarchical context prefix and the text content under its title as the context to perform semantic understanding on the image and generate an image vector; S4: Vectorize the text content of each segment to generate a text vector; for parent title block segments containing images, concatenate the image vector and text vector corresponding to the parent title block segment to generate a composite vector of the parent title block segment; for subtitle block segments and parent title block segments that do not contain images, use their text vectors directly as the composite vector of the segment. S5: Store the composite vector of each fragment and its corresponding fragment metadata in the vector database. The fragment metadata includes at least a hierarchical path identifier for identifying the position of the fragment in the document hierarchy. S6: Receive user query information, vectorize the query information into query vectors, use the query vectors to perform similarity retrieval in the vector database, and obtain a preliminary matching composite vector set; S7: Based on the hierarchical path identifiers corresponding to each composite vector in the preliminarily matched composite vector set, perform hierarchical aggregation and sorting on the search results to generate a structured search context; S8: Return the structured search context to the user.

2. The knowledge base parent-child segmentation method based on graph-text composite vectors as described in claim 1, characterized in that, Its own hierarchical context prefix and the text content under its heading serve as context, specifically including: For the current parent title block segment to be processed, extract the ancestor title text of each level in its hierarchical context prefix to form an ancestor title sequence [T1,T2,...,T]. k ], where T1 is the top-level ancestor title, T k Set the title of the subsection for the parent title block; Retrieve the text content C under the parent title block title, and compare the text content C with each ancestor title T in the ancestor title sequence. i Each input to a relevance scoring model yields a set of relevance scores [S1, S2, ..., S]. k ], where i takes the value [1, k]; The correlation scores [S1,S2,...,S] are used to... k Input a weight assignment function to generate a set of normalized dynamic weights [W1, W2, ..., W...]. k ], where the dynamic weight W i Correlation score S i Positive correlation; The corresponding ancestor title texts are weighted and fused using the dynamic weights to generate a weighted ancestor title context C. ancestor The calculation formula is: C ancestor =Σ(W i ×Embed(T i ), where Embed(·) is the text embedding function; The weighted ancestor title context C ancestor This is concatenated with the text content C to form the final weighted context C. final The calculation formula is as follows: C final =Concat(C ancestor ,C), where Concat(·) represents the concatenation function.

3. The knowledge base parent-child segmentation method based on graph-text composite vectors as described in claim 1, characterized in that, The following steps are included after step S5: After the vector database is stored, all fragments are traversed to identify the set of child fragments with the same direct parent header block fragment identifier; For each pair of image vectors V in the sub-segment set i and V j Calculate the semantic similarity or relationship category between the two, where the relationship category includes continuous illustrations, contrast charts, or unrelated illustrations; The calculated semantic relationships are stored as cross-shard association metadata in the form of a relationship graph or a relationship list. In the index of the vector database, backlinks are established for image vectors with strong semantic associations. This allows the retrieval of an image vector to be directly located through the backlinks when an image vector is found. Then, based on the fragment metadata stored in the vector database that is bound to each image vector, the fragment to which these associated image vectors belong can be determined.

4. The knowledge base parent-child segmentation method based on graph-text composite vectors as described in claim 1, characterized in that, The image is semantically understood to generate an image vector, specifically including the following fine-grained alignment steps: For a parent title block slice containing at least one image, the text content C under the title of the parent title block slice is divided into several text segments [Seg1, Seg2, ..., Seg...]. m ]; For each image Img in the parent title block segment n Using a visual-language alignment model, the image is computed to match each text segment Seg. p The correlation score, where p ranges from [1, m]; Img per image n Assign one or more text fragments with the highest relevance as the unique descriptive text (Desc) for the image. m ; Image Img n The specific descriptive text Desc corresponding to this image m A large visual language model is input together to generate a fine-grained aligned image vector V. img-m ; Perform mean pooling on all fine-grained aligned image vectors corresponding to the parent title block slice, and use the pooling result as the image vector corresponding to the parent title block slice.

5. The knowledge base parent-child segmentation method based on graph-text composite vectors as described in claim 1, characterized in that, The method further includes: After generating the image vectors for all slices, perform the following steps for each slice: Based on the text vector of this segment, the text content topic of this segment is extracted through semantic clustering. text ; Based on all image vectors in this segment, semantic topics are extracted from the image vectors of this segment through semantic clustering. img ; Calculate Topic text and Topic img The semantic consistency score between shards is used. When the semantic consistency score of a shard is lower than a preset threshold, cross-shard retrieval is performed, specifically including: Calculate the semantic topic of the image vector of this segment. img Text content topics in other segments text semantic consistency score; When the semantic topic of the image vector of this segment is... img If the similarity between the text topic of another segment and the text content of another segment exceeds a preset similarity threshold, it is determined that the image set in that segment is semantically highly related to the text content of another segment, and a hierarchical structure correction record is generated. The hierarchical structure correction record includes: the image set identifier suspected of misattribution, the segment identifier, the identifier of another segment with semantically high relevance, and the Topic in that segment. text and Topic img The semantic consistency score between the segments, and the semantic topic of the image vectors in that segment. img Similarity to the text topic of another segment and the confidence score of attribution; The attribution confidence score is based on the Topic in the segment. text and Topic img The semantic consistency score between the segments and the semantic topic of the image vectors of the segment. img The semantic consistency score with the text topic of another segment is calculated.

6. The knowledge base parent-child fragmentation method based on graph-text composite vectors as described in claim 1, characterized in that, Perform semantic understanding on the image to generate an image vector, specifically including: Determine whether the text content C under the title of the parent title block segment meets the preset image description sufficiency condition. The image description sufficiency condition is configured as follows: when the length of the text content C is lower than a first length threshold, or when the text content C is judged to be insufficiently described by a pre-trained text sufficiency evaluation model, it is determined that it is not satisfied. If the condition is not met, then based on enhanced contextual information, a hypothetical image description text C related to the semantics of the segment is generated using a text generation language model. h The enhanced context information includes at least one of the following: the hierarchical context prefix of the segment, the summary information of the multimodal document, and the topic keywords extracted from the multimodal document; The text content C is compared with the hypothetical image description text C. h The text is then concatenated to generate an enhanced text context. The enhanced text context and the image are input together into the visual language large model to generate the image vector.

7. The knowledge base parent-child segmentation method based on graph-text composite vectors as described in claim 1, characterized in that, The composite vector of each slice and its corresponding slice metadata are stored in the vector database, specifically including: For each slice, its composite vector is parsed into independent text component vectors and image component vectors; The text component vector is associated with the fragment metadata and stored as a record in the text vector sub-library of the vector database; the image component vector is associated with the fragment metadata and stored as a record in the image vector sub-library of the vector database. Among them, text component vector records and image component vector records belonging to the same segment are associated with a unique segment identifier corresponding to that segment; Using the query vector to perform similarity retrieval in the vector database specifically includes: The query vector is used for similarity retrieval in the text vector sub-database to obtain a preliminary matching set of text component vectors and its corresponding first similarity score set; The query vector is used for similarity retrieval in the image vector sub-database to obtain a preliminary matching set of image component vectors and its corresponding second similarity score set; Based on the unique segment identifier, the set of text component vectors and the set of image component vectors are associated. For a segment that appears in any preliminary matching set, its corresponding component vector and its similarity score are retrieved and associated from another sub-database to ensure that complete text and image component vector information and corresponding scores are recalled for each matching segment. For each associated segment, a comprehensive similarity score is calculated based on the first similarity score of the text component vector and the second similarity score of the image component vector of the segment, using a preset score fusion rule. All matching segments are then sorted according to the comprehensive similarity score. The text component vectors, image component vectors, and their complete metadata corresponding to the sorted segments are integrated to generate a sorted preliminary matching composite vector set.

8. The knowledge base parent-child segmentation method based on graph-text composite vectors as described in claim 1, characterized in that, Following step S2, the method further includes a step of semantically refining the parent title block fragments, specifically including: For the parent title block segment, semantic segmentation point detection is performed on the text content it contains to identify the boundary positions where the topic changes significantly, and thus obtain the semantic boundary. Based on the detected semantic boundaries, the parent title block segment is divided into multiple semantically coherent micro-block segments. Each micro-block segment inherits the hierarchical context prefix of the parent title block segment and generates a derived title based on the title of the parent title block segment and the sub-topic content corresponding to the micro-block segment. The images in the parent title block are reassigned to the corresponding micro-blocks based on the semantic relationship between the images and the text content within the micro-blocks. After the semantic refinement step is completed, the micro-block fragment replaces its original parent title block fragment and serves as the execution unit for subsequent steps S3 to S8.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the knowledge base parent-child sharding method based on graph-text composite vectors as described in any one of claims 1 to 8.

10. An electronic device having a computer program stored thereon, characterized in that, It includes a processor and a storage medium, wherein a computer program is stored on the storage medium, and when the computer program is executed by the processor, it implements the knowledge base parent-child sharding method based on graph-text composite vector as described in any one of claims 1 to 8.

Images