A unified vectorization construction method and system for multimodal media knowledge base
By incorporating multimodal media resources into a unified vector space and using semantic segmentation and a large visual model to generate visual semantic descriptions, the problems of fragmented cross-modal retrieval and incomplete semantic segments in the construction of a multimodal media knowledge base are solved, achieving high efficiency and reliability in text query hits and database processing of pure visual content.
Patent Information
- Application Number
- CN202611248835.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-18
- Publication Date
- 2026-11-10
AI Technical Summary
Existing technologies for constructing multimodal media knowledge bases suffer from problems such as fragmented cross-modal retrieval, inability of purely visual content to be hit by text queries, incomplete semantic segmentation, and high cost of recovery from database entry processing failures.
By incorporating four types of media resources—video, audio, images, and documents—into a unified vector space, semantic segmentation and a large visual model are used to generate visual semantic descriptions. Role prefixes are defined and role-labeled embedded text is constructed. A unified input single text embedding model is used to achieve cross-modal unified recall. A two-stage pipeline of concurrent prefetching and serial data entry is used for processing.
This enables text queries to find purely visual content, improves the completeness of search results and the efficiency and reliability of data entry, and reduces system complexity and maintenance costs.
Smart Images

Figure CN123068660A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data management technology, and in particular to a unified vectorization construction method and system for a multimodal media knowledge base. Background Technology
[0002] With the continuous improvement of the digitalization of integrated media content production and media asset management, thematic knowledge bases have become the infrastructure for accumulating historical materials and supporting intelligent question answering and precise retrieval. The four types of heterogeneous media resources—video, audio, images, and documents—continue to grow, and users expect to retrieve relevant content in one go based on natural language descriptions, making cross-modal retrieval a growing demand.
[0003] In existing technologies, the construction of media knowledge bases typically involves deploying independent embedding models and independent vector indexes for different modalities; that is, video, audio, images, and documents each employ their own dedicated processing and indexing chains. For textual information in visual content, most solutions rely on optical character recognition (OCR) engines to extract text before vectorizing it. Some solutions introduce contrast alignment models based on large-scale pre-trained image-text pairs to attempt to align the representations of images and text. The material slicing stage generally employs fixed-duration segmentation or segmentation based on simple punctuation marks.
[0004] The aforementioned existing technologies have the following shortcomings in practical applications: First, the vector indexes of each modality are independent of each other, and the system complexity and operation and maintenance costs increase linearly with the increase of the number of modalities. It is difficult for users to recall relevant content of different modalities such as video, audio, images, and documents in a single query, especially for needs such as "finding silent video footage using text descriptions". At the same time, the general image and text alignment model is not well adapted to long-tail scenarios such as layout, tables, and scanned documents that are common in the Chinese media field.
[0005] Secondly, a large amount of material falls into the category of "rich in visual information but lacking textual content," such as video clips without audio narration, scanned documents, and presentations primarily consisting of charts and layouts. This type of content lacks textual entry points for matching with search queries; key visual information such as chart trends, layouts, and diagrams are lost during the retrieval process and cannot be found by users using natural language queries.
[0006] Third, fixed-duration segmentation can easily truncate complete semantic units, while segmentation based on simple punctuation can generate a large number of text fragments that are too short. The semantic expression of vectors generated by such fragments is weak, which not only makes it difficult to match the user's query intent, but also introduces noise and reduces the overall retrieval accuracy.
[0007] Fourth, the process of importing multimodal materials into the database is time-consuming. When a failure occurs during the process, the cost of recovery and retry is high, and the overall efficiency and reliability of the import process need to be improved.
[0008] In summary, there is an urgent need for a unified vectorized construction method and system for multimodal media knowledge bases to solve the technical problems existing in the construction of multimodal media knowledge bases, such as fragmented cross-modal retrieval, inability of purely visual content to be hit by text queries, incomplete semantic segments, and high cost of recovery from failures in the data entry process. Summary of the Invention
[0009] To address the aforementioned technical issues, this invention provides a unified vectorization construction method and system for a multimodal media knowledge base. The aim is to incorporate four types of media resources—video, audio, images, and documents—into a unified vector space, enabling unified cross-modal retrieval in a single query. This also improves the semantic quality of segments and the robustness and efficiency of data entry processing. Ultimately, this ensures segment quality during the construction of the multimodal media knowledge base, allows purely visual content to be effectively retrieved by text queries, enhances the completeness of search results, and balances the efficiency and reliability of data entry processing.
[0010] To achieve the above objectives, this application proposes a unified vectorization construction method for multimodal media knowledge bases, comprising the following steps: It receives four types of media resources: video, audio, images, and documents. It routes the resources to the corresponding processing services according to their types, obtains at least one type of related information from speech recognition text, scene analysis results, image addresses, or document files, and uniformly registers them as knowledge base document records and marks them as pending processing status. The media resources are semantically segmented according to their resource types to generate semantically complete logical slices. The logical slices are then associated with the knowledge base document records, and the slice type is recorded. Among them, audio and video resources are determined to be compact or semantic based on the statistical characteristics of speech recognition text, document resources are divided according to their document structure, and image resources are treated as a whole image as a logical slice. Call the large visual model on the logical slice containing visual content to generate a visual semantic description, and convert the pure visual content into text; Four types of role prefixes are defined: screen role prefix, voice role prefix, visual role prefix and text role prefix. Role-labeled embedded text with role prefixes is constructed according to the modality to which the logical slice belongs. The role-labeled embedded text of all modalities is uniformly input into a single text embedding model and mapped to a unified vector space to obtain the dense vector of each logical slice in the unified vector space. For each logical slice, establish a display field and a description field. The display field stores text for front-end display and reference tracing, and the description field stores the visual semantic description generated by the visual big model. Write the dense vector into the vector field. The visual semantic description generation, vectorization, and index writing are executed using a two-stage pipeline of concurrent prefetching and serial ingestion. The visual semantic description generation is executed concurrently in stage one, while the vectorization and index writing are executed serially in stage two, and the execution is resumed from the breakpoint by a state machine.
[0011] As a further solution, the embedded text for the role label is constructed according to the modality to which the logical slice belongs as follows: The embedded text of the role label in the video logical slice consists of a text segment obtained by splicing the role prefix in the picture with the visual semantic description of the video picture, and a text segment obtained by splicing the speech role prefix with the speech recognition text. The embedded text for the role label of the audio logic slice is composed of the speech role prefix and the speech recognition text. The embedded text for the role label of the image logical slice is composed of a visual role prefix and a visual semantic description of the image; The embedded text of the role label in the document's logical slice consists of a text segment obtained by concatenating the main text role prefix with the main text, and a text segment obtained by concatenating the visual role prefix with the visual semantic description of the document page. Among them, the screen role prefix is used to annotate the visual semantic description of the video screen, the voice role prefix is used to annotate the speech recognition text of the audio and video, the visual role prefix is used to annotate the visual semantic description of the image and document page, and the text role prefix is used to annotate the document extraction text and optical character recognition text; the text corresponding to the screen role prefix and the visual role prefix is taken from the description field, and the text corresponding to the voice role prefix and the text role prefix is taken from the display field. When the description field is empty, the role annotation embedded text is constructed only from the content of the display field. The role prefix serves as a semantic soft cue for modality awareness, explicitly declaring the semantic role and source modality of the corresponding text segment to the single text embedding model.
[0012] As a further solution, the character annotation embedded text is mapped into an N-dimensional dense vector by the single text embedding model according to the embedding function E; Where E = Embed(T_role), and T_role is the embedded text of the role annotation; After the user query text is encoded by the single text embedding model, the cosine similarity between it and the dense vector of each logical slice is calculated using the similarity function sim(Q,E) to achieve cross-modal unified recall. Where sim(Q,E)=(Q·E) / (‖Q‖·‖E‖); Q is the vector obtained by encoding the user query text through the single text embedding model, and E is the dense vector of the corresponding logical slice.
[0013] As a further solution, the display field stores speech recognition text or document extraction text with timestamps; wherein, the speech recognition text is stored one by one according to timestamps, and each timestamp corresponds to the speech recognition text of the corresponding time period. The display field is used for front-end display and retrieval enhancement during the generation of reference traceability. It also includes: generating independent image vectors for keyframes and scene frames for video logical slices and storing them in an image vector field family; synchronizing the dense vectors of the logical slices to the nested vector field of the search engine for precise time interval retrieval.
[0014] As a further solution, purely visual content can be converted into text through the following steps: For video logical slices, call external segmentation services to perform physical slicing and extract key frames and scene frames, and call the visual big model to generate visual semantic descriptions for each physical slice. For logical slices of an image, the visual big model is invoked to generate a visual semantic description of the image; wherein, when the image file exceeds a preset size, it is first compressed to not exceed the preset size before the visual big model is invoked; For logical slices of the document, execute the document page rendering pipeline: If the document logical slice originates from a document saved in an office suite format, the office software running in headless mode is first called to convert it into a PDF intermediate format. If it originates from a PDF document, the PDF document is directly used as the rendering object. Then, a PDF rendering library is used to render each page of the PDF document into an independent page PNG image at a preset resolution. Finally, the visual big model is called to generate a visual semantic description for each page PNG image.
[0015] As a further solution, the large visual model simultaneously performs the functions of optical character recognition and visual semantic description: For each page's PNG image, the visual big model describes the chart data trends, layout, and schematic content, generating a visual semantic description; When the number of characters in the PDF text layer of the page is less than a preset threshold, the page is determined to be a scanned page, and the optical character recognition of the visual big data model is triggered to intelligently recognize the visible text in the page, and the optical character recognition text is used as the main text. When the number of characters in the PDF text layer of the page is not less than the preset threshold, the text extracted from the PDF text layer is used as the main text; at most one logical slice record is generated per page. For scanned pages: when the visual semantic description function succeeds but the optical character recognition function fails, only the visual semantic description is written to the knowledge base, and the logical slice is marked as lacking optical character recognition text; When the optical character recognition function succeeds but the visual semantic description function fails, only the optical character recognition text is written into the knowledge base, and the logical slice is marked as lacking visual semantic description. If both functions fail, the page PNG image will not generate a logical slice record, and the failed page number will be recorded.
[0016] As a further solution, semantic segmentation of the media resources is performed according to resource type through the following steps: For audio and video resources: Statistical analysis of the total number of characters in the full text of speech recognition, average sentence length, and the proportion of short sentences; When the total number of characters in the full text is less than the compact total character threshold, or the average sentence length is less than the compact average sentence length threshold and the proportion of short sentences is greater than the compact short sentence threshold, it is determined to be compact mode, and all sentences are aggregated into one or more logical slices. Otherwise, it is determined to be a semantic pattern, and it is divided into multiple logical slices at the semantic boundary to ensure that each logical slice is a complete semantic unit; For document resources, a word-aware segmentation planning method is adopted: PDFs are divided by page, and pages exceeding the preset page length threshold are further segmented; word processing documents are divided by title level, and when there is no title, they are divided by preset word window and adjacent windows overlap by 10%; presentations are divided by slides; and spreadsheets are divided by row window.
[0017] As a further solution, under the semantic mode, the lexical estimation formula T_est=N_cjk×0.6+N_other×0.25 of the CJK Unified Ideographs is used to determine the soft upper limit of lexical units. The soft upper limit of lexical units is determined based on the input constraints of the single text embedding model. Wherein, N_cjk is the number of characters in the Unicode encoding range U+4E00 to U+9FFF, and N_other is the number of other characters. Segmentation occurs when any of the following conditions are met: the cumulative number of words exceeds the soft limit, the pause between adjacent sentences exceeds the hard pause threshold, the previous sentence ends with a punctuation mark and the pause exceeds the soft pause threshold, or the cumulative number of characters exceeds the preset character limit. Single sentences exceeding the soft limit of word length are segmented separately and an alarm is generated, without being segmented within the sentence; For excessively short logical slices with a character count below the minimum segment character threshold, aggregate them to adjacent logical slices with a smaller difference in character count. When adjacent logical slices on both sides have the same character count, prioritize aggregation to the adjacent logical slice with the earlier timing to maintain timing.
[0018] As a further solution, in the two-stage pipeline: in stage one, a bounded thread pool is used to concurrently call the large visual model to generate visual semantic descriptions; in stage two, vectorization and index writing are executed serially; a distributed lock is used throughout the pipeline execution process to prevent repeated processing of the same media resource; the state machine includes a pending state, a planning state, a segmented submission state, a segmented state, a vectorized state, and a completed state, and proceeds in the order of the above states, continuing execution from the last successful state in case of failure.
[0019] On the other hand, the present invention also provides a unified vectorization construction system for a multimodal media knowledge base, for executing a unified vectorization construction method for a multimodal media knowledge base as described in any of the preceding claims, comprising: The multimodal media acquisition and registration module is used to receive four types of media resources: video, audio, images, and documents. It routes the resources to the corresponding processing services according to their types, obtains at least one type of related information from speech recognition text, scene analysis results, image addresses, or document files, and uniformly registers them as knowledge base document records and marks them as pending processing status. An adaptive semantic segmentation module is used to perform semantic segmentation on the media resources according to resource type, generate semantically complete logical slices, associate the logical slices with the knowledge base document records and record the slice type; wherein, audio and video resources are determined to be compact mode or semantic mode based on the statistical features of speech recognition text, document resources are divided according to their document structure, and image resources are treated as a whole image as a logical slice. The visual semantic bridging module is used to call the large visual model on logical slices containing visual content, generate visual semantic descriptions, and convert pure visual content into text. The role labeling and unified vectorization module is used to define four types of role prefixes: screen role prefix, voice role prefix, visual role prefix and text role prefix. It constructs role label embedded text with role prefixes according to the modality to which the logical slice belongs. The role label embedded text of all modalities is uniformly input into a single text embedding model and mapped to a unified vector space to obtain the dense vector of each logical slice in the unified vector space. A dual-field index building module is used to create a display field and a description field for each logical slice. The display field stores text for front-end display and reference tracing, and the description field stores the visual semantic description generated by the visual big model; the dense vector is written into the vector field; The concurrent pipelined data entry module is used to execute the visual semantic description generation, vectorization, and index writing using a two-stage pipeline of concurrent prefetching and serial data entry. The visual semantic description generation is executed concurrently in stage one, while the vectorization and index writing are executed serially in stage two, and the execution is resumed from the breakpoint by a state machine.
[0020] Compared with related technologies, the unified vectorization construction method and system for multimodal media knowledge base provided by this invention has the following advantages: 1. This invention uses four types of role prefixes—image role prefix, audio role prefix, visual role prefix, and text role prefix—to standardize the logical slices of four types of media resources (video, audio, images, and documents) into role-labeled embedded text. These are then uniformly input into a single text embedding model and mapped to a unified vector space, allowing direct comparison of dense vectors from different modalities. Users can retrieve all four modalities in a single query within the same vector space, eliminating the need to maintain separate embedding models and vector indexes for each modality, thus reducing the system complexity and operational costs of the knowledge base. Furthermore, the role prefixes enable modality-based filtering and attribution of the retrieved results.
[0021] 2. This invention generates visual semantic descriptions through a large visual model and, via a bridging pipeline of "office suite format conversion—PDF page-by-page rendering—dual-use of large visual model," enables purely visual content such as silent video footage, scanned document pages, and chart pages to obtain textual representations that can be matched by text queries, allowing them to be hit by natural language queries. At the same time, the display field and description field are decoupled. The retrieval is based on the visual semantics of the description field, while the front-end display and retrieval enhancement referencing is based on the readable text of the display field. This avoids semantic misalignment caused by the same text field serving both display and retrieval purposes, and the hit results can still be traced back to the timestamp or text position of the original material.
[0022] 3. This invention adaptively determines compact or semantic patterns based on the statistical features of speech recognition text, and combines word estimation based on the perception of the unified Chinese-Japanese-Korean ideographic characters, multi-dimensional semantic boundary segmentation, and short segment aggregation. This ensures that the boundaries of logical slices coincide with the semantic units of the material itself, and that the word scale is moderate. This avoids the problems of semantic unit truncation caused by fixed-duration segmentation, and the problems of thin semantics and large similarity deviations in short text slice vectors caused by simple punctuation segmentation. It improves the quality of vector indexing from the source and reduces the interference of noisy slices on retrieval accuracy.
[0023] 4. This invention adopts a two-stage pipeline of concurrent prefetching and serial data entry. In the first stage, a bounded thread pool concurrently calls the large visual model to hide the time consumption of a single call. In the second stage, vectorization and index writing are executed serially to ensure the consistency of index writing. A distributed lock is used to prevent the same media resource from being processed concurrently and repeatedly. A state machine drives breakpoint recovery. If any stage fails, execution can continue from the last successful state. The completed visual semantic description, segmentation results and vectorization results can be reused, which reduces the overall time consumption of multimodal data entry and improves failure recovery capability. Attached Figure Description
[0024] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0025] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0026] Figure 1 A schematic diagram illustrating the steps of a unified vectorization construction method for a multimodal media knowledge base provided by this invention; Figure 2 A schematic diagram of a unified vectorized construction system for a multimodal media knowledge base provided by this invention; The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.
[0028] Please see Figure 1 This invention provides a unified vectorization construction method for a multimodal media knowledge base, comprising the following steps: It receives four types of media resources: video, audio, images, and documents. It routes the resources to the corresponding processing services according to their types, obtains at least one type of related information from speech recognition text, scene analysis results, image addresses, or document files, and uniformly registers them as knowledge base document records and marks them as pending processing status. The media resources are semantically segmented according to their resource types to generate semantically complete logical slices. The logical slices are then associated with the knowledge base document records, and the slice type is recorded. Among them, audio and video resources are determined to be compact or semantic based on the statistical characteristics of speech recognition text, document resources are divided according to their document structure, and image resources are treated as a whole image as a logical slice. Call the large visual model on the logical slice containing visual content to generate a visual semantic description, and convert the pure visual content into text; Four types of role prefixes are defined: screen role prefix, voice role prefix, visual role prefix and text role prefix. Role-labeled embedded text with role prefixes is constructed according to the modality to which the logical slice belongs. The role-labeled embedded text of all modalities is uniformly input into a single text embedding model and mapped to a unified vector space to obtain the dense vector of each logical slice in the unified vector space. For each logical slice, establish a display field and a description field. The display field stores text for front-end display and reference tracing, and the description field stores the visual semantic description generated by the visual big model. Write the dense vector into the vector field. The visual semantic description generation, vectorization, and index writing are executed using a two-stage pipeline of concurrent prefetching and serial ingestion. The visual semantic description generation is executed concurrently in stage one, while the vectorization and index writing are executed serially in stage two, and the execution is resumed from the breakpoint by a state machine.
[0029] To more clearly illustrate this application, the construction process of a certain broadcasting-related knowledge base is used as an example throughout the text. The materials for this knowledge base include: an interview video (some segments of which are unnarrated outdoor scenes with instrumental music), a live audio recording, several on-site photos, and a policy document (containing both text and scanned pages). The following section, using this example, first explains the overall architecture and process, then details the specific implementation of each key step, and finally provides a modified implementation method.
[0030] In addition, for ease of understanding, we will first establish a unified convention for several terms: The logical slice referred to in this application refers to a semantically complete fragment unit obtained after performing semantic segmentation on media resources, which is the basic unit for subsequent role labeling, vectorization and index construction. The so-called role-labeled embedded text refers to the text used as input to a single text embedding model after appending the corresponding role prefix according to the modality to which the logical slice belongs. The so-called visual semantic description refers to the semantic description text generated by the visual big model for visual content, covering the summary description of video physical slices, the visual semantic description of images, and the visual semantic description of document pages. The so-called single text embedding model refers to a single text embedding model that uniformly encodes the embedded text of role labels for all modalities. The so-called search-enhanced generation refers to a generation method that uses search results as a basis and references the retrieved text content during the generation process to enhance the credibility of the generated results.
[0031] Example 1 This embodiment provides a unified vectorization construction method for a multimodal media knowledge base, which includes six steps: multimodal media acquisition and registration, adaptive semantic segmentation, visual semantic bridging, role labeling and unified vectorization, dual-field index construction, and concurrent pipelined database entry, forming a closed loop of "acquisition-segmentation-bridging-vectorization-indexing-database entry".
[0032] The first step is multimodal media acquisition and registration. It receives four types of media resources: video, audio, images, and documents. These resources are routed to the corresponding processing services based on their type. At least one piece of related information is obtained from speech recognition text, scene analysis results, image addresses, or document files. All of these are then uniformly registered as knowledge base document records and marked as pending processing.
[0033] In this example: after the interview video is transcoded for video on demand, the speech recognition timeline and scene analysis results are loaded. The scene analysis results can be used to assist in the extraction of keyframes and scene frames from video logic slices. The on-site recordings are loaded with speaker logs and speech recognition text. The speaker logs can be used to distinguish speech recognition text segments from different speakers. Accessible image addresses are obtained from on-site photos. Policy documents are downloaded to a temporary directory.
[0034] The four types of materials mentioned above are registered as four knowledge base document records. In existing technologies, each modality is usually deployed with an independent processing link, and the system complexity and operation and maintenance costs increase linearly with the number of modalities. In this step, the four types of media resources are uniformly registered as knowledge base document records, providing a unified resource entry point for subsequent unified semantic segmentation, vectorization and index construction, thereby avoiding the duplication of construction according to modality in subsequent steps.
[0035] The second step: Adaptive semantic segmentation. Media resources are semantically segmented according to resource type, generating semantically complete logical slices. These logical slices are then associated with document records in the knowledge base, and their types are recorded. Specifically, audio and video resources are determined to have either a compact or semantic pattern based on the statistical characteristics of the speech recognition text; document resources are segmented according to their document structure; and image resources are treated as a single logical slice for the entire image. See Example 3 for specific rules.
[0036] In this example, the interview video and on-site audio recordings are each segmented into several logical slices, each on-site photo constitutes its own logical slice, and the policy document is segmented according to its page structure. In existing technologies, fixed-duration segmentation (e.g., every 30 seconds) easily truncates complete semantic units, while simple punctuation segmentation generates a large number of short text slices. The vectors generated by these short text slices have thin semantics and large similarity deviations, polluting the vector index and reducing retrieval accuracy. This step performs semantic segmentation according to resource type and produces unified logical slices, ensuring the semantic integrity of the subsequent vectorized input text and reducing the interference of short text noise on retrieval accuracy from the source.
[0037] The third step: Visual semantic bridging. The logical slices containing visual content are invoked using the large visual model to generate visual semantic descriptions, converting the pure visual content into text. This allows the pure visual content to subsequently participate in the same vectorization process as text resources. See Example 4 for specific implementation details.
[0038] In this example, the unnarrated segment of the interview video featuring a sunset at the beach with instrumental music, along with on-site photos and scanned pages of policy documents, all obtained their respective visual semantic descriptions at this stage. Existing technologies primarily rely on optical character recognition (OCR) engines to extract text from purely visual content before vectorizing it. However, OCR can only recognize text and cannot understand the trends and layout semantics of charts, diagrams, and visual compositions, leading to the loss of crucial information such as charts, diagrams, and visual compositions during retrieval. This stage generates visual semantic descriptions through a large visual model, compensating for the lack of text retrieval entry points for purely visual content.
[0039] The fourth step: Role labeling and unified vectorization. Four types of role prefixes are defined: visual role prefix, voice role prefix, visual role prefix, and text role prefix. Role-labeled embedded text with these prefixes is constructed according to the modality to which the logical slice belongs. All role-labeled embedded text from all modalities is uniformly input into a single text embedding model and mapped to a unified vector space, obtaining dense vectors for each logical slice in the unified vector space. See Example 2 for specific implementation details.
[0040] Since the logical segments of interview videos, on-site recordings, on-site photos, and policy documents are all encoded using the same single text embedding model, the dense vectors of the four types of materials are comparable. Existing technologies typically employ a fragmented strategy of "one modality, one model; one modality, one vector library," resulting in inconsistent semantic spaces for vectors from different modalities. This prevents users from retrieving data across modalities in a single search. This approach constructs role-labeled embedded text using four role prefixes and uniformly inputs it into a single text embedding model to map it to a unified vector space. This allows for direct similarity comparison of dense vectors from different modalities, enabling unified cross-modal retrieval in a single query.
[0041] Fifth step: Constructing a dual-field index. A display field and a description field are created for each logical slice. The display field stores text used for front-end display and citation tracing, while the description field stores the visual semantic description generated by the large visual model. Dense vectors are written into the vector field. See Example 5 for specific implementation details.
[0042] Existing technologies often use the same text field for both front-end display and source tracing, as well as for vectorization, resulting in a misalignment between display semantics and retrieval semantics. This step separates the display field and the description field, allowing retrieval and source tracing to each perform their respective functions.
[0043] The sixth step: Concurrent pipelined data import. A two-stage pipeline of concurrent prefetching and serial data import is used to perform visual semantic description generation, vectorization, and index writing. Visual semantic description generation is executed concurrently in stage one, while vectorization and index writing are executed serially in stage two, with breakpoint recovery driven by a state machine. See Example 6 for specific implementation. In existing technologies, calling large visual models is time-consuming, and the completed processing results are difficult to reuse if any stage of the data import process fails. This step uses a two-stage pipeline of concurrent prefetching and serial data import, with breakpoint recovery driven by a state machine, reducing data import time and improving failure recovery capabilities.
[0044] Of the six steps mentioned above, adaptive semantic segmentation ensures the semantic integrity of logical slices, enabling the embedded text of role annotations to produce semantically dense vectors; visual semantic bridging provides a vectorizable visual semantic source for the construction of dual-field indexes; dual-field index construction allows for the separate storage of screen descriptions and display texts, with retrieval and tracing each performing their respective functions; the four work together to form a complete closed loop, and implementing any one of them alone cannot completely solve the problem of unified cross-modal data entry.
[0045] Example 2 In the overall process of Example 1, the fourth step is responsible for transforming the logical slices of each modality into dense vectors in a unified vector space, which is the key to realizing cross-modal unified recall. This example further explains the specific implementation described, first explaining the semantics of the four types of role prefixes and the construction method of role label embedding text, and then explaining the unified vector space mapping and cross-modal unified recall calculation process.
[0046] The semantics of the four role prefixes are as follows: the "Screen" prefix is used to annotate the visual semantic description of video frames; the "Speech" prefix is used to annotate the speech recognition text of audio and video; the "Visual" prefix is used to annotate the visual semantic description of images and document pages; and the "Text" prefix is used to annotate the extracted text of documents and the text recognized by optical character recognition. In specific implementations, the four role prefixes can be represented in symbolic form, for example, as
Screen
Speech
Visual
Text
[0047] Role prefixes serve as semantic soft cues for modality awareness, explicitly declaring the semantic role and source modality of a corresponding text segment to a single text embedding model. This allows the single text embedding model to perceive the modal context to which the text segment belongs during encoding. The reason for using the same single text embedding model to encode text of all modalities is that the semantic spaces of vectors produced by different embedding models are usually inconsistent and cannot be directly compared. However, by using the same model to encode text of all four modalities uniformly, it can be ensured that all dense vectors are located in a unified vector space, and semantic relevance can be directly measured by cosine similarity.
[0048] The embedded text for character labels is constructed according to the modality to which the logical slice belongs: The embedded text for character labels in video logical slices is composed of a text segment obtained by splicing the character prefix of the screen with the visual semantic description of the video screen, and a text segment obtained by splicing the character prefix of the voice with the voice recognition text; The embedded text for character labels in audio logical slices is composed of a text segment obtained by splicing the character prefix of the voice with the voice recognition text; The embedded text for character labels in image logical slices is composed of a text segment obtained by splicing the character prefix of the visual image with the visual semantic description of the image; The embedded text for character labels in document logical slices is composed of a text segment obtained by splicing the character prefix of the text with the text, and a text segment obtained by splicing the character prefix of the visual image with the visual semantic description of the document page.
[0049] Returning to this example: In a logical segment of an interview video with narration, the embedded text for the role label consists of two parts: "[Visual] <Visual Visual Semantic Description>" and "[Audio] <Speech Recognition Text>"; in a logical segment of an on-site recording, the embedded text for the role label consists only of "[Audio] <Speech Recognition Text>"; in a logical segment of an on-site photograph, the embedded text for the role label consists of "[Visual] <Image Visual Semantic Description>"; in a single page of a policy document (text version), the embedded text for the role label consists of two parts: "[Body Text] <Body Text>" and "[Visual] <Page Visual Semantic Description>"; and in a scanned page, the body text in the embedded text for the role label comes from optical character recognition (OCR) text.
[0050] By explicitly declaring the role prefix, the single text embedding model can distinguish the semantic roles of different modal texts, such as "description of a sunset scene at the beach" and "speech recognition text of pure music audio", thus avoiding semantic confusion between different modal texts.
[0051] Regarding field sources, the text corresponding to the visual role prefix and the speech role prefix is taken from the description field, while the text corresponding to the speech role prefix and the text role prefix is taken from the display field. When the description field is empty, the role annotation embedding text is constructed solely from the content of the display field. For example, if the visual big data model fails to generate a visual description for a certain video logical slice, the description field of that logical slice is empty. In this case, its role annotation embedding text is constructed solely from the speech role prefix and the speech recognition text, preventing the logical slice from being unable to be vectorized due to the lack of visual description. This degradation mechanism ensures that any logical slice can continue to participate in vectorization and database entry as long as any available text source exists, and will not be discarded entirely due to the failure of a single function.
[0052] Regarding the unified vector space mapping, the character annotation embedded text is mapped to an N-dimensional dense vector using a single text embedding model according to the embedding function E, where E = Embed(T_role), T_role is the character annotation embedded text, and N ranges from 1024 to 3072. The specific value of N depends on the output dimension of the single text embedding model used; for example, a 1024-dimensional, 2048-dimensional, or 3072-dimensional single text embedding model can be used. Since the character annotation embedded text of all modalities is encoded by the same single text embedding model, the dense vectors of different modalities reside in a unified vector space, allowing for direct similarity comparison.
[0053] In terms of cross-modal unified recall, after the user query text is encoded by a single text embedding model, the cosine similarity between the user query text and the dense vector of each logical slice is calculated using the similarity function sim(Q,E) to achieve cross-modal unified recall; where sim(Q,E)=(Q·E) / (‖Q‖·‖E‖), Q is the vector obtained by encoding the user query text by a single text embedding model, and E is the dense vector of the corresponding logical slice.
[0054] In this example, when a user queries "sunset at the beach with instrumental music," the query vector is compared and uniformly ranked within the same calculation process with logical slices of silent footage from an interview video, logical slices of instrumental music segments, logical slices of photos of the sunset, and logical slices of relevant policy document pages. This achieves text description matching silent video footage. Compared to existing solutions that require "one modality, one model; one modality, one vector library," users do not need to switch to different modal indexes to perform multiple queries; a single query can obtain unified recall results covering four modalities.
[0055] Example 3 The second step of Example 1 provides a general overview of the role of adaptive semantic segmentation, the key point of which is to ensure that the boundaries of logical slices coincide with the semantic units of the material itself. This example further explains the specific implementation described, first explaining the mode determination of audio and video resources and the segmentation rules under semantic modes, and then explaining the block division methods for document and image resources.
[0056] For audio and video resources, we statistically analyze the total number of characters, average sentence length, and proportion of short sentences in the speech recognition text. For example, the total number of characters can be denoted as C_total; the average sentence length can be denoted as C_avg, which is the total number of characters divided by the number of sentences; and the proportion of short sentences can be denoted as R_short, which is the percentage of sentences with a length less than twice the average sentence length. These statistical features are introduced because the speech recognition text materials vary significantly in size and sentence structure: short texts with densely packed fragments, if segmented according to semantic boundaries, will produce a large number of excessively short and semantically weak segments; long texts with complete sentences, if not segmented, will exceed the input limits of a single text embedding model. Determining the pattern based on statistical features before segmentation allows the segmentation strategy to adapt to the text's inherent characteristics.
[0057] When the total number of characters in the full text is less than the compact total character threshold, or the average sentence length is less than the compact average sentence length threshold and the proportion of short sentences is greater than the compact short sentence threshold, it is determined to be a compact mode, and all sentences are aggregated into one or more logical slices; otherwise, it is determined to be a semantic mode, and it is divided into multiple logical slices at the semantic boundaries, ensuring that each logical slice is a complete semantic unit. For example, the compact total character threshold can be set to 2000 characters, the compact average sentence length threshold can be set to 15 characters, and the compact short sentence threshold can be set to 0.6.
[0058] In this example, the speech recognition text from the on-site recording contains approximately 900 characters and 60 sentences, with an average sentence length of about 15 characters. Sentences shorter than 30 characters (i.e., twice the average sentence length) account for approximately 0.85%. Since the total number of characters in the text is less than 2000, it is determined to be in compact mode, and all sentences are aggregated into a single logical slice to avoid semantically sparse short text vectors. The speech recognition text from the interview video contains approximately 15,000 characters, with an average sentence length of about 25 characters. Short sentences account for approximately 0.3%. It does not meet the compact mode condition and is determined to be in semantic mode. It is segmented into multiple logical slices at semantic boundaries to avoid truncating complete sentences with fixed-duration segments.
[0059] In semantic mode, the soft upper limit of lexical estimation is determined using the CJK Unified Ideographs perception formula T_est=N_cjk×0.6+N_other×0.25, where N_cjk is the number of characters within the Unicode encoding range U+4E00 to U+9FFF, and N_other is the number of other characters. The soft upper limit of lexical ...
[0060] The reason for using the model-independent estimation formula is that different word segmenters handle the CJK Unified Ideographs differently. Estimating directly by the number of characters avoids the overhead of calling each word segmenter individually, while also providing a stable upper bound on the word size. For example, when the maximum input length of the single text embedding model used is 512 words, the soft upper limit on the word size can be set to 512 words.
[0061] Segmentation occurs when any of the following conditions are met: the cumulative number of words exceeds the soft word limit, the pause between adjacent sentences exceeds the hard pause threshold, the previous sentence ends with a punctuation mark and the pause exceeds the soft pause threshold, or the cumulative number of characters exceeds the preset character limit. For example, the pause between adjacent sentences can be obtained from the time information corresponding to the speech-recognized text; sentence punctuation marks include periods, exclamation marks, question marks, etc.; the preset character limit serves as a fallback segmentation condition to ensure that the character size of any logical slice is controlled.
[0062] The above multidimensional segmentation conditions take into account three types of boundaries: the soft upper bound of lexical units ensures that the segments do not exceed the input limit of the model, the pause threshold corresponds to the natural semantic pauses in spoken expression, and the combination of sentence-end punctuation and pauses corresponds to the semantic boundary of the complete sentence, so that the segmentation position coincides with the semantic unit of the material itself as much as possible.
[0063] The aforementioned hard pause threshold, soft pause threshold, preset character upper limit, and minimum segment character threshold can be set based on the corpus statistics or empirical values of the speech recognition text, and this application does not impose any restrictions on them.
[0064] For single statements exceeding the soft limit on the length of a lexical unit, separate segments are created and alerts are generated. These segments are not split within the statement itself to avoid disrupting statement-level semantics. For excessively short logical slices with a character count below the minimum segment character threshold, aggregation is performed on adjacent logical slices with a smaller character count difference. Furthermore, when adjacent logical slices have the same character count, aggregation is prioritized towards the earlier-ordered adjacent logical slice to maintain temporal order. Through these processes, a balance is achieved between semantic integrity and lexical unit size for each logical slice, reducing the pollution of the vector index by semantically sparse slices.
[0065] For document resources, a word-aware segmentation planning approach is adopted: PDFs are divided by page, and pages exceeding a preset page length threshold are further segmented; word processing documents are divided by heading level, and when there are no headings, they are divided by preset word windows with adjacent windows overlapping by 10%; presentations are divided by slides; and spreadsheets are divided by row windows. The preset page length threshold and preset word windows can be determined based on the input constraints of the single text embedding model used. The reason for dividing according to document structure is that pages, headings, slides, and row windows are themselves natural semantic units of the document, and dividing along the document structure can avoid breaking continuous arguments across pages and headings; maintaining a 10% overlap between adjacent windows can reduce the possibility of window boundaries cutting off sentences when there are no headings.
[0066] In this example, policy documents are logically sliced page by page, with pages exceeding a preset page length threshold undergoing secondary segmentation. For image resources, each entire image is treated as a single logical slice. All modalities produce uniform logical slices, and the slice type is recorded.
[0067] Example 4 The third step of Example 1 generates visual semantic descriptions for purely visual content using a large visual model, enabling materials that originally lacked textual carriers, such as silent images and scanned pages, to obtain text retrieval entry points. This example further explains the specific implementation described, successively covering the bridging methods of video logical slices, image logical slices, and document logical slices, as well as the dual-use of the large visual model and the handling of partial failure degradation.
[0068] For video logical slices, external segmentation services are invoked to extract physical slices, keyframes, and scene frames. For each physical slice, a visual big model is invoked to generate a visual semantic description. The visual semantic descriptions of each physical slice within the same video logical slice can be sequentially concatenated and merged according to the temporal order of the physical slices to form the visual semantic description of that video logical slice, used for the character annotation embedded text construction in Example 2. Physical slices are only used for the analysis granularity of the visual big model; logical slices remain the basic unit for index construction.
[0069] The reason for performing physical slicing under logical slicing is that the visual large model has actual constraints on the duration of a single input image. By breaking down long logical slices into short physical slices for segmented analysis, the quality and stability of the image description can be guaranteed without changing the basic index unit.
[0070] In this example, the silent segment of the interview video with "sunset at the beach with instrumental music" was physically sliced and then used to generate visual semantic descriptions such as "the sea under the setting sun, the sky is orange-red, and there are waves on the sea" by calling the visual big model segment by segment.
[0071] For logical image slices, a large visual model is invoked to generate visual semantic descriptions of the images. When an image file exceeds a preset size, it is first compressed to within the preset size before invoking the large visual model. The preset size can be set to 10MB; compression reduces the latency and resource consumption of the large visual model. After this step, purely visual content such as on-site photos obtains text representations that can be matched with text queries, no longer relying on text extracted by optical character recognition as the sole vectorization source.
[0072] For document logical slices, the document page rendering pipeline is executed: if the document logical slice originates from a document saved in an office suite format, the office software running in headless mode (such as LibreOffice headless mode) is first called to convert it to a PDF intermediate format; if it originates from a PDF document, the PDF document is directly used as the rendering object. Then, a PDF rendering library (such as PDFBox) is used to render each page of the PDF document as an independent page PNG image at a preset resolution, which can be set to 150 DPI; finally, a visual semantic description is generated for each page PNG image using a large visual model.
[0073] The reason for setting up pipelines for "office suite format, intermediate PDF format, and page-by-page PNG images" is that the layout parsing results of different document formats vary significantly. Using page-by-page PNG images as input for the overall visual model ensures that visual information such as layout, charts, and diagrams are fully integrated into the semantic description generation process, avoiding the loss of layout semantics when only the main text is extracted. In this example, both the Chinese text version and the scanned pages of the policy document are processed using this pipeline.
[0074] In the aforementioned bridging process, the visual big data model simultaneously undertakes two functions: optical character recognition (OCR) and visual semantic description. For each page's PNG image, the visual big data model describes the chart data trends, layout, and diagram content, generating a visual semantic description. When the number of characters in the PDF text layer is less than a preset threshold, the page is determined to be a scanned page, and the visual big data model's OCR is triggered to intelligently identify the visible text on the page. The OCR-recognized text is used as the main text. The preset threshold can be set to 50 characters. When the number of characters in the PDF text layer is not less than the preset threshold, the text extracted from the PDF text layer is used as the main text. Regardless of whether it is a scanned page or not, a page layout and chart semantic description are generated.
[0075] Each page generates at most one logical slice record. The text extracted from the optical character recognition (OCR) text or PDF text layer serves as the main text, while the layout and chart semantic descriptions serve as the visual semantic descriptions, falling into the text sources corresponding to the main text role prefix and the visual role prefix, respectively. By using the same large visual model to perform both functions, the cost of deploying an independent OCR engine can be saved, and the generation style of the text recognition results and visual semantic descriptions can be kept consistent.
[0076] For scanned pages: When the visual semantic description function succeeds but the optical character recognition function fails, only the visual semantic description is written to the knowledge base, and the logical slice is marked as lacking optical character recognition text (this mark can be denoted as vlm_skipped); when the optical character recognition function succeeds but the visual semantic description function fails, only the optical character recognition text is written to the knowledge base, and the logical slice is marked as lacking visual semantic description (this mark can be denoted as ocr_skipped); when both functions fail, no logical slice record is generated for the page PNG image, and the failed page number is recorded for retry or manual verification.
[0077] The names of the aforementioned flags are for identification purposes only, and their meanings are subject to the description in this application. The reason for setting up partial failure degradation is that the two functions are independent of each other, and the failure of a single function should not lead to the complete discarding of the entire page of information; by marking and retaining the output of the successful function, the page can still be hit by queries of the corresponding type, improving the fault tolerance of the inbound process.
[0078] In this example, even if optical character recognition fails on a scanned page of a policy document, it can still be retrieved by a text query based on the layout and the visual semantic description of the charts.
[0079] Example 5 In the fifth step of Example 1, a display field and a description field are established for each logical slice, thus separating the retrieval semantics from the display semantics. This example further explains the specific implementation described, including the storage content of the two types of fields, the correspondence between the two types of fields and the role prefix during vectorization, the construction of image vectors and nested vector fields, and modal filtering and modal weight adjustment during the retrieval period.
[0080] The display field stores timestamped speech-recognized text or extracted document text. The speech-recognition text is stored sequentially according to timestamps, with each timestamp corresponding to a specific time period. The display field is used for front-end display and citation tracing during retrieval enhancement generation. For example, the display field for a logical slice of an interview video can store records in the format "00:01:20 Welcome guests...", where timestamps and speech-recognition text are connected by a preset separator. This can be directly presented on the front-end, and the retrieval enhancement can locate the specific time period. The display field for a document logical slice stores extracted document text for both display and citation tracing.
[0081] The description field stores the visual semantic description generated by the large visual model. During vectorization, the text corresponding to the on-screen character prefix and the visual character prefix is taken from the description field, while the text corresponding to the voice character prefix and the text character prefix is taken from the display field. When the description field is empty, only the content of the display field is used to construct the character annotation embedded text. The reason for storing the display and description fields separately is that in existing technologies, the same text field is used for both display tracing and vectorization, leading to a misalignment between display semantics and retrieval semantics: videos without narration cannot be retrieved by on-screen semantics because the voice text is empty, while videos with narration may be recalled at a different rate due to inconsistencies between the voice text and on-screen semantics. After the two fields are decoupled, retrieval depends on the on-screen semantics of the description field, and display depends on the readable text of the display field. Logical segments of videos without narration can be retrieved by relying on the on-screen visual semantic description, and the display content of the retrieved results is still the readable tracing text.
[0082] Simultaneously, the dense vectors of each logical slice are written into the vector field; independent image vectors for keyframes and scene frames are generated for each video logical slice and stored in the image vector field family. These independent image vectors can be obtained by encoding keyframes and scene frames using an image encoding model. These independent image vectors are used for image-level retrieval of keyframes and scene frames and do not participate in the unified recall of text queries in the unified vector space. The dense vectors of the logical slices are synchronized to the nested vector field of the search engine for precise retrieval within a time interval. For example, the nested vector field allows vector retrieval of speech recognition text segments within a specified time interval, supporting requirements such as "retrieving the context of a sentence in an interview."
[0083] During the retrieval period, modal filtering can be applied to the unified recall results based on role prefixes. For example, when querying "sunset at the beach with instrumental music," only logical slices corresponding to the visual role prefixes are retained for sorting. The similarity weights of different modalities can also be adjusted based on role prefixes. For instance, the similarity of visual, audio, visual, and text role prefixes can be multiplied by preset weights before sorting, thus obtaining interpretable recall results with modal attribution. This function is an optional retrieval-period processing feature and does not change the unified vector space constructed during the database entry phase.
[0084] Example 6 The sixth step of Example 1 employs a two-stage pipeline for database entry processing to address the issues of time-consuming large-scale visual model calls and link failure recovery. This example provides further details on its implementation, including the division between stage one and stage two, the role of distributed locks, and the state machine-driven breakpoint recovery method.
[0085] In the two-stage pipeline: Stage 1 uses a bounded thread pool to concurrently call the large visual model to generate visual semantic descriptions. The number of threads in the bounded thread pool can be set to 20 to achieve a balance between concurrent throughput and rate limiting of the large visual model calls; Stage 2 executes vectorization and index writing serially to avoid index contention caused by concurrent writes. The reason for separating the large visual model call and vectorization into two stages is that a single call to the large visual model is time-consuming. If processed serially, the overall data throughput will be limited by the call time. Stage 1 uses a bounded thread pool to concurrently prefetch visual semantic descriptions, which can hide the call latency, while the serial writing in Stage 2 ensures the consistency and stability of index writing. Throughout the pipeline execution process, distributed locks are used to prevent the same media resource from being processed repeatedly and to prevent the same resource from being concurrently and repeatedly written to the database in multi-instance deployment scenarios.
[0086] The state machine includes a pending state, a planning state, a segmented submission state, a segmented state, a vectorization state, and a completed state, progressing in the order described above. For example, after a media resource is registered as a knowledge base document record, it enters the pending state; after completing segmented planning, it enters the planning state; after submitting the segmented task, it enters the segmented submission state; during segmented execution, it enters the segmented state; during vectorization execution, it enters the vectorization state; and after all vectors and indexes are written, it enters the completed state.
[0087] The reason for introducing a state machine is that the multimodal data entry process is long and involves many steps. In existing technologies, if any step fails, the process must be restarted from the beginning, and the results of completed steps are wasted. By recording the progress status of each resource with a state machine, if any step fails, the process can continue from the last successful state. The completed visual semantic descriptions, segmentation results, and vectorization results can be reused without having to start from the beginning, thereby reducing data entry time and improving failure recovery capabilities.
[0088] In this example, if both optical character recognition and visual semantic description fail for a page of a policy document, resulting in no logical slice being generated for that page, the failed page number can be recorded and the document containing that page can be restored from the segmentation state for a retry, without affecting the processing results of other pages already in the database.
[0089] Example 7 The foregoing embodiments illustrate the implementation of this application in the form of method steps. Based on this, this embodiment provides a unified vectorization construction system for a multimodal media knowledge base, used to execute a unified vectorization construction method for a multimodal media knowledge base as described in any of the preceding embodiments, enabling the modular implementation of the aforementioned steps.
[0090] Please see Figure 2 The system includes: The multimodal media acquisition and registration module is used to receive four types of media resources: video, audio, images, and documents. It routes the resources to the corresponding processing services according to their types, obtains at least one type of related information from speech recognition text, scene analysis results, image addresses, or document files, and uniformly registers them as knowledge base document records and marks them as pending processing status. The adaptive semantic segmentation module is used to perform semantic segmentation on media resources according to resource type, generate semantically complete logical slices, associate logical slices with knowledge base document records and record slice type. Among them, audio and video resources are determined to be compact mode or semantic mode based on the statistical features of speech recognition text, document resources are divided according to their document structure, and image resources are treated as a whole image as a logical slice. The visual semantic bridging module is used to call the large visual model on logical slices containing visual content, generate visual semantic descriptions, and convert pure visual content into text. The Role Labeling and Unified Vectorization module is used to define four types of role prefixes: screen role prefix, voice role prefix, visual role prefix, and text role prefix. It constructs role labeling embedded text with role prefixes according to the modality to which the logical slice belongs. The role labeling embedded text of all modalities is uniformly input into a single text embedding model and mapped to a unified vector space to obtain dense vectors of each logical slice in the unified vector space. The dual-field index building module is used to create a display field and a description field for each logical slice. The display field stores the text used for front-end display and reference tracing, and the description field stores the visual semantic description generated by the visual big model and writes the dense vector into the vector field. The concurrent pipelined ingestion module is used to perform visual semantic description generation, vectorization, and index writing in a two-stage pipeline of concurrent prefetching and serial ingestion. Visual semantic description generation is executed concurrently in stage one, while vectorization and index writing are executed serially in stage two, and the execution is resumed from breakpoints by a state machine.
[0091] The collaboration process of the six modules is the same as in Example 1, and will not be repeated here.
[0092] The scope of protection of this application is not limited to the above embodiments. Exemplarily, it may also include the following modified embodiments: Parameter Variation: All of the above preset parameters can be adjusted. For example, the compact total character threshold, compact average sentence length threshold, and compact short sentence threshold can be adjusted according to the corpus size; the preset size, preset resolution, and preset threshold can be adjusted according to the interface constraints of the large visual model; the number of threads in the bounded thread pool can be adjusted according to machine resources; the coefficients 0.6 and 0.25 in the word estimation formula can be fine-tuned according to the word segmentation behavior of the single text embedding model used.
[0093] Role prefix symbol variations: In addition to being denoted as [Image], [Speech], [Visual], and [Text], the four types of role prefixes can also use other symbols or keywords, as long as they can declare the semantic role and source modality of the text segment to a single text embedding model.
[0094] Embedding model variations: A single text embedding model can adopt any text embedding model with an output dimension ranging from 1024 to 3072; headless office software and PDF rendering libraries can also adopt other implementations with equivalent functions (LibreOffice headless mode and PDFBox are just examples).
[0095] Document processing transformation: For a complete PDF page with a text layer, the text extracted from the PDF text layer can be used directly as the main text without triggering optical character recognition; for image resources, even if the file exceeds the preset size, the entire image is still treated as a logical slice after compression.
[0096] For example, the preset parameters of this application can take the following values: the dimension N of the dense vector is between 1024 and 3072; the total character threshold for compactness can be set to 2000 characters; the average sentence length threshold for compactness can be set to 15 characters; the short sentence threshold for compactness can be set to 0.6; the preset size for image compression can be set to 10MB; the preset resolution for page rendering can be set to 150 DPI; the preset threshold for scanned page determination can be set to 50 characters; the overlap ratio of adjacent windows when the text processing document has no title can be set to 10%; the coefficients of the lexical estimation formula are 0.6 and 0.25 respectively; and the number of threads in the bounded thread pool can be set to 20. The above values are all exemplary values, and those skilled in the art can adjust them according to actual application scenarios, which do not constitute a limitation on the scope of protection of this application.
[0097] Through the above technical solutions, this application has at least the following beneficial effects: four types of media resources—video, audio, images, and documents—are mapped to a unified vector space through the same single text embedding model, enabling unified retrieval across modalities with a single query, eliminating the need to maintain independent embedding models and independent vector indexes for each modality; pure visual content, after being transformed into visual semantic descriptions through visual semantic bridging, can be hit by text queries, while the display field and description field are separated, and the hit results can still display readable source text; adaptive semantic segmentation ensures the semantic integrity and appropriateness of logical slices, reducing the interference of short text noise on retrieval accuracy; the two-stage pipeline of concurrent prefetching and serial data entry, combined with state machine breakpoint recovery, improves the throughput and robustness of unified multimodal data entry.
[0098] The above are only some embodiments of this application and do not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.
Claims
1. A unified vectorization construction method for a multimodal media knowledge base, characterized in that, Includes the following steps: It receives four types of media resources: video, audio, images, and documents. It routes the resources to the corresponding processing services according to their types, obtains at least one type of related information from speech recognition text, scene analysis results, image addresses, or document files, and uniformly registers them as knowledge base document records and marks them as pending processing status. The media resources are semantically segmented according to their resource types to generate semantically complete logical slices. The logical slices are then associated with the knowledge base document records, and the slice type is recorded. Among them, audio and video resources are determined to be compact or semantic based on the statistical characteristics of speech recognition text, document resources are divided according to their document structure, and image resources are treated as a whole image as a logical slice. Call the large visual model on the logical slice containing visual content to generate a visual semantic description, and convert the pure visual content into text; Four types of role prefixes are defined: screen role prefix, voice role prefix, visual role prefix and text role prefix. Role-labeled embedded text with role prefixes is constructed according to the modality to which the logical slice belongs. The role-labeled embedded text of all modalities is uniformly input into a single text embedding model and mapped to a unified vector space to obtain the dense vector of each logical slice in the unified vector space. For each logical slice, establish a display field and a description field. The display field stores text for front-end display and reference tracing, and the description field stores the visual semantic description generated by the visual big model. Write the dense vector into the vector field. The visual semantic description generation, vectorization, and index writing are executed using a two-stage pipeline of concurrent prefetching and serial ingestion. The visual semantic description generation is executed concurrently in stage one, while the vectorization and index writing are executed serially in stage two, and the execution is resumed from the breakpoint by a state machine.
2. The unified vectorization construction method for a multimodal media knowledge base according to claim 1, characterized in that, The embedded text for the role label is constructed according to the modality to which the logical slice belongs as follows: The embedded text of the role label in the video logical slice consists of a text segment obtained by splicing the role prefix in the picture with the visual semantic description of the video picture, and a text segment obtained by splicing the speech role prefix with the speech recognition text. The embedded text for the role label of the audio logic slice is composed of the speech role prefix and the speech recognition text. The embedded text for the role label of the image logical slice is composed of a visual role prefix and a visual semantic description of the image; The embedded text of the role label in the document's logical slice consists of a text segment obtained by concatenating the main text role prefix with the main text, and a text segment obtained by concatenating the visual role prefix with the visual semantic description of the document page. Among them, the screen role prefix is used to annotate the visual semantic description of the video screen, the voice role prefix is used to annotate the speech recognition text of the audio and video, the visual role prefix is used to annotate the visual semantic description of the image and document page, and the text role prefix is used to annotate the document extraction text and optical character recognition text; the text corresponding to the screen role prefix and the visual role prefix is taken from the description field, and the text corresponding to the voice role prefix and the text role prefix is taken from the display field. When the description field is empty, the role annotation embedded text is constructed only from the content of the display field. The role prefix serves as a semantic soft cue for modality awareness, explicitly declaring the semantic role and source modality of the corresponding text segment to the single text embedding model.
3. The unified vectorization construction method for a multimodal media knowledge base according to claim 2, characterized in that, The character annotation embedded text is mapped into an N-dimensional dense vector by the single text embedding model according to the embedding function E; Where E = Embed(T_role), and T_role is the embedded text of the role annotation; After the user query text is encoded by the single text embedding model, the cosine similarity between it and the dense vector of each logical slice is calculated using the similarity function sim(Q,E) to achieve cross-modal unified recall. Where sim(Q,E)=(Q·E) / (‖Q‖·‖E‖); Q is the vector obtained by encoding the user query text through the single text embedding model, and E is the dense vector of the corresponding logical slice.
4. The unified vectorization construction method for a multimodal media knowledge base according to claim 1, characterized in that, The display field stores timestamped speech recognition text or document extraction text; wherein, the speech recognition text is stored one by one according to the timestamp, and each timestamp corresponds to the speech recognition text of the corresponding time period. The display field is used for front-end display and retrieval enhancement during the generation of reference tracing. It also includes: generating independent image vectors for keyframes and scene frames for video logical slices and storing them in an image vector field family; synchronizing the dense vectors of the logical slices to the nested vector field of the search engine for precise time interval retrieval.
5. The unified vectorization construction method for a multimodal media knowledge base according to claim 1, characterized in that, Convert purely visual content into text using the following steps: For video logical slices, call external segmentation services to perform physical slicing and extract key frames and scene frames, and call the visual big model to generate visual semantic descriptions for each physical slice. For logical slices of an image, the visual big model is invoked to generate a visual semantic description of the image; wherein, when the image file exceeds a preset size, it is first compressed to not exceed the preset size before the visual big model is invoked; For logical slices of the document, execute the document page rendering pipeline: If the document logical slice originates from a document saved in an office suite format, the office software running in headless mode is first called to convert it into a PDF intermediate format. If it originates from a PDF document, the PDF document is directly used as the rendering object. Then, a PDF rendering library is used to render each page of the PDF document into an independent page PNG image at a preset resolution. Finally, the visual big model is called to generate a visual semantic description for each page PNG image.
6. The unified vectorization construction method for a multimodal media knowledge base according to claim 5, characterized in that, The large-scale visual model simultaneously performs two functions: optical character recognition and visual semantic description. For each page's PNG image, the visual big model describes the chart data trends, layout, and schematic content, generating a visual semantic description; When the number of characters in the PDF text layer of the page is less than a preset threshold, the page is determined to be a scanned page, and the optical character recognition of the visual big data model is triggered to intelligently recognize the visible text in the page, and the optical character recognition text is used as the main text. When the number of characters in the PDF text layer of the page is not less than the preset threshold, the text extracted from the PDF text layer is used as the main text. At most one logical slice record is generated per page; For scanned pages: when the visual semantic description function succeeds but the optical character recognition function fails, only the visual semantic description is written to the knowledge base, and the logical slice is marked as lacking optical character recognition text; When the optical character recognition function succeeds but the visual semantic description function fails, only the optical character recognition text is written into the knowledge base, and the logical slice is marked as lacking visual semantic description. If both functions fail, the page PNG image will not generate a logical slice record, and the failed page number will be recorded.
7. The unified vectorization construction method for a multimodal media knowledge base according to claim 1, characterized in that, The media resources are semantically segmented by resource type using the following steps: For audio and video resources: Statistical analysis of the total number of characters in the full text of speech recognition, average sentence length, and the proportion of short sentences; When the total number of characters in the full text is less than the compact total character threshold, or the average sentence length is less than the compact average sentence length threshold and the proportion of short sentences is greater than the compact short sentence threshold, it is determined to be compact mode, and all sentences are aggregated into one or more logical slices. Otherwise, it is determined to be a semantic pattern, and it is divided into multiple logical slices at the semantic boundary to ensure that each logical slice is a complete semantic unit; For document resources, a word-aware segmentation planning method is adopted: PDFs are divided by page, and pages exceeding the preset page length threshold are further segmented; word processing documents are divided by title level, and when there is no title, they are divided by preset word window and adjacent windows overlap by 10%; presentations are divided by slides; and spreadsheets are divided by row window.
8. The unified vectorization construction method for a multimodal media knowledge base according to claim 7, characterized in that, In the semantic mode, the soft upper limit of the word is determined by the word estimation formula T_est=N_cjk×0.6+N_other×0.25 of the CJK Unified Ideographs perception. The soft upper limit of the word is determined based on the input constraints of the single text embedding model. Wherein, N_cjk is the number of characters in the Unicode encoding range U+4E00 to U+9FFF, and N_other is the number of other characters. Segmentation occurs when any of the following conditions are met: the cumulative number of words exceeds the soft limit, the pause between adjacent sentences exceeds the hard pause threshold, the previous sentence ends with a punctuation mark and the pause exceeds the soft pause threshold, or the cumulative number of characters exceeds the preset character limit. Single sentences exceeding the soft limit of word length are segmented separately and an alarm is generated, without being segmented within the sentence; For excessively short logical slices with a character count below the minimum segment character threshold, aggregate them to adjacent logical slices with a smaller difference in character count. When adjacent logical slices on both sides have the same character count, prioritize aggregation to the adjacent logical slice with the earlier timing to maintain timing.
9. The unified vectorization construction method for a multimodal media knowledge base according to claim 1, characterized in that, In the two-stage pipeline: Stage 1 uses a bounded thread pool to concurrently call the large visual model to generate visual semantic descriptions; Stage 2 performs vectorization and index writing serially. The entire pipeline execution process employs distributed locks to prevent repeated processing of the same media resource; the state machine includes a pending state, a planning state, a segmented submission state, a segmented state, a vectorized state, and a completed state, and proceeds in the order of the above states, continuing execution from the last successful state in case of failure.
10. A unified vectorization construction system for a multimodal media knowledge base, used to execute a unified vectorization construction method for a multimodal media knowledge base as described in any one of claims 1 to 9, characterized in that, include: The multimodal media acquisition and registration module is used to receive four types of media resources: video, audio, images, and documents. It routes the resources to the corresponding processing services according to their types, obtains at least one type of related information from speech recognition text, scene analysis results, image addresses, or document files, and uniformly registers them as knowledge base document records and marks them as pending processing status. An adaptive semantic segmentation module is used to perform semantic segmentation on the media resources according to resource type, generate semantically complete logical slices, associate the logical slices with the knowledge base document records and record the slice type; wherein, audio and video resources are determined to be compact mode or semantic mode based on the statistical features of speech recognition text, document resources are divided according to their document structure, and image resources are treated as a whole image as a logical slice. The visual semantic bridging module is used to call the large visual model on logical slices containing visual content, generate visual semantic descriptions, and convert pure visual content into text. The role labeling and unified vectorization module is used to define four types of role prefixes: screen role prefix, voice role prefix, visual role prefix and text role prefix. It constructs role label embedded text with role prefixes according to the modality to which the logical slice belongs. The role label embedded text of all modalities is uniformly input into a single text embedding model and mapped to a unified vector space to obtain the dense vector of each logical slice in the unified vector space. A dual-field index building module is used to create a display field and a description field for each logical slice. The display field stores text for front-end display and reference tracing, and the description field stores the visual semantic description generated by the visual big model; the dense vector is written into the vector field; The concurrent pipelined data entry module is used to execute the visual semantic description generation, vectorization, and index writing using a two-stage pipeline of concurrent prefetching and serial data entry. The visual semantic description generation is executed concurrently in stage one, while the vectorization and index writing are executed serially in stage two, and the execution is resumed from the breakpoint by a state machine.
