Method and device for constructing multi-modal knowledge base, electronic equipment and storage medium
By segmenting and clustering multimodal data, target semantic vectors are generated, and a multimodal knowledge base is constructed. This solves the problem of insufficient semantic relevance in existing technologies and improves the generation accuracy of the multimodal knowledge base.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SEAWAY TECHNOLOGY CO LTD
- Filing Date
- 2025-05-29
- Publication Date
- 2026-08-04
AI Technical Summary
Existing multimodal knowledge bases ignore semantic relevance during text block processing, resulting in low accuracy of generated answers.
By acquiring multimodal data, parsing and generating multimodal text, segmenting it into text block groups according to the segmentation strategy, clustering and merging to generate aggregated text clusters, determining the target semantic vector, and constructing a multimodal knowledge base.
It improves the semantic relevance between text blocks in the multimodal knowledge base, thereby enhancing the accuracy of generated answers.
Smart Images
Figure CN120873102B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of database technology, and in particular to a method, apparatus, electronic device and storage medium for constructing a multimodal knowledge base. Background Technology
[0002] With the rapid development of artificial intelligence technology, large-scale models have demonstrated powerful capabilities in fields such as natural language processing. However, the knowledge of large-scale models is usually based on static data from training, making real-time updates difficult. This can lead to inaccurate or inaccurate responses when generating answers. Therefore, it is necessary to construct a multimodal knowledge base to assist large-scale models in providing answers.
[0003] In related technologies, existing methods for constructing multimodal knowledge bases often divide text data into multiple text blocks of fixed size during text block processing. This "hard cutting" ignores the semantic relationships between text blocks. When facing complex scenarios, due to insufficient semantic relationships between text blocks in the multimodal knowledge base, the multimodal knowledge base exhibits "semantic fragmentation," resulting in low accuracy of the answers generated by large models based on the retrieved multimodal knowledge base. Summary of the Invention
[0004] The purpose of this application is to provide a method, apparatus, electronic device, and storage medium for constructing a multimodal knowledge base, so as to improve the semantic correlation between text blocks in the multimodal knowledge base. The specific technical solution is as follows:
[0005] In a first aspect of this application, a method for constructing a multimodal knowledge base is provided, the method comprising:
[0006] Acquire multimodal data and parse the multimodal data to generate multimodal text;
[0007] According to the segmentation strategy corresponding to the multimodal data, the multimodal text is segmented to generate at least one text block group, wherein the text block group contains M text blocks, and M is a positive integer;
[0008] For any of the text block groups, cluster the M text blocks in the text block group to generate N aggregated text clusters, where N is a positive integer and is less than or equal to M;
[0009] For any of the aggregated text clusters, all text blocks corresponding to the aggregated text clusters are merged to obtain merged text, and word segmentation vectors for each word in the merged text are generated;
[0010] Based on the word segmentation vector, determine the target semantic vector corresponding to each text block in the aggregated text cluster;
[0011] A multimodal knowledge base is constructed based on the target semantic vector and the multimodal data.
[0012] In an optional implementation, parsing the multimodal data to generate multimodal text includes:
[0013] When the multimodal data is document data, text content is extracted from the document data, and the presence of images in the document data is identified.
[0014] If images exist in the document data, the images in the document data are parsed to generate image content;
[0015] The text content and the image content are integrated to generate the multimodal text.
[0016] In an optional implementation, the image content includes image information and an image description, and the step of parsing the images in the document data to generate image content includes:
[0017] If the image contains text content, then the text content in the image is extracted to obtain the image information;
[0018] If the image does not contain text content, the image is input into a multimodal model to generate an image description.
[0019] In an optional implementation, the step of clustering the M text blocks in the text block group to generate N aggregated text clusters includes:
[0020] The M text blocks in the text block group are input into the embedding model to generate an initial semantic vector corresponding to each text block;
[0021] Cluster all the initial semantic vectors to generate N aggregated semantic vector groups;
[0022] Based on the N aggregated semantic vector groups, the M text blocks in the text block group are clustered to generate N aggregated text clusters, and the aggregated semantic vector groups correspond one-to-one with the aggregated text clusters.
[0023] In an optional implementation, determining the target semantic vector corresponding to each text block in the aggregated text cluster based on the word segmentation vector includes:
[0024] For any text block in the aggregated text cluster, determine the text block segmentation based on the text block;
[0025] The target semantic vector corresponding to the text block is determined based on the word segmentation of the text block and the word segmentation vector of each word in the merged text.
[0026] In an optional implementation, determining the target semantic vector corresponding to the text block based on the word segmentation of the text block and the word segmentation vector of each word in the merged text includes:
[0027] Based on the word segmentation of the text block, a target word segmentation vector group is determined from the word segmentation vector of each word in the merged text;
[0028] The target semantic vector corresponding to the text block is determined based on the target word segmentation vector group.
[0029] In an optional implementation, constructing a multimodal knowledge base based on the target semantic vector and the multimodal data includes:
[0030] The target semantic vector is associated with the multimodal data to generate an index table;
[0031] The multimodal knowledge base is constructed based on the index table.
[0032] In a second aspect of this application, an apparatus for constructing a multimodal knowledge base is also provided, the apparatus comprising:
[0033] The initial module acquires multimodal data and parses the multimodal data to generate multimodal text;
[0034] The segmentation module segments the multimodal text according to the segmentation strategy corresponding to the multimodal data, generating at least one text block group, wherein the text block group contains M text blocks, and M is a positive integer;
[0035] The first processing module clusters M text blocks in any given text block group to generate N aggregated text clusters, where N is a positive integer and is less than or equal to M.
[0036] The second processing module, for any of the aggregated text clusters, merges all the text blocks corresponding to the aggregated text clusters to obtain merged text, and generates a word segmentation vector for each word in the merged text;
[0037] The third processing module determines the target semantic vector corresponding to each text block in the aggregated text cluster based on the word segmentation vector;
[0038] The module constructs a multimodal knowledge base based on the target semantic vector and the multimodal data.
[0039] In a third aspect of the embodiments of this application, an electronic device is also provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;
[0040] Memory, used to store computer programs;
[0041] When a processor executes a program stored in memory, it implements the method for constructing a multimodal knowledge base as described in any of the first aspects above.
[0042] In a fourth aspect of the embodiments of this application, a storage medium is also provided, wherein the storage medium stores instructions that, when run on a computer, cause the computer to execute the method for constructing a multimodal knowledge base as described in any of the first aspects above.
[0043] In a fifth aspect of the embodiments of this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute the method for constructing a multimodal knowledge base as described in any of the first aspects above.
[0044] The technical solution provided in this application involves acquiring multimodal data, parsing the multimodal data to generate multimodal text, segmenting the multimodal text according to the segmentation strategy corresponding to the multimodal data, generating at least one text block group, where each text block group contains M text blocks (M is a positive integer), clustering the M text blocks in any text block group to generate N aggregated text clusters (N is a positive integer less than or equal to M), merging all text blocks corresponding to any aggregated text cluster to obtain merged text, and generating a segmentation vector for each word in the merged text. Based on the segmentation vectors, the target semantic vector corresponding to each text block in the aggregated text cluster is determined, and a multimodal knowledge base is constructed based on the target semantic vector and the multimodal data. This construction of a multimodal knowledge base based on the target semantic vector and multimodal data improves the semantic correlation between text blocks in the multimodal knowledge base. Attached Figure Description
[0045] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0046] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0047] Figure 1 A schematic diagram illustrating the implementation process of a method for constructing a multimodal knowledge base provided in this application embodiment;
[0048] Figure 2A schematic diagram illustrating the implementation process of another method for constructing a multimodal knowledge base provided in this application embodiment;
[0049] Figure 3 A schematic diagram illustrating the implementation process of a method for generating multimodal text provided in this application embodiment;
[0050] Figure 4 A schematic diagram illustrating the implementation process of a method for generating image content provided in this application embodiment;
[0051] Figure 5 A schematic diagram illustrating the implementation process of a method for determining a target semantic vector provided in this application embodiment;
[0052] Figure 6 A schematic diagram of the structure of a multimodal knowledge base construction device provided in an embodiment of this application;
[0053] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0054] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0055] To facilitate understanding of the embodiments of this application, the following will provide further explanation and description with reference to the accompanying drawings and specific embodiments. These embodiments do not constitute a limitation on the embodiments of this application.
[0056] like Figure 1 The diagram shown illustrates the implementation flow of a method for constructing a multimodal knowledge base according to an embodiment of this application. The method may specifically include the following steps:
[0057] S101: Acquire multimodal data, parse the multimodal data, and generate multimodal text.
[0058] In this embodiment of the application, multimodal data is acquired and parsed to generate multimodal text. The multimodal data may include document data, image data, audio data, video data, etc. After parsing the multimodal data, the corresponding text content is merged to obtain unified multimodal text. This embodiment of the application does not limit this process.
[0059] For example, parsing document data yields document text and image text 1; parsing image data yields image text 2; parsing audio data yields audio text 1; parsing video data yields audio text 2 and image text 3; aligning and merging document text, image text 1, image text 2, audio text 1, audio text 2, and image text 3 yields multimodal text.
[0060] S102, according to the segmentation strategy corresponding to the multimodal data, segment the multimodal text to generate at least one text block group, the text block group contains M text blocks, M is a positive integer.
[0061] In this embodiment, different segmentation strategies are formulated based on the structural and semantic characteristics of multimodal data. Multimodal text is segmented according to the segmentation strategy to generate at least one text block group. Different text block groups correspond to different types of multimodal data. For multimodal data of image type, the segmentation strategy can be one text block per image. For multimodal data of document type, the segmentation strategy can be one text block per image in the document data, and the document content in the document data can be segmented according to the preset text block size and segmentation methods such as paragraphs, punctuation, keywords, line breaks, and page breaks to obtain at least one text block. This embodiment does not limit the specific segmentation strategy.
[0062] For example, for multimodal text corresponding to image data in multimodal data, the segmentation strategy can be that one image corresponds to one text block, and all text blocks corresponding to the images are grouped together as a text block group. For multimodal text corresponding to document data in multimodal data, the segmentation strategy can be that each image in the document data corresponds to one text block, and the document content in the document data is segmented into multiple text blocks according to the preset text block size (fixed number of characters) and paragraphs. All text blocks corresponding to the images and text blocks corresponding to the document content together form a text block group.
[0063] S103. For any group of text blocks, cluster the M text blocks in the group to generate N aggregated text clusters, where N is a positive integer and is less than or equal to M.
[0064] In the embodiments of this application, for any group of text blocks, clustering of the M text blocks in the group can be performed by using clustering algorithms such as K-Means to generate N aggregated text clusters, where N is less than or equal to M.
[0065] For example, a text block group contains four text blocks (M = 4): Text block 1 "The phone's camera function is excellent"; Text block 2 "It is equipped with a 48-megapixel main camera and optical image stabilization"; Text block 3 "The battery capacity reaches 5000mAh and supports 65W fast charging"; Text block 4 "A full charge only takes 30 minutes, and the battery lasts all day." Clustering these four text blocks results in two aggregated text clusters (N = 2): Cluster 1 (camera function): Text block 1, Text block 2; Cluster 2 (battery life function): Text block 3, Text block 4.
[0066] S104: For any aggregated text cluster, merge all the text blocks corresponding to the aggregated text cluster to obtain the merged text, and generate the word segmentation vector for each word in the merged text.
[0067] In this embodiment, for any aggregated text cluster, all text blocks corresponding to the same aggregated text cluster are merged to generate a merged text. At the same time, a word segmentation vector for each word in the merged text is generated. The merging order can be based on timestamps, document structure, etc. Redundancy processing is performed to remove duplicate content and retain key information. Then, a pre-trained model (such as BERT, GPT, etc.) is used to generate a word segmentation vector for each word in the merged text. These word segmentation vectors are low-dimensional semantic representations of each word in the merged text. This embodiment does not limit this.
[0068] For example, consider two aggregated text clusters: Cluster 1 (Camera Function): Text Block 1, Text Block 2; Cluster 2 (Battery Life Function): Text Block 3, Text Block 4. Merge the text content corresponding to Text Block 1 and Text Block 2 in Cluster 1, merge the text content corresponding to Text Block 3 and Text Block 4 in Cluster 2, and through redundancy processing, we can obtain merged text 1 (The phone's camera function is excellent, equipped with a 48MP main camera and optical image stabilization) and merged text 2 (Battery capacity reaches 5000mAh, supports 65W fast charging, fully charges in just 30 minutes, and lasts a whole day). Then, perform word segmentation on the merged text content to obtain the word vector corresponding to each word.
[0069] S105, Based on the word segmentation vector, determine the target semantic vector corresponding to each text block in the aggregated text cluster.
[0070] In this embodiment, the target semantic vector corresponding to each text block in the aggregated text cluster is determined based on the word segmentation vector of each word in the merged text. For example, the word segmentation vector corresponding to the text block is found from the word segmentation vector in the merged text, thereby assigning a target semantic vector to each text block. This embodiment does not limit this.
[0071] For example, an aggregated text cluster contains two text blocks: text block 1 "The phone's camera function is excellent" and text block 2 "Equipped with a 48-megapixel main camera and optical image stabilization". The merged text is "The phone's camera function is excellent, equipped with a 48-megapixel main camera and optical image stabilization". The corresponding word segmentation vectors are "photo":[0.2,0.5,-0.1], "48-megapixel":[0.6,0.3,0.2], and "optical image stabilization":[0.3,0.4,0.1]. The word segmentation vector corresponding to text block 1 is found from the word segmentation vectors, thus determining the target semantic vector of text block 1. The word segmentation vector corresponding to text block 2 is found, thus determining the target semantic vector of text block 2.
[0072] S106, Construct a multimodal knowledge base based on the target semantic vector and multimodal data.
[0073] In this embodiment, a multimodal knowledge base is constructed based on the target semantic vector and multimodal data. This multimodal knowledge base stores and manages the multimodal data and its corresponding target semantic vector for efficient subsequent multimodal retrieval and reasoning. During construction, database technologies (such as relational databases, NoSQL databases, etc.) or vector database technologies (such as Faiss, Milvus, etc.) can be used to achieve efficient data storage and retrieval.
[0074] For example, when the multimodal data is document data, the document data is parsed to obtain multimodal text. The multimodal text is then segmented to obtain a text block group, which contains text block 1, text block 2, text block 3, and text block 4. These four text blocks are clustered to obtain two aggregated text clusters: Cluster 1: text block 1 and text block 2, and Cluster 2: text block 3 and text block 4. Text block 1 and text block 2 are merged to generate merged text 1, and text block 3 and text block 4 are merged to generate merged text 2. The word segmentation vector of each word in merged text 1 and merged text 2 are determined. Thus, the target semantic vector corresponding to text block 1, the target semantic vector corresponding to text block 2, the target semantic vector corresponding to text block 3, and the target semantic vector corresponding to text block 4 are determined. Based on the target semantic vectors 1, 2, 3, and 4 and the document data, a multimodal knowledge base is constructed to support multimodal retrieval and reasoning functions based on semantic vectors.
[0075] Based on the above description of the technical solution provided in the embodiments of this application, multimodal data is acquired and parsed to generate multimodal text. According to the segmentation strategy corresponding to the multimodal data, the multimodal text is segmented to generate at least one text block group. Each text block group contains M text blocks, where M is a positive integer. For any text block group, the M text blocks in the group are clustered to generate N aggregated text clusters, where N is a positive integer less than or equal to M. For any aggregated text cluster, all text blocks corresponding to the cluster are merged to obtain merged text, and a segmentation vector for each word in the merged text is generated. Based on the segmentation vectors, the target semantic vector corresponding to each text block in the aggregated text cluster is determined. A multimodal knowledge base is constructed based on the target semantic vector and the multimodal data. Constructing a multimodal knowledge base based on the target semantic vector and multimodal data in this way can improve the semantic correlation between text blocks in the multimodal knowledge base.
[0076] like Figure 2 The diagram shown illustrates the implementation flow of another method for constructing a multimodal knowledge base provided in this application, which may specifically include the following:
[0077] S201: Acquire multimodal data, parse the multimodal data, and generate multimodal text.
[0078] In this application embodiment, multimodal data can be obtained from various channels, including but not limited to local storage (such as document libraries, media folders), web crawlers (such as web pages, social media platforms), API interfaces (such as enterprise systems, cloud storage services), and real-time acquisition devices (such as cameras, microphones). Based on the data format and characteristics of the acquired multimodal data, it is automatically identified and classified into document data (such as PDF, Word, PPT, etc., which may contain mixed content such as text, tables, and images), image data (such as JPG, PNG, SVG, etc., which may contain text, charts, or scene information), audio data (such as MP3, WAV, AAC, etc., which may contain semantics, music, or ambient sound), and video data (such as MP4, AVI, MOV, etc., which contain audio and video frame sequences). This application embodiment does not limit this classification.
[0079] For details on how to parse multimodal data and generate multimodal text, please refer to [reference needed]. Figure 3 The method shown. (As shown) Figure 3 The diagram shown illustrates the implementation flow of a method for generating multimodal text according to an embodiment of this application, which may specifically include the following steps:
[0080] S301, when the multimodal data is document data, extract text content from the document data and identify whether there are images in the document data.
[0081] In this embodiment of the application, when the multimodal data is document data, text content is extracted from the document data, and the presence of images in the document data is identified. The document data can be in the format of PDF, Word, PPT, etc. When the multimodal data is document data, the document data is parsed, and readable text content is identified and extracted from the document data, while the presence of images in the document data is also identified. This embodiment of the application does not limit this.
[0082] For example, Word document data 1 records information about smartphone usage. It is parsed, identified, and the readable text content 1 is extracted. In addition, images 1, 2, and 3 are also identified in Word document data 1.
[0083] S302, If images exist in the document data, parse the images in the document data and generate image content.
[0084] In this embodiment of the application, the multimodal data is document data, and the document data contains images. OCR (Optical Character Recognition) technology can convert the text in the image into an editable text format, which is convenient for subsequent processing and analysis. By using OCR (Optical Character Recognition) technology, the images in the multimodal data are parsed to obtain the image content.
[0085] For example, OCR technology can be used to process images 1, 2, and 3 in Word document data 1 to obtain the image content.
[0086] For details on how to parse images from document data and generate image content, please refer to [reference needed]. Figure 4 The method shown. (As shown) Figure 4 The diagram shown illustrates the implementation flow of a method for generating image content according to an embodiment of this application, which may specifically include the following steps:
[0087] S401, If the image contains text content, extract the text content from the image to obtain the image information;
[0088] In this embodiment, the image content includes image information. When text content exists in the image, the image information can be obtained by parsing the image in the document data using OCR (Optical Character Recognition) technology.
[0089] For example, in Word document data 1, there are images 1, 2, and 3. Image 1 contains text content, while images 2 and 3 do not. In this case, OCR (Optical Character Recognition) technology can be used to recognize image 1 and extract the image information 1 corresponding to image 1.
[0090] S402, If there is no text content in the image, then input the image into the multimodal model to generate an image description.
[0091] In this embodiment, the image content includes an image description. Since OCR technology can only process images containing explicit text information, when the image does not contain text information, it can be input into a multimodal model to generate an image description. The multimodal model can be a large model such as Alibaba's or Tongyi Qianwen, and this embodiment does not limit this.
[0092] For example, Word document data 1 contains images 1, 2, and 3, but images 2 and 3 do not contain any text content. Inputting images 2 and 3 into Tongyi Qianwen will cause the large model to output image descriptions.
[0093] S303 integrates the text content and image content to generate the first text data.
[0094] In this embodiment, to ensure that text and image content convey more complete and accurate information, they are integrated to generate first text data. This can be achieved by obtaining the specific location information of the text content and images within the document data, including the start and end positions of text paragraphs and the insertion position of images. Based on the obtained location information, the text and image content are aligned, associating the image corresponding to the image content with its adjacent or related text paragraphs in the document data, thereby obtaining the first text data.
[0095] Furthermore, when using OCR technology, it's important to note that its accuracy can be affected by various factors such as image quality, font type, and text layout. To improve recognition accuracy, preprocessing techniques (such as noise reduction and binarization) can be used to optimize the image. The system's processing speed can be improved by optimizing algorithms and using parallel computing. This method can also be applied to other types of data, such as tabular data and 3D model data. For tabular data, text information can be extracted from the table, and images or charts within the table can be recognized; for 3D model data, geometric and texture information of the model can be extracted, and corresponding descriptive text can be generated.
[0096] The above process is for handling multimodal data as document data, but similar methods can be used to handle multimodal data of other data types.
[0097] In another embodiment of this application, when the multimodal data is image data, the image can be directly parsed using methods such as S401 and S402 to obtain multimodal text.
[0098] In another embodiment of this application, when the multimodal data is audio data, an ASR model can be invoked to recognize the audio data and generate multimodal text.
[0099] In another embodiment of this application, when the multimodal data is video data, the video data can be parsed to obtain audio data and image data. The audio data can be identified using an ASR model to obtain audio text. The image data can be parsed using methods such as S401 and S402 to obtain image content and perform time alignment to ensure that the audio text and the image content corresponding to the video frame images are synchronized on the timeline, thereby generating multimodal text.
[0100] S202, according to the segmentation strategy corresponding to the multimodal data, segment the multimodal text to generate at least one text block group, and the text block group contains M text blocks, where M is a positive integer.
[0101] In this embodiment, different segmentation strategies are formulated based on the structural and semantic characteristics of multimodal data. Multimodal text is segmented according to the segmentation strategy to generate at least one text block group. Different text block groups correspond to different types of multimodal data. For multimodal data of image type, the segmentation strategy can be one text block per image. For multimodal data of document type, the segmentation strategy can be a preset text block size, and segmentation can be performed according to segmentation methods such as paragraph, punctuation, keywords, line breaks, and page breaks to obtain at least one text block. This embodiment does not limit the specific segmentation strategy.
[0102] For example, for multimodal text corresponding to image data in multimodal data, the segmentation strategy could be to assign one text block to each image, with all text blocks corresponding to the images forming a text block group. For multimodal text corresponding to document data in multimodal data, the segmentation strategy could assign one text block to each image in the document data, and segment the document content according to a preset text block size (fixed number of characters) and paragraphs to obtain multiple text blocks. All text blocks corresponding to the images and the text blocks corresponding to the document content together form a text block group. For multimodal data corresponding to audio data, the segmentation strategy could be to segment according to timestamps to obtain multiple text blocks (e.g., converting 10 seconds of audio to text into one text block), with all text blocks forming a text block group.
[0103] S203: For any group of text blocks, input the M text blocks in the group into the embedding model to generate an initial semantic vector corresponding to each text block.
[0104] In this embodiment, the embedding model can convert the semantic information in the text block into a vector representation in a high-dimensional space. The M text blocks in the text block group are input into the embedding model to obtain the initial semantic vector corresponding to each text block.
[0105] For example, if there are three text blocks in a text block group: “Smartphones are developing rapidly”, “Mobile phone functions are becoming increasingly powerful”, and “Mobile devices are being updated quickly”, after inputting them into the embedding model, three initial semantic vectors are obtained, each of which contains the semantic features of the corresponding text block.
[0106] S204, cluster all initial semantic vectors to generate N aggregated semantic vector groups.
[0107] In this embodiment, all initial semantic vectors are clustered to generate N aggregated semantic vector groups. Each aggregated semantic vector group contains at least one initial semantic vector. The initial semantic vectors can be clustered using K-Means, hierarchical clustering, etc., to obtain N aggregated semantic vector groups. For example, one group contains features related to mobile phone type, another group contains features related to speed, and yet another group contains features related to change. This embodiment does not limit the specific features in this embodiment.
[0108] Specifically, when using the K-Means clustering algorithm, the first step is to determine the number of clusters, K. This value of K can be set according to actual needs and data characteristics. For example, if we want to roughly divide the initial semantic vectors into several large categories based on semantics, we can set K=3. Then, the algorithm randomly initializes K cluster centers (i.e., the initial aggregated word segmentation vector groups). Next, it calculates the distance between each initial semantic vector and these K cluster centers (commonly using Euclidean distance or the reciprocal of cosine similarity as the distance metric), and assigns each initial semantic vector to the cluster represented by the nearest cluster center. After that, it recalculates the cluster center of each cluster based on all the initial semantic vectors in each cluster (i.e., the average or weighted average of all initial semantic vectors within the cluster), as a new aggregated semantic vector group. The above steps of assigning and updating cluster centers are repeated until the cluster centers no longer change significantly or the preset number of iterations is reached. Finally, K aggregated semantic vector groups are obtained, each representing a set of concentrated features in the semantic space. For example, one set of aggregated semantic vectors might represent semantic features related to phone type ("smartphone", "mobile device"), while another might represent semantic features related to speed ("rapid", "fast-updating").
[0109] For example, given text blocks 1, 2, and 3, and their corresponding initial semantic vectors 1, 2, and 3, K-Means is used to cluster these three initial semantic vectors, resulting in aggregated semantic vector group 1 (containing initial semantic vector 1 and initial semantic vector 2) and aggregated semantic vector group 2 (containing initial semantic vector 3).
[0110] S205, cluster the M text blocks in the text block group according to the N aggregate semantic vector groups to generate N aggregate text clusters. The aggregate semantic vector groups correspond one-to-one with the aggregate text clusters. N is a positive integer and is less than or equal to M.
[0111] In this embodiment, M text blocks in a text block group are clustered according to N aggregated semantic vector groups to generate N aggregated text clusters, and the aggregated semantic vector groups correspond one-to-one with the aggregated text clusters.
[0112] For example, as shown in the above example, two aggregated semantic vector groups are ultimately generated. Based on the aggregated semantic vector groups, the text blocks in the text block group are clustered. Each text block is assigned to the aggregated text cluster corresponding to the aggregated semantic vector group that is closest to its initial semantic vector. In the end, two aggregated text clusters are generated. One aggregated text cluster contains the text blocks "rapid development of smartphones" and "fast updates of mobile devices", which are semantically closer to the aggregated semantic vector group about the speed of mobile phone development. The other aggregated text cluster contains the text block "the functions of mobile phones are becoming increasingly powerful", which is closer to the aggregated semantic vector group about the functions of mobile phones.
[0113] S206: For any aggregated text cluster, merge all the text blocks corresponding to the aggregated text cluster to obtain the merged text, and generate the word segmentation vector for each word in the merged text.
[0114] In this embodiment, for any aggregated text cluster, all text blocks corresponding to the same aggregated text cluster are merged to generate a merged text. At the same time, a word segmentation vector for each word in the merged text is generated. The merging order can be based on timestamps, document structure, etc. Redundancy processing is performed to remove duplicate content and retain key information. Then, a pre-trained model (such as BERT, GPT, etc.) is used to generate a word segmentation vector for each word in the merged text. These word segmentation vectors are low-dimensional semantic representations of each word in the merged text. This embodiment does not limit this.
[0115] For example, consider two aggregated text clusters: Cluster 1 (Camera Function): Text Block 1, Text Block 2; Cluster 2 (Battery Life Function): Text Block 3, Text Block 4. Merge the text content corresponding to Text Block 1 and Text Block 2 in Cluster 1, merge the text content corresponding to Text Block 3 and Text Block 4 in Cluster 2, and through redundancy processing, we can obtain merged text 1 (The phone's camera function is excellent, equipped with a 48MP main camera and optical image stabilization) and merged text 2 (Battery capacity reaches 5000mAh, supports 65W fast charging, fully charges in just 30 minutes, and lasts a whole day). Then, perform word segmentation on the merged text content to obtain the word vector corresponding to each word.
[0116] S207, for any text block in the aggregated text cluster, determine the word segmentation of the text block based on the text block.
[0117] In this embodiment of the application, for each text block in the aggregated text cluster, the corresponding text block word segmentation is determined.
[0118] For example, aggregate text cluster 1 corresponds to text block 1 and text block 2, determine the text block segmentation 1 corresponding to text block 1, and determine the text block segmentation 2 corresponding to text block 2.
[0119] S208. Based on the word segmentation vectors of each word in the text block and the merged text, determine the target semantic vector corresponding to the text block.
[0120] In this embodiment of the application, the word segmentation vector corresponding to the word segmentation of the text block is found from the word segmentation vector of each word in the merged text, thereby determining the target semantic vector corresponding to the text block.
[0121] For details on how to determine the target semantic vector corresponding to a text block based on the segmentation vectors of each segmented word in the text and the merging of the segmentation vectors, please refer to [reference needed]. Figure 5 The method shown. (As shown) Figure 5 The diagram shown illustrates the implementation flow of a method for determining a target semantic vector according to an embodiment of this application, which may specifically include the following steps:
[0122] S501, Based on the word segmentation of the text block, determine the target word segmentation vector group from the word segmentation vector of each word in the merged text.
[0123] In this embodiment of the application, based on text block word segmentation, at least one target word corresponding to the text block word is found from the merged text, and the word vectors corresponding to all target words are combined into a target word vector group.
[0124] For example, if text block 1 is "mobile phone screen anti-blue light", its corresponding text block word segmentation is "mobile phone", "screen", "anti-blue light". Find the target word segmentation corresponding to "mobile phone", "screen", "anti-blue light" from the merged text, and use the word segmentation vector corresponding to the target word segmentation as the target word segmentation vector group.
[0125] S502, determine the target semantic vector corresponding to the text block based on the target word segmentation vector group.
[0126] In this embodiment of the application, the word segmentation vectors within the obtained target word segmentation vector group are aggregated to obtain the target semantic vector corresponding to the text block.
[0127] For example, text block 1 is "mobile phone screen anti-blue light", and its corresponding target words are "mobile phone", "screen" and "anti-blue light". The target word vector group corresponding to the target words includes target word vector 1, target word vector 2 and target word vector 3. By aggregating these three target word vectors, the target semantic vector corresponding to text block 1 is obtained.
[0128] S209: Associate the target semantic vector with multimodal data to generate an index table.
[0129] In this embodiment, the target semantic vector is associated with the multimodal data based on the correspondence between multimodal data and text block groups, and the correspondence between the target semantic vector and text blocks in the text block group. The association between the target semantic vector and the multimodal data can be established through manual annotation or automatic matching algorithms, thereby generating an index table. The index table can be designed as a table with multiple fields, such as vector ID, text block content, a list of associated image IDs, a list of associated audio IDs, a list of associated video IDs, etc. This embodiment does not limit this.
[0130] S210, Construct a multimodal knowledge base based on the index table.
[0131] In this embodiment, different database management systems, such as relational databases (e.g., MySQL) or non-relational databases (e.g., MongoDB), are selected based on the characteristics of multimodal data, thereby designing the corresponding database structure, importing the index table into the database, and constructing the corresponding multimodal knowledge base.
[0132] For example, if you choose MySQL, you need to design the database table structure. You can create multiple tables, such as a semantic vector table, a text block table, an image table, an audio table, and a video table. The semantic vector table stores vector IDs and vector data; the text block table stores vector IDs (as a foreign key related to the semantic vector table), text block content, and other information; the image table stores image IDs, image data, image descriptions, and related vector IDs (as a foreign key related to the semantic vector table); the audio and video tables are similar, storing audio and video information and related vector IDs respectively. Foreign key constraints ensure data consistency and integrity. If you choose MongoDB, you can use a document model to store data. You can create a collection to store all related data, with each document containing semantic vectors, text block content, related image information (stored as an array of image IDs and other related attributes), related audio information, related video information, etc. MongoDB's flexibility allows it to handle complex data structures and nested relationships more easily. Then, you can import the index table into the database.
[0133] Furthermore, to facilitate user access to and use of data in the multimodal knowledge base, a suitable query interface can be designed, such as using a RESTful API to query data via HTTP requests. Simultaneously, the multimodal knowledge base should be regularly maintained and updated. When new text blocks and multimodal data are added, a corresponding target semantic vector should first be generated. Then, the target semantic vector is associated with the multimodal data, the index table is updated, and the new data is imported into the database. At the same time, database performance monitoring and optimization are necessary to ensure the stable operation and efficient querying of the knowledge base.
[0134] Corresponding to the above method embodiments, this application also provides a multimodal knowledge base construction apparatus, such as... Figure 6 As shown, the device may include an initialization module 601, a segmentation module 602, a first processing module 603, a second processing module 604, a third processing module 605, and a construction module 606.
[0135] Initial module 601 acquires multimodal data, parses the multimodal data, and generates multimodal text;
[0136] The segmentation module 602 segments the multimodal text according to the segmentation strategy corresponding to the multimodal data, generating at least one text block group, where each text block group contains M text blocks, and M is a positive integer.
[0137] The first processing module 603 clusters M text blocks in any text block group to generate N aggregated text clusters, where N is a positive integer and is less than or equal to M.
[0138] The second processing module 604 merges all the text blocks corresponding to any aggregated text cluster to obtain merged text, and generates the word segmentation vector of each word in the merged text.
[0139] The third processing module 605 determines the target semantic vector corresponding to each text block in the aggregated text cluster based on the word segmentation vector;
[0140] Module 606 is used to construct a multimodal knowledge base based on the target semantic vector and multimodal data.
[0141] This application also provides an electronic device, such as... Figure 7 As shown, it includes a processor 701, a communication interface 702, a memory 703, and a communication bus 704, wherein the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704.
[0142] Memory 703 is used to store computer programs;
[0143] In one embodiment of this application, when the processor 701 executes a program stored in the memory 703, it performs the following steps:
[0144] Acquire multimodal data and parse it to generate multimodal text. Based on the segmentation strategy corresponding to the multimodal data, segment the multimodal text to generate at least one text block group. Each text block group contains M text blocks, where M is a positive integer. For any text block group, cluster the M text blocks to generate N aggregated text clusters, where N is a positive integer less than or equal to M. For any aggregated text cluster, merge all the text blocks corresponding to the cluster to obtain merged text, and generate a segmentation vector for each word in the merged text. Based on the segmentation vectors, determine the target semantic vector corresponding to each text block in the aggregated text cluster. Construct a multimodal knowledge base based on the target semantic vectors and the multimodal data.
[0145] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not indicate that there is only one bus or one type of bus.
[0146] The communication interface is used for communication between the aforementioned electronic devices and other devices.
[0147] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.
[0148] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0149] In another embodiment provided in this application, a storage medium is also provided, which stores instructions that, when run on a computer, cause the computer to execute the modal knowledge base construction method described in any of the above embodiments.
[0150] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute the modal knowledge base construction method described in any of the above embodiments.
[0151] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a storage medium or transmitted from one storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).
[0152] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0153] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.
[0154] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.
Claims
1. A method for constructing a multimodal knowledge base, characterized in that, The method includes: Acquire multimodal data and parse the multimodal data to generate multimodal text; According to the segmentation strategy corresponding to the multimodal data, the multimodal text is segmented to generate at least one text block group, wherein the text block group contains M text blocks, and M is a positive integer; For any of the text block groups, cluster the M text blocks in the text block group to generate N aggregated text clusters, where N is a positive integer and is less than or equal to M; For any of the aggregated text clusters, all text blocks corresponding to the aggregated text clusters are merged to obtain merged text, and word segmentation vectors for each word in the merged text are generated; Based on the word segmentation vector, determine the target semantic vector corresponding to each text block in the aggregated text cluster; A multimodal knowledge base is constructed based on the target semantic vector and the multimodal data.
2. The method according to claim 1, characterized in that, The process of parsing multimodal data to generate multimodal text includes: When the multimodal data is document data, text content is extracted from the document data, and the presence of images in the document data is identified. If images exist in the document data, the images in the document data are parsed to generate image content; The text content and the image content are integrated to generate the multimodal text.
3. The method according to claim 2, characterized in that, The image content includes image information and image description. Parsing the images in the document data to generate image content includes: If the image contains text content, then the text content in the image is extracted to obtain the image information; If the image does not contain text content, the image is input into a multimodal model to generate an image description.
4. The method according to claim 1, characterized in that, The step of clustering M text blocks in the text block group to generate N aggregated text clusters includes: The M text blocks in the text block group are input into the embedding model to generate an initial semantic vector corresponding to each text block; Cluster all the initial semantic vectors to generate N aggregated semantic vector groups; Based on the N aggregated semantic vector groups, the M text blocks in the text block group are clustered to generate N aggregated text clusters, and the aggregated semantic vector groups correspond one-to-one with the aggregated text clusters.
5. The method according to claim 1, characterized in that, The step of determining the target semantic vector corresponding to each text block in the aggregated text cluster based on the word segmentation vector includes: For any text block in the aggregated text cluster, determine the text block segmentation based on the text block; The target semantic vector corresponding to the text block is determined based on the word segmentation of the text block and the word segmentation vector of each word in the merged text.
6. The method according to claim 5, characterized in that, The step of determining the target semantic vector corresponding to the text block based on the word segmentation of the text block and the word segmentation vector of each word in the merged text includes: Based on the word segmentation of the text block, a target word segmentation vector group is determined from the word segmentation vector of each word in the merged text; The target semantic vector corresponding to the text block is determined based on the target word segmentation vector group.
7. The method according to claim 1, characterized in that, The step of constructing a multimodal knowledge base based on the target semantic vector and the multimodal data includes: The target semantic vector is associated with the multimodal data to generate an index table; The multimodal knowledge base is constructed based on the index table.
8. A device for constructing a multimodal knowledge base, characterized in that, The device includes: The initial module acquires multimodal data and parses the multimodal data to generate multimodal text; The segmentation module segments the multimodal text according to the segmentation strategy corresponding to the multimodal data, generating at least one text block group, wherein the text block group contains M text blocks, and M is a positive integer; The first processing module clusters M text blocks in any given text block group to generate N aggregated text clusters, where N is a positive integer and is less than or equal to M. The second processing module, for any of the aggregated text clusters, merges all the text blocks corresponding to the aggregated text clusters to obtain merged text, and generates a word segmentation vector for each word in the merged text; The third processing module determines the target semantic vector corresponding to each text block in the aggregated text cluster based on the word segmentation vector; The module constructs a multimodal knowledge base based on the target semantic vector and the multimodal data.
9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-7.