Multi-modal data hybrid retrieval method and device, equipment and storage medium

Abstract

The invention provides a multi-modal data mixed retrieval method and device, equipment and a storage medium, and the method comprises the steps: obtaining query information of a user and a modal of the query information, and converting the query information into a text feature matched with the modal, the modal comprising at least one of a text, an image, an audio and a video; the similarity between the text feature and each multi-modal comprehensive index entry pre-constructed in a database is calculated, and the multi-modal comprehensive index entry comprises the text feature of at least one modal; based on the similarity between the text feature and each multi-modal comprehensive index entry, screening out a target multi-modal comprehensive index entry; and returning multi-modal data associated with the target multi-modal comprehensive index entry as a retrieval result corresponding to the query information. Through the method disclosed by the invention, efficient and accurate multi-modal mixed retrieval can be realized.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a multimodal data hybrid retrieval method, apparatus, device, and storage medium. Background Technology

[0002] In today's era of digital information explosion, data exhibits multimodal characteristics, including but not limited to images, audio, video, and traditional text. However, traditional retrieval methods are often limited to a single modality, making it difficult to effectively support users' comprehensive retrieval needs for cross-modal information. For example, when searching for information about a product, users often need to simultaneously obtain multimodal data such as appearance images, introductory audio, and related documentation. Existing retrieval methods cannot efficiently integrate this type of information, resulting in incomplete and inefficient search results. Therefore, how to achieve efficient and accurate multimodal hybrid retrieval has become a pressing technical problem to be solved in this field. Summary of the Invention

[0003] In view of this, this disclosure proposes a multimodal data hybrid retrieval method, apparatus, device and storage medium, which can achieve efficient and accurate multimodal hybrid retrieval.

[0004] According to a first aspect of this disclosure, a multimodal data hybrid retrieval method is provided, comprising: The system acquires user query information and the modality of the query information, and converts the query information into text features that match the modality, wherein the modality includes at least one of text, image, audio, and video; Calculate the similarity between the text features and each pre-built multimodal comprehensive index item in the database, wherein the multimodal comprehensive index item includes text features of at least one modality; Based on the similarity between the text features and each of the multimodal comprehensive index items, target multimodal comprehensive index items are selected; The multimodal data associated with the target multimodal comprehensive index item is returned as the retrieval result corresponding to the query information.

[0005] In one possible implementation, the construction of the multimodal comprehensive index item includes: Retrieve multimodal text fragments; The multimodal text fragment is parsed to obtain at least two modal data among text modal data, image modal data, audio modal data, and video modal data; Each modal data is converted into text features that match its modality to obtain the text features of each modal data; The text features of each modality data are combined in sequence to obtain the multimodal comprehensive index item of the multimodal text fragment.

[0006] In one possible implementation, the process of converting text modal data into text features that match its modality includes: Keywords were extracted from the text modality data using natural language processing algorithms; All extracted keywords are combined in sequence to obtain the text features of the text modality data.

[0007] In one possible implementation, converting image modal data into text features that match its modality includes: Image recognition algorithms are used to extract key feature descriptions of objects from the image modal data; The extracted key feature descriptions are combined in sequence to obtain the text features of the image modality data.

[0008] In one possible implementation, converting audio modal data into text features that match its modality includes: The audio modal data is converted into text modal data, and the keywords in the text modal data are extracted as the keywords of the audio modal data. Perform audio feature analysis on the audio modal data to generate an audio feature description of the audio modal data; The keywords and audio feature descriptions of the audio modality data are combined in sequence to obtain the text features of the audio modality data.

[0009] In one possible implementation, converting video modal data into text features that match its modality includes: The video modal data is split into image modal data and audio modal data, and multiple scenes included in the video modal data and the start and end times of each scene are identified; Iterate through each of the aforementioned scenes. For the current scene, extract the start and end times of the current scene. Based on the start and end times of the current scene, extract image modal data segments and audio modal data segments of the corresponding time periods from the image modal data and the audio modal data, respectively. Based on the start and end times of the current scene and the image modal data segments and audio modal data segments of the corresponding time periods, generate a timeline summary of the current scene. After the traversal is completed, the timeline summary of each scene is obtained, and the timeline summaries of each scene are combined in order to obtain the timeline summary of the video modal data; The timeline summary of the video modal data is used as the text feature of the video modal data.

[0010] In one possible implementation, calculating the similarity between the text features and various pre-built multimodal comprehensive index items in the database includes: Traverse each of the aforementioned multimodal comprehensive index items in the database; For each traversed multimodal index item, the similarity between the text feature and the text features of each modality in the current multimodal index item is calculated sequentially. Based on the similarity between the text feature and the text features of each modality in the current multimodal index item, the similarity between the text feature and the current multimodal index item is determined. After the traversal is complete, the similarity between the text features and each multimodal comprehensive index item in the database is obtained.

[0011] According to a second aspect of this disclosure, a multimodal data hybrid retrieval apparatus is provided, comprising: The query information acquisition module is used to acquire the user's query information and the modality of the query information, and convert the query information into text features that match the modality, wherein the modality includes at least one of text, image, audio and video; A similarity calculation module is used to calculate the similarity between the text features and various pre-built multimodal comprehensive index items in the database, wherein the multimodal comprehensive index items include text features of at least one modality; The filtering module is used to filter out target multimodal comprehensive index items based on the similarity between the text features and each of the multimodal comprehensive index items; The result feedback module is used to return multimodal data associated with the target multimodal comprehensive index item as the retrieval result corresponding to the query information.

[0012] According to a third aspect of this disclosure, a multimodal data hybrid retrieval device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to perform the method described in the first aspect of this disclosure.

[0013] According to a fourth aspect of this disclosure, a non-volatile computer-readable storage medium is provided that stores computer program instructions thereon, wherein the computer program instructions, when executed by a processor, implement the method described in the first aspect of this disclosure.

[0014] This disclosure provides a method, apparatus, device, and storage medium for multimodal data hybrid retrieval. The method includes: acquiring user query information and the modality of the query information, and converting the query information into text features matching the modality, wherein the modality includes at least one of text, image, audio, and video; calculating the similarity between the text features and various pre-constructed multimodal comprehensive index items in a database, wherein the multimodal comprehensive index items include text features of at least one modality; filtering target multimodal comprehensive index items based on the similarity between the text features and each multimodal comprehensive index item; and returning multimodal data associated with the target multimodal comprehensive index item as the retrieval result corresponding to the query information. This disclosure achieves semantic integration and comparability of cross-modal data by uniformly converting multimodal data such as text, image, audio, and video into text features of the same modality and constructing a unified multimodal comprehensive index item. This method effectively overcomes the semantic barriers caused by modality differences in traditional retrieval, enabling similarity calculations for different modal data in the same feature space, thereby achieving efficient and accurate responses to multimodal queries and improving the overall performance and user experience of the retrieval system.

[0015] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0016] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0017] Figure 1 A flowchart is shown for a multimodal data hybrid retrieval method according to an embodiment of the present disclosure; Figure 2 A schematic block diagram of a multimodal data hybrid retrieval apparatus according to an embodiment of the present disclosure is shown; Figure 3 A schematic block diagram of a multimodal data hybrid retrieval device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0018] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0019] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0020] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0021] <Method Implementation> It should first be noted that before implementing the methods disclosed herein, a database including multimodal composite index entries needs to be constructed. In one possible implementation, constructing the multimodal composite index entries may include the following steps: First, acquire multimodal text fragments. A multimodal text fragment refers to a semantically coherent, content-aggregated data unit containing at least two different types of modal data (e.g., text, images, audio, or video). These modal data are highly semantically related, collectively describing the same entity, event, or topic, and can therefore be processed as a unified index item during the retrieval process. Specifically, text modal data can be product manuals, academic papers, etc.; image modal data can be product design drawings, product appearance images, experimental images, etc.; audio modal data can be meeting recordings, product explanation audio, etc.; and video modal data can be product demonstration videos, instructional videos, etc.

[0022] Taking a smartwatch as an example, a typical multimodal text fragment may include: Text modal data: paragraphs in the product manual that describe the functions; Image modal data: Images of the smartwatch's appearance; Audio modal data: includes audio clips introducing product features; Video modal data: Video clips demonstrating how the watch is used or its features.

[0023] These data not only complement each other in terms of content, but also maintain semantic consistency, together forming a multi-dimensional description of the smartwatch, providing basic data for the construction of subsequent multimodal comprehensive index items and hybrid retrieval.

[0024] Second, the multimodal text fragments are parsed to obtain at least two modalities of data, including text modal data, image modal data, audio modal data, and video modal data.

[0025] Third, the modal data is converted into text features that match its modality, thus obtaining the text features of each modal data.

[0026] In one possible implementation, the process of converting text modal data into text features that match its modality can include the following steps: First, using a natural language processing algorithm to extract keywords from the text modal data; then, combining all the extracted keywords in sequence to obtain the text features of the text modal data.

[0027] For example, if the text modal data of a smartwatch mentions that "the smartwatch has heart rate monitoring and sleep monitoring functions, a stylish appearance, and a round dial", and the keywords extracted using natural language processing algorithms include "smartwatch", "heart rate monitoring", "sleep monitoring", and "round dial", then the text features corresponding to this text modal data are ["smartwatch", "heart rate monitoring", "sleep monitoring", "round dial"].

[0028] Furthermore, the process of extracting keywords from text modal data using natural language processing algorithms can also include the following steps: identifying and recording entities appearing in the text modal data; calculating the weight of each keyword using the Term Frequency-Inverse Document Frequency (TF-IDF) algorithm; and determining the topic distribution information of the text modal data based on the extracted keywords using a topic model (such as LDA). For example, for the text modal data of the aforementioned smartwatch, the extracted keywords include "smartwatch," "heart rate monitoring," "sleep monitoring," and "round watch face." The topic distribution information determined by the topic model is "the topic revolves around the functions and appearance design of the smartwatch." Finally, the text modal data itself, the extracted text features (keyword sequences), the identified entities, the keywords and their weights, and the topic distribution information are associated and stored, thereby establishing a complete and efficient data foundation for subsequent multimodal hybrid retrieval.

[0029] In one possible implementation, converting image modal data into text features that match its modality can include the following steps: First, using an image recognition algorithm, key feature descriptions of objects within the image are extracted from the image modal data, such as visual attributes like contours, colors, and materials. Then, these extracted key feature descriptions are combined sequentially to form the text features of the image modal data. Taking a smartwatch's appearance image as an example, the image recognition algorithm can extract key feature descriptions such as "circular contour" and "metallic material." Combining these key feature descriptions sequentially yields the text features of the smartwatch's appearance image ["circular contour," "metallic material"].

[0030] Furthermore, based on the obtained key feature descriptions, these key feature descriptions are input into a pre-trained deep learning model (using an architecture combining Generative Adversarial Networks (GANs) and Recurrent Neural Networks (RNNs) or a Transformer architecture) to generate fluent and semantically complete image-text descriptions. For example, based on the three key feature descriptions of "circular outline" and "metallic material," the deep learning model will output the image-text description "a round metal smartwatch."

[0031] Furthermore, after generating the image text description of the image modality data, natural language processing technology will be used to perform deep processing on the image text description to obtain the entity, keywords and weights, and topic distribution information corresponding to the image text description (see above for specific steps). The image modality data itself, the text features of the image modality data, the image text description, and the entity, keywords and their weights, and topic distribution information further parsed from the image text description will be associated and stored, thereby establishing a complete and efficient data foundation for subsequent multimodal hybrid retrieval.

[0032] In one possible implementation, converting audio modal data into text features that match its modality can include the following steps: Using an Automatic Speech Recognition (ASR) algorithm, the audio modal data is converted into text modal data, and keywords from the text modal data are extracted as keywords for the audio modal data. Simultaneously, the ASR algorithm is used to perform audio feature analysis on the audio modal data, generating an audio feature description describing the characteristics of the audio signal. This audio feature description may include one or more attributes such as emotional intensity, volume fluctuations (to reflect the speaker's emphasis), and spectral energy concentration points. Finally, the keywords and audio feature descriptions of the audio modal data are combined sequentially to obtain the text features of the audio modal data.

[0033] Taking an audio clip that includes an introduction to the functions of a smartwatch as an example, the converted text modal data is "This smartwatch supports accurate heart rate monitoring and scientific sleep monitoring functions". The keywords extracted from it include "accurate heart rate monitoring" and "scientific sleep monitoring". The audio feature analysis results show that "the volume of the function introduction clip is relatively large". Therefore, the text features of this audio modal data can be represented as ["accurate heart rate monitoring", "scientific sleep monitoring", "volume (relatively large in the function introduction section)"].

[0034] Based on the extraction of keywords from the text modal data, the entity, keyword and weight, and topic distribution information from the text modal data will also be extracted simultaneously (see above for specific steps). Finally, the audio modal data itself, the text features of the audio modal data, the text modal data converted from the audio modal data, and the entity, keyword and weight, and topic distribution information further extracted from the text modal data will be associated and stored, thereby establishing a complete and efficient data foundation for subsequent multimodal hybrid retrieval.

[0035] In one possible implementation, the process of converting video modal data into text features that match its modality may include the following steps: First, the video modal data is split into image modal data and audio modal data. Then, multiple scenes contained in the video modal data are identified by inter-frame difference calculation or a deep learning-based scene segmentation model, and the start and end times of each scene are determined.

[0036] Secondly, for each identified scene, based on the start and end times of the current scene, image and audio modal data segments corresponding to the specified time periods are extracted from both the image and audio modal data. These segments are then parsed to generate corresponding text features. Specifically, text features are extracted from each image frame in the image modal segment (see above), and the union of the text features from all image frames is used as the text features for that image modal data segment. For the audio modal data segment, corresponding text features are generated through speech recognition and audio feature analysis (see above). Subsequently, based on the start and end times of the current scene, the text features of the image and audio modal data segments, a pre-trained large language model is used for semantic fusion and summarization to generate a summary of the current scene. The start and end times are then combined with the summary content to form a timeline summary of the current scene.

[0037] After the traversal is complete, a timeline summary of each scene is obtained. The timeline summaries of each scene are combined in order to obtain a timeline summary of the video modal data.

[0038] In a specific example, a smartwatch product introduction video was analyzed into three scenarios: Scenario 1 (0:00-0:30): The text features of the image modal data segment are ["round dial", "metal material"], and the text features of the audio modal data segment are ["introducing the product appearance"]. Therefore, the timeline summary of Scenario 1 is "0:00-0:30 Showing a round metal dial and introducing the product appearance". Scenario 2 (0:30-1:30): The text features of the image modal data segment are ["heart rate sensor", "screen data"], and the text features of the audio modal data segment are ["accurate heart rate monitoring"]. Therefore, the timeline summary of Scenario 2 is "0:30-1:30 Display heart rate sensor and monitoring data, explain the accurate heart rate function"; Scenario 3 (1:30-2:00): The text features of the image modal data fragment are ["sleep report", "chart"], and the audio text is ["sleep monitoring science"]. Therefore, the timeline summary of Scenario 3 is "1:30-2:00 Show sleep report charts and explain scientific sleep monitoring".

[0039] The final video timeline summary is an ordered combination of the above three summaries, namely "0:00-0:30 Showing the round metal dial and introducing the product's appearance; 0:30-1:30 Displaying the heart rate sensor and monitoring data, explaining the accurate heart rate function; 1:30-2:00 Showing the sleep report chart and explaining scientific sleep monitoring." This timeline summary is the textual feature of the video modality data.

[0040] Furthermore, based on the generated video timeline summary, natural language processing technology will be used to perform deep processing on the video timeline summary to obtain the entity, keyword and weight, and topic distribution information of the video timeline summary (see above for specific steps). The video modal data itself, the text features of the video modal data, the video timeline summary, and the entity, keyword and weight, and topic distribution information further parsed from the video timeline summary will be associated and stored, thereby establishing a complete and efficient data foundation for subsequent multimodal hybrid retrieval.

[0041] Fourth, the text features of each modality are combined sequentially to obtain a multimodal comprehensive index item for the multimodal text fragment. Specifically, the text features of each modality are combined in the order of text features, image features, audio features, and video features to obtain a multimodal comprehensive index item for the multimodal text fragment.

[0042] For example, in a smart product knowledge retrieval scenario, after parsing a multimodal text fragment of a smartwatch, the text features of the resulting text modal data are ["smartwatch", "heart rate monitoring", "sleep monitoring", "round dial"], the text features of the image modal data are ["round outline", "metal material"], the text features of the audio modal data are ["accurate heart rate monitoring", "scientific sleep monitoring", volume (louder in the function description)], and the text features of the video modal data are ["0:00 - 0:30 showcasing the watch's appearance, 0:30 - 1:30 introducing the heart rate monitoring function, 1:30 - 2:00 introducing the sleep monitoring function"]. Therefore, the multimodal comprehensive index item for this multimodal text fragment is {"smartwatch", "heart rate monitoring", "sleep monitoring", "round dial", "round outline", "metal material", "accurate heart rate monitoring", "scientific sleep monitoring", volume (louder in the function description), "0:00 - 0:30 showcasing the watch's appearance, 0:30 - 1:30 introducing the heart rate monitoring function, 1:30 - 2:00 introducing the sleep monitoring function"]. 2:00 Introduction to the sleep monitoring function. This multimodal comprehensive index integrates the multimodal feature information of the smartwatch. When searching, if a user queries "smartwatch heart rate monitoring function", the system can match this multimodal comprehensive index and use the text features of relevant modalities such as text, images, audio and video to find the corresponding multimodal data more comprehensively and accurately.

[0043] Following the method described above, the multimodal comprehensive index items of each multimodal text fragment can be stored in the database, which facilitates the subsequent use of the stored multimodal comprehensive index items for fast and accurate mixed retrieval of multimodal data.

[0044] It should be noted that the number of modalities of text features included in a multimodal text fragment will determine the number of modalities included in its corresponding multimodal comprehensive index item.

[0045] After constructing the database, multimodal hybrid retrieval of multimodal data can be performed based on the multimodal comprehensive index items in the database. Specifically, Figure 1 A flowchart illustrating a multimodal data hybrid retrieval method according to an embodiment of the present disclosure is shown. Figure 1 As shown, the method includes steps S1100-S1400.

[0046] S1100: Obtain the user's query information and the modality of the query information, and convert the query information into text features that match the modality. The modality includes at least one of text, image, audio, and video. Specifically, after obtaining the user's query information, the modality of the query information is first identified, and then the query information is converted into text features corresponding to its modality. The specific conversion methods for each modality's text features are described above and will not be repeated here.

[0047] It should be noted that the user's query information may include multimodal query information (for example, when entering text and images at the same time). In this case, after converting the query information of each modality into corresponding text features, the text features corresponding to each modality query information are combined in sequence as the text features of the query information.

[0048] Furthermore, after obtaining the text features of the query information, a large language model will be used to generate a text description corresponding to the text features of the query information. Natural language processing will be performed on the text description to obtain information such as entities, keywords, weights, and topic distribution information corresponding to the text description. The query information, the text features of the query information, the text description of the query information, and the entities, keywords, weights, and topic distribution information of the text description will be stored together to facilitate subsequent similarity calculations.

[0049] S1200: Calculate the similarity between the text features of the query information and each pre-built multimodal comprehensive index item in the database, wherein each multimodal comprehensive index item includes text features of at least one modality. Specifically, this may include the following steps: Traverse each multimodal comprehensive index item in the database. For each traversed multimodal comprehensive index item, sequentially calculate the similarity between the text features of the query information and the text features of each modality in the current multimodal comprehensive index item, and determine the similarity between the text features of the query information and the text features of each modality in the current multimodal comprehensive index item based on the similarity between the text features of the query information and the text features of each modality in the current multimodal comprehensive index item; after traversal, obtain the similarity between the text features of the query information and each multimodal comprehensive index item in the database.

[0050] In one possible implementation, the similarity score (Score) between the text features of the query information and the text features of each modality in the current multimodal comprehensive index is determined based on the matching score (M) and relevance score (r) between the text features of the query information and the text features of each modality in the current multimodal comprehensive index. Specifically, the similarity score (Score) between the text features of the query information and the text features of each modality is calculated as: Score = λ1 × M + (1 - λ1) × r, where λ1 is the fusion weight of the matching score and the relevance score. This weight can be set according to the specific scenario; preferably, λ1 can be set to 0.7, which emphasizes both matching degree and relevance.

[0051] It should be noted that, in order to improve the accuracy of matching score calculation for different modal data, different matching score calculation methods were set for each modal data. The matching score calculation methods for each modal data will be introduced below.

[0052] In one possible implementation, calculating the matching score between the text features of the query information and the text features of the text modality data in the current multimodal comprehensive index item may include the following steps: First, calculate the keyword coverage score. Specifically, obtain the text description of the query information associated with the text features of the query information, and extract the keyword set Q corresponding to this text description. Obtain the keyword set D associated with the text features of the text modality data. Calculate the keyword coverage R based on the keyword set Q and the keyword set D. The formula for calculating the keyword coverage R is as follows: Secondly, calculate the cumulative score of keyword weights. Specifically, calculate the intersection Q of keyword set Q and keyword set D. D, calculate the intersection Q The sum of the weights of each keyword in D is used as the cumulative score S for keyword weights.

[0053] Next, the angle between the semantic center vectors is calculated. Specifically, the text description associated with the text features of the query information and the original text modality data associated with the text features of the text modality data are obtained. A pre-trained sentence encoder model (such as a lightweight encoder based on Transformer) is used to encode the obtained text description and text modality data into high-dimensional vector representations, and the cosine similarity between the two vectors is calculated. , which serves as the angle between the semantic center vectors.

[0054] Finally, by combining the keyword coverage score, cumulative keyword weight score, and semantic center vector angle, the matching score M between the text features of the query information and the text features of the text modality data in the current multimodal comprehensive index item is calculated.text-text Among them, the matching score M text-text The calculation formula is as follows: In the formula, R is the keyword coverage score, and S is the cumulative keyword weight score. Let α be the angle between the semantic center vectors, and let α, β, and γ be the weight coefficients, satisfying α + β + γ = 1. These coefficients can be determined based on the actual application scenario through experience or machine learning methods.

[0055] In one possible implementation, the following steps may be included when calculating the matching score between the text features of the query information and the text features of the image modality data in the current multimodal comprehensive index item: First, calculate the global matching score between the text features of the query information and the text features of the image modality data. global-similarity(v t , v i ) Specifically, a self-developed lightweight text-image interaction encoder is first used to convert the text features of the query information into vectors. v t Convert text features of image modal data into vectors v i This is done to align the text features of the query information with the text features of the image modal data; then, vectors are calculated. v t sum vector v i The cosine similarity between the two is used as a global matching score between the text features of the query information and the text features of the image modality data.

[0056] Secondly, calculate the local matching score between the text features of the query information and the text features of the image modality data. mean_local_similaritySpecifically, the process involves: acquiring the original image modality data associated with the text features of the image modality data; segmenting the image modality data into multiple local regions according to preset rules; generating a corresponding text feature for each local region (see the process of generating text features for image modality data); acquiring the text description associated with the text features of the query information; segmenting the text description into multiple text segments (such as phrases or sentences); generating a corresponding text feature for each text segment (see the process of generating text features for text modality data); traversing each text segment; calculating the cosine similarity between the text features of the current text segment and the text features of each local region for the current text segment (see the calculation process of the global matching score); completing the traversal and obtaining the cosine similarity between the text features of each text segment and the text features of each local region; and taking the maximum value among all cosine similarities as the local matching score between the text features of the query information and the text features of the image modality data.

[0057] Finally, the local matching scores and global matching scores are weighted and summed to obtain the matching degree score between the text features of the query information and the text features of the image modality data. M text-image Among them, the matching score M text-image The calculation formula is as follows: M text-image =λ2× global-similarity(v t , v i )+（1- λ2 ） × mean_local_similarity In the formula, λ2 is the fusion weight between the global matching score and the local matching score, which can be set according to the specific scenario.

[0058] In one possible implementation, calculating the matching score between the text features of the query information and the text features of the audio modality data in the current multimodal comprehensive index item may include the following steps: First, obtain the text description of the query information associated with the text features of the query information, and the text modality data converted from the audio modality data associated with the text features of the audio modality data. It also calculates the text description of the query information and the text modal data converted from the audio modal data. Text matching similarity between For the detailed calculation process, please refer to the matching score M. text-text This will not be elaborated upon here.

[0059] Secondly, sentiment words (such as "excited" and "calm") are extracted from the text description of the query information, and audio feature descriptions are extracted from the text features of the audio modality data. The sentiment matching degree between the extracted sentiment words and the audio feature descriptions is then calculated. If the text description of the query information emphasizes "excitement," then the emotional intensity, volume fluctuation, and spectral energy fluctuation in the audio feature description will exceed the preset threshold.

[0060] Finally, based on text matching similarity And emotional compatibility Calculate the matching score M between the text features of the query information and the text features of the audio modality data in the current multimodal comprehensive index item. text-audio Among them, M text-audio The calculation formula is as follows: +(1- ) In the formula, Match text similarity And emotional compatibility The fusion weight can be set according to the specific scenario.

[0061] In one possible implementation, the calculation of the matching score between the text features of the query information and the text features of the video modal data in the current multimodal comprehensive index item may include the following steps: First, extract the timeline summary of each scene from the text features of the video modal data, and then calculate the similarity between the text features of the query information and the timeline summaries of each scene in turn (see Matching Score M for the specific calculation process). text-text The maximum similarity is taken as the matching score between the text features of the query information and the text features of the video modal data in the current multimodal comprehensive index item.

[0062] It should be noted here that the calculation process for the relevance score between the text features of the query information and the text features of each modality is the same, and may specifically include the following steps: First, obtain the text description of the query information associated with the text features of the query information, and extract the entities, keywords, weights, topic distribution information, and time information corresponding to the text description. Then, obtain the entities, keywords, weights, topic distribution information, and time information associated with the text features of each modality.

[0063] Secondly, based on the topic distribution information corresponding to the text description and the topic distribution information associated with the text features of each modality, a topic relevance score is calculated. Specifically, the cosine similarity between the topic distribution information corresponding to the text description and the topic distribution information associated with the text features of each modality is calculated and used as the topic relevance score. .

[0064] Next, based on the keywords and weights corresponding to the text description and the keywords and weights associated with the text features of each modality, the keyword relevance score is calculated. Specifically, based on the keywords and weights corresponding to the text description and the keywords and weights associated with the text features of each modality, a keyword coverage score and a cumulative keyword weight score are calculated. The specific calculation process is described above and will not be repeated here. The calculated keyword coverage score and cumulative keyword weight score are then weighted and fused to obtain the keyword relevance score. .

[0065] Next, based on the entities corresponding to the text descriptions and the entities associated with the text features of each modality, entity relevance scores are calculated. Specifically, the Jaccard similarity (intersection / union) between the entity set corresponding to the text description and the entity set associated with the text features of each modality is calculated as the entity relevance score. The entities are calculated using the Named Entity Recognition (NER) algorithm and can include names of people, places, organizations, products, and specific terms.

[0066] Next, based on the time information corresponding to the text description and the time information associated with the text features of each modality, the time relevance score is calculated. Specifically, the degree of temporal overlap between the temporal information corresponding to the text description and the temporal information associated with the text features of each modality is calculated, and the calculated degree of temporal overlap is used as the temporal relevance score. .

[0067] Finally, the topic relevance score, keyword relevance score, entity relevance score, and time relevance score are weighted and summed to obtain the relevance score R between the text features of the query information and the text features of each modality. The formula for calculating this relevance score R is shown below: In the formula, , , , The weighting coefficients for each sub-relevance score can be set based on scenario experience or obtained through training. Preferably, they can be set as follows: =0.4、 =0.3、 , =0.1.

[0068] In one possible implementation, when determining the similarity between the text features of the query information and the text features of each modality in the current multimodal comprehensive index item based on the similarity between the text features of the query information and the text features of each modality in the current multimodal comprehensive index item, a weighted average of the similarities between the text features of the query information and the text features of each modality in the current multimodal comprehensive index item can be used as the similarity between the text features of the query information and the current multimodal comprehensive index item.

[0069] Referring to the method described above, the similarity between the text features of the query information and each multimodal comprehensive index item in the database can be calculated.

[0070] S1300: Based on the similarity between text features and each multimodal index item, target multimodal index items are selected. Specifically, multimodal index items whose similarity meets a preset similarity threshold are selected as target multimodal index items, and the target multimodal index items are sorted in descending order of similarity to obtain a list of target multimodal index items.

[0071] S1400: Return the multimodal data associated with the target multimodal composite index item as the retrieval result corresponding to the query information. Specifically, after obtaining the list of target multimodal composite index items, for each target multimodal composite index item in the list, extract the original modal data of each modality associated with the target multimodal composite index item and return it as the retrieval result corresponding to that target multimodal composite index item.

[0072] Furthermore, the system presents the sorted search results to the user in an integrated format, including text modal data, image modal data (if present), audio modal data (if present), and video modal data (if present) associated with the target multimodal comprehensive index item. For example, when a user searches for a product, the system can simultaneously display product description text, product images, and related product introduction audio clips, achieving a unified and structured output of multimodal search results.

[0073] Furthermore, after users view the search results, the system automatically collects user feedback, such as positive or negative evaluations of a particular search result (e.g., "likes" or "dislikes"). By analyzing user feedback, the system can identify the degree of matching between the user's true intent and the search results. Based on this, it dynamically adjusts the search model parameters, optimizes the multimodal feature extraction and similarity matching algorithms, updates the multimodal comprehensive index structure, and continuously optimizes the result ranking logic, thereby continuously improving the accuracy and efficiency of multimodal hybrid retrieval. This disclosure provides a multimodal data hybrid retrieval method, comprising: acquiring user query information and the modality of the query information, and converting the query information into text features matching the modality, wherein the modality includes at least one of text, image, audio, and video; calculating the similarity between the text features and various pre-constructed multimodal comprehensive index items in the database, wherein the multimodal comprehensive index items include text features of at least one modality; filtering target multimodal comprehensive index items based on the similarity between the text features and each multimodal comprehensive index item; and returning multimodal data associated with the target multimodal comprehensive index item as the retrieval result corresponding to the query information. This disclosure achieves semantic integration and comparability of cross-modal data by uniformly converting multimodal data such as text, image, audio, and video into text features of the same modality and constructing a unified multimodal comprehensive index item. This method effectively overcomes the semantic barriers caused by modal differences in traditional retrieval, enabling similarity calculations for different modal data in the same feature space, thereby achieving efficient and accurate responses to multimodal queries and improving the overall performance and user experience of the retrieval system.

[0074] <Device Embodiment> Figure 2 A schematic block diagram of a multimodal data hybrid retrieval apparatus according to an embodiment of the present disclosure is shown. Figure 2 As shown, the device 100 includes: The query information acquisition module 110 is used to acquire the user's query information and the modality of the query information, and convert the query information into text features that match the modality, wherein the modality includes at least one of text, image, audio and video; The similarity calculation module 120 is used to calculate the similarity between text features and various pre-built multimodal comprehensive index items in the database, wherein the multimodal comprehensive index items include text features of at least one modality; The filtering module 130 is used to filter out target multimodal comprehensive index items based on the similarity between text features and each multimodal comprehensive index item; The result feedback module 140 is used to return multimodal data associated with the target multimodal comprehensive index item as the retrieval result corresponding to the query information.

[0075] <Equipment Example> Figure 3 A schematic block diagram of a multimodal data hybrid retrieval device according to an embodiment of the present disclosure is shown. Figure 3 As shown, the multimodal data hybrid retrieval device 200 includes a processor 210 and a memory 220 for storing executable instructions of the processor 210. The processor 210 is configured to implement any of the aforementioned multimodal data hybrid retrieval methods when executing the executable instructions.

[0076] It should be noted here that the number of processors 210 can be one or more. Furthermore, the multimodal data retrieval device 200 of this embodiment may also include an input device 230 and an output device 240. The processors 210, memory 220, input device 230, and output device 240 can be connected via a bus or other means, without specific limitations here.

[0077] The memory 220, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and various modules, such as the program or module corresponding to the multimodal data hybrid retrieval method of this disclosure embodiment. The processor 210 executes various functional applications and data processing of the multimodal data hybrid retrieval device 200 by running the software program or module stored in the memory 220.

[0078] Input device 230 can be used to receive input digital numbers or signals. These signals may include key signals related to user settings and function control of the device / terminal / server. Output device 240 may include a display device such as a screen.

[0079] <Storage Medium Examples> According to a fourth aspect of this disclosure, a non-volatile computer-readable storage medium is also provided, on which computer program instructions are stored, which, when executed by processor 210, implement any of the preceding multimodal data hybrid retrieval methods.

[0080] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A multimodal data hybrid retrieval method, characterized in that, include: The system acquires user query information and the modality of the query information, and converts the query information into text features that match the modality, wherein the modality includes at least one of text, image, audio, and video; Calculate the similarity between the text features and each pre-built multimodal comprehensive index item in the database, wherein the multimodal comprehensive index item includes text features of at least one modality; Based on the similarity between the text features and each of the multimodal comprehensive index items, target multimodal comprehensive index items are selected; The multimodal data associated with the target multimodal comprehensive index item is returned as the retrieval result corresponding to the query information.

2. The method according to claim 1, characterized in that, When constructing multimodal comprehensive index items, the following are included: Retrieve multimodal text fragments; The multimodal text fragment is parsed to obtain at least two modal data among text modal data, image modal data, audio modal data, and video modal data; Each modal data is converted into text features that match its modality to obtain the text features of each modal data; The text features of each modality data are combined in sequence to obtain the multimodal comprehensive index item of the multimodal text fragment.

3. The method according to claim 2, characterized in that, When converting text modal data into text features that match its modality, the following steps are included: Keywords were extracted from the text modality data using natural language processing algorithms; All extracted keywords are combined in sequence to obtain the text features of the text modality data.

4. The method according to claim 2, characterized in that, When converting image modal data into text features that match its modality, the following steps are included: Image recognition algorithms are used to extract key feature descriptions of objects from the image modal data; The extracted key feature descriptions are combined in sequence to obtain the text features of the image modality data.

5. The method according to claim 2, characterized in that, When converting audio modal data into text features that match its modality, the following steps are included: The audio modal data is converted into text modal data, and the keywords in the text modal data are extracted as the keywords of the audio modal data. Perform audio feature analysis on the audio modal data to generate an audio feature description of the audio modal data; The keywords and audio feature descriptions of the audio modality data are combined in sequence to obtain the text features of the audio modality data.

6. The method according to claim 2, characterized in that, When converting video modal data into text features that match its modality, the following steps are included: The video modal data is split into image modal data and audio modal data, and multiple scenes included in the video modal data and the start and end times of each scene are identified; Iterate through each of the aforementioned scenes. For the current scene, extract the start and end times of the current scene. Based on the start and end times of the current scene, extract image modal data segments and audio modal data segments of the corresponding time periods from the image modal data and the audio modal data, respectively. Based on the start and end times of the current scene and the image modal data segments and audio modal data segments of the corresponding time periods, generate a timeline summary of the current scene. After the traversal is completed, the timeline summary of each scene is obtained, and the timeline summaries of each scene are combined in order to obtain the timeline summary of the video modal data; The timeline summary of the video modal data is used as the text feature of the video modal data.

7. The method according to claim 1, characterized in that, Calculating the similarity between the text features and the various pre-built multimodal comprehensive index items in the database includes: Traverse each of the aforementioned multimodal comprehensive index items in the database; For each traversed multimodal index item, the similarity between the text feature and the text features of each modality in the current multimodal index item is calculated sequentially. Based on the similarity between the text feature and the text features of each modality in the current multimodal index item, the similarity between the text feature and the current multimodal index item is determined. After the traversal is complete, the similarity between the text features and each multimodal comprehensive index item in the database is obtained.

8. A multimodal data hybrid retrieval device, characterized in that, include: The query information acquisition module is used to acquire the user's query information and the modality of the query information, and convert the query information into text features that match the modality, wherein the modality includes at least one of text, image, audio and video; A similarity calculation module is used to calculate the similarity between the text features and various pre-built multimodal comprehensive index items in the database, wherein the multimodal comprehensive index items include text features of at least one modality; The filtering module is used to filter out target multimodal comprehensive index items based on the similarity between the text features and each of the multimodal comprehensive index items; The result feedback module is used to return multimodal data associated with the target multimodal comprehensive index item as the retrieval result corresponding to the query information.

9. A multimodal data hybrid retrieval device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the method of any one of claims 1 to 7 when executing the executable instructions.

10. A non-volatile computer-readable storage medium storing computer program instructions thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 7.

Images