Multimodal retrieval feature library construction method, multimodal retrieval method and related device

By extracting and fusing multi-dimensional features from multimodal data, a high-quality multimodal retrieval feature library is constructed, which solves the problems of low retrieval recall and accuracy in existing technologies and achieves more efficient multimodal data retrieval.

CN118797115BActive Publication Date: 2026-11-17ANHUI IFLYTEK INTELLIGENT SYST
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411127541.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-16
Publication Date
2026-11-17
Estimated Expiration
2044-08-16

AI Technical Summary

Technical Problem

The existing multimodal retrieval feature database is of low quality, resulting in low retrieval recall and accuracy.

Method used

By acquiring multi-dimensional features from multimodal data, including visual, auditory, and content aspects, we perform intra-modal and inter-modal feature fusion to ultimately build a high-quality multimodal retrieval feature library.

Benefits of technology

It improves the recall and accuracy of multimodal retrieval.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118797115B_ABST
    Figure CN118797115B_ABST
Patent Text Reader

Abstract

This application discloses a method for constructing a multimodal retrieval feature library, a multimodal retrieval method, and related apparatus, relating to the field of multimodal retrieval technology. The method includes: acquiring multimodal data; acquiring features of multiple dimensions for each modality in the multimodal data to obtain a feature set corresponding to each modality; fusing the features in each feature set; fusing the fused features corresponding to the data of multiple modalities to obtain multimodal fused features; storing the multimodal fused features in a multimodal retrieval feature library; after constructing the feature library, acquiring query data and processing it into query features; determining the N features with the highest similarity to the query features from the multimodal retrieval feature library, and determining the multimodal data corresponding to the N features as the retrieval results and outputting them. The feature library construction method disclosed in this application can construct a high-quality multimodal retrieval feature library, and the retrieval method provided in this application has high retrieval recall and retrieval accuracy.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of multimodal retrieval technology, and in particular to a method for constructing a multimodal retrieval feature library, a multimodal retrieval method, and related apparatus. Background Technology

[0002] Multimodal data retrieval technology is a technique that allows users to input query data and retrieve related data in multiple modalities.

[0003] Current multimodal retrieval methods are mostly feature-based, which means that the query data is processed into query features, features similar to the query features are found in the constructed multimodal retrieval feature library, corresponding multimodal data are found based on the features similar to the query features, and then the found multimodal data is output.

[0004] As can be seen from the above process, a multimodal retrieval feature library is crucial, and its quality directly affects the retrieval results. How to construct a high-quality multimodal retrieval feature library is a problem that urgently needs to be solved. Summary of the Invention

[0005] In view of this, this application provides a method for constructing a multimodal retrieval feature library, a multimodal retrieval method, and related apparatus, to construct a high-quality multimodal retrieval feature library, thereby achieving high retrieval recall and retrieval accuracy during retrieval. The technical solution is as follows:

[0006] The first aspect of this application provides a method for constructing a multimodal retrieval feature library, including:

[0007] Acquire multimodal data, wherein the multimodal data contains data related to multiple modalities;

[0008] For each modality of the multimodal data, features of multiple dimensions are obtained to obtain a feature set corresponding to each modality of the multimodal data; wherein, the feature set corresponding to any modality of the data contains features of one or more of the following three levels: visual level, auditory level, and content level;

[0009] The features in the feature set corresponding to each modality of the multimodal data are fused to obtain the target features corresponding to each modality of the multimodal data.

[0010] The target features corresponding to the multiple modalities contained in the multimodal data are fused to obtain the target features corresponding to the multimodal data.

[0011] The target features corresponding to the multimodal data are stored in the multimodal retrieval feature library.

[0012] In one possible implementation, for any modality of data, if the data is text, the corresponding feature set includes content-level features; if the data is an image, the corresponding feature set includes visual and content-level features; if the data is audio, the corresponding feature set includes auditory and content-level features; and if the data is video, the corresponding feature set includes visual, auditory, and content-level features.

[0013] In one possible implementation, obtaining features of multiple dimensions from the data of each modality included in the multimodal data to obtain a feature set corresponding to the data of each modality included in the multimodal data includes:

[0014] For each modality contained in the multimodal data:

[0015] If the data for this modality is text, then based on the data for this modality, features of multiple dimensions are obtained to get the feature set corresponding to the data for this modality;

[0016] If the data for this modality is an image, audio, or video, then the data for this modality is processed into data for other modalities. Based on the data for this modality and the processing results, features of multiple dimensions are obtained to obtain the feature set corresponding to the data for this modality.

[0017] In one possible implementation, if the data of the modality is text, then based on the data of the modality, features of multiple dimensions are obtained, including:

[0018] If the language to which the text belongs is not the specified language, then the text is translated into the specified language to obtain the translated text.

[0019] The text and its translation are segmented into words to obtain the segmentation results of the text and the translation.

[0020] The translated text of the text is subjected to element information recognition and semantic information extraction to obtain the element information and semantic information of the translated text.

[0021] Feature extraction is performed on the text, the word segmentation result of the text, the translated text, the word segmentation result of the translated text, the element information of the translated text, and the semantic information of the translated text to obtain features in multiple dimensions.

[0022] In one possible implementation, if the data of the modality is an image, audio, or video, then the data of the modality is processed into data of other modalities, and multiple-dimensional features are obtained based on the data of the modality and the processing result of the data of the modality, including:

[0023] If the data of this modality is an image, then content recognition is performed on the image to obtain the content text of the image, and features of multiple dimensions are obtained based on the image and the content text of the image;

[0024] If the data of this modality is audio, then the audio is transcribed to obtain the audio content text, and features of multiple dimensions are obtained based on the audio and the audio content text;

[0025] If the data for this modality is video, then the video is split into audio and multiple images, and features of multiple dimensions are obtained based on the video and the split audio and multiple images.

[0026] In one possible implementation, obtaining multi-dimensional features based on the image and the text content of the image includes:

[0027] If the language of the content text of the image is not the specified language, then the content text of the image is translated into the specified language to obtain the translated content text of the image.

[0028] The content text and the translated content text of the image are segmented into words to obtain the word segmentation results of the content text and the translated content text of the image.

[0029] The translated text of the image content is subjected to element information recognition and semantic information extraction to obtain the element information and semantic information of the translated text of the image content;

[0030] Feature extraction is performed on the content text of the image, the word segmentation result of the content text of the image, the translated text of the image, the word segmentation result of the translated text of the image, the element information of the translated text of the image, and the semantic information of the translated text of the image to obtain features at the content level. Feature extraction is also performed on the image to obtain features at the visual level.

[0031] In one possible implementation, obtaining multi-dimensional features based on the audio and the text content of the audio includes:

[0032] If the language of the audio content text is not the specified language, then the audio content text is translated into the specified language to obtain the translated audio content text;

[0033] The audio content text and the translated content text are segmented into words to obtain the word segmentation results of the audio content text and the translated content text.

[0034] The translated text of the audio content is subjected to element information recognition and semantic information extraction to obtain the element information and semantic information of the translated text of the audio content;

[0035] Feature extraction is performed on the audio content text, the word segmentation result of the audio content text, the audio content translated text, the word segmentation result of the audio content translated text, the element information of the audio content translated text, and the semantic information of the audio content translated text to obtain features at the content level in multiple dimensions. Feature extraction is also performed on the audio to obtain features at the auditory level.

[0036] In one possible implementation, obtaining multi-dimensional features based on the video, the split audio, and multiple images includes:

[0037] The audio obtained from the split audio is transcribed to obtain the content text of the split audio; if the language of the content text of the split audio is not the specified language, the content text of the split audio is translated into the specified language to obtain the translated content text of the split audio.

[0038] For each image obtained from the segmentation, content recognition is performed on the image to obtain the content text of the image. If the language of the content text of the image is not the specified language, the content text of the image is translated into the specified language to obtain the translated content text of the image.

[0039] Based on the video, obtain video visual features; based on each image obtained from the splitting, obtain image visual features; based on the audio obtained from the splitting, obtain auditory features; based on the content text and translated text of the split audio, obtain audio content features; based on the content text and translated text of each image obtained from the splitting, obtain image content features.

[0040] In one possible implementation, fusing the target features corresponding to the multiple modalities included in the multimodal data to obtain the target features corresponding to the multimodal data includes:

[0041] Align the target features corresponding to the data of the multiple modalities contained in the multimodal data to obtain the aligned target features corresponding to the data of the multiple modalities contained in the multimodal data.

[0042] Based on the aligned target features corresponding to the multiple modalities contained in the multimodal data, the weights corresponding to the multiple modalities contained in the multimodal data are determined.

[0043] Based on the weights corresponding to the multiple modalities contained in the multimodal data, the aligned target features corresponding to the multiple modalities contained in the multimodal data are weighted and summed to obtain the target features corresponding to the multimodal data.

[0044] A second aspect of this application provides a multimodal retrieval method, including:

[0045] Retrieve query data;

[0046] The query data is processed into features to obtain query features;

[0047] The N features with the highest similarity to the query feature are determined from the multimodal retrieval feature library, wherein the multimodal retrieval feature library is constructed using any of the multimodal retrieval feature library construction methods described above;

[0048] The multimodal data corresponding to the N features are determined as the retrieval results, where N is an integer greater than 0;

[0049] Output the search results.

[0050] In one possible implementation, the multimodal retrieval method also includes:

[0051] The search results are reviewed to obtain the review results of the search results;

[0052] Based on the review results of the search results, the features in the multimodal search feature library are optimized.

[0053] A third aspect of this application provides a multimodal retrieval feature library construction apparatus, comprising: a data acquisition module, a feature acquisition module, a first feature fusion module, a second feature fusion module, and a feature storage module;

[0054] The data acquisition module is used to acquire multimodal data, wherein the multimodal data contains data related to multiple modalities;

[0055] The feature acquisition module is used to acquire features of multiple dimensions for each modality of the multimodal data to obtain a feature set corresponding to each modality of the multimodal data; wherein, the feature set corresponding to any modality of the data contains features of one or more of the following three levels: visual level, auditory level and content level;

[0056] The first feature fusion module is used to fuse the features in the feature set corresponding to the data of each modality contained in the multimodal data to obtain the target features corresponding to the data of each modality contained in the multimodal data;

[0057] The second feature fusion module is used to fuse the target features corresponding to the multiple modalities contained in the multimodal data to obtain the target features corresponding to the multimodal data.

[0058] The feature storage module is used to store the target features corresponding to the multimodal data into the multimodal retrieval feature library.

[0059] The fourth aspect of this application provides a multimodal retrieval device, including: a query data acquisition module, a query data processing module, a multimodal retrieval module, and a retrieval result output module;

[0060] The query data acquisition module is used to acquire query data;

[0061] The query data processing module is used to process the query data into features to obtain query features;

[0062] The multimodal retrieval module is used to determine the N features with the highest similarity to the query feature from the multimodal retrieval feature library, and to determine the multimodal data corresponding to the N features as the retrieval result. The multimodal retrieval feature library is constructed using the multimodal retrieval feature library construction device described above, and N is an integer greater than 0.

[0063] The search result output module is used to output the search results.

[0064] A fifth aspect of this application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0065] The memory is used to store computer programs;

[0066] The processor is used to execute the computer program so that the electronic device can implement the multimodal retrieval feature library construction method described above, and / or implement the steps of the multimodal retrieval method described above.

[0067] The sixth aspect of this application provides a computer storage medium carrying one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the multimodal retrieval feature library construction method described in any one of the above claims, and / or to implement the steps of the multimodal retrieval method described in any one of the above claims.

[0068] This application provides a computer program product comprising computer-readable instructions that, when executed on an electronic device, enable the electronic device to implement the multimodal retrieval feature library construction method described in any one of the preceding claims, and / or to implement the steps of the multimodal retrieval method described in any one of the preceding claims.

[0069] By employing the above technical solution, the multimodal retrieval feature library construction method provided in this application, after obtaining multimodal data, firstly acquires multi-dimensional features for each modality of the multimodal data to obtain a feature set corresponding to each modality of the multimodal data. Then, it fuses the features in the feature sets corresponding to each modality of the multimodal data to obtain the target features corresponding to each modality. Next, it fuses the target features corresponding to the data from multiple modalities of the multimodal data to obtain the target features corresponding to the multimodal data. Finally, it stores the target features corresponding to the multimodal data in the multimodal retrieval feature library. The multimodal retrieval feature library construction method provided in this application can acquire relatively rich features for each modality of data. Based on this, through intra-modal feature fusion and inter-modal feature fusion, it can ultimately obtain multimodal fusion features that can better represent multimodal data, thereby constructing a high-quality multimodal retrieval feature library. Based on the multimodal retrieval feature library construction method provided in this application, this application also provides a multimodal retrieval method. This retrieval method determines the N features with the highest similarity to the query feature from the multimodal retrieval feature library, and then finds the multimodal data corresponding to the N features with the highest similarity to the query feature, thereby obtaining the retrieval results. Since the features contained in the multimodal retrieval feature library can well express the corresponding multimodal data, the retrieval method based on the multimodal retrieval feature library provided in this application has a high retrieval recall and retrieval accuracy. Attached Figure Description

[0070] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0071] Figure 1 This is a schematic diagram of a system architecture involved in this application;

[0072] Figure 2 A schematic diagram of a hardware structure of a terminal provided in an embodiment of this application;

[0073] Figure 3 A schematic diagram of a server hardware structure provided in an embodiment of this application;

[0074] Figure 4 A flowchart illustrating the method for constructing a multimodal retrieval feature library provided in this application embodiment;

[0075] Figure 5A schematic diagram illustrating the multimodal retrieval feature library construction method and the multimodal retrieval method provided in the embodiments of this application;

[0076] Figure 6 A flowchart illustrating the multimodal retrieval method provided in this application embodiment;

[0077] Figure 7 This is a schematic diagram of the structure of the multimodal retrieval feature library construction device provided in the embodiments of this application;

[0078] Figure 8 This is a schematic diagram of the structure of the multimodal retrieval device provided in the embodiments of this application. Detailed Implementation

[0079] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0080] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0081] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0082] Before introducing the multimodal retrieval feature library construction method and multimodal retrieval method provided in this application, the system architecture involved in this application will be explained first.

[0083] like Figure 1 As shown, the system architecture involved in this application may include a terminal device 101 and a server 102. The terminal device 101 can interact with the server 102 via a network (wired network or wireless network). The server 102 may include one or more servers (…). Figure 1 (This example uses a server as an illustration.)

[0084] The product form of the aforementioned terminal will be described next.

[0085] The aforementioned terminals can be mobile phones, tablets, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), etc., and this application embodiment does not impose any limitations on them.

[0086] Figure 2 A schematic diagram of one possible hardware architecture for the terminal is shown.

[0087] refer to Figure 2 As shown, the terminal may include components such as a radio frequency unit 210, a memory 220, an input unit 230, a display unit 240, a camera 250 (optional), an audio circuit 260 (optional), a speaker 261 (optional), a microphone 262 (optional), a headphone jack 263 (optional), a processor 270, an external interface 280, and a power supply 290. Those skilled in the art will understand that... Figure 2 This is merely an example of a terminal and does not constitute a limitation on the terminal. It may include more or fewer components than shown in the illustration, or combine certain components, or use different components.

[0088] The input unit 230 can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of the terminal. Specifically, the input unit 230 may include a touch screen 231 (optional) and / or other input devices 232. The touch screen 231 can collect touch operations performed by the user on or near it (such as operations performed by the user using fingers, knuckles, styluses, or any suitable object on or near the touch screen), and drive the corresponding connection devices according to a pre-set program. The touch screen can detect the user's touch actions, convert the touch actions into touch signals and send them to the processor 270, and can receive and execute commands sent by the processor 270; the touch signal includes at least touch point coordinate information. The touch screen 231 can provide an input interface and an output interface between the terminal and the user. In addition, various types of touch screens, such as resistive, capacitive, infrared, and surface acoustic wave, can be used to implement the touch screen. Besides the touch screen 231, the input unit 230 may also include other input devices. Specifically, other input devices 232 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0089] The display unit 240 can be used to display information input by the user or information provided to the user, various menus of the terminal, interactive interfaces, file display and / or playback of any multimedia file.

[0090] The memory 220 can be used to store instructions and data. The memory 220 may primarily include an instruction storage area and a data storage area. The data storage area can store various types of data, such as multimedia files and text. The instruction storage area can store software units such as operating systems, applications, and instructions required for at least one function, or subsets or extended sets thereof. It may also include non-volatile random access memory. It provides the processor 270 with hardware, software, and data resources for managing the computing device, supporting control software and applications. It is also used for storing multimedia files, as well as storing running programs and applications.

[0091] The processor 270 is the control center of the terminal, connecting various parts of the terminal through various interfaces and lines. It executes instructions stored in the memory 220 and calls data stored in the memory 220 to perform various functions and process data, thereby controlling the terminal as a whole. Optionally, the processor 270 may include one or more processing units; preferably, the processor 270 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 270. In some embodiments, the processor and memory can be implemented on a single chip; in some embodiments, they can also be implemented on separate chips. The processor 270 can also be used to generate corresponding operation control signals, send them to the corresponding components of the computing device, read and process data in the software, especially read and process data and programs in the memory 220, so that the various functional modules therein perform corresponding functions, thereby controlling the corresponding components to act according to the instructions.

[0092] The radio frequency (RF) unit 210 (optional) can be used for receiving and transmitting signals during information transmission or calls. For example, it can receive downlink information from the base station and process it for the processor 270; additionally, it can transmit uplink data to the base station. Typically, the RF unit 210 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), a duplexer, etc. Furthermore, the RF unit 210 can also communicate wirelessly with network devices and other devices. This wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.

[0093] In this embodiment, the radio frequency unit 210 can transmit data to other devices and receive data from other devices. It should be understood that the radio frequency unit 210 is optional and can be replaced with other communication interfaces, such as a network port.

[0094] The terminal also includes a power supply 290 (such as a battery) that supplies power to various components. Preferably, the power supply can be logically connected to the processor 270 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0095] The terminal also includes an external interface 280, which can be a standard Micro USB interface or a multi-pin connector, which can be used to connect the terminal to other devices for communication or to connect a charger to charge the terminal.

[0096] Although not shown, the terminal may also include a flash, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which will not be described in detail here.

[0097] The product form of the above-mentioned server will be described next.

[0098] Figure 3 A schematic diagram of the above-mentioned server structure is provided, such as... Figure 3As shown, the server may include a bus 301, a processor 302, a communication interface 303, and a memory 304. The processor 302, the memory 304, and the communication interface 303 communicate with each other via the bus 301.

[0099] Bus 301 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0100] The processor 302 can be any one or more of the following processors: central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP), or digital signal processor (DSP).

[0101] Memory 304 may include volatile memory, such as random access memory (RAM). Memory 304 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0102] The memory 304 can be used to store the multimodal retrieval feature library construction method and related software code. The processor 302 can call the software code stored in the memory 304, and can also schedule other units to implement the corresponding functions.

[0103] The processors in the aforementioned terminals and servers (e.g., processor 270 and processor 302) can be hardware circuits (such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), general-purpose processors, digital signal processors (DSPs), microprocessors, or microcontrollers, etc.) or combinations of these hardware circuits. For example, a processor can be a hardware system with instruction execution capabilities, such as a CPU or DSP, or a hardware system without instruction execution capabilities, such as an ASIC or FPGA, or a combination of the aforementioned hardware systems without instruction execution capabilities and hardware systems with instruction execution capabilities.

[0104] In the process of realizing this case, the inventors discovered that current methods for constructing multimodal retrieval feature libraries extract relatively single features when extracting features from each modality of the multimodal data (for example, using convolutional neural networks to extract image features for images, and obtaining Mel-frequency cepstral coefficients as audio features for audio). This results in the multimodal fusion features obtained by fusing the features corresponding to each modality of the multimodal data not being able to represent the multimodal data well. In other words, the quality of the finally constructed multimodal retrieval feature library is not high. Understandably, a low quality multimodal retrieval feature library will lead to low retrieval recall and retrieval accuracy.

[0105] To address the problem that current multimodal retrieval feature library construction methods fail to adequately represent multimodal data, resulting in low retrieval recall and accuracy, the inventors of this application have conducted research and, through continuous efforts, proposed a more effective multimodal retrieval feature library construction method. This method can construct a high-quality multimodal retrieval feature library. In addition to this method, this application also provides a multimodal retrieval method with high retrieval recall and accuracy.

[0106] The following embodiments will introduce the method for constructing a multimodal retrieval feature library and the multimodal retrieval method provided in this application.

[0107] Please see Figure 4 The diagram illustrates a flowchart of a method for constructing a multimodal retrieval feature library according to an embodiment of this application. This method may include:

[0108] Step S401: Acquire multimodal data.

[0109] Multimodal data contains data related across multiple modalities. For example, if multimodal data includes text, images, and audio, then the data of these three modalities are related. For instance, the text might be "a dog is running in the park," the image might be a picture of a dog running on the grass, and the audio might be the dog barking and ambient sounds from the park.

[0110] The multimodal data in this embodiment may be, but is not limited to, data crawled from web pages, data downloaded from an FTP server based on a file transfer protocol, or data obtained from an API interface.

[0111] Step S402: Obtain features of multiple dimensions for each modality contained in the multimodal data to obtain the feature set corresponding to each modality contained in the multimodal data.

[0112] Specifically, such as Figure 5 As shown, the multimodal data can first be classified into text, image, audio, and video to obtain the data for each modality. Then, features of multiple dimensions can be obtained from the data of each modality to obtain the feature set corresponding to the data of each modality.

[0113] For example, multimodal data includes text, images, audio, and video. After acquiring the multimodal data, it can first be classified into text, images, audio, and video to obtain text, images, audio, and video. Then, features of multiple dimensions are obtained for the text to obtain the feature set corresponding to the text, features of multiple dimensions are obtained for the images to obtain the feature set corresponding to the images, features of multiple dimensions are obtained for the audio to obtain the feature set corresponding to the audio, and features of multiple dimensions are obtained for the video to obtain the feature set corresponding to the video.

[0114] In this embodiment, the feature set corresponding to the data of any modality includes features from one or more of the following three levels: visual level, auditory level, and content level.

[0115] In one possible implementation, for any modality of data, if the data is text, the corresponding feature set includes content-level features; if the data is an image, the corresponding feature set includes visual and content-level features; if the data is audio, the corresponding feature set includes auditory and content-level features; and if the data is video, the corresponding feature set includes visual, auditory, and content-level features.

[0116] Step S403: Fuse the features in the feature set corresponding to each modality of the multimodal data to obtain the target features corresponding to each modality of the multimodal data.

[0117] For example, multimodal data includes text, images, audio, and video. After obtaining the feature sets corresponding to text, images, audio, and video respectively, such as... Figure 5 As shown, intramodal feature fusion is performed, that is, the features in the feature set corresponding to the text are fused to obtain the target features corresponding to the text, the features in the feature set corresponding to the image are fused to obtain the target features corresponding to the image, the features in the feature set corresponding to the audio are fused to obtain the target features corresponding to the audio, and the features in the feature set corresponding to the video are fused to obtain the target features corresponding to the video.

[0118] Step S404: Fuse the target features corresponding to the multiple modalities contained in the multimodal data to obtain the target features corresponding to the multimodal data.

[0119] For example, multimodal data includes text, images, audio, and video. After obtaining the target features corresponding to the text, images, audio, and video respectively, such as... Figure 5 As shown, feature fusion between modalities is performed, that is, the target features corresponding to text, images, audio and video are fused.

[0120] In one possible implementation, a weighted summation fusion method can be used to fuse the target features corresponding to the multiple modalities contained in the multimodal data to obtain the multimodal fused features, that is, the target features corresponding to the multimodal data.

[0121] Step S405: Store the target features corresponding to the multimodal data into the multimodal retrieval feature library.

[0122] It should be noted that when constructing the multimodal retrieval feature library, multiple multimodal data will be acquired. For each acquired multimodal data, the multimodal data will be processed according to the processing method provided in this application to obtain the target feature corresponding to the multimodal data. Then, the target feature corresponding to the multimodal data will be stored in the multimodal retrieval feature library. In this way, a multimodal retrieval feature library containing the target features corresponding to multiple multimodal data will be constructed.

[0123] The multimodal retrieval feature library construction method provided in this application first obtains features of multiple dimensions for each modality of the multimodal data after obtaining the multimodal data, so as to obtain the feature set corresponding to each modality of the multimodal data. Then, the features in the feature set corresponding to each modality of the multimodal data are fused to obtain the target features corresponding to each modality of the multimodal data. Next, the target features corresponding to the multiple modalities of the multimodal data are fused to obtain the target features corresponding to the multimodal data. Finally, the target features corresponding to the multimodal data are stored in the multimodal retrieval feature library. The multimodal retrieval feature library construction method provided in this application can obtain features in multiple dimensions for each modality of data. For example, for text, it can obtain features in multiple dimensions at the content level; for images, it can obtain features in multiple dimensions at the visual and content levels; for audio, it can obtain features in multiple dimensions at the auditory and content levels; and for video, it can obtain features in multiple dimensions at the visual, auditory, and content levels. It can be seen that the multimodal retrieval feature library construction method provided in this application can obtain relatively rich features for each modality of data. On this basis, through intra-modal feature fusion and inter-modal feature fusion, multimodal fusion features that can better represent multimodal data can be obtained, thereby constructing a high-quality multimodal retrieval feature library.

[0124] In another embodiment of this application, the specific implementation process of "step S402: obtaining features of multiple dimensions for each modality contained in the multimodal data, and obtaining the feature set corresponding to each modality contained in the multimodal data" in the above embodiment will be described.

[0125] The process of obtaining multi-dimensional features from each modality of the multimodal data to obtain the feature set corresponding to each modality of the multimodal data can include:

[0126] For each modality of data contained in the multimodal data, perform the following steps:

[0127] Step a: If the data of this modality is text, then based on the data of this modality, obtain features in multiple dimensions to obtain the feature set corresponding to the data of this modality.

[0128] When the data for this modality is text, the process of obtaining multi-dimensional features based on this modality data may include:

[0129] Step a1: Identify the language of the text.

[0130] Existing language identification methods can be used to identify the language to which the text belongs. For example, a language identification model can be used to identify the language to which the text belongs.

[0131] Step a2: If the language of the text is not the specified language, then translate the text into the specified language to obtain the translated text.

[0132] For example, if the language of the text is English and the specified language is Chinese, then the English text will be translated into Chinese to obtain the Chinese text.

[0133] It should be noted that if the text is in a specified language, then no translation is required.

[0134] Step a3: Perform word segmentation on the text and the translated text respectively to obtain the word segmentation results of the text and the translated text.

[0135] Existing word segmentation tools (such as Jieba, SnowNlp, nltk, NLPIR, etc.) can be used to segment the text and its translation separately to obtain the word segmentation results of the text and the translation.

[0136] Step a4: Perform element information recognition and semantic information extraction on the translated text to obtain the element information and semantic information of the translated text.

[0137] The element information can be, but is not limited to, entities, summaries, keywords, events, scenes, etc. It should be noted that the element information in this embodiment can be element information with one or more dimensions.

[0138] Step a5: Extract features from the text, the word segmentation result of the text, the translated text, the word segmentation result of the translated text, the element information of the translated text, and the semantic information of the translated text to obtain features in multiple dimensions.

[0139] By extracting features from the text, we can obtain the text content features; by extracting features from the word segmentation results, we can obtain the word segmentation features; by extracting features from the translated text, we can obtain the text content features of the translated text; by extracting features from the word segmentation results of the translated text, we can obtain the word segmentation features of the translated text; by extracting features from the element information of the translated text, we can obtain the element features of the translated text; and by extracting features from the semantic information of the translated text, we can obtain the semantic features of the translated text. Thus, features from multiple dimensions can be obtained. It should be noted that this embodiment is not limited to obtaining all of the above features, but may also obtain some of the features mentioned above.

[0140] Step b: If the data of this modality is an image, audio or video, then process the data of this modality into data of other modalities, obtain features of multiple dimensions based on the data of this modality and the processing results of the data of this modality, and obtain the feature set corresponding to the data of this modality.

[0141] When the data for this modality is an image, the process of processing the data for this modality into data for other modalities, and obtaining multi-dimensional features based on the data for this modality and the processing results, may include:

[0142] Step b11: Perform content recognition on the image to obtain the image's text content.

[0143] For example, OCR recognition can be performed on an image to obtain the OCR recognition result, which is the text content of the image.

[0144] Step b12: Obtain multi-dimensional features based on the image and the text content of the image.

[0145] In one possible implementation, step b12 may include: identifying the language of the image content text; if the language of the image content text is not a specified language, translating the image content text into the specified language to obtain the translated image content text; performing word segmentation on the image content text and the translated content text to obtain the word segmentation results of the image content text and the translated image content text; performing element information recognition and semantic information extraction on the translated image content text to obtain the element information and semantic information of the translated image content text; performing feature extraction on the image content text, the word segmentation results of the image content text, the translated image content text, the word segmentation results of the translated image content text, the element information of the translated image content text, and the semantic information of the translated image content text to obtain features at the content level, and performing feature extraction on the image to obtain features at the visual level.

[0146] By extracting features from the text content of an image, we can obtain the text content features of the image's text. Similarly, by extracting features from the word segmentation results of the image's text, we can obtain the word segmentation features of the image's text. Furthermore, by extracting features from the semantic information of the image's translated text, we can obtain the semantic features of the image's translated text. Thus, we can obtain features at multiple dimensions of the content level. By extracting features from the image, we can obtain the image features (which can be extracted based on convolutional neural networks). It should be noted that this embodiment is not limited to obtaining all the features at the content level; some features at the content level can also be obtained.

[0147] When the data of this modality is audio, the process of processing the data of this modality into data of other modalities, and obtaining multi-dimensional features based on the data of this modality and the processing results of the data of this modality, may include:

[0148] Step b21: Transcribe the audio to obtain the audio content text.

[0149] Transcription of audio yields the audio transcription result, which is the text content of the audio.

[0150] Step b22: Obtain features in multiple dimensions based on the audio and the text content of the audio.

[0151] In one possible implementation, step b22 may include: if the language of the audio content text is not a specified language, translating the audio content text into the specified language to obtain the audio content translation text; performing word segmentation on the audio content text and the content translation text to obtain the word segmentation results of the audio content text and the audio content translation text; performing element information recognition and semantic information extraction on the audio content translation text to obtain the element information and semantic information of the audio content translation text; performing feature extraction on the audio content text, the word segmentation results of the audio content text, the audio content translation text, the word segmentation results of the audio content translation text, the element information of the audio content translation text, and the semantic information of the audio content translation text to obtain features at the content level in multiple dimensions, and performing feature extraction on the audio (such as extracting the audio sampling rate, Mel frequency cepstral coefficients, etc.) to obtain features at the auditory level.

[0152] By extracting features from the audio content text, we can obtain the textual content features of the audio content text. Similarly, by extracting features from the word segmentation results of the audio content text, we can obtain the word segmentation features of the audio content text. Furthermore, by extracting features from the semantic information of the audio content translated text, we can obtain the semantic features of the audio content translated text. Thus, we can obtain features at multiple dimensions of the content level. Additionally, by extracting features from the audio, we can obtain the audio features. It should be noted that this embodiment is not limited to obtaining all the above-mentioned content-level features; some of the content-level features can also be obtained.

[0153] When the data of this modality is video, the process of processing the data of this modality into data of other modalities, and obtaining multi-dimensional features based on the data of this modality and the processing results of the data of this modality, may include:

[0154] Step b31: Split the video into audio and multiple images.

[0155] It can perform frame segmentation and audio extraction processing on videos to obtain audio and multiple images.

[0156] Step b32: Obtain multi-dimensional features based on the video, the audio obtained from splitting the video, and multiple images.

[0157] In one possible implementation, step b23 may include: transcribing the split audio to obtain the content text of the split audio; if the language of the content text of the split audio is not a specified language, translating the content text of the split audio into the specified language to obtain the translated content text of the split audio; for each split image, performing content recognition on the image to obtain the content text of the image; if the language of the content text of the image is not a specified language, translating the content text of the image into the specified language to obtain the translated content text of the image; obtaining video visual features based on the video, obtaining image visual features based on each split image, obtaining auditory features based on the split audio, obtaining audio content features based on the content text and translated content text of the split audio, and obtaining image content features based on the content text and translated content text of each split image.

[0158] A three-dimensional convolutional network can be used to extract features from the video to obtain visual features. A two-dimensional convolutional network can be used to extract features from each of the split images to obtain visual features. Steps b11 and b12 can be used to obtain content-level features for each of the split images to obtain content-level features. For the split audio, features such as sampling rate and Mel-frequency cepstral coefficients can be obtained to obtain auditory features. Steps b21 and b22 can be used to obtain content-level features for the split audio to obtain audio content-level features.

[0159] It should be noted that this embodiment is not limited to obtaining features at the visual level, features at the content level, features at the auditory level, or features at the audio content level; it may also obtain some of these features.

[0160] In another embodiment of the present invention, the implementation process of "step S403: fusing the features in the feature set corresponding to the data of each modality contained in the multimodal data to obtain the target features corresponding to the data of each modality contained in the multimodal data" in the above embodiment will be described.

[0161] The process of fusing features from the feature set corresponding to each modality of the multimodal data to obtain the target feature corresponding to each modality of the multimodal data may include: for each modality of the multimodal data, concatenating the features from the feature set corresponding to that modality of the data to obtain the target feature corresponding to that modality of the data.

[0162] For example, the acquired multimodal data is the i-th multimodal data, which includes text(i), image(i), audio(i), and video(i). Then, the target features corresponding to the text... Target features corresponding to the image Target features corresponding to audio Target features corresponding to the video It can be represented as:

[0163]

[0164] Among them, f 1 text(i) f represents the concatenation result of multiple dimensions of features of text(i). 2 image(i) The visual features of image(i) are represented by f. 2 text(i) f represents the concatenation result of multiple dimensions of content-level features of image image(i). 3 audio(i) The audio features represent the auditory characteristics of audio(i), f 3 text(i) f represents the concatenation result of multiple dimensions of content-level features of audio(i). 4 video(i) The video visual features, f, represent the video features, i.e., the video characteristics of the video(i). 4 image(i) f represents the image visual features of video(i), i.e., the result of stitching together image features. 4 text(i) f represents the concatenation result of multiple dimensions of features at the image content level and audio content level of video(i). 4 audio(i) The audio features represent the auditory dimension of the video (i).

[0165] In another embodiment of the present invention, the specific implementation process of "step S404: fusing the target features corresponding to the multiple modal data contained in the multimodal data to obtain the target features corresponding to the multimodal data" in the above embodiment will be described.

[0166] The process of fusing the target features corresponding to multiple modalities contained in multimodal data to obtain the target features corresponding to the multimodal data can include:

[0167] Step c1: Align the target features corresponding to the data of the multiple modalities contained in the multimodal data to obtain the aligned target features corresponding to the data of the multiple modalities contained in the multimodal data.

[0168] In one possible implementation, a pre-trained multimodal feature alignment model can be used to align the target features corresponding to the multiple modalities contained in the multimodal data, thereby obtaining the aligned target features corresponding to the multiple modalities contained in the multimodal data.

[0169] It should be noted that a multimodal training sample set can be used, combined with a contrastive loss function, to train the multimodal feature alignment model. When training the multimodal feature alignment model, positive training samples (such as those containing text t) can be used. i Video v i and audio a i (samples) and multimodal training negative samples (such as text t) i Video v j and audio a k The training is performed on a sample (where j and k are not equal to i), with the training objective being to make the data of each modality (e.g., text t) contained in the positive training samples of the multimodal training be such that the data of each modality are obtained. i Video v i and audio a i In the embedding space, they are close, while multimodal training negative samples contain data from each modality (e.g., text t). i Video v j and audio a k It is far away in the embedded space.

[0170] Step c2: Determine the weights of the multiple modalities contained in the multimodal data based on the aligned target features corresponding to the data of each modality.

[0171] In one possible implementation, the weights of the multiple modalities contained in the multimodal data can be determined based on the aligned target features corresponding to the data of each modality, using an attention mechanism.

[0172] Step c3: According to the weights corresponding to the data of each of the multiple modalities contained in the multimodal data, the aligned target features corresponding to each of the multiple modalities contained in the multimodal data are weighted and summed to obtain the target features corresponding to the multimodal data.

[0173] For example, the multimodal data is the i-th multimodal data item, which contains data of four modalities: text(i), image(i), audio(i), and video(i). The target feature corresponding to text(i) is... The target features corresponding to image image(i) are The target feature corresponding to audio(i) is The target features corresponding to the video are Assuming that step c2 determines the weights as follows: text(i) = α1, image(i) = α2, audio(i) = α3, and video(i) = α4, then the target features corresponding to the multimodal data are... It can be represented as:

[0174] (5)

[0175] In one possible implementation, weight determination and feature fusion can be achieved based on a multimodal feature fusion model. Specifically, the multimodal feature fusion model can include an attention module, a normalization module, and a feature fusion module. Aligned target features corresponding to the multiple modalities contained in the multimodal data are input into the attention module. The attention module performs attention calculations on the input features and outputs weights corresponding to the multiple modalities contained in the multimodal data. The weights output by the attention module are input into a normalization module (e.g., a softmax layer). The normalization module normalizes the input weights and outputs normalized weights corresponding to the multiple modalities contained in the multimodal data. The aligned target features corresponding to the multiple modalities contained in the multimodal data and the normalized weights corresponding to the multiple modalities contained in the multimodal data are input into the feature fusion module. The feature fusion module performs a weighted sum of the aligned target features corresponding to the multiple modalities according to the input weights and outputs the multimodal fused features, i.e., the target features corresponding to the multimodal data.

[0176] After obtaining the target features corresponding to the multimodal data, the target features corresponding to the multimodal data can be stored in the multimodal retrieval feature library.

[0177] The multimodal retrieval feature library construction method provided in this application can obtain relatively rich features for each modality of the multimodal data. Based on this, through intra-modal feature fusion and inter-modal feature fusion, multimodal fusion features that can better represent the multimodal data can be obtained, thereby constructing a high-quality multimodal retrieval feature library. In addition, when obtaining content-level features for each modality of data, this application embodiment can also obtain relevant features of the translated text. The introduction of relevant features of the translated text enables the constructed multimodal retrieval feature library to support cross-language multimodal retrieval.

[0178] Based on the multimodal retrieval feature library construction method provided in the embodiments of this application, the embodiments of this application also provide a multimodal retrieval method. Please refer to [link to relevant documentation]. Figure 6 The flowchart of the multimodal retrieval method is shown, which may include:

[0179] Step S601: Obtain query data.

[0180] The query data in this embodiment can be either unimodal or multimodal.

[0181] Step S602: Process the query data into features to obtain query features.

[0182] like Figure 5 As shown, after obtaining the query data, the query data is processed into query features.

[0183] If the query data is unimodal data, the query features can be obtained in accordance with the method for obtaining target features corresponding to one modality of data provided in the above embodiments. If the query data is multimodal data, the query features can be obtained in accordance with the method for obtaining target features corresponding to multimodal data provided in the above embodiments.

[0184] Step S603: Determine the N features with the highest similarity to the query feature from the multimodal retrieval feature library.

[0185] Where N is an integer greater than 0, and the specific value of N can be determined according to the actual application scenario.

[0186] The multimodal retrieval feature library in this embodiment is constructed using the multimodal retrieval feature library construction method provided in the above embodiment.

[0187] In one possible implementation, the NDSort (Neural Damping Sort) algorithm can be used to sort the features in the multimodal retrieval feature library according to the similarity between the query feature and each feature in the multimodal retrieval feature library, so as to find the N features with the highest similarity to the query feature.

[0188] NDSort is a neural network-based sorting algorithm that combines the ideas of gradient descent and simulated annealing. Its main purpose is to optimize the loss function by adjusting the weights during neural network training, thereby achieving data sorting. During training, the neural network weights are first initialized, then the sorting error under the current weights is calculated. Next, gradient descent is used to update the weights to reduce the sorting error. Then, the weights are randomly perturbed with a certain probability, and the simulated annealing process is repeated. This process of "calculating the sorting error under the current weights" and subsequent steps is repeated until a predetermined number of iterations is reached or the convergence condition is met.

[0189] Step S604: Determine the multimodal data corresponding to the identified N features as the retrieval results.

[0190] Based on the identified N features, the corresponding multimodal data is found, and the retrieval results are obtained.

[0191] Step S605: Output the search results.

[0192] When outputting multimodal data corresponding to N features, the multimodal data can be sorted in descending order of feature similarity.

[0193] The multimodal retrieval method provided in this application, after obtaining query data, first processes the query data into features to obtain query features. Then, it determines the N features with the highest similarity to the query features from a multimodal retrieval feature library. Subsequently, it identifies the multimodal data corresponding to the determined N features as the retrieval results, and finally outputs the retrieval results. Since the features in the multimodal retrieval feature library can effectively represent the corresponding multimodal data, the retrieval method based on the multimodal retrieval feature library provided in this application has high retrieval recall and retrieval accuracy.

[0194] To further improve the quality of features in the multimodal retrieval feature library, thereby enhancing retrieval recall and accuracy, in another embodiment of this application, the multimodal retrieval method may further include: reviewing the retrieval results to obtain a review result; and optimizing the features in the multimodal retrieval feature library based on the review result.

[0195] When reviewing search results, manual review methods (such as manually scoring search results, or manually liking or disliking search results) and / or automatic review methods (such as automatically scoring search results) can be used to review search results.

[0196] In one possible implementation, the process of optimizing features in the multimodal retrieval feature library based on the review results of the retrieval results may include: determining whether feature adjustment is needed based on the review results of the retrieval results; if feature adjustment is needed, adjusting the N feature vectors based on the similarity between the query feature vector and N features (the N features with the highest similarity to the query feature).

[0197] The process of adjusting N feature vectors based on the similarity between the query feature vector and N features (the N features with the highest similarity to the query feature) may include: determining feature adjustment values ​​based on the similarity between the query feature vector and the N features (the N features with the highest similarity to the query feature), and adjusting the N feature vectors based on the feature adjustment values.

[0198] In another possible implementation, the process of optimizing features in the multimodal retrieval feature library based on the review results of the retrieval results may include: optimizing the multimodal feature fusion model based on the review results of the retrieval results; and after obtaining the optimized multimodal feature fusion model, optimizing features in the multimodal retrieval feature library based on the optimized multimodal feature fusion model.

[0199] The process of optimizing features in the multimodal retrieval feature library based on the optimized multimodal feature fusion model includes: redetermining the weights of various modalities contained in the multimodal data based on the optimized multimodal feature fusion model; weighting and summing the various modalities contained in the multimodal data according to the redetermined weights to obtain the optimized target features corresponding to the multimodal data; and updating the optimized target features corresponding to the multimodal data to the multimodal feature database.

[0200] The above describes the method for constructing a multimodal retrieval feature library provided by the embodiments of this application. The following will describe the apparatus corresponding to the above-described method for constructing a multimodal retrieval feature library.

[0201] Please see Figure 7 , Figure 7 This is a schematic diagram of a multimodal retrieval feature library construction device provided in an embodiment of this application. The multimodal retrieval feature library construction device may include: a data acquisition module 701, a feature acquisition module 702, a first feature fusion module 703, a second feature fusion module 704, and a feature storage module 705.

[0202] The data acquisition module 701 is used to acquire multimodal data.

[0203] Multimodal data includes data related to multiple modalities.

[0204] The feature acquisition module 702 is used to acquire features of multiple dimensions from the data of each modality contained in the multimodal data, and obtain the feature set corresponding to the data of each modality contained in the multimodal data.

[0205] The feature set corresponding to any modality of data contains features from one or more of the following three levels: visual level, auditory level, and content level.

[0206] The first feature fusion module 703 is used to fuse the features in the feature set corresponding to each modality of the multimodal data to obtain the target features corresponding to each modality of the multimodal data.

[0207] The second feature fusion module 704 is used to fuse the target features corresponding to the multiple modalities contained in the multimodal data to obtain the target features corresponding to the multimodal data.

[0208] The feature storage module 705 is used to store the target features corresponding to the multimodal data into the multimodal retrieval feature library.

[0209] In one possible implementation, for any modality of data, if the data is text, the corresponding feature set includes content-level features; if the data is an image, the corresponding feature set includes visual and content-level features; if the data is audio, the corresponding feature set includes auditory and content-level features; and if the data is video, the corresponding feature set includes visual, auditory, and content-level features.

[0210] In one possible implementation, when the feature acquisition module 702 acquires features of multiple dimensions from the data of each modality included in the multimodal data to obtain the feature set corresponding to the data of each modality included in the multimodal data, it is specifically used for:

[0211] For each modality contained in the multimodal data:

[0212] If the data for this modality is text, then based on the data for this modality, features of multiple dimensions are obtained to get the feature set corresponding to the data for this modality;

[0213] If the data for this modality is an image, audio, or video, then the data for this modality is processed into data for other modalities. Based on the data for this modality and the processing results, features of multiple dimensions are obtained to obtain the feature set corresponding to the data for this modality.

[0214] In one possible implementation, the feature acquisition module 702 may include: a first feature acquisition module.

[0215] The first feature acquisition module may include a text preprocessing module and a first feature extraction module.

[0216] For each modality of the multimodal data: when the data of that modality is text, the text preprocessing module is used to translate the text into the specified language if the language to which the text belongs is not the specified language, to obtain the translated text of the text; to perform word segmentation on the text and the translated text of the text, to obtain the word segmentation results of the text and the translated text of the text; and to perform element information recognition and semantic information extraction on the translated text of the text, to obtain the element information and semantic information of the translated text of the text.

[0217] The first feature extraction module is used to extract features from the text, the word segmentation result of the text, the translated text, the word segmentation result of the translated text, the element information of the translated text, and the semantic information of the translated text, respectively, to obtain features in multiple dimensions.

[0218] In one possible implementation, the feature acquisition module 702 may include a second feature acquisition module, a third feature acquisition module, and a fourth feature acquisition module.

[0219] For each modality contained in the multimodal data:

[0220] When the data in this modality is an image, the second feature acquisition module is used to perform content recognition on the image to obtain the content text of the image, and to obtain features in multiple dimensions based on the image and the content text of the image.

[0221] When the data in this modality is audio, the third feature acquisition module is used to transcribe the audio to obtain the audio content text, and to acquire features in multiple dimensions based on the audio and the audio content text.

[0222] When the data in this modality is video, the fourth feature acquisition module is used to split the video into audio and multiple images, and to acquire features of multiple dimensions based on the video and the split audio and multiple images.

[0223] In one possible implementation, the second feature acquisition module may include an image preprocessing module and a second feature extraction module.

[0224] The image preprocessing module is used to perform content recognition on the image to obtain the image content text. If the language of the image content text is not the specified language, the image content text is translated into the specified language to obtain the image content translation text. The image content text and the content translation text are segmented separately to obtain the segmentation results of the image content text and the image content translation text. The image content translation text is then subjected to element information recognition and semantic information extraction to obtain the element information and semantic information of the image content translation text.

[0225] The second feature extraction module is used to extract features from the image content text, the word segmentation result of the image content text, the translated text of the image content, the word segmentation result of the translated text of the image content, the element information of the translated text of the image content, and the semantic information of the translated text of the image content, to obtain features at the content level in multiple dimensions, and to extract features from the image to obtain features at the visual level.

[0226] In one possible implementation, the third feature acquisition module may include an audio preprocessing module and a third feature extraction module.

[0227] The audio preprocessing module is used to transcribe the audio to obtain the audio content text. If the language of the audio content text is not the specified language, the audio content text is translated into the specified language to obtain the audio content translation text. The audio content text and the content translation text are segmented separately to obtain the segmentation results of the audio content text and the audio content translation text. The audio content translation text is then subjected to element information recognition and semantic information extraction to obtain the element information and semantic information of the audio content translation text.

[0228] The third feature extraction module is used to extract features from the audio content text, the word segmentation results of the audio content text, the audio content translated text, the word segmentation results of the audio content translated text, the element information of the audio content translated text, and the semantic information of the audio content translated text, respectively, to obtain features at the content level in multiple dimensions, and to extract features from the audio to obtain features at the auditory level.

[0229] In one possible implementation, the third feature acquisition module, when acquiring multi-dimensional features based on the video, the split audio, and multiple images, is specifically used for:

[0230] The audio obtained from the split audio is transcribed to obtain the content text of the split audio; if the language of the content text of the split audio is not the specified language, the content text of the split audio is translated into the specified language to obtain the translated content text of the split audio.

[0231] For each image obtained from the segmentation, content recognition is performed on the image to obtain the content text of the image. If the language of the content text of the image is not the specified language, the content text of the image is translated into the specified language to obtain the translated content text of the image.

[0232] Based on the video, obtain video visual features; based on each image obtained from the splitting, obtain image visual features; based on the audio obtained from the splitting, obtain auditory features; based on the content text and translated text of the split audio, obtain audio content features; based on the content text and translated text of each image obtained from the splitting, obtain image content features.

[0233] In one possible implementation, the second feature fusion module 704, when fusing the target features corresponding to the multiple modalities contained in the multimodal data to obtain the target features corresponding to the multimodal data, is specifically used for:

[0234] Align the target features corresponding to the data of each of the multiple modalities contained in the multimodal data to obtain the aligned target features corresponding to the data of each of the multiple modalities contained in the multimodal data.

[0235] Based on the aligned target features corresponding to the multiple modalities contained in the multimodal data, determine the weights corresponding to the multiple modalities contained in the multimodal data.

[0236] The target features corresponding to the multimodal data are obtained by weighting the aligned target features corresponding to the multiple modalities contained in the multimodal data according to their respective weights.

[0237] The multimodal retrieval feature library construction device provided in this application can obtain relatively rich features for each modality of the multimodal data. Based on this, through feature fusion within the modality and feature fusion between the modalities, multimodal fusion features that can better represent multimodal data can be obtained, thereby constructing a high-quality multimodal retrieval feature library.

[0238] The following section will introduce the apparatus corresponding to the aforementioned multimodal retrieval method. Please refer to [link / reference]. Figure 8 The diagram shows a schematic of the structure of a multimodal retrieval device provided in an embodiment of this application. The multimodal retrieval device may include: a query data acquisition module 801, a query data processing module 802, a multimodal retrieval module 803, and a retrieval result output module 804.

[0239] The query data acquisition module 801 is used to acquire query data.

[0240] The query data processing module 802 is used to process the query data into features to obtain query features.

[0241] The multimodal retrieval module 803 is used to determine the N features with the highest similarity to the query features from the multimodal retrieval feature library, and to determine the multimodal data corresponding to the N features as the retrieval results.

[0242] The multimodal retrieval feature library is constructed using the multimodal retrieval feature library construction device provided in the above embodiments, where N is an integer greater than 0.

[0243] The search results output module 804 is used to output search results.

[0244] In one possible implementation, the multimodal retrieval device may further include a retrieval result review module and a feature optimization module.

[0245] The search results review module is used to review the search results and obtain the review results.

[0246] The feature optimization module is used to optimize the features in the multimodal retrieval feature library based on the review results of the search results.

[0247] The multimodal retrieval device provided in this application embodiment is based on a high-quality multimodal retrieval feature library to achieve retrieval, thus having a high retrieval recall rate and retrieval accuracy rate.

[0248] This application also provides an electronic device, which may include: at least one processor, at least one communication interface, at least one memory, and at least one communication bus.

[0249] In this embodiment, the number of processor, communication interface, memory, and communication bus is at least one, and the processor, communication interface, and memory communicate with each other through the communication bus.

[0250] The processor may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0251] The memory may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device;

[0252] The memory stores a program, and the processor can call the program stored in the memory. The program is used to implement the steps of the multimodal retrieval feature library construction method provided in the above embodiments, and / or to implement the steps of the multimodal retrieval method provided in the above embodiments.

[0253] This application also provides a computer storage medium carrying one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the steps of the multimodal retrieval feature library construction method provided in the above embodiments, and / or implement the steps of the multimodal retrieval method provided in the above embodiments.

[0254] This application also provides a computer program product, including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement the steps of the multimodal retrieval feature library construction method provided in the above embodiments, and / or implement the steps of the multimodal retrieval method provided in the above embodiments.

[0255] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0256] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0257] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0258] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A method for constructing a multimodal retrieval feature library, characterized in that, include: Acquire multimodal data, wherein the multimodal data contains data related to multiple modalities; For each modality of the multimodal data, features of multiple dimensions are obtained to obtain a feature set corresponding to each modality of the multimodal data. The feature set corresponding to any modality of data includes features from one or more of the following three levels: visual level, auditory level, and content level. For any modality of data, if the data is text, the corresponding feature set includes features from the content level; if the data is an image, the corresponding feature set includes features from both the visual and content levels; if the data is audio, the corresponding feature set includes features from both the auditory and content levels; and if the data is video, the corresponding feature set includes features from the visual, auditory, and content levels. The features in the feature set corresponding to each modality of the multimodal data are fused to obtain the target features corresponding to each modality of the multimodal data. The target features corresponding to the multiple modalities contained in the multimodal data are fused to obtain the target features corresponding to the multimodal data. The target features corresponding to the multimodal data are stored in the multimodal retrieval feature library; The step of acquiring features of multiple dimensions from the data of each modality included in the multimodal data to obtain a feature set corresponding to the data of each modality included in the multimodal data includes: For each modality contained in the multimodal data: If the data for this modality is text, then based on the data for this modality, features of multiple dimensions are obtained to get the feature set corresponding to the data for this modality; If the data of this modality is an image, audio, or video, then the data of this modality is processed into data of other modalities. Based on the data of this modality and the processing results of the data of this modality, features of multiple dimensions are obtained to obtain the feature set corresponding to the data of this modality. If the data for this modality is text, then features in multiple dimensions are obtained based on the data for this modality, including: If the language to which the text belongs is not the specified language, then the text is translated into the specified language to obtain the translated text. The text and its translation are segmented into words to obtain the segmentation results of the text and the translation. The translated text of the text is subjected to element information recognition and semantic information extraction to obtain the element information and semantic information of the translated text. Feature extraction is performed on the text, the word segmentation result of the text, the translated text, the word segmentation result of the translated text, the element information of the translated text, and the semantic information of the translated text to obtain features in multiple dimensions.

2. The method for constructing a multimodal retrieval feature library according to claim 1, characterized in that, If the data for this modality is an image, audio, or video, then the data for this modality is processed into data for other modalities. Based on the data for this modality and the processing results, multi-dimensional features are obtained, including: If the data of this modality is an image, then content recognition is performed on the image to obtain the content text of the image, and features of multiple dimensions are obtained based on the image and the content text of the image; If the data of this modality is audio, then the audio is transcribed to obtain the audio content text, and features of multiple dimensions are obtained based on the audio and the audio content text; If the data for this modality is video, then the video is split into audio and multiple images, and features of multiple dimensions are obtained based on the video and the split audio and multiple images.

3. The method for constructing a multimodal retrieval feature library according to claim 2, characterized in that, The step of obtaining multi-dimensional features based on the image and the text content of the image includes: If the language of the content text of the image is not the specified language, then the content text of the image is translated into the specified language to obtain the translated content text of the image. The content text and the translated content text of the image are segmented into words to obtain the word segmentation results of the content text and the translated content text of the image. The translated text of the image content is subjected to element information recognition and semantic information extraction to obtain the element information and semantic information of the translated text of the image content; Feature extraction is performed on the content text of the image, the word segmentation result of the content text of the image, the translated text of the image, the word segmentation result of the translated text of the image, the element information of the translated text of the image, and the semantic information of the translated text of the image to obtain features at the content level. Feature extraction is also performed on the image to obtain features at the visual level.

4. The method for constructing a multimodal retrieval feature library according to claim 2, characterized in that, The step of obtaining multi-dimensional features based on the audio and the text content of the audio includes: If the language of the audio content text is not the specified language, then the audio content text is translated into the specified language to obtain the translated audio content text; The audio content text and the translated content text are segmented into words to obtain the word segmentation results of the audio content text and the translated content text. The translated text of the audio content is subjected to element information recognition and semantic information extraction to obtain the element information and semantic information of the translated text of the audio content; Feature extraction is performed on the audio content text, the word segmentation result of the audio content text, the audio content translated text, the word segmentation result of the audio content translated text, the element information of the audio content translated text, and the semantic information of the audio content translated text to obtain features at the content level in multiple dimensions. Feature extraction is also performed on the audio to obtain features at the auditory level.

5. The method for constructing a multimodal retrieval feature library according to claim 2, characterized in that, The step of obtaining multi-dimensional features based on the video, the split audio, and multiple images includes: The audio obtained from the split audio is transcribed to obtain the content text of the split audio; if the language of the content text of the split audio is not the specified language, the content text of the split audio is translated into the specified language to obtain the translated content text of the split audio. For each image obtained from the segmentation, content recognition is performed on the image to obtain the content text of the image. If the language of the content text of the image is not the specified language, the content text of the image is translated into the specified language to obtain the translated content text of the image. Based on the video, obtain the visual features of the video; based on each image obtained from the splitting, obtain the visual features of the image; based on the audio obtained from the splitting, obtain the auditory features; based on the content text and translated text of the split audio, obtain the audio content features; based on the content text and translated text of each image obtained from the splitting, obtain the image content features.

6. The method for constructing a multimodal retrieval feature library according to claim 1, characterized in that, The step of fusing the target features corresponding to the multiple modalities contained in the multimodal data to obtain the target features corresponding to the multimodal data includes: Align the target features corresponding to the data of the multiple modalities contained in the multimodal data to obtain the aligned target features corresponding to the data of the multiple modalities contained in the multimodal data. Based on the aligned target features corresponding to the multiple modalities contained in the multimodal data, the weights corresponding to the multiple modalities contained in the multimodal data are determined. Based on the weights corresponding to the multiple modalities contained in the multimodal data, the aligned target features corresponding to the multiple modalities contained in the multimodal data are weighted and summed to obtain the target features corresponding to the multimodal data.

7. A multimodal retrieval method, characterized in that, include: Retrieve query data; The query data is processed into features to obtain query features; The N features with the highest similarity to the query feature are determined from the multimodal retrieval feature library, wherein the multimodal retrieval feature library is constructed using the multimodal retrieval feature library construction method as described in any one of claims 1 to 6; The multimodal data corresponding to the N features are determined as the retrieval results, where N is an integer greater than 0; Output the search results.

8. The multimodal retrieval method according to claim 7, characterized in that, Also includes: The search results are reviewed to obtain the review results of the search results; Based on the review results of the search results, the features in the multimodal search feature library are optimized.

9. A multimodal retrieval feature library construction device, characterized in that, include: The system comprises a data acquisition module, a feature acquisition module, a first feature fusion module, a second feature fusion module, and a feature storage module. The data acquisition module is used to acquire multimodal data, wherein the multimodal data contains data related to multiple modalities; The feature acquisition module is used to acquire features of multiple dimensions for each modality of the multimodal data to obtain a feature set corresponding to each modality of the multimodal data. The feature set corresponding to any modality of data includes features from one or more of the following three levels: visual level, auditory level, and content level. For any modality of data, if the data is text, the corresponding feature set includes features from the content level; if the data is an image, the corresponding feature set includes features from both the visual and content levels; if the data is audio, the corresponding feature set includes features from both the auditory and content levels; and if the data is video, the corresponding feature set includes features from the visual, auditory, and content levels. The first feature fusion module is used to fuse the features in the feature set corresponding to the data of each modality contained in the multimodal data to obtain the target features corresponding to the data of each modality contained in the multimodal data; The second feature fusion module is used to fuse the target features corresponding to the multiple modalities contained in the multimodal data to obtain the target features corresponding to the multimodal data. The feature storage module is used to store the target features corresponding to the multimodal data into the multimodal retrieval feature library; Specifically, when the feature acquisition module acquires features of multiple dimensions from the data of each modality included in the multimodal data to obtain the feature set corresponding to the data of each modality included in the multimodal data, it is used for: For each modality contained in the multimodal data: If the data for this modality is text, then based on the data for this modality, features of multiple dimensions are obtained to get the feature set corresponding to the data for this modality; If the data of this modality is an image, audio, or video, then the data of this modality is processed into data of other modalities. Based on the data of this modality and the processing results of the data of this modality, features of multiple dimensions are obtained to obtain the feature set corresponding to the data of this modality. Wherein, if the data of the modality is text, the feature acquisition module, when acquiring features of multiple dimensions based on the data of the modality, is specifically used for: If the language to which the text belongs is not the specified language, then the text is translated into the specified language to obtain the translated text. The text and its translation are segmented into words to obtain the segmentation results of the text and the translation. The translated text of the text is subjected to element information recognition and semantic information extraction to obtain the element information and semantic information of the translated text. Feature extraction is performed on the text, the word segmentation result of the text, the translated text, the word segmentation result of the translated text, the element information of the translated text, and the semantic information of the translated text to obtain features in multiple dimensions.

10. A multimodal retrieval device, characterized in that, include: The system includes a data acquisition module, a data processing module, a multimodal retrieval module, and a retrieval result output module. The query data acquisition module is used to acquire query data; The query data processing module is used to process the query data into features to obtain query features; The multimodal retrieval module is used to determine the N features with the highest similarity to the query feature from the multimodal retrieval feature library, and to determine the multimodal data corresponding to the N features as the retrieval result. The multimodal retrieval feature library is constructed using the multimodal retrieval feature library construction device as described in claim 9, and N is an integer greater than 0. The search result output module is used to output the search results.

11. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the multimodal retrieval feature library construction method as described in any one of claims 1 to 6, and / or to implement the steps of the multimodal retrieval method as described in any one of claims 7 to 8.

12. A computer storage medium, characterized in that, The storage medium carries one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the multimodal retrieval feature library construction method as described in any one of claims 1 to 6, and / or to implement the steps of the multimodal retrieval method as described in any one of claims 7 to 8.

13. A computer program product, characterized in that, It includes computer-readable instructions that, when executed on an electronic device, enable the electronic device to implement the multimodal retrieval feature library construction method as described in any one of claims 1 to 6, and / or to implement the steps of the multimodal retrieval method as described in any one of claims 7 to 8.

Citation Information

Patent Citations

  • Cross-modal retrieval method and device and storage medium

    CN114861016A

  • Referring image segmentation

    US20210390700A1