A multimodal image fusion method, apparatus, and computer device

Abstract

This application provides a multimodal image fusion method, apparatus, and computer device, relating to the field of image processing technology, used to improve the accuracy of data retrieval while enhancing the efficiency and accuracy of multimodal image fusion. The method mainly includes: extracting a first feature matrix, a second feature matrix, and a third feature matrix from images in a first data format, a second data format, and a third data format, respectively; fusing the first feature matrix, the second feature matrix, and the third feature matrix to obtain a fused image; determining a feature fusion matrix based on the fused image and the text data and audio data corresponding to each format image, respectively; obtaining retrieval data content corresponding to the original data based on the feature fusion matrix, wherein the retrieval data content includes retrieval images, retrieval text, and / or retrieval audio; and searching for data corresponding to the original data based on the retrieval data content.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a multimodal image fusion method, apparatus and computer equipment. Background Technology

[0002] Multimodal image fusion refers to extracting important information from images of different modalities and synthesizing a clearer and more comprehensive image to facilitate further image processing. To overcome the limitations of single images and obtain more accurate image descriptions, multimodal image fusion technology has become a current research hotspot.

[0003] The rich features in multimodal images can provide valuable information for semantic segmentation tasks in large-scale applications. However, existing semantic segmentation methods have the following drawbacks: the network model is usually a single-modal deep neural network, which is difficult to adapt to image semantic segmentation in complex environments. Furthermore, existing network models fail to make full use of contextual information, making it difficult to design better modules for multi-scale contextual relationship aggregation. Summary of the Invention

[0004] This application provides a multimodal image fusion method, apparatus, and computer device, which improves the accuracy of data retrieval while enhancing the efficiency and accuracy of multimodal image fusion.

[0005] This invention provides a multimodal image fusion method, the method comprising:

[0006] Obtain the raw data to be retrieved, which includes images in a first data format, images in a second data format, and images in a third data format, as well as text data and audio data corresponding to each image format;

[0007] Extract the first feature matrix, the second feature matrix, and the third feature matrix from the first data format image, the second data format image, and the third data format image, respectively;

[0008] The first feature matrix, the second feature matrix, and the third feature matrix are fused to obtain a fused image;

[0009] Based on the text data and audio data corresponding to the fused image and the images of each format, a feature fusion matrix is ​​determined. The feature fusion matrix includes the fused image features corresponding to each pixel, as well as the text features and audio features corresponding to each image format.

[0010] Based on the feature fusion matrix, retrieved data content corresponding to the original data is obtained, and the retrieved data content includes retrieved images, retrieved text, and / or retrieved audio.

[0011] Based on the retrieved data content, find the data corresponding to the original data.

[0012] This invention provides a multimodal image fusion apparatus, the apparatus comprising:

[0013] The acquisition module is used to acquire the raw data to be retrieved, which includes images in a first data format, images in a second data format, and images in a third data format, as well as text data and audio data corresponding to each image format.

[0014] The extraction module is used to extract a first feature matrix, a second feature matrix, and a third feature matrix from images in a first data format, a second data format, and a third data format, respectively.

[0015] The fusion module is used to fuse the first feature matrix, the second feature matrix, and the third feature matrix to obtain a fused image;

[0016] The determining module is used to determine a feature fusion matrix based on the text data and audio data corresponding to the fused image and the images of each format, respectively. The feature fusion matrix includes the fused image features corresponding to each pixel, as well as the text features and audio features corresponding to each image format.

[0017] The acquisition module is further configured to acquire retrieval data content corresponding to the original data based on the feature fusion matrix, wherein the retrieval data content includes retrieval images, retrieval text, and / or retrieval audio.

[0018] The search module is used to find data corresponding to the original data based on the search data content.

[0019] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described multimodal image fusion method.

[0020] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described multimodal image fusion method.

[0021] A computer program product comprising a computer program that, when executed by a processor, implements the above-described multimodal image fusion method.

[0022] This invention provides a multimodal image fusion method, apparatus, and computer device. The method involves acquiring raw data to be retrieved, including images in a first, second, and third data format, as well as text and audio data corresponding to each format image. A first, second, and third feature matrices are extracted from the first, second, and third data format images, respectively. These matrices are then fused to obtain a fused image. A feature fusion matrix is ​​determined based on the fused image and the text and audio data corresponding to each format image. This feature fusion matrix includes fused image features corresponding to each pixel, as well as text and audio features corresponding to each format image. Retrieval data content corresponding to the raw data is obtained based on the feature fusion matrix. This retrieval data content includes retrieval images, retrieval text, and / or retrieval audio. Data corresponding to the raw data is then searched based on the retrieval data content. Therefore, this invention improves the efficiency and accuracy of multimodal image fusion, thereby increasing the accuracy of data retrieval. Attached Figure Description

[0023] Figure 1 A flowchart of a multimodal image fusion method provided in this application;

[0024] Figure 2 A schematic diagram of the structure of the multimodal image fusion device provided in this application;

[0025] Figure 3 A schematic diagram of the computer device provided in this application. Detailed Implementation

[0026] To better understand the above technical solutions, the technical solutions of the embodiments of this application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of this application and the specific features in the embodiments are detailed descriptions of the technical solutions of the embodiments of this application, rather than limitations on the technical solutions of this application. In the absence of conflict, the embodiments of this application and the technical features in the embodiments can be combined with each other.

[0027] Please see Figure 1 As shown, this is a multimodal image fusion method provided by an embodiment of the present invention. This method can be applied to map retrieval, map drawing, etc., and is used to execute steps S101-S106:

[0028] Step S101: Obtain the original data to be retrieved. The original data includes a first data format image, a second data format image, and a third data format image, as well as text data and audio data corresponding to each format image.

[0029] The images in the first, second, and third data formats can be remote sensing data, point cloud data, infrared and thermal imaging, lidar images, microwave imaging, etc. This embodiment does not specifically limit them.

[0030] Step S102: Extract the first feature matrix, the second feature matrix, and the third feature matrix from the first data format image, the second data format image, and the third data format image, respectively.

[0031] Among them, the first data format image, the second data format image, and the third data format image are images of different data formats taken from the same shooting angle (including close-up and distant views), and the size of each format image is the same.

[0032] Step S103: The first feature matrix, the second feature matrix, and the third feature matrix are fused to obtain a fused image.

[0033] In this embodiment, the first, second, and third data format images can be panchromatic, multispectral, and hyperspectral images, respectively. The feature matrices corresponding to the panchromatic, multispectral, and hyperspectral images are then fused to obtain a fused image. The fusion of high-resolution and low-resolution hyperspectral images can significantly improve the spatial resolution of the hyperspectral image. It is also possible to fuse panchromatic / multispectral images with SRA (Active Microwave Imaging) synthetic aperture radar images. SRA imaging is easily affected by noise. Fusing SRA images with panchromatic and multispectral images combines the advantages of both, generating higher-quality images. Fusion of SRA with infrared images is also possible. SAR images exhibit isolated discontinuities at edge points. Infrared images can approximately reflect the temperature and radiation gradients of the observed object, providing relatively complete edge and texture information. Fusing SAR images with infrared images adds the edge and texture information obtained from the infrared image to the SAR image, resulting in more integrated edges and textures in the fused image while maintaining the frequency characteristics of the SAR image, making the fused image more readable. Finally, fusion of lidar point cloud data with RGB graphics is also possible: lidar point cloud data has precise ensemble information, but the data is very sparse. The image data has high resolution and rich texture features.

[0034] In an optional embodiment of the present invention, fusing the first feature matrix, the second feature matrix, and the third feature matrix to obtain a fused image includes:

[0035] Step S1031: Based on the first feature matrix, the second feature matrix, and the third feature matrix, filter out the outliers in the feature matrix by comparing the corresponding feature points in each feature matrix.

[0036] All feature matrices have the same matrix dimension, which is n*m.

[0037] Specifically, the step of filtering outliers in the feature matrices based on the comparison results of the corresponding feature points in each feature matrix, according to the first feature matrix, the second feature matrix, and the third feature matrix, includes: feature point A in each feature matrix ij B ij C ij Perform comparisons separately and obtain the X features that are inconsistent with the feature points of the other two feature matrices in the comparison results. ij X ij For A ij B ij C ij One of them. X in the feature matrix. ij Filtering continues until all feature points in the feature matrix have been compared. The first data format image, the second data format image, and the third data format image are images of different data formats from the same viewpoint.

[0038] Among them, A ij Let B be the feature point in the i-th row and j-th column of the first feature matrix. ij C represents the feature point in the i-th row and j-th column of the second feature matrix. ij Let i be the feature point in the i-th row and j-th column of the third feature matrix, where i takes the value from 1 to n and j takes the value from 1 to m.

[0039] Step S1032: The filtered first feature matrix, second feature matrix and third feature matrix are fused to obtain a fused image.

[0040] Specifically, fusing the filtered first feature matrix, second feature matrix, and third feature matrix to obtain a fused image includes: inputting the filtered first feature matrix, second feature matrix, and third feature matrix into an image feature fusion model to obtain a fused image, wherein the fused image includes fused depth features and texture features; wherein the image feature fusion model is trained using a neural network model based on the first sample feature matrix, second sample feature matrix, and third sample feature matrix, as well as the corresponding fused image label feature matrix.

[0041] Furthermore, the image feature fusion model includes three channels, each corresponding to two convolutional layers, one pooling layer, and one stack; the step of inputting the filtered first feature matrix, second feature matrix, and third feature matrix into the image feature fusion model includes:

[0042] The filtered first feature matrix is ​​input into the first channel, and the depth features and texture features are obtained by passing through two convolutional layers and one pooling layer in the first channel. The obtained depth features and texture features are then transferred to the stack.

[0043] The filtered second feature matrix is ​​input into the second channel, and the depth features and texture features are obtained through two convolutional layers and one pooling layer in the second channel. The obtained depth features and texture features are then transferred to the stack.

[0044] The filtered third feature matrix is ​​input into the third channel, and after passing through two convolutional layers and one pooling layer in the third channel, depth features and texture features are obtained. The obtained depth features and texture features are then transferred to the stack.

[0045] The depth and texture features within each stack are fused to obtain a fused image.

[0046] Specifically, in this implementation, the fused image can be obtained using the following formula:

[0047]

[0048]

[0049] in, To fuse the depth features of the pixel in the i-th row and j-th column of the image, The depth feature of the pixel in the i-th row and j-th column of the first feature matrix. The depth feature of the pixel in the i-th row and j-th column of the second feature matrix. This represents the depth feature of the pixel in the i-th row and j-th column of the third feature matrix.

[0050] To fuse the texture features of the pixel in the i-th row and j-th column of the image, The texture feature of the pixel in the i-th row and j-th column of the first feature matrix. The texture features of the pixel in the i-th row and j-th column of the second feature matrix. This represents the texture feature of the pixel in the i-th row and j-th column of the third feature matrix. ω1-ω6 are the weight values, which are user-defined.

[0051] Step S104: Determine the feature fusion matrix based on the text data and audio data corresponding to the fused image and the images of each format, respectively.

[0052] The feature fusion matrix includes the fused image features corresponding to each pixel, as well as the text features and audio features corresponding to each image format.

[0053] In an optional embodiment of the present invention, determining the feature fusion matrix based on the text data and audio data corresponding to the fused image and the images of each format respectively includes:

[0054] Step S1041: Convert the fused image, text data and audio data corresponding to each format image into image feature matrix, text feature matrix and audio feature matrix respectively.

[0055] Step S1042: The image feature matrix, the text feature matrix, and the audio feature matrix are fused to obtain the feature fusion matrix.

[0056] Step S105: Obtain the retrieval data content corresponding to the original data based on the feature fusion matrix. The retrieval data content includes retrieval images, retrieval text, and / or retrieval audio.

[0057] In an optional embodiment of the present invention, obtaining the retrieval data content corresponding to the original data according to the feature fusion matrix includes: obtaining data from the database whose similarity to the feature fusion matrix is ​​greater than a preset value, and determining the data whose similarity is greater than the preset value as the retrieval data content corresponding to the original data; or inputting the feature fusion matrix into a keyword recognition model to obtain keywords corresponding to the original data, and retrieving the retrieval data content corresponding to the original data through the keywords.

[0058] The database stores the retrieval data content corresponding to multiple feature fusion matrices. Preset values ​​can be set according to actual needs, such as 85%, 90%, 95%, etc. In this embodiment, by searching the database, data with a similarity greater than the feature fusion matrix can be obtained, and then the data retrieval content corresponding to the found fusion matrix is ​​determined as the retrieval data content corresponding to the original data.

[0059] Step S106: Based on the retrieved data content, find the data corresponding to the original data.

[0060] In this embodiment, considering a practical usage scenario, the user can acquire raw data, which includes images in three data formats, along with corresponding text and audio data. Based on this raw data, the user can determine the search data content to find relevant data. For example, if the user captures images of urban roads in three data formats, along with corresponding audio data, text data can be extracted from the images and audio data. Then, the search data content can be obtained based on this data. This search data content can be data related to the image, audio, or text, such as historical and cultural information related to the urban road image, or information about urban roads from a better perspective. This embodiment does not specifically limit this.

[0061] This invention provides a multimodal image fusion method. The method involves acquiring raw data to be retrieved, including images in a first, second, and third data format, as well as text and audio data corresponding to each format image. A first feature matrix, a second feature matrix, and a third feature matrix are extracted from the first, second, and third data format images, respectively. These feature matrices are then fused to obtain a fused image. A feature fusion matrix is ​​determined based on the fused image and the text and audio data corresponding to each format image. This feature fusion matrix includes fused image features for each pixel, as well as text and audio features corresponding to each format image. Retrieval data content corresponding to the raw data is obtained based on the feature fusion matrix. This retrieval data content includes retrieval images, retrieval text, and / or retrieval audio. Data corresponding to the raw data is then searched based on the retrieval data content. Therefore, this invention improves the efficiency and accuracy of multimodal image fusion, thereby increasing the accuracy of data retrieval.

[0062] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0063] In one embodiment, a multimodal image fusion apparatus is provided, which corresponds one-to-one with the multimodal image fusion method described in the above embodiments. For example... Figure 2 As shown, the functional modules of the device are described in detail below:

[0064] The acquisition module 21 is used to acquire the raw data to be retrieved, the raw data including a first data format image, a second data format image and a third data format image, as well as text data and audio data corresponding to each format image respectively;

[0065] Extraction module 22 is used to extract a first feature matrix, a second feature matrix, and a third feature matrix from the first data format image, the second data format image, and the third data format image, respectively;

[0066] The fusion module 23 is used to fuse the first feature matrix, the second feature matrix and the third feature matrix to obtain a fused image;

[0067] The determining module 24 is used to determine a feature fusion matrix based on the text data and audio data corresponding to the fused image and the images of each format, respectively. The feature fusion matrix includes the fused image features corresponding to each pixel, as well as the text features and audio features corresponding to each format image.

[0068] The acquisition module 21 is further configured to acquire the retrieval data content corresponding to the original data according to the feature fusion matrix, wherein the retrieval data content includes retrieval images, retrieval text and / or retrieval audio;

[0069] The search module 25 is used to search for data corresponding to the original data based on the search data content.

[0070] In an optional embodiment provided by the present invention, the fusion module 23 is specifically used for:

[0071] Based on the first feature matrix, the second feature matrix, and the third feature matrix, the comparison results of the corresponding feature points in each feature matrix are used to filter out outliers in the feature matrix; wherein the matrix dimensions of each feature matrix are the same.

[0072] The filtered first feature matrix, second feature matrix, and third feature matrix are fused to obtain a fused image.

[0073] In an optional embodiment of the present invention, the fusion module 23 is specifically used for:

[0074] Feature points A in each feature matrix ij B ij C ij Perform comparisons separately and obtain the X features that are inconsistent with the feature points of the other two feature matrices in the comparison results. ij X ij For A ij B ij Or C ij Among them, A ij Let B be the feature point in the i-th row and j-th column of the first feature matrix. ij C represents the feature point in the i-th row and j-th column of the second feature matrix. ijLet i be the feature point in the i-th row and j-th column of the third feature matrix, where i takes the value of 1-n and j takes the value of 1-m.

[0075] X in the feature matrix ij Filtering continues until all feature points in the feature matrix have been compared. The first data format image, the second data format image, and the third data format image are images of different data formats from the same viewpoint.

[0076] In an optional embodiment of the present invention, the fusion module 23 is specifically used for:

[0077] The filtered first feature matrix, second feature matrix, and third feature matrix are input into the image feature fusion model to obtain a fused image, which includes fused depth features and texture features.

[0078] The image feature fusion model is trained using a neural network model based on the first sample feature matrix, the second sample feature matrix, the third sample feature matrix, and the corresponding fused image label feature matrix.

[0079] In an optional embodiment provided by the present invention, the image feature fusion model includes three channels, each channel corresponding to two convolutional layers, one pooling layer, and one stack; the fusion module 23 is specifically used for:

[0080] The filtered first feature matrix is ​​input into the first channel, and the depth features and texture features are obtained by passing through two convolutional layers and one pooling layer in the first channel. The obtained depth features and texture features are then transferred to the stack.

[0081] The filtered second feature matrix is ​​input into the second channel, and the depth features and texture features are obtained through two convolutional layers and one pooling layer in the second channel. The obtained depth features and texture features are then transferred to the stack.

[0082] The filtered third feature matrix is ​​input into the third channel, and after passing through two convolutional layers and one pooling layer in the third channel, depth features and texture features are obtained. The obtained depth features and texture features are then transferred to the stack.

[0083] The depth and texture features within each stack are fused to obtain a fused image.

[0084] In an optional embodiment of the present invention, the determining module 24 is specifically used for:

[0085] The fused image, text data, and audio data corresponding to each image format are respectively converted into image feature matrix, text feature matrix, and audio feature matrix;

[0086] The feature fusion matrix is ​​obtained by fusing the image feature matrix, the text feature matrix, and the audio feature matrix.

[0087] In an optional embodiment provided by the present invention, the acquisition module 21 is specifically used for:

[0088] Data with a similarity greater than a preset value to the feature fusion matrix is ​​retrieved from the database, and this data is identified as the retrieval data content corresponding to the original data; or

[0089] The feature fusion matrix is ​​input into the keyword recognition model to obtain keywords corresponding to the original data, and the search data content corresponding to the original data is retrieved through the keywords.

[0090] For specific limitations regarding the device, please refer to the limitations of the multimodal image fusion method above, which will not be repeated here. Each module in the aforementioned device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0091] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 3 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a multimodal image fusion method.

[0092] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0093] Obtain the raw data to be retrieved, which includes images in a first data format, images in a second data format, and images in a third data format, as well as text data and audio data corresponding to each image format;

[0094] Extract the first feature matrix, the second feature matrix, and the third feature matrix from the first data format image, the second data format image, and the third data format image, respectively;

[0095] The first feature matrix, the second feature matrix, and the third feature matrix are fused to obtain a fused image;

[0096] Based on the text data and audio data corresponding to the fused image and the images of each format, a feature fusion matrix is ​​determined. The feature fusion matrix includes the fused image features corresponding to each pixel, as well as the text features and audio features corresponding to each image format.

[0097] Based on the feature fusion matrix, retrieved data content corresponding to the original data is obtained, and the retrieved data content includes retrieved images, retrieved text, and / or retrieved audio.

[0098] Based on the retrieved data content, find the data corresponding to the original data.

[0099] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0100] Obtain the raw data to be retrieved, which includes images in a first data format, images in a second data format, and images in a third data format, as well as text data and audio data corresponding to each image format;

[0101] Extract the first feature matrix, the second feature matrix, and the third feature matrix from the first data format image, the second data format image, and the third data format image, respectively;

[0102] The first feature matrix, the second feature matrix, and the third feature matrix are fused to obtain a fused image;

[0103] Based on the text data and audio data corresponding to the fused image and the images of each format, a feature fusion matrix is ​​determined. The feature fusion matrix includes the fused image features corresponding to each pixel, as well as the text features and audio features corresponding to each image format.

[0104] Based on the feature fusion matrix, retrieved data content corresponding to the original data is obtained, and the retrieved data content includes retrieved images, retrieved text, and / or retrieved audio.

[0105] Based on the retrieved data content, find the data corresponding to the original data.

[0106] In one embodiment, a computer program product is provided, the computer program product comprising a computer program that is executed by a processor to perform the following steps:

[0107] Obtain the raw data to be retrieved, which includes images in a first data format, images in a second data format, and images in a third data format, as well as text data and audio data corresponding to each image format;

[0108] Extract the first feature matrix, the second feature matrix, and the third feature matrix from the first data format image, the second data format image, and the third data format image, respectively;

[0109] The first feature matrix, the second feature matrix, and the third feature matrix are fused to obtain a fused image;

[0110] Based on the text data and audio data corresponding to the fused image and the images of each format, a feature fusion matrix is ​​determined. The feature fusion matrix includes the fused image features corresponding to each pixel, as well as the text features and audio features corresponding to each image format.

[0111] Based on the feature fusion matrix, retrieved data content corresponding to the original data is obtained, and the retrieved data content includes retrieved images, retrieved text, and / or retrieved audio.

[0112] Based on the retrieved data content, find the data corresponding to the original data.

[0113] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A multi-modal image fusion method, characterized in that, The method comprises: obtaining original data to be retrieved, the original data comprising first data format images, second data format images and third data format images, and text data and audio data corresponding to each format image respectively; extracting first feature matrices, second feature matrices and third feature matrices from the first data format images, the second data format images and the third data format images respectively; The filtering of the abnormal points in the feature matrix based on the comparison results of the corresponding feature points in each feature matrix includes: the feature points in each feature matrix respectively, and obtaining the feature points in the comparison results that are inconsistent with the feature points of the other two feature matrices , are ; wherein, is the feature point in the i-th row and the j-th column of the first feature matrix, is the feature point in the i-th row and the j-th column of the second feature matrix, is the feature point in the i-th row and the j-th column of the third feature matrix, the value of i is 1-n, and the value of j is 1-m; the feature points in the feature matrix that are filtered out until all the feature points in the feature matrix are compared, the first data format image, the second data format image, and the third data format image are images of different data formats of the same perspective; wherein the matrix dimensions of each feature matrix are the same, and are n*m dimensions; fusing the filtered first feature matrices, the second feature matrices and the third feature matrices to obtain a fused image, comprising: inputting the filtered first feature matrices, the second feature matrices and the third feature matrices into an image feature fusion model to obtain a fused image, wherein the fused image comprises fused depth features and texture features; wherein the image feature fusion model is trained by a neural network model according to first sample feature matrices, second sample feature matrices and third sample feature matrices, and corresponding fused image label feature matrices; the image feature fusion model comprises three channels, each channel corresponding to 2 convolution layers, 1 pooling layer and 1 stack; the inputting of the filtered first feature matrices, the second feature matrices and the third feature matrices into the image feature fusion model comprises: inputting the filtered first feature matrices into a first channel, obtaining depth features and texture features through the 2 convolution layers and 1 pooling layer of the first channel, and transmitting the obtained depth features and texture features to the stack; inputting the filtered second feature matrices into a second channel, obtaining depth features and texture features through the 2 convolution layers and 1 pooling layer of the second channel, and transmitting the obtained depth features and texture features to the stack; inputting the filtered third feature matrices into a third channel, obtaining depth features and texture features through the 2 convolution layers and 1 pooling layer of the third channel, and transmitting the obtained depth features and texture features to the stack; fusing the depth features and the texture features in each stack to obtain a fused image; determining a feature fusion matrix according to the fused image and the text data and the audio data corresponding to each format image, wherein the feature fusion matrix comprises fused image features corresponding to each pixel point and text features and audio features corresponding to each format image respectively; obtaining retrieval data content corresponding to the original data according to the feature fusion matrix, wherein the retrieval data content comprises a retrieval image, retrieval text and / or retrieval audio; finding data corresponding to the original data based on the retrieval data content.

2. The method of claim 1, wherein, The determining of the feature fusion matrix according to the fused image and the text data and the audio data corresponding to each format image comprises: converting the fused image, the text data and the audio data corresponding to each format image into an image feature matrix, a text feature matrix and an audio feature matrix respectively; fusing the image feature matrix, the text feature matrix and the audio feature matrix to obtain the feature fusion matrix.

3. The method of claim 1, wherein, The obtaining of the retrieval data content corresponding to the original data according to the feature fusion matrix comprises: acquire data with a similarity to the feature fusion matrix greater than a preset value from a database, and determine the data with the similarity greater than the preset value as search data content corresponding to the original data; or input the feature fusion matrix to a keyword recognition model to obtain a keyword corresponding to the original data, and search for search data content corresponding to the original data through the keyword.

4. A multi-modal image fusion apparatus characterized by comprising: The apparatus comprises: an acquisition module configured to acquire original data to be searched, the original data comprising first, second and third data format images, and text data and audio data corresponding to each format image; an extraction module configured to extract first, second and third feature matrices from the first, second and third data format images, respectively; a fusion module configured to fuse the first feature matrix, the second feature matrix and the third feature matrix to obtain a fused image; specifically comprising: based on the first feature matrix, the second feature matrix and the third feature matrix, filtering out abnormal points in the feature matrix according to the matching results of the corresponding feature points in each feature matrix, including: filtering out the feature points in each feature matrix respectively to obtain the feature points in the matching results that are inconsistent with the feature points in the other two feature matrices , ; wherein, is the feature point in the i-th row and the j-th column of the first feature matrix, is the feature point in the i-th row and the j-th column of the second feature matrix, is the feature point in the i-th row and the j-th column of the third feature matrix, the value of i is 1-n, and the value of j is 1-m; filtering out the feature points in the feature matrix until all the feature points in the feature matrix are matched, the first data format image, the second data format image and the third data format image are images of different data formats of the same perspective; wherein the matrix dimensions of each feature matrix are the same, and are n*m dimensions;​ fusing the filtered first, second and third feature matrices to obtain a fusion image, comprising inputting the filtered first, second and third feature matrices to an image feature fusion model to obtain a fusion image, the fusion image comprising fused depth features and texture features; wherein the image feature fusion model is trained by a neural network model according to first, second and third sample feature matrices and corresponding fusion image label feature matrices; the image feature fusion model comprises three channels, each channel corresponding to 2 convolution layers, 1 pooling layer and 1 stack; inputting the filtered first, second and third feature matrices to the image feature fusion model comprises inputting the filtered first feature matrix to a first channel, obtaining depth features and texture features through the 2 convolution layers and 1 pooling layer of the first channel, and transmitting the obtained depth features and texture features to the stack; inputting the filtered second feature matrix to a second channel, obtaining depth features and texture features through the 2 convolution layers and 1 pooling layer of the second channel, and transmitting the obtained depth features and texture features to the stack; inputting the filtered third feature matrix to a third channel, obtaining depth features and texture features through the 2 convolution layers and 1 pooling layer of the third channel, and transmitting the obtained depth features and texture features to the stack; fusing the depth features and texture features in each stack to obtain a fusion image; a determination module configured to determine a feature fusion matrix according to the fusion image and the text data and audio data corresponding to each format image, the feature fusion matrix comprising fusion image features corresponding to each pixel point, and text features and audio features corresponding to each format image; the acquisition module is further configured to acquire search data content corresponding to the original data according to the feature fusion matrix, the search data content comprising search images, search text and / or search audio; a search module configured to search for data corresponding to the original data based on the search data content.

5. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The computer program is executed by the processor to implement the multi-modal image fusion method according to any one of claims 1 to 3.

6. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 5. The computer program is executed by the processor to implement the multi-modal image fusion method according to any one of claims 1 to 3.

Images