Multimodal image fusion methods, apparatuses, electronic devices and storage media

Abstract

The present invention is mainly used in the field of computer vision technology. The present invention discloses a multimodal image fusion method, device, electronic device and storage medium, the method comprising acquiring multiple images, and extracting features from each image to obtain multiple image features; generating text information corresponding to each image; generating text fusion features based on multiple text information; generating a fused image based on multiple image features and text fusion features by presetting a cross-modal fusion model. The present application can generate high-quality images by image fusion.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a multimodal image fusion method, device, electronic device and storage medium. Background Art

[0002] In recent years, artificial intelligence technology has developed rapidly, especially in the field of artificial intelligence image generation. Artificial intelligence based on deep learning models can not only generate images from scratch, but also fuse multiple images to obtain new images. Image fusion solutions based on deep learning models can extract features from different images and then fuse different features to generate new images. However, due to excessive reliance on the visual features of the image, the deeper semantic information of the image is ignored, resulting in low resolution and lack of texture details in the fused image, which reduces the image quality. Summary of the invention

[0003] The present invention provides a multimodal image fusion method, device, electronic device and storage medium, which can generate high-quality images by image fusion.

[0004] The present invention provides a multimodal image fusion method, the method comprising:

[0005] Acquire multiple images, and perform feature extraction on each of the images to obtain multiple image features;

[0006] generating text information corresponding to each of the images;

[0007] Based on the plurality of text information, generating text fusion features;

[0008] Based on the multiple image features and the text fusion features, a fused image is generated by using a preset cross-modal fusion model.

[0009] Optionally, generating text information corresponding to each of the images includes:

[0010] When multiple types of images are acquired, each of the images is classified based on the type of the image to obtain multiple image groups;

[0011] Based on each image group, call the target preset text format;

[0012] Based on the target preset text format, text information corresponding to each image in the target image group is generated, wherein the target image group is an image group corresponding to the target preset text template.

[0013] Optionally, the step of generating the text information includes:

[0014] Based on a preset text format, semantic extraction is performed on the image to obtain semantic information of the image;

[0015] Based on the semantic information, text information corresponding to the image is generated through a preset natural language model.

[0016] Optionally, the plurality of images include a first image and a second image;

[0017] Generating text fusion features based on the plurality of text information includes:

[0018] Acquire first text information corresponding to the first image and second text information corresponding to the second image;

[0019] Performing feature extraction on the first text information and the second text information respectively to obtain a first text feature and a second text feature;

[0020] The first text feature and the second text feature are subjected to feature fusion processing to obtain a text fusion feature.

[0021] Optionally, the generating a fused image based on the multiple image features and the text fusion feature by using a preset cross-modal fusion model includes:

[0022] Acquire a first image feature corresponding to the first image and a second image feature corresponding to the second image;

[0023] Using the preset cross-modal fusion model, the text fusion feature and the target image feature are subjected to feature fusion processing to obtain a target text-image fusion feature, wherein the target image feature is the first image feature or the second image feature, and the target text-image fusion feature is the first text-image fusion feature or the second text-image fusion feature;

[0024] Performing feature fusion processing on the first image-text fusion feature and the second image-text fusion feature to obtain a fusion feature;

[0025] The fused features are decoded to obtain a fused image.

[0026] Optionally, the using the preset cross-modal fusion model to perform feature fusion processing on the text fusion feature and the target image feature to obtain the target image-text fusion feature includes:

[0027] Acquire an image token corresponding to the target image feature and a text token corresponding to the text fusion feature;

[0028] Based on the preset Mamba model, after the image token and the text token are fused, a parallel scanning operation is performed to obtain the target image-text fusion feature.

[0029] Optionally, the step of fusing the image token and the text token comprises:

[0030] The image token and the text token are interacted with each other through a preset multi-layer perceptron model to fuse the image token and the text token.

[0031] In a second aspect, the present invention provides a multimodal image fusion device, the device comprising:

[0032] An acquisition module, used for acquiring a plurality of images and performing feature extraction on each of the images to obtain a plurality of image features;

[0033] A generating module, used for generating text information corresponding to each of the images;

[0034] A first fusion module, used for generating a text fusion feature based on the plurality of text information;

[0035] The second fusion module is used to generate a fused image based on the multiple image features and the text fusion feature through a preset cross-modal fusion model.

[0036] In a third aspect, the present invention provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the multimodal image fusion method as described in any one of the above items when executing the computer program.

[0037] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the multimodal image fusion method as described in any one of the above items is implemented.

[0038] The present invention has at least the following beneficial effects:

[0039] The technical solution of the present application generates text information corresponding to each image and generates text fusion features based on the text information. When fusing multiple images, it not only relies on the visual features of the images, but also considers the text fusion features of these images. It utilizes the deeper semantic features between multiple images that go beyond the visual information to fuse the multiple images, thereby obtaining a more advanced fusion result and a fused image with better quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings are used to provide a further understanding of the technical solution of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the technical solution of the present invention and do not constitute a limitation on the technical solution of the present invention.

[0041] Figure 1 is a flow chart of a multimodal image fusion method provided by an embodiment;

[0042] Figure 2 yes Figure 1 Flow chart of step S200 in FIG.

[0043] Figure 3 yes Figure 1 Flow chart of step S300 in FIG.

[0044] Figure 4 yes Figure 1 Flow chart of step S400 in FIG.

[0045] Figure 5 yes Figure 4 Flow chart of step S420 in FIG.

[0046] Figure 6 is a schematic diagram of information flow in a multimodal image fusion method provided by an embodiment;

[0047] Figure 7 is a schematic diagram of the structure of a multimodal image fusion device provided by an embodiment;

[0048] Figure 8 The present invention is a schematic diagram of the hardware structure of an electronic device provided by an embodiment. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0050] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0051] Image fusion is an important image enhancement technique. Image fusion usually adopts three main feature fusion strategies: first, using element-level operations such as addition, multiplication, or concatenation to merge features from different modes. However, element-level operations ignore the relationship between modalities, resulting in poor fusion performance. Second, focusing on deeper feature extraction operations lacks effective inter-modal interactions and texture detail emphasis. Finally, image fusion methods using cross-modal fusion, including convolution-based and Transformer-based methods, although they solve the problem of lack of interaction between features to some extent, the method still has limitations. In general, contemporary fusion methods have difficulty in optimally utilizing modal features and highlighting key information, indicating the need to improve modal fusion and emphasize key image details.

[0052] Mamba Model,The development of the Mamba model provides a promising path to achieve a balance between global receptive field and computational efficiency.,By formalizing the discrete Mamba state space equation into a recursive form,,combined with a specially designed structured reparameterization, it is able to model very,long-range dependencies. In addition, the implementation of a parallel scanning algorithm,enables Mamba to process each token in a parallel manner,,facilitating efficient training on modern hardware such as GPUs.,Mamba surpasses traditional natural language tasks with its significant,data-dependent mechanism, efficient hardware, and superior language processing capabilities.,In addition to language tasks, Mamba has also been successfully applied to vision tasks,such as image classification, video understanding, and biomedical image,segmentation. This success has inspired a wave of research,focusing on applying Mamba-based models to specialized vision applications,including medical image segmentation.,In addition, Mamba has been integrated into graph representation tasks,enhancing the graph embedding and processing capabilities through the,model. Mamba's versatility and efficiency make it a strong choice for a wide range of,applications, from language processing to computer vision representation tasks.,These encouraging features motivate us to further explore the potential of Mamba for,effective long-range modeling in image fusion networks.

[0053] Different sensors can capture unique information. Infrared sensors excel at capturing thermal radiation, visible light sensors excel at detailed texture information, MRI and CT provide structural insights, and PET provides functional data. Understandably, due to hardware limitations, one sensor cannot capture the full complexity of an image, so it is necessary to fuse the images to create informative images. Fusion of infrared and visible images with medical images is an important subcategory that can enhance images and aid in diagnosis.

[0054] In recent years, the application of deep learning models in multimodal image fusion has increased significantly, mainly using CNNs and Transformer architectures for feature extraction and reconstruction. However, these models also face limitations. Specifically, the use of static convolutional layers in these fusion methods leads to a uniform traversal of the entire image, which limits their ability to capture global contextual information due to the limitation of the receptive field. In addition, the indiscriminate processing of each pixel by static convolutional layers ignores the subtle spatial differences between different locations. Due to the urgency of the image fusion task, different input data processing is required. Current convolution-only based techniques have difficulty in effectively extracting features from different modalities, resulting in degraded fusion performance.

[0055] Transformer-based models excel in global modeling, but due to their self-attention mechanism, they have quadratic complexity with respect to image size, resulting in a large amount of computational overhead. In addition, Transformer is less accurate in capturing local features than CNN. Although some fusion models combine convolutional layers and Transformer layers to leverage their respective strengths and mitigate weaknesses, computational overhead remains a significant issue. Despite the widespread use of Transformer-based models, image fusion methods based on Transformer models suffer from an obvious limitation: over-reliance on visual features. This method mainly revolves around the extraction, alignment, fusion, and reconstruction of visual features, prioritizing aspects such as texture, contrast, highlight information, and pixel registration, while ignoring deeper semantic information of the image. As a result, the fused image has problems such as low resolution and lack of texture details, which degrades image quality.

[0056] In order to solve the above problems, the technical solution of the present application generates text information corresponding to each image, and generates text fusion features based on the text information. When fusing multiple images, it not only relies on the visual features of the images, but also considers the text fusion features of these images, thereby utilizing the deeper semantic features between multiple images that go beyond the visual information to fuse the multiple images, thereby obtaining a more advanced fusion result and a fused image with better quality.

[0057] First, please refer to Figure 1 , Figure 1 This embodiment provides a multimodal image fusion method, including:

[0058] S100: Acquire multiple images, and perform feature extraction on each image to obtain multiple image features.

[0059] S200: Generate text information corresponding to each image.

[0060] S300: Generate text fusion features based on multiple text information.

[0061] S400: Based on multiple image features and text fusion features, a fused image is generated by using a preset cross-modal fusion model.

[0062] In step S200 of some embodiments, when generating text information of an image, semantic information is first extracted from the image, and then text information is generated based on the semantic information. For example, in a specific embodiment, the semantic information of the image is input into ChatGPT, and then a text segment describing the image is output.

[0063] In step S300 of some embodiments, the multiple text information generated may be multiple phrases, multiple sentences or multiple paragraphs, or any combination of multiple phrases, multiple sentences or multiple paragraphs. First, these phrases, sentences or paragraphs are generated into paragraphs according to the grammar of natural language, and then text fusion features are extracted from the paragraphs.

[0064] Please refer to Figure 2 In some embodiments, step S200 includes:

[0065] S210: When multiple types of images are acquired, classify each image based on the type of the image to obtain multiple image groups.

[0066] S220: Based on each image group, call a target preset text format.

[0067] S230 . Generate text information corresponding to each image in a target image group based on a target preset text format, wherein the target image group is an image group corresponding to a target preset text template.

[0068] In step S210 of some embodiments, for example, when three types of images are acquired, they are divided into three image groups: a first group, a second group, and a third group. Each image that meets the characteristics of the first group is classified into the first group. Similarly, the corresponding images are classified into the second group and the third group, so that each group has at least one image.

[0069] In step S220 of some embodiments, a preset text format is used to limit the generated text information. For example, the text information limited by one preset text format is that the first sentence of the text is a title describing the image, and the second sentence of the text is a summary describing the image; the text information limited by another preset text format is to generate a semantic mask corresponding to the image.

[0070] In some embodiments, the step of generating text information includes: based on a preset text format, performing semantic extraction on the image to obtain semantic information of the image; based on the semantic information, generating text information corresponding to the image through a preset natural language model.

[0071] In a specific embodiment, semantic extraction is performed on an image through BLIP2, GRIT and Segment Anything models, and image titles, dense titles and semantic masks are output respectively, thereby providing semantic information from coarse-grained to fine-grained, such as input and representation of a sentence title, object-level information and semantic masks.

[0072] It can be understood that this embodiment provides a convenient and easy method for generating text from an image, which greatly simplifies the steps of generating text from an image.

[0073] Please refer to Figure 3 In some embodiments, the multiple images include a first image and a second image; it can be understood that, in some embodiments, the multiple images to be fused are classified, and the image classified into the first category is the first image, and the image classified into the second category is the second image.

[0074] In some embodiments, step S300 includes:

[0075] S310: Acquire first text information corresponding to the first image and second text information corresponding to the second image.

[0076] S320: Perform feature extraction on the first text information and the second text information respectively to obtain a first text feature and a second text feature.

[0077] S330: Perform feature fusion processing on the first text feature and the second text feature to obtain a text fusion feature.

[0078] In some embodiments, a text segment generated based on a first image is input into a text encoder to output text features corresponding to the first image, and a text segment generated based on a second image is input into another text encoder to output text features corresponding to the second image. Then, the text features corresponding to the first image and the text features corresponding to the second image are fused. It can be understood that the fused text features can further be fused with other text features.

[0079] In other embodiments, the text segment generated according to the first image and the text segment generated according to the second image may be input into a CLIP model, and the CLIP model may generate text features of different modalities and fuse the text features.

[0080] It can be understood that this embodiment fuses the semantic information of the images by fusing the text features of the two images, which is beneficial to fusing the two images according to the fused semantic information of the two images in the subsequent steps, thereby improving the fusion effect.

[0081] Please refer to Figure 4 In some embodiments, step S400 includes:

[0082] S410: Acquire a first image feature corresponding to the first image and a second image feature corresponding to the second image.

[0083] S420. Using a preset cross-modal fusion model, perform feature fusion processing on the text fusion feature and the target image feature to obtain a target text-image fusion feature, wherein the target image feature is the first image feature or the second image feature, and the target text-image fusion feature is the first text-image fusion feature or the second text-image fusion feature.

[0084] S430: Perform feature fusion processing on the first image-text fusion feature and the second image-text fusion feature to obtain a fusion feature.

[0085] S440: Decode the fused features to obtain a fused image.

[0086] In a preferred embodiment, the preset cross-modal fusion model is a Mamba model, and the Mamba model is used to perform cross-modal interaction and feature fusion processing on text features and different image features. For example, first, the first image feature and the text fusion feature are subjected to feature fusion processing to obtain a first image-text fusion feature, and then, the second image feature and the text fusion feature are subjected to feature fusion processing to obtain a second image-text fusion feature. Finally, the Mamba model is used to perform deeper feature fusion processing on the first image-text fusion feature and the second image-text fusion feature.

[0087] After the first image-text fusion feature and the second image-text fusion feature are subjected to feature fusion processing, a fusion result, that is, a fused image, is generated through an image decoder.

[0088] Optionally, the fused image will be further subjected to tasks such as classification, segmentation, and object detection.

[0089] Please refer to Figure 5 In some embodiments, step S420 includes:

[0090] S421, obtaining image tokens corresponding to target image features and text tokens corresponding to text fusion features.

[0091] S422. Based on the preset Mamba model, after fusing the image token and the text token, a parallel scanning operation is performed to obtain the target image-text fusion feature.

[0092] It can be understood that the parallel scanning operation performed using the Mamba model includes a space scanning operation and a channel scanning operation.

[0093] In some embodiments, the step of fusing the image token with the text token includes:

[0094] The image token and the text token are interacted through a preset multi-layer perceptron model to fuse the image token and the text token.

[0095] In some embodiments, the process of the Mamba model processing text features and different image features is, for example, for the first image feature and the text fusion feature, the image token corresponding to the first image feature and the text token corresponding to the text fusion feature are processed by the preset multi-layer perceptron model MLP respectively, and then the processed image token and text token are convoluted by a 3×3 convolution layer to obtain an intermediate variable; through the intermediate variable, the image token and the text token are subjected to feature connection operation to obtain the fused token, and then, the space scanning step and the channel scanning step are performed to output the first image fusion feature. Similarly, the second image feature can be output using the Mamba model. Finally, the first image fusion feature and the second image fusion feature are subjected to feature fusion processing to obtain the fusion feature, and the fusion feature is decoded to obtain the fused image.

[0096] It can be understood that this embodiment uses a preset cross-modal fusion model to enable text fusion features and image features to achieve cross-modal interaction, which is conducive to enhancing the correlation between text features and image features, and further improving the fusion effect between features.

[0097] See also Figure 6 , Figure 6 This is a specific embodiment of a multimodal image fusion method. In conjunction with the specific embodiment, the multimodal image fusion method of the present application is introduced and explained in detail:

[0098] After obtaining the CT image and MRI image in the medical field, image feature extraction is first performed to extract the image features of the CT image and the image features of the MRI image respectively. Then, semantic information is extracted from the CT image and the MRI image and a text paragraph of the CT image and a text paragraph of the MRI image are generated. Preferably, ChatGPT can be used to generate the text paragraph of the CT image and the text paragraph of the MRI image. Next, feature extraction is performed on the text paragraph of the CT image and the text paragraph of the MRI image respectively to obtain the text features of the corresponding CT image and the text features of the corresponding MRI image, and feature fusion is performed to obtain text fusion features. Subsequently, the text fusion features and the image features of the CT image are input into the Mamba model to output the text-image fusion features of the CT image, and the text fusion features and the image features of the MRI image are input into the Mamba model to output the text-image fusion features of the MRI image, and the text-image fusion features of the CT image and the text-image fusion features of the MRI image are subjected to feature fusion processing to obtain fusion features, and a medical image after the fusion of the CT image and the MRI image is generated through an image encoder.

[0099] Second, reference Figure 7 , Figure 7 is a schematic diagram of the structure of a multimodal image fusion device. This embodiment provides a multimodal image fusion device, including:

[0100] The acquisition module 701 is used to acquire multiple images and perform feature extraction on each image to obtain multiple image features.

[0101] The generating module 702 is used to generate text information corresponding to each image.

[0102] The first fusion module 703 is used to generate text fusion features based on multiple text information.

[0103] The second fusion module 704 is used to generate a fused image based on multiple image features and text fusion features through a preset cross-modal fusion model.

[0104] It will be appreciated by those skilled in the art that all or some of the steps and devices in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or transient medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. It is well known to those skilled in the art that communication media generally contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and may include any information delivery media.

[0105] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0106] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing any of the above multimodal image fusion methods when executing the computer program.

[0107] refer to Figure 8 , Figure 8 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0108] The processor 801 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0109] The memory 802 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 802 can store operating devices and other applications. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 802, and the processor 801 calls and executes the multimodal image fusion method of the embodiment of this application;

[0110] Input / output interface 803, used to implement information input and output;

[0111] The communication interface 804 is used to realize the communication interaction between the device and other devices. The communication can be realized through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.);

[0112] A bus 805 that transmits information between various components of the device (e.g., the processor 801, the memory 802, the input / output interface 803, and the communication interface 804);

[0113] The processor 801 , the memory 802 , the input / output interface 803 and the communication interface 804 are connected to each other in communication within the device via a bus 805 .

[0114] It can be understood that the contents of the above method embodiments are all applicable to the electronic device embodiment, the functions specifically implemented by the electronic device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0115] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a program executable by a processor, and the program executable by the processor is used to implement the multimodal image fusion method as described in any one of the above specific embodiments when executed by the processor.

[0116] An embodiment of the present application also discloses a computer program product, including a computer program or computer instructions, wherein the computer program or computer instructions are stored in a computer-readable storage medium, a processor of a computer device reads the computer program or computer instructions from the computer-readable storage medium, and the processor executes the computer program or computer instructions, so that the computer device executes the multimodal image fusion method described in any of the previous embodiments.

[0117] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0118] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, device, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. It should be understood that in the present application, "at least one (item)" refers to one or more, and "a plurality" refers to two or more.

[0119] In the several embodiments provided in the present application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0120] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0121] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0122] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0123] Although the description of the present application has been quite detailed and specifically describes several described embodiments, it is not intended to be limited to any of these details or embodiments or any particular embodiment, but should be regarded as providing a broad possible interpretation of these claims by reference to the attached claims, taking into account the prior art, so as to effectively cover the intended scope of the present application. In addition, the above description of the present application is based on the embodiments foreseeable by the inventor, and its purpose is to provide a useful description, and those non-substantial changes to the present application that have not yet been foreseen may still represent equivalent changes to the present application.

Claims

1. A multimodal image fusion method, characterized in that: The method comprises: Acquire multiple images, and perform feature extraction on each of the images to obtain multiple image features, wherein the multiple images include a first image and a second image; generating text information corresponding to each of the images; Based on the plurality of text information, generating text fusion features; Acquire a first image feature corresponding to the first image and a second image feature corresponding to the second image; Acquire an image token corresponding to a target image feature and a text token corresponding to the text fusion feature, wherein the target image feature is the first image feature or the second image feature; Based on a preset Mamba model, after fusing the image token and the text token, a parallel scanning operation is performed to obtain a target image-text fusion feature, wherein the target image-text fusion feature is a first image-text fusion feature or a second image-text fusion feature; Performing feature fusion processing on the first image-text fusion feature and the second image-text fusion feature to obtain a fusion feature; The fused features are decoded to obtain a fused image.

2. A multimodal image fusion method according to claim 1, characterized in that: Generating text information corresponding to each of the images includes: When multiple types of images are acquired, each of the images is classified based on the type of the image to obtain multiple image groups; Based on each image group, call the target preset text format; Based on the target preset text format, text information of each image in a corresponding target image group is generated, wherein the target image group is an image group corresponding to the target preset text format.

3. The multimodal image fusion method according to claim 1, characterized in that: The steps of generating the text information include: Based on a preset text format, semantic extraction is performed on the image to obtain semantic information of the image; Based on the semantic information, text information corresponding to the image is generated through a preset natural language model.

4. The multimodal image fusion method according to claim 1, characterized in that: Generating text fusion features based on the plurality of text information includes: Acquire first text information corresponding to the first image and second text information corresponding to the second image; Performing feature extraction on the first text information and the second text information respectively to obtain a first text feature and a second text feature; The first text feature and the second text feature are subjected to feature fusion processing to obtain a text fusion feature.

5. The multimodal image fusion method according to claim 1, characterized in that: The step of fusing the image token and the text token comprises: The image token and the text token are interacted with each other through a preset multi-layer perceptron model to fuse the image token and the text token.

6. A multimodal image fusion device, characterized in that: The device comprises: An acquisition module, used for acquiring a plurality of images and performing feature extraction on each of the images to obtain a plurality of image features, wherein the plurality of images include a first image and a second image; A generating module, used for generating text information corresponding to each of the images; A first fusion module, used for generating a text fusion feature based on the plurality of text information; A second fusion module, configured to generate a fused image based on the multiple image features and the text fusion feature by using a preset cross-modal fusion model; The second fusion module is also used for: Acquire a first image feature corresponding to the first image and a second image feature corresponding to the second image; Acquire an image token corresponding to a target image feature and a text token corresponding to the text fusion feature, wherein the target image feature is the first image feature or the second image feature; Based on a preset Mamba model, after fusing the image token and the text token, a parallel scanning operation is performed to obtain a target image-text fusion feature, wherein the target image-text fusion feature is a first image-text fusion feature or a second image-text fusion feature; Performing feature fusion processing on the first image-text fusion feature and the second image-text fusion feature to obtain a fusion feature; The fused features are decoded to obtain a fused image.

7. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the multimodal image fusion method according to any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the multimodal image fusion method according to any one of claims 1 to 5 is implemented.

Images