Image fusion method and device, electronic equipment and medium

Abstract

The invention relates to an image fusion method and device, electronic equipment and a medium, and the method comprises the steps: obtaining an original image sequence and an intermediate image sequence corresponding to the original image sequence; the original images in the original image sequence are the same in size; an intermediate image in the intermediate image sequence is determined and obtained according to a dimension value of each pixel point in each original image on a channel dimension; performing feature extraction processing on the original image sequence and the intermediate image sequence to obtain a spatial feature image and a time feature image; determining a fusion image corresponding to the original image sequence according to the spatial feature image and the time feature image; wherein the spatial feature image and the time feature image are respectively extracted, so that the displacement and deformation among the plurality of original images can be determined by combining the time feature image under the condition of accurately extracting the spatial features, thereby improving the quality of the determined fusion image and improving the fusion efficiency.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technologies, and in particular, to an image fusion method, apparatus, electronic device, and medium. Background Art

[0002] Currently, the main image fusion method is to obtain an original image sequence; perform fusion processing on multiple original images in the original image sequence to obtain a fused image. In the above solution, due to shooting jitter or object movement, there may be large displacements and deformations between multiple original images. Directly performing fusion processing on multiple original images results in poor quality and low fusion efficiency of the obtained fused image. Summary of the Invention

[0003] The present disclosure provides an image fusion method, apparatus, electronic device, and medium.

[0004] According to a first aspect of an embodiment of the present disclosure, an image fusion method is provided. The method includes: obtaining an original image sequence and an intermediate image sequence corresponding to the original image sequence; the sizes of the original images in the original image sequence are the same; the intermediate images in the intermediate image sequence are determined according to the dimensional values of each pixel point in each original image in one channel dimension; performing feature extraction processing on the original image sequence and the intermediate image sequence to obtain a spatial feature image and a temporal feature image; and determining a fused image corresponding to the original image sequence according to the spatial feature image and the temporal feature image.

[0005] In an embodiment of the present disclosure, the obtaining the original image sequence and the intermediate image sequence corresponding to the original image sequence includes: obtaining the original image sequence; for each channel dimension of the original images in the original image sequence, for each pixel point, performing splicing processing on the dimensional values of the pixel point in the channel dimension in each original image in the original image sequence to obtain an intermediate value of the pixel point; determining an intermediate image in the channel dimension according to the intermediate values of each pixel point; and determining the intermediate image sequence according to the intermediate images in each channel dimension.

[0006] In an embodiment of the present disclosure, the performing feature extraction processing on the original image sequence and the intermediate image sequence to obtain a spatial feature image and a temporal feature image includes: inputting the original image sequence into a spatial feature extraction network in an image fusion model to obtain the spatial feature image output by the spatial feature extraction network; and inputting the intermediate image sequence into a temporal feature extraction network in the image fusion model to obtain the temporal feature image output by the temporal feature extraction network.

[0007] In one embodiment of the present disclosure, the spatial feature image includes the spatial features of each pixel point in the original image; the temporal feature image includes the temporal features of each pixel point in the original image; determining the fused image corresponding to the original image sequence according to the spatial feature image and the temporal feature image includes: for each pixel point, splicing the spatial feature and the temporal feature of the pixel point to obtain the spatio-temporal feature of the pixel point; determining a fused feature image according to the spatio-temporal features of each pixel point; and determining the fused image corresponding to the original image sequence according to the fused feature image.

[0008] In one embodiment of the present disclosure, determining the fused image corresponding to the original image sequence according to the fused feature image includes: inputting the fused feature image into a fusion network in an image fusion model to obtain the fused image output by the fusion network.

[0009] In one embodiment of the present disclosure, the fusion network is a U-Net convolutional neural network.

[0010] In one embodiment of the present disclosure, the image format of the original image is the same as or different from the image format of the fused image; when the image format of the original image is different from the image format of the fused image, the image format of the original image is a RAW image; and the image format of the fused image is an RGB image.

[0011] According to a second aspect of the embodiments of the present disclosure, there is also provided a method for training an image fusion model, the method including: obtaining training data; the training data includes a sample original image sequence, a sample intermediate image sequence corresponding to the sample original image sequence, and a sample fused image; the sizes of the sample original images in the sample original image sequence are the same; the sample intermediate images in the sample intermediate image sequence are determined according to the dimensional values of each pixel point in each sample original image in one channel dimension; obtaining an initial image fusion model; inputting the sample original image sequence and the sample intermediate image sequence corresponding to the sample original image sequence into the image fusion model to obtain a predicted fused image output by the image fusion model; determining a fused image loss according to the sample fused image and the predicted fused image; and adjusting the parameters of the image fusion model according to the fused image loss to achieve training.

[0012] In one embodiment of the present disclosure, the image fusion model includes a spatial feature extraction network, a temporal feature extraction network, a splicing network, and a fusion network; the spatial feature extraction network is configured to perform spatial feature extraction processing on the sample original image sequence to obtain a spatial feature image; the temporal feature extraction network is configured to perform temporal feature extraction processing on the sample intermediate image sequence to obtain a temporal feature image; the splicing network is configured to perform splicing processing on the spatial feature image and the temporal feature image to obtain a fused feature image; the fusion network is configured to process the fused feature image to obtain a predicted fused image.

[0013] In one embodiment of the present disclosure, the sample fused image includes a first sample fused image and / or a second sample fused image; the image format of the first sample fused image is different from the image format of the sample original image; the image format of the second sample fused image is the same as the image format of the sample original image.

[0014] In one embodiment of the present disclosure, the sample fused image includes a first sample fused image and a second sample fused image; the predicted fused image includes a first predicted fused image and a second predicted fused image; determining the fused image loss according to the sample fused image and the predicted fused image includes: determining a first image loss according to the first sample fused image and the first predicted fused image; determining a second image loss according to the second sample fused image and the second predicted fused image; performing an addition process on the first image loss and the second image loss to obtain the fused image loss.

[0015] According to a third aspect of the embodiments of the present disclosure, there is also provided an image fusion device, the device includes: an acquisition module, configured to acquire an original image sequence and an intermediate image sequence corresponding to the original image sequence; the sizes of the original images in the original image sequence are the same; the intermediate images in the intermediate image sequence are determined according to the dimensional values of each pixel point in each original image in one channel dimension; a feature extraction module, configured to perform feature extraction processing on the original image sequence and the intermediate image sequence to obtain a spatial feature image and a temporal feature image; a determination module, configured to determine a fused image corresponding to the original image sequence according to the spatial feature image and the temporal feature image.

[0016] In one embodiment of the present disclosure, the obtaining module is specifically configured to obtain the original image sequence; for each channel dimension of the original images in the original image sequence, for each pixel, splice the dimension values of the pixel in the channel dimension in each original image in the original image sequence to obtain the intermediate value of the pixel; determine the intermediate image in the channel dimension according to the intermediate values of the pixels; and determine the intermediate image sequence according to the intermediate images in each channel dimension.

[0017] In one embodiment of the present disclosure, the feature extraction module is specifically configured to input the original image sequence into the spatial feature extraction network in the image fusion model to obtain the spatial feature image output by the spatial feature extraction network; input the intermediate image sequence into the temporal feature extraction network in the image fusion model to obtain the temporal feature image output by the temporal feature extraction network.

[0018] In one embodiment of the present disclosure, the spatial feature image includes the spatial features of each pixel in the original image; the temporal feature image includes the temporal features of each pixel in the original image; the determining module is specifically configured to, for each pixel, splice the spatial feature and the temporal feature of the pixel to obtain the spatio-temporal feature of the pixel; determine the fusion feature image according to the spatio-temporal features of the pixels; and determine the fusion image corresponding to the original image sequence according to the fusion feature image.

[0019] In one embodiment of the present disclosure, the determining module is further specifically configured to input the fusion feature image into the fusion network in the image fusion model to obtain the fusion image output by the fusion network.

[0020] In one embodiment of the present disclosure, the fusion network is a U-Net convolutional neural network.

[0021] In one embodiment of the present disclosure, the image format of the original image is the same as or different from the image format of the fusion image; when the image format of the original image is different from the image format of the fusion image, the image format of the original image is a RAW image; and the image format of the fusion image is an RGB image.

[0022] According to a fourth aspect of the embodiments of the present disclosure, there is also provided a training device for an image fusion model. The device includes: a first acquisition module configured to acquire training data. The training data includes a sample original image sequence, a sample intermediate image sequence corresponding to the sample original image sequence, and a sample fusion image. The sizes of the sample original images in the sample original image sequence are the same. The sample intermediate images in the sample intermediate image sequence are determined according to the dimensional values of each pixel point in each sample original image in a channel dimension. A second acquisition module configured to acquire an initial image fusion model. A third acquisition module configured to input the sample original image sequence and the sample intermediate image sequence corresponding to the sample original image sequence into the image fusion model to obtain a predicted fusion image output by the image fusion model. A determination module configured to determine a fusion image loss according to the sample fusion image and the predicted fusion image. A training module configured to adjust the parameters of the image fusion model according to the fusion image loss to implement training.

[0023] In an embodiment of the present disclosure, the image fusion model includes a spatial feature extraction network, a temporal feature extraction network, a splicing network, and a fusion network. The spatial feature extraction network is configured to perform spatial feature extraction processing on the sample original image sequence to obtain a spatial feature image. The temporal feature extraction network is configured to perform temporal feature extraction processing on the sample intermediate image sequence to obtain a temporal feature image. The splicing network is configured to perform splicing processing on the spatial feature image and the temporal feature image to obtain a fused feature image. The fusion network is configured to process the fused feature image to obtain a predicted fusion image.

[0024] In an embodiment of the present disclosure, the sample fusion image includes a first sample fusion image and / or a second sample fusion image. The image format of the first sample fusion image is different from the image format of the sample original image. The image format of the second sample fusion image is the same as the image format of the sample original image.

[0025] In an embodiment of the present disclosure, the sample fusion image includes a first sample fusion image and a second sample fusion image. The predicted fusion image includes a first predicted fusion image and a second predicted fusion image. The determination module is specifically configured to determine a first image loss according to the first sample fusion image and the first predicted fusion image. Determine a second image loss according to the second sample fusion image and the second predicted fusion image. Perform an addition process on the first image loss and the second image loss to obtain the fusion image loss.

[0026] According to a fifth aspect of the embodiments of the present disclosure, an electronic device is further provided, including: a processor; a memory for storing executable instructions of the processor; wherein, the processor is configured to: implement the steps of the image fusion method as described above; or, implement the steps of the training method of the image fusion model as described above.

[0027] According to a sixth aspect of the embodiments of the present disclosure, a non-transitory computer-readable storage medium is further provided. When the instructions in the storage medium are executed by a processor, the processor is enabled to execute the image fusion method as described above; or, execute the training method of the image fusion model as described above.

[0028] According to a fifth aspect of the embodiments of the present disclosure, a chip is further provided, including one or more interface circuits and one or more processors; the interface circuit is configured to receive a signal from a memory of an electronic device and send the signal to the processor, the signal including computer instructions stored in the memory. When the processor executes the computer instructions, the electronic device is enabled to execute the image fusion method as described above; or, execute the training method of the image fusion model as described above.

[0029] The technical solutions provided by the embodiments of the present disclosure at least bring the following beneficial effects:

[0030] By obtaining an original image sequence and an intermediate image sequence corresponding to the original image sequence; the sizes of the original images in the original image sequence are the same; the intermediate images in the intermediate image sequence are determined according to the dimensional values of each pixel point in each original image in one channel dimension; performing feature extraction processing on the original image sequence and the intermediate image sequence to obtain a spatial feature image and a temporal feature image; determining a fused image corresponding to the original image sequence according to the spatial feature image and the temporal feature image; wherein, the separate extraction of the spatial feature image and the temporal feature image can, while accurately extracting spatial features, determine the displacement and deformation between multiple original images in combination with the temporal feature image, thereby improving the quality of the determined fused image and improving the fusion efficiency.

[0031] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings herein are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with the present disclosure and used together with the specification to explain the principles of the present disclosure, and do not constitute an improper limitation to the present disclosure.

[0033] Figure 1 It is a flowchart of an image fusion method according to an embodiment of the present disclosure;

[0034] Figure 2 Flow chart of an image fusion method according to another embodiment of the present disclosure;

[0035] Figure 3 Flow chart of a training method of an image fusion model according to an embodiment of the present disclosure;

[0036] Figure 4 Schematic structural diagram of an image fusion device according to an embodiment of the present disclosure;

[0037] Figure 5 Schematic structural diagram of a training device of an image fusion model according to an embodiment of the present disclosure;

[0038] Figure 6 Block diagram of a structure of an electronic device shown according to an exemplary embodiment of the present disclosure;

[0039] Figure 7 Schematic structural diagram of a chip according to an embodiment of the present disclosure. Detailed implementation manners

[0040] In order to enable those of ordinary skill in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0041] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that such used data may be interchanged under appropriate circumstances so that the embodiments of the present disclosure described herein can be implemented in an order different from those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0042] The current image fusion method mainly includes: obtaining an original image sequence; performing fusion processing on a plurality of original images in the original image sequence to obtain a fused image. In the above solution, due to the situation of shooting jitter or object movement, there may be large displacements and deformations between multiple original images. Directly performing fusion processing on multiple original images results in poor quality and low fusion efficiency of the obtained fused image.

[0043] Figure 1 Flow chart of an image fusion method according to an embodiment of the present disclosure. It should be noted that the image fusion method of this embodiment can be applied to an image fusion device, and the device can be configured in an electronic device or a chip so that the electronic device or the chip can perform the image fusion function.

[0044] Among them, the electronic device can be any device with computing capabilities, such as a personal computer (PC), a mobile terminal, a server, a controller in a vehicle, etc. The mobile terminal can be, for example, a vehicle-mounted device, a mobile phone, a tablet computer, a personal digital assistant, a wearable device, etc., which are hardware devices with various operating systems, touch screens, and / or display screens.

[0045] In addition, the image fusion device can also be software in the electronic device, etc. Among them, the software can be, for example, image processing software, etc. Among the following embodiments, the electronic device is taken as an example of the execution subject for illustration.

[0046] As Figure 1 shown, the method includes the following steps:

[0047] Step 101, obtain an original image sequence and an intermediate image sequence corresponding to the original image sequence; the sizes of the original images in the original image sequence are the same; the intermediate images in the intermediate image sequence are determined according to the dimensional values of each pixel point in each original image in one channel dimension.

[0048] In the embodiments of the present disclosure, the process of the electronic device executing step 101 can be, for example, obtaining the original image sequence; for each channel dimension of the original images in the original image sequence, for each pixel point, performing splicing processing on the dimensional values of the pixel points in the channel dimension of each original image in the original image sequence to obtain the intermediate value of the pixel point; determining the intermediate image in the channel dimension according to the intermediate values of each pixel point; and determining the intermediate image sequence according to the intermediate images in each channel dimension.

[0049] Among them, the image format of the original image and the image format of the fused image can be the same or different. When the image format of the original image and the image format of the fused image are the same, in one example, the image format of the original image and the image format of the fused image can both be the RAW format. In another example, the image format of the original image and the image format of the fused image can both be the RGB format. In addition, it should be noted that the image format of the original image and the image format of the fused image can also be other image formats, such as the GPEG format, the GIF format, the PNG format, etc.

[0050] Among them, when the image format of the original image is different from the image format of the fused image, the image format of the original image can be the RAW format; the image format of the fused image can be the RGB format.

[0051] Among them, taking the example that the original image sequence includes 5 original images and the number of channel dimensions of the pixel points in the original images is 4, the process by which the electronic device obtains the intermediate image sequence can be, for example, for the first channel dimension, for each pixel point, splicing the dimension values of the pixel point in the first channel dimension in the 5 original images to obtain the intermediate value of the pixel point; according to the intermediate values of each pixel point, determining the intermediate image in the first channel dimension; repeating the above steps to obtain the intermediate image in the second channel dimension, the intermediate image in the third channel dimension, and the intermediate image in the fourth channel dimension; that is, obtaining 4 intermediate images.

[0052] Among them, each intermediate image in the determined intermediate image sequence is determined by the pixel point values on the same channel dimension in each original image; thus, the displacement trend and deformation trend between each original image can be reflected. Therefore, performing feature extraction processing on the intermediate image sequence can extract temporal features.

[0053] Step 102, perform feature extraction processing on the original image sequence and the intermediate image sequence to obtain a spatial feature image and a temporal feature image.

[0054] In the embodiment of the present disclosure, the process by which the electronic device executes step 102 can be, for example, inputting the original image sequence into the spatial feature extraction network in the image fusion model to obtain the spatial feature image output by the spatial feature extraction network; inputting the intermediate image sequence into the temporal feature extraction network in the image fusion model to obtain the temporal feature image output by the temporal feature extraction network.

[0055] Among them, the spatial feature extraction network and the temporal feature extraction network can be at least one of the following networks: 3D convolutional neural network, recurrent neural network, convolutional LSTM, Transformer model, etc., and can be selected according to actual needs.

[0056] Among them, the separate extraction of the spatial feature image and the temporal feature image can reduce the number of parameters of the spatial feature extraction network and the temporal feature extraction network, and reduce the processing amount of the spatial feature extraction network and the temporal feature extraction network, thereby improving the image fusion speed.

[0057] Step 103, determine the fused image corresponding to the original image sequence according to the spatial feature image and the temporal feature image.

[0058] In an embodiment of the present disclosure, a spatial feature image can reflect the spatial position relationship of objects in each original image in an original image sequence; a temporal feature image can reflect the displacement trend and deformation trend between each original image in the original image sequence; by combining the spatial feature image and the temporal feature image, the position mapping relationship between objects in each original image in the original image sequence can be reflected. Furthermore, the fused image determined by combining the spatial feature image and the temporal feature image can consider the displacement and deformation between multiple original images, thereby improving the quality of the fused image and the fusion efficiency.

[0059] In the image fusion method of the embodiment of the present disclosure, an original image sequence and an intermediate image sequence corresponding to the original image sequence are obtained; the sizes of all original images in the original image sequence are the same; the intermediate images in the intermediate image sequence are determined according to the dimensional values of each pixel point in each original image in one channel dimension; feature extraction processing is performed on the original image sequence and the intermediate image sequence to obtain a spatial feature image and a temporal feature image; a fused image corresponding to the original image sequence is determined according to the spatial feature image and the temporal feature image; wherein, the separate extraction of the spatial feature image and the temporal feature image can, while accurately extracting the spatial features, combine the temporal feature image to determine the displacement and deformation between multiple original images, thereby improving the quality of the determined fused image and the fusion efficiency.

[0060] Figure 2 It is a flowchart of an image fusion method according to another embodiment of the present disclosure. It should be noted that the image fusion method of this embodiment can be applied to an image fusion device, which can be configured in an electronic device or a chip so that the electronic device or the chip can perform the image fusion function.

[0061] Among them, the electronic device can be any device with computing capabilities, such as a personal computer (PC for short), a mobile terminal, a server, a controller in a vehicle, etc. The mobile terminal can be, for example, a vehicle-mounted device, a mobile phone, a tablet computer, a personal digital assistant, a wearable device, etc., which are hardware devices with various operating systems, touch screens, and / or display screens.

[0062] In addition, the image fusion device can also be software in an electronic device, etc. Among them, the software can be, for example, image processing software, etc. In the following embodiments, the execution entity is taken as an electronic device for illustration.

[0063] As Figure 2 shown, the method includes the following steps:

[0064] Step 201: Obtain the original image sequence and the intermediate image sequence corresponding to the original image sequence; the sizes of the original images in the original image sequence are the same; the intermediate images in the intermediate image sequence are determined according to the dimensional values of each pixel point in each original image in one channel dimension.

[0065] Step 202: Perform feature extraction processing on the original image sequence and the intermediate image sequence to obtain a spatial feature image and a temporal feature image.

[0066] Step 203: For each pixel point, splice the spatial feature and the temporal feature of the pixel point to obtain the spatio-temporal feature of the pixel point.

[0067] Among them, for example, if the number of dimensions of the spatial feature of a pixel point is 20 dimensions, and the number of dimensions of the temporal feature of the pixel point is 20 dimensions; after splicing the spatial feature and the temporal feature of the pixel point, the number of dimensions of the obtained spatio-temporal feature is 40 dimensions.

[0068] Step 204: Determine a fused feature image according to the spatio-temporal features of each pixel point.

[0069] In the embodiments of the present disclosure, the electronic device may arrange the spatio-temporal features of each pixel point according to the positions of the pixel points to obtain a fused feature image. That is to say, the fused feature image includes multiple pixel points, and each pixel point is set with a spatio-temporal feature.

[0070] Step 205: Determine a fused image corresponding to the original image sequence according to the fused feature image.

[0071] In the embodiments of the present disclosure, the process of the electronic device executing Step 205 may be, for example, inputting the fused feature image into a fusion network in an image fusion model to obtain the fused image output by the fusion network.

[0072] Among them, the fusion network may be, for example, a U-Net convolutional neural network. The U-Net convolutional neural network may include an encoder, a decoder, and skip connections between the encoder and the decoder. Among them, the encoder is composed of multiple convolutional layers and pooling layers, and these layers gradually reduce the spatial resolution of the input image while increasing the number of feature channels. The decoder is composed of multiple deconvolutional layers, and these layers upsample the feature map to the original size of the input image. The skip connections are used to connect the low-level features in the encoder and the high-level features in the decoder, and then fuse the low-level features in the encoder and the high-level features in the decoder to make full use of the feature information at different levels, so as to be able to extract more features and further improve the accuracy of the determined fused image.

[0073] Among them, it should be noted that for the detailed descriptions of step 201 and step 202, reference can be made to Figure 1 step 101 and step 102 in the illustrated embodiment, and no detailed description will be given here.

[0074] In the image fusion method of the present disclosure embodiment, an original image sequence and an intermediate image sequence corresponding to the original image sequence are obtained; the sizes of the original images in the original image sequence are the same; the intermediate images in the intermediate image sequence are determined according to the dimensional values of each pixel point in each original image in one channel dimension; feature extraction processing is performed on the original image sequence and the intermediate image sequence to obtain a spatial feature image and a temporal feature image; for each pixel point, the spatial feature and the temporal feature of the pixel point are spliced to obtain the spatio-temporal feature of the pixel point; according to the spatio-temporal features of each pixel point, a fusion feature image is determined; according to the fusion feature image, a fusion image corresponding to the original image sequence is determined; among them, the separate extraction of the spatial feature image and the temporal feature image can, while accurately extracting the spatial feature, combine the temporal feature image to determine the displacement and deformation between multiple original images, thereby improving the quality of the determined fusion image and improving the fusion efficiency.

[0075] Figure 3 It is a flowchart of a method for training an image fusion model according to an embodiment of the present disclosure. Among them, it should be noted that the method for training the image fusion model in this embodiment can be applied to a training device for the image fusion model, and the device can be configured in an electronic device or a chip so that the electronic device or the chip can perform the training function of the image fusion model.

[0076] Among them, the electronic device can be any device with computing power, such as a personal computer (PC for short), a mobile terminal, a server, a controller in a vehicle, etc. The mobile terminal can be, for example, a vehicle-mounted device, a mobile phone, a tablet computer, a personal digital assistant, a wearable device, etc., which are hardware devices with various operating systems, touch screens, and / or display screens.

[0077] In addition, the image fusion device can also be software in the electronic device, etc. Among them, the software is, for example, software for training the image fusion model, etc. Among them, in the following embodiments, the execution entity is taken as an example of an electronic device for description.

[0078] As Figure 3 shown, the method includes the following steps:

[0079] Step 301: Obtain training data. The training data includes a sample original image sequence, a sample intermediate image sequence corresponding to the sample original image sequence, and a sample fused image. The sizes of all sample original images in the sample original image sequence are the same. The sample intermediate images in the sample intermediate image sequence are determined based on the dimensional values of each pixel point in each sample original image in one channel dimension.

[0080] In an embodiment of the present disclosure, the process by which the electronic device obtains the sample intermediate image sequence corresponding to the sample original image sequence may be, for example, for each channel dimension of the sample original images in the sample original image sequence, for each pixel point, splicing the dimensional values of the pixel point in the channel dimension of each sample original image in the sample original image sequence to obtain the intermediate value of the pixel point; determining the sample intermediate image on the channel dimension based on the intermediate values of each pixel point; and determining the sample intermediate image sequence based on the sample intermediate images on each channel dimension.

[0081] Step 302: Obtain an initial image fusion model.

[0082] Step 303: Input the sample original image sequence and the sample intermediate image sequence corresponding to the sample original image sequence into the image fusion model, and obtain the predicted fused image output by the image fusion model.

[0083] In an embodiment of the present disclosure, the image fusion model may include a spatial feature extraction network, a temporal feature extraction network, a splicing network, and a fusion network. The spatial feature extraction network is used to perform spatial feature extraction processing on the sample original image sequence to obtain a spatial feature image. The temporal feature extraction network is used to perform temporal feature extraction processing on the sample intermediate image sequence to obtain a temporal feature image. The splicing network is used to splice the spatial feature image and the temporal feature image to obtain a fused feature image. The fusion network is used to process the fused feature image to obtain the predicted fused image.

[0084] Among them, the spatial feature extraction network and the temporal feature extraction network may be at least one of the following networks: 3D convolutional neural network, recurrent neural network, convolutional LSTM, Transformer model, etc., which can be selected according to actual needs.

[0085] Among them, the separate extraction of the spatial feature image and the temporal feature image can reduce the number of parameters of the spatial feature extraction network and the temporal feature extraction network, as well as reduce the processing amount of the spatial feature extraction network and the temporal feature extraction network, and improve the image fusion speed.

[0086] Among them, the fusion network can be, for example, a U-Net convolutional neural network. The U-Net convolutional neural network can include an encoder, a decoder, and skip connections between the encoder and the decoder. Among them, the encoder is composed of multiple convolutional layers and pooling layers, and these layers gradually reduce the spatial resolution of the input image while increasing the number of feature channels. The decoder is composed of multiple deconvolutional layers, and these layers upsample the feature map to the original size of the input image. The skip connections are used to connect the low-level features in the encoder and the high-level features in the decoder, and then fuse the low-level features in the encoder and the high-level features in the decoder to make full use of the feature information at different levels, so as to be able to extract more features and further improve the accuracy of the determined fusion image.

[0087] Step 304, determine the fusion image loss according to the sample fusion image and the predicted fusion image.

[0088] In the embodiments of the present disclosure, the sample fusion image may include a first sample fusion image and / or a second sample fusion image; the image format of the first sample fusion image is different from the image format of the sample original image; the image format of the second sample fusion image is the same as the image format of the sample original image.

[0089] Among them, in one example, the sample fusion image may include a first sample fusion image. Correspondingly, the predicted fusion image may include a first predicted fusion image. The process of the electronic device executing step 304 may be, for example, to determine a first image loss according to the first sample fusion image and the first predicted fusion image; and determine the first image loss as the fusion image loss.

[0090] Among them, in one case, the process of the electronic device determining the first image loss may be, for example, for each pixel point in the first sample fusion image, obtain the difference between the value of the pixel point in the first sample fusion image and the value of the pixel point in the first predicted fusion image; according to the average value of the differences of each pixel point; and determine the average value as the first image loss.

[0091] In another case, the process of the electronic device determining the first image loss may be, for example, to determine the image similarity between the first sample fusion image and the first predicted fusion image; and determine the image similarity as the first image loss.

[0092] Among them, in another example, the sample fusion image may include a second sample fusion image. Correspondingly, the predicted fusion image may include a second predicted fusion image. The process of the electronic device executing step 304 may be, for example, to determine a second image loss according to the second sample fusion image and the second predicted fusion image; and determine the second image loss as the fusion image loss.

[0093] Among them, for the determination process of the second image loss, reference can be made to the determination process of the first image loss, and no detailed description will be given here.

[0094] Among them, in another example, the sample fusion image may include a first sample fusion image and a second sample fusion image. Correspondingly, the predicted fusion image may include a first predicted fusion image and a second predicted fusion image. The process of the electronic device executing step 304 may be, for example, to determine the first image loss according to the first sample fusion image and the first predicted fusion image; to determine the second image loss according to the second sample fusion image and the second predicted fusion image; and to perform an addition process on the first image loss and the second image loss to obtain the fusion image loss.

[0095] Among them, the calculation formula of the fusion image loss may be, for example, as shown in the following formula.

[0096]

[0097]

[0098] L total = L rgb + αL raw

[0099] Among them, L total represents the fusion image loss, L rgb represents the first image loss, L raw represents the second image loss, represents the first sample fusion image, represents the second sample fusion image. α is the ratio between the two image losses and is generally set to 1.

[0100] Step 305: Adjust the parameters of the image fusion model according to the fusion image loss to achieve training.

[0101] In the training method of the image fusion model according to the embodiments of the present disclosure, training data is obtained; the training data includes a sample original image sequence, a sample intermediate image sequence corresponding to the sample original image sequence, and a sample fusion image; the sizes of the sample original images in the sample original image sequence are the same; the sample intermediate images in the sample intermediate image sequence are determined according to the dimensional values of each pixel point in each sample original image in a channel dimension; an initial image fusion model is obtained; the sample original image sequence and the sample intermediate image sequence corresponding to the sample original image sequence are input into the image fusion model to obtain a predicted fusion image output by the image fusion model; according to the sample fusion image and the predicted fusion image, a fusion image loss is determined; according to the fusion image loss, the parameters of the image fusion model are adjusted to achieve training; wherein, the setting of the sample original image sequence and the sample intermediate image sequence facilitates the extraction of spatial feature images and temporal feature images, so that during the training process of the image fusion model, the displacement and deformation between multiple sample original images can be considered, thereby extracting more features, and further improving the accuracy of the trained image fusion model, and thus improving the image fusion efficiency.

[0102] Figure 4 FIG. is a schematic structural diagram of an image fusion device according to an embodiment of the present disclosure.

[0103] As Figure 4 shown, the image fusion device may include: an acquisition module 401, a feature extraction module 402, and a determination module 403.

[0104] Among them, the acquisition module 401 is configured to acquire an original image sequence and an intermediate image sequence corresponding to the original image sequence; the sizes of the original images in the original image sequence are the same; the intermediate images in the intermediate image sequence are determined according to the dimensional values of each pixel point in each original image in a channel dimension; the feature extraction module 402 is configured to perform feature extraction processing on the original image sequence and the intermediate image sequence to obtain a spatial feature image and a temporal feature image; the determination module 403 is configured to determine a fusion image corresponding to the original image sequence according to the spatial feature image and the temporal feature image.

[0105] In an embodiment of the present disclosure, the acquisition module 401 is specifically configured to acquire the original image sequence; for each channel dimension of the original images in the original image sequence, for each pixel point, perform splicing processing on the dimensional values of the pixel point in the channel dimension of each original image in the original image sequence to obtain an intermediate value of the pixel point; determine an intermediate image on the channel dimension according to the intermediate values of each pixel point; and determine the intermediate image sequence according to the intermediate images on each channel dimension.

[0106] In one embodiment of the present disclosure, the feature extraction module 402 is specifically configured to input the original image sequence into the spatial feature extraction network in the image fusion model to obtain the spatial feature image output by the spatial feature extraction network; input the intermediate image sequence into the temporal feature extraction network in the image fusion model to obtain the temporal feature image output by the temporal feature extraction network.

[0107] In one embodiment of the present disclosure, the spatial feature image includes the spatial features of each pixel point in the original image; the temporal feature image includes the temporal features of each pixel point in the original image; the determination module 403 is specifically configured to, for each pixel point, splice the spatial feature and the temporal feature of the pixel point to obtain the spatio-temporal feature of the pixel point; determine the fused feature image according to the spatio-temporal features of each pixel point; and determine the fused image corresponding to the original image sequence according to the fused feature image.

[0108] In one embodiment of the present disclosure, the determination module 403 is further specifically configured to input the fused feature image into the fusion network in the image fusion model to obtain the fused image output by the fusion network.

[0109] In one embodiment of the present disclosure, the fusion network is a U-Net convolutional neural network.

[0110] In one embodiment of the present disclosure, the image format of the original image and the image format of the fused image are the same or different; when the image format of the original image and the image format of the fused image are different, the image format of the original image is the RAW format; and the image format of the fused image is the RGB format.

[0111] In the image fusion device according to the embodiments of the present disclosure, by obtaining the original image sequence and the intermediate image sequence corresponding to the original image sequence; the sizes of the original images in the original image sequence are the same; the intermediate images in the intermediate image sequence are determined according to the dimensional values of each pixel point in each original image in one channel dimension; performing feature extraction processing on the original image sequence and the intermediate image sequence to obtain a spatial feature image and a temporal feature image; determining the fused image corresponding to the original image sequence according to the spatial feature image and the temporal feature image; wherein, the separate extraction of the spatial feature image and the temporal feature image can, on the premise of accurately extracting the spatial feature, combine the temporal feature image to determine the displacement and deformation between multiple original images, thereby improving the quality of the determined fused image and improving the fusion efficiency.

[0112] Figure 5Schematic structural diagram of a training device for an image fusion model according to an embodiment of the present disclosure.

[0113] As Figure 5 shown, the training device for the image fusion model may include: a first acquisition module 501, a second acquisition module 502, a third acquisition module 503, a determination module 504, and a training module 505.

[0114] Among them, the first acquisition module 501 is configured to acquire training data; the training data includes a sample original image sequence, a sample intermediate image sequence corresponding to the sample original image sequence, and a sample fusion image; the sizes of the sample original images in the sample original image sequence are the same; the sample intermediate images in the sample intermediate image sequence are determined according to the dimensional values of each pixel point in each sample original image in one channel dimension; the second acquisition module 502 is configured to acquire an initial image fusion model; the third acquisition module 503 is configured to input the sample original image sequence and the sample intermediate image sequence corresponding to the sample original image sequence into the image fusion model to obtain a predicted fusion image output by the image fusion model; the determination module 504 is configured to determine a fusion image loss according to the sample fusion image and the predicted fusion image; the training module 505 is configured to adjust the parameters of the image fusion model according to the fusion image loss to achieve training.

[0115] In an embodiment of the present disclosure, the image fusion model includes a spatial feature extraction network, a temporal feature extraction network, a splicing network, and a fusion network; the spatial feature extraction network is configured to perform spatial feature extraction processing on the sample original image sequence to obtain a spatial feature image; the temporal feature extraction network is configured to perform temporal feature extraction processing on the sample intermediate image sequence to obtain a temporal feature image; the splicing network is configured to perform splicing processing on the spatial feature image and the temporal feature image to obtain a fused feature image; the fusion network is configured to process the fused feature image to obtain a predicted fusion image.

[0116] In an embodiment of the present disclosure, the sample fusion image includes a first sample fusion image and / or a second sample fusion image; the image format of the first sample fusion image is different from the image format of the sample original image; the image format of the second sample fusion image is the same as the image format of the sample original image.

[0117] In one embodiment of the present disclosure, the sample fusion image includes a first sample fusion image and a second sample fusion image; the predicted fusion image includes a first predicted fusion image and a second predicted fusion image; the determining module 504 is specifically configured to determine a first image loss according to the first sample fusion image and the first predicted fusion image; determine a second image loss according to the second sample fusion image and the second predicted fusion image; perform an addition process on the first image loss and the second image loss to obtain the fusion image loss.

[0118] In the training device of the image fusion model according to the embodiment of the present disclosure, training data is obtained; the training data includes a sample original image sequence, a sample intermediate image sequence corresponding to the sample original image sequence, and a sample fusion image; the sizes of the sample original images in the sample original image sequence are the same; the sample intermediate images in the sample intermediate image sequence are determined according to the dimensional values of each pixel point in each sample original image in one channel dimension; an initial image fusion model is obtained; the sample original image sequence and the sample intermediate image sequence corresponding to the sample original image sequence are input into the image fusion model to obtain a predicted fusion image output by the image fusion model; a fusion image loss is determined according to the sample fusion image and the predicted fusion image; the parameters of the image fusion model are adjusted according to the fusion image loss to achieve training; wherein, the setting of the sample original image sequence and the sample intermediate image sequence facilitates the extraction of spatial feature images and temporal feature images, so that during the training process of the image fusion model, the displacement and deformation between multiple sample original images can be considered, thereby extracting more features, and further improving the accuracy of the trained image fusion model, and thus improving the image fusion efficiency.

[0119] According to the fifth aspect of the embodiment of the present disclosure, an electronic device is further provided, including: a processor; a memory for storing processor-executable instructions, wherein the processor is configured to: implement the above-mentioned image fusion method or the training method of the image fusion model.

[0120] To implement the above embodiments, the present disclosure also proposes a non-transitory computer-readable storage medium.

[0121] Wherein, when the instructions in the storage medium are executed by the processor, the processor can execute the above-mentioned image fusion method or the training method of the image fusion model.

[0122] To implement the above embodiments, the present disclosure also provides a computer program product.

[0123] Wherein, when the computer program product is executed by the processor of the electronic device, the electronic device can execute the above method.

[0124] Figure 6 The block diagram of an electronic device shown according to an exemplary embodiment. Figure 6 The electronic device shown is merely an example and should not impose any limitations on the functions and usage scope of the embodiments of the present disclosure.

[0125] As Figure 6 shown, the electronic device 1000 includes a processor 111, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM, Read Only Memory) 112 or the program loaded from the memory 116 into the random access memory (RAM, Random Access Memory) 113. In the RAM 113, various programs and data required for the operation of the electronic device 1000 are also stored. The processor 111, the ROM 112, and the RAM 113 are connected to each other through a bus 114. The input / output (I / O, Input / Output) interface 115 is also connected to the bus 114.

[0126] The following components are connected to the I / O interface 115: a memory 116 including a hard disk, etc.; and a communication part 117 including a network interface card such as a local area network (LAN) card, a modem, etc., and the communication part 117 performs communication processing via a network such as the Internet; a driver 118 is also connected to the I / O interface 115 as needed.

[0127] Specifically, according to the embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments of the present disclosure include a computer program carried on a computer-readable medium, and the computer program includes program codes for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 117. When the computer program is executed by the processor 111, the above functions defined in the method of the present disclosure are executed.

[0128] In an exemplary embodiment, a storage medium including instructions is also provided, for example, a memory including instructions, and the above instructions can be executed by the processor 111 of the electronic device 1000 to complete the above method. Optionally, the storage medium can be a non-transitory computer-readable storage medium. For example, the non-transitory computer-readable storage medium can be ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage devices, etc.

[0129] In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program, which can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination of the above.

[0130] Figure 7 Schematic diagram of the structure of a chip according to an embodiment of the present disclosure. As Figure 7 shown, the chip includes a processor 701 and an interface circuit 702. Among them, the number of processors 701 may be one or more, and the number of interface circuits 702 may be one or more.

[0131] Optionally, the chip further includes a memory 703 for storing necessary computer programs and data; the interface circuit 702 is used to receive signals from the memory 703 and send signals to the processor 701. The signals include computer instructions stored in the memory 703. When the processor 701 executes the computer instructions, the electronic device is caused to execute the image fusion method or the training method of the image fusion model described in the above embodiments of the present disclosure.

[0132] In addition, the word "exemplary" is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as "exemplary" is not necessarily to be understood as being advantageous compared to other aspects or designs. Instead, the use of the word exemplary is intended to present concepts in a specific manner. As used herein, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or". That is, unless otherwise specified or clear from the context, "X applies A or B" is intended to mean any permutation in a natural inclusive arrangement. That is, if X applies A; X applies B; or X applies both A and B, then "X applies A or B" is satisfied in any of the foregoing instances. Additionally, unless otherwise specified or clear from the context referring to the singular form, the articles "a" and "an" as used in this application and the appended claims are generally understood to mean "one or more".

[0133] Likewise, although the present disclosure has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art upon reading and understanding this specification and the drawings. The present disclosure includes all such modifications and variations and is limited only by the scope of the claims. Specifically with respect to the various functions performed by the components described above (e.g., elements, resources, etc.), unless otherwise indicated, the terms used to describe such components are intended to correspond to any component (functionally equivalent) that performs the specific function of the described component, even if not structurally equivalent to the disclosed structure. Additionally, although certain features of the present disclosure may have been disclosed with respect to only one of several implementations, such features may, as may be desired and advantageous for any given or particular application, be combined with one or more other features of the other implementations. Further, with respect to the use of "comprising," "having," "including," "containing," or variants thereof in the detailed description or claims, such terms are intended to be inclusive in a manner similar to the term "including."

[0134] Other embodiments of the present disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include known or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are pointed out by the following claims.

[0135] It should be understood that the present disclosure is not limited to the exact structures described above and shown in the drawings, and various modifications and changes may be made without departing from its scope. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image fusion method, characterized in that, The method includes: Obtaining an original image sequence and an intermediate image sequence corresponding to the original image sequence; the sizes of the original images in the original image sequence are the same; the intermediate images in the intermediate image sequence are determined according to the dimensionality values of each pixel point in each original image in one channel dimension. Performing feature extraction processing on the original image sequence and the intermediate image sequence to obtain a spatial feature image and a temporal feature image. Determining a fusion image corresponding to the original image sequence according to the spatial feature image and the temporal feature image.

2. The method according to claim 1, wherein The obtaining of the original image sequence and the intermediate image sequence corresponding to the original image sequence includes: Obtaining the original image sequence. For each channel dimension of the original images in the original image sequence and for each pixel point, splicing the dimensionality values of the pixel point in the channel dimension in each original image in the original image sequence to obtain an intermediate value of the pixel point. Determining an intermediate image in the channel dimension according to the intermediate values of each pixel point. Determining the intermediate image sequence according to the intermediate images in each channel dimension.

3. The method according to claim 1, wherein The performing of feature extraction processing on the original image sequence and the intermediate image sequence to obtain a spatial feature image and a temporal feature image includes: Inputting the original image sequence into a spatial feature extraction network in an image fusion model to obtain the spatial feature image output by the spatial feature extraction network. Inputting the intermediate image sequence into a temporal feature extraction network in the image fusion model to obtain the temporal feature image output by the temporal feature extraction network.

4. The method according to claim 1, wherein The spatial feature image includes the spatial features of each pixel point in the original image; the temporal feature image includes the temporal features of each pixel point in the original image; the determining of the fusion image corresponding to the original image sequence according to the spatial feature image and the temporal feature image includes: For each pixel point, splicing the spatial feature and the temporal feature of the pixel point to obtain the spatio-temporal feature of the pixel point. Determining a fusion feature image according to the spatio-temporal features of each pixel point. Determining the fusion image corresponding to the original image sequence according to the fusion feature image.

5. The method according to claim 4, wherein The determining of the fusion image corresponding to the original image sequence according to the fusion feature image includes: Inputting the fusion feature image into a fusion network in the image fusion model to obtain the fusion image output by the fusion network.

6. The method according to claim 5, wherein The fusion network is a U-Net convolutional neural network.

7. The method according to claim 1, characterized in that, The image format of the original image is the same as or different from the image format of the fusion image. When the image format of the original image is different from the image format of the fusion image, the image format of the original image is the RAW format. The image format of the fusion image is the RGB format.

8. A training method for an image fusion model, characterized in that, The method includes: Obtain training data; the training data includes a sample original image sequence, a sample intermediate image sequence corresponding to the sample original image sequence, and a sample fusion image; the sizes of the sample original images in the sample original image sequence are the same; the sample intermediate images in the sample intermediate image sequence are determined according to the dimensional values of each pixel point in each sample original image in one channel dimension. Obtain an initial image fusion model. Input the sample original image sequence and the sample intermediate image sequence corresponding to the sample original image sequence into the image fusion model, and obtain a predicted fusion image output by the image fusion model. Determine a fusion image loss according to the sample fusion image and the predicted fusion image. Adjust the parameters of the image fusion model according to the fusion image loss to achieve training.

9. The method according to claim 8, characterized in that The image fusion model includes a spatial feature extraction network, a temporal feature extraction network, a splicing network, and a fusion network. The spatial feature extraction network is used to perform spatial feature extraction processing on the sample original image sequence to obtain a spatial feature image. The temporal feature extraction network is used to perform temporal feature extraction processing on the sample intermediate image sequence to obtain a temporal feature image. The splicing network is used to splice the spatial feature image and the temporal feature image to obtain a fused feature image. The fusion network is used to process the fused feature image to obtain a predicted fusion image.

10. The method according to claim 8, characterized in that The sample fusion image includes a first sample fusion image and / or a second sample fusion image. The image format of the first sample fusion image is different from the image format of the sample original image. The image format of the second sample fusion image is the same as the image format of the sample original image.

11. The method according to claim 10, characterized in that The sample fusion image includes a first sample fusion image and a second sample fusion image; the predicted fusion image includes a first predicted fusion image and a second predicted fusion image. The determining the fusion image loss according to the sample fusion image and the predicted fusion image includes: Determine a first image loss according to the first sample fusion image and the first predicted fusion image. Determine a second image loss according to the second sample fusion image and the second predicted fusion image. Perform an addition process on the first image loss and the second image loss to obtain the fusion image loss.

12. An image fusion device, characterized in that, The device includes: An obtaining module, configured to obtain an original image sequence and an intermediate image sequence corresponding to the original image sequence; the sizes of the original images in the original image sequence are the same; the intermediate images in the intermediate image sequence are determined according to the dimensional values of each pixel point in each original image in one channel dimension. A feature extraction module, configured to perform feature extraction processing on the original image sequence and the intermediate image sequence to obtain a spatial feature image and a temporal feature image. A determining module, configured to determine a fusion image corresponding to the original image sequence according to the spatial feature image and the temporal feature image.

13. The device according to claim 12, wherein, The obtaining module is specifically configured to Obtain the original image sequence; For each channel dimension of the original images in the original image sequence, for each pixel, splice the dimension values of the pixel in the channel dimension in each original image in the original image sequence to obtain the intermediate value of the pixel; Determine the intermediate image in the channel dimension according to the intermediate values of the respective pixels; Determine the intermediate image sequence according to the intermediate images in each channel dimension.

14. The device according to claim 12, characterized in that, The feature extraction module is specifically configured to Input the original image sequence into the spatial feature extraction network in the image fusion model to obtain the spatial feature image output by the spatial feature extraction network; Input the intermediate image sequence into the temporal feature extraction network in the image fusion model to obtain the temporal feature image output by the temporal feature extraction network.

15. The device according to claim 12, characterized in that, The spatial feature image includes the spatial features of each pixel in the original image; the temporal feature image includes the temporal features of each pixel in the original image; the determination module is specifically configured to For each pixel, splice the spatial feature and the temporal feature of the pixel to obtain the spatio-temporal feature of the pixel; Determine the fused feature image according to the spatio-temporal features of the respective pixels; Determine the fused image corresponding to the original image sequence according to the fused feature image.

16. The device according to claim 15, characterized in that, The determination module is specifically further configured to Input the fused feature image into the fusion network in the image fusion model to obtain the fused image output by the fusion network.

17. The device according to claim 16, characterized in that, The fusion network is a U-Net convolutional neural network.

18. The device according to claim 12, characterized in that, The image format of the original image is the same as or different from the image format of the fused image; When the image format of the original image is different from the image format of the fused image, the image format of the original image is the RAW format; The image format of the fused image is the RGB format.

19. A training device for an image fusion model, characterized in that, The device includes: The first acquisition module is used to acquire training data; the training data includes a sample original image sequence, a sample intermediate image sequence corresponding to the sample original image sequence, and a sample fused image; the sizes of the sample original images in the sample original image sequence are the same; the sample intermediate images in the sample intermediate image sequence are determined according to the dimension values of each pixel in each sample original image in one channel dimension; The second acquisition module is used to acquire an initial image fusion model; The third acquisition module is used to input the sample original image sequence and the sample intermediate image sequence corresponding to the sample original image sequence into the image fusion model to obtain the predicted fused image output by the image fusion model; The determination module is used to determine the fused image loss according to the sample fused image and the predicted fused image; The training module is used to adjust the parameters of the image fusion model according to the fused image loss to achieve training.

20. The device according to claim 19, characterized in that, The image fusion model includes a spatial feature extraction network, a temporal feature extraction network, a splicing network, and a fusion network; The spatial feature extraction network is used to perform spatial feature extraction processing on the original sample image sequence to obtain a spatial feature image; The temporal feature extraction network is used to perform temporal feature extraction processing on the intermediate sample image sequence to obtain a temporal feature image; The splicing network is used to splice the spatial feature image and the temporal feature image to obtain a fused feature image; The fusion network is used to process the fused feature image to obtain a predicted fusion image.

21. The device according to claim 19, characterized in that, The sample fusion image includes a first sample fusion image and / or a second sample fusion image; The image format of the first sample fusion image is different from the image format of the original sample image; The image format of the second sample fusion image is the same as the image format of the original sample image.

22. The device according to claim 21, characterized in that, The sample fusion image includes a first sample fusion image and a second sample fusion image; the predicted fusion image includes a first predicted fusion image and a second predicted fusion image; the determining module is specifically configured to: Determine a first image loss according to the first sample fusion image and the first predicted fusion image; Determine a second image loss according to the second sample fusion image and the second predicted fusion image; Perform an addition process on the first image loss and the second image loss to obtain the fusion image loss.

23. An electronic device, characterized in that, Comprising: A processor; A memory for storing instructions executable by the processor; Wherein, the processor is configured to: Implement the steps of the image fusion method according to any one of claims 1 to 7; or, implement the steps of the training method of the image fusion model according to any one of claims 8 to 11.

24. A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor, enabling the processor to execute the image fusion method according to any one of claims 1 to 7; or, execute the training method of the image fusion model according to any one of claims 8 to 11.

25. A chip, characterized in that, Comprising one or more interface circuits and one or more processors; the interface circuit is used to receive signals from the memory of the electronic device and send the signals to the processor, the signals include computer instructions stored in the memory, when the processor executes the computer instructions, enabling the electronic device to execute the image fusion method according to any one of claims 1 to 7; or, execute the training method of the image fusion model according to any one of claims 8 to 11.

Images