Training method and device of image fusion model, electronic device and storage medium

Abstract

The application relates to a training method and device of an image fusion model, an electronic device and a storage medium. The method comprises the following steps: acquiring a first sample image and a second sample image; determining a first loss function based on a first initial network in an initial image fusion model; determining a second loss function based on a second initial network in the initial image fusion model; determining a third loss function based on a third initial network in the initial image fusion model; and training the initial image fusion model based on the first loss function, the second loss function and the third loss function to obtain a target image fusion model. Through the application, the technical problem of low image quality after security X-ray image fusion in the related art is solved, the data loss of the fused image is reduced, and the quality of the fused image is improved.

Description

Technical Field

[0001] This application relates to the field of image processing, and in particular to a training method, apparatus, electronic device, and storage medium for an image fusion model. Background Technology

[0002] In the field of image processing, classic image datasets often contain hundreds of thousands or even millions of sample images, such as the COCO dataset. However, these image datasets are generally used for the detection of ordinary images. In the field of security inspection, the number of security X-ray images available for training, especially those containing contraband, is relatively small, making it difficult to meet the training requirements of network models in terms of diversity.

[0003] To obtain a sufficient number of diverse X-ray security inspection sample images, image fusion is typically used to generate sample images. In related technologies, image fusion is performed based on image information such as color and texture. However, this method is more suitable for ordinary images. Since security X-ray images do not have rich image information such as texture and color, the quality of the fused image is low. Furthermore, for security X-ray images, related technologies also acquire images of contraband and background images separately, directly extracting the contraband area from the background image for fusion. This method directly removes the background image portion of the contraband area, resulting in the loss of local background information in the fused image. Therefore, the image quality of the fused security X-ray images obtained using these technologies is relatively low.

[0004] There is currently no effective solution to the technical problem of low image quality after fusion of X-ray images used in security inspections. Summary of the Invention

[0005] This embodiment provides a training method, apparatus, electronic device, and storage medium for an image fusion model to address the problem of low image quality after fusion of X-ray images used in security inspections in related technologies.

[0006] Firstly, this embodiment provides a method for training an image fusion model, including:

[0007] Acquire a first sample image and a second sample image, wherein the first sample image includes the target object to be fused, and the second sample image includes a target background associated with the target object;

[0008] A first loss function is determined based on the first initial network in the initial image fusion model, and the first initial network is used to extract the first sample features of the first sample image.

[0009] A second loss function is determined based on the second initial network in the initial image fusion model, and the second initial network is used to extract the second sample features of the second sample image;

[0010] A third loss function is determined based on the third initial network in the initial image fusion model, and the third initial network is used to fuse the first sample features and the second sample features;

[0011] The initial image fusion model is trained based on the first loss function, the second loss function, and the third loss function to obtain the target image fusion model.

[0012] In some embodiments, determining the first loss function based on the first initial network in the initial image fusion model includes:

[0013] Based on the first initial network, obtain the features of the first sample;

[0014] The first sample features are used to reconstruct the image, resulting in a first reconstructed image.

[0015] The first loss function is determined based on the first sample image and the first reconstructed image.

[0016] In some embodiments, determining the second loss function based on the second initial network in the initial image fusion model includes:

[0017] Based on the second initial network, the second sample features are obtained;

[0018] The second sample features are used to reconstruct the image, resulting in a second reconstructed image;

[0019] The second loss function is determined based on the second sample image and the second reconstructed image.

[0020] In some embodiments, determining the third loss function based on the third initial network in the initial image fusion model includes:

[0021] Based on the third initial network, the overlapping regions in the first sample image and the second sample image are obtained;

[0022] The overlapping region is compared with the first sample image to obtain a first similarity function;

[0023] The overlapping region is compared with the second sample image to obtain a second similarity function;

[0024] The third loss function is determined based on the first similarity function and the second similarity function.

[0025] In some embodiments, training the initial image fusion model based on the first loss function, the second loss function, and the third loss function includes:

[0026] The first loss function, the second loss function, and the third loss function are weighted and summed to determine the target loss function;

[0027] The initial image fusion model is trained based on the target loss function.

[0028] In some embodiments, the fusion of the first sample features and the second sample features includes:

[0029] Based on the first sample features and the second sample features, the overlapping region of the first sample image and the second sample image is determined;

[0030] The overlapping regions are normalized to generate a fusion result;

[0031] The fusion result is then reconstructed to obtain a fused sample image.

[0032] In some embodiments, the normalization process for the overlapping regions includes:

[0033] Based on the features of the first sample, a mask image of the first sample image is determined;

[0034] Based on the mask image, the first sample features, and the second sample features, normalization parameters are determined.

[0035] Based on the normalization parameters, the overlapping regions are normalized.

[0036] In some embodiments, the process of obtaining the target image fusion model further includes:

[0037] Acquire a first image to be fused and a second image to be fused, wherein the first image to be fused includes a fusion target and the second image to be fused includes a fusion background associated with the fusion target;

[0038] Based on the target image fusion model, the first image to be fused and the second image to be fused are fused to obtain the target fused image.

[0039] Secondly, this embodiment provides a training device for an image fusion model, comprising:

[0040] An acquisition module is used to acquire a first sample image and a second sample image, wherein the first sample image includes the target object to be fused and the second sample image includes a target background associated with the target object;

[0041] The first determining module is used to determine a first loss function based on a first initial network in the initial image fusion model, wherein the first initial network is used to extract a first sample feature of the first sample image.

[0042] The second determining module is used to determine a second loss function based on the second initial network in the initial image fusion model, wherein the second initial network is used to extract second sample features of the second sample image;

[0043] The third determining module is used to determine a third loss function based on the third initial network in the initial image fusion model, wherein the third initial network is used to fuse the first sample features and the second sample features;

[0044] The training module is used to train the initial image fusion model based on the first loss function, the second loss function, and the third loss function to obtain the target image fusion model.

[0045] Thirdly, this embodiment provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the training method of the image fusion model described in the first aspect above.

[0046] Fourthly, this embodiment provides a storage medium storing a computer program that, when executed by a processor, implements the training method for the image fusion model described in the first aspect above.

[0047] Compared with related technologies, this application provides a training method, apparatus, electronic device, and storage medium for an image fusion model. The method includes: acquiring a first sample image and a second sample image, wherein the first sample image includes a target object to be fused, and the second sample image includes a target background associated with the target object; determining a first loss function based on a first initial network in an initial image fusion model, wherein the first initial network is used to extract first sample features from the first sample image; determining a second loss function based on a second initial network in the initial image fusion model, wherein the second initial network is used to extract second sample features from the second sample image; determining a third loss function based on a third initial network in the initial image fusion model, wherein the third initial network is used to fuse the first sample features and the second sample features; and training the initial image fusion model based on the first loss function, the second loss function, and the third loss function to obtain a target image fusion model. In the initial image fusion model, the features of the first sample and the second sample are fused to ensure that the image information of both the first and second sample images is retained in the fused image. At the same time, the first loss function generated during the feature extraction process of the first sample image, the second loss function generated during the feature extraction process of the second sample image, and the third loss function generated during the feature fusion process are combined to extract all loss data in the image fusion process. This allows the initial image fusion model to adjust the loss, enabling the trained image fusion model to accurately fuse images. This solves the technical problem of low image quality after fusion of security X-ray images in related technologies, reduces data loss in the fused image, and improves the quality of the fused image.

[0048] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0049] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0050] Figure 1 This is a block diagram of the terminal hardware structure of a training method for an image fusion model according to an embodiment of this application;

[0051] Figure 2 This is a schematic flowchart of a training method for an image fusion model according to an embodiment of this application;

[0052] Figure 3 This is a schematic diagram of the structure of an initial image fusion model according to an embodiment of this application;

[0053] Figure 4 This is a schematic diagram of the structure of an image fusion module according to an embodiment of this application;

[0054] Figure 5 This is a structural block diagram of a training device for an image fusion model according to an embodiment of this application. Detailed Implementation

[0055] To better understand the purpose, technical solution, and advantages of this application, the application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0056] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning as understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these,” used in this application, do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to such processes, methods, products, or devices. The terms “connected,” “linked,” and “coupled,” used in this application, are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. The term “multiple” used in this application refers to two or more. The "and / or" operator describes the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: A alone, A and B simultaneously, and B alone. Typically, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," and "third," etc., used in this application are merely for distinguishing similar objects and do not represent a specific ordering of the objects.

[0057] The method embodiments provided in this example can be executed on a terminal, computer, or similar computing device. For example, it can run on a terminal. Figure 1 This is a hardware structure block diagram of the terminal for the training method of the image fusion model in this embodiment. For example... Figure 1 As shown, a terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 and a memory 104 for storing data are also included. The processor 102 may be, but is not limited to, a microprocessor (MCU) or a programmable logic device (FPGA). The terminal may also include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that… Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the terminal described above. For example, the terminal may also include components that are larger than... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown are illustrated.

[0058] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the training method of the image fusion model in this embodiment. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0059] The transmission device 106 is used to receive or send data via a network. This network includes a wireless network provided by the terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 can be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0060] In the field of security X-ray image detection, the scarcity and limited variety of X-ray images containing prohibited items make it difficult to meet the training requirements of network detection models. Therefore, constructing a comprehensive dataset of security X-ray images with diverse types of prohibited items, a large number of images, and varied target features is a primary task for prohibited item detection in the security field. Since obtaining security X-ray images containing prohibited items is difficult, fusing images of prohibited items with images of luggage and packages is an important method for acquiring sample images. However, due to the randomness and diversity of the types and arrangements of items in luggage and packages, and the significant differences between the imaging principles of security X-ray images and ordinary images, security X-ray image fusion presents a significant challenge.

[0061] In related technologies, a first image including image texture information and a second image including image color information are acquired. Single-channel image data of the first and second images are extracted respectively, and the image data of the same channel in the first and second images are fused. However, the above method is generally applicable to ordinary images, while the imaging principle of security X-ray images is different from that of ordinary images, and their texture and color information is far less than that of ordinary images. Therefore, the above method is not suitable for the fusion of security X-ray images.

[0062] Alternatively, in related technologies, an image of the contraband with the contraband and a background image without the contraband are acquired. The location of the contraband is extracted from the contraband image, and random noise is added to the same location in the background image to obtain a noisy background image. The contraband image and the noisy background image are then input into a generative adversarial synthesis network for image fusion. However, the above method adds random noise to the location of the contraband in the background image, causing a loss of local information in the background image, which in turn affects the quality of the fused image.

[0063] In addition to the aforementioned problem of poor image fusion performance, the feature fusion strategies used in related technologies for security X-ray images are relatively simple, such as employing an averaging strategy to fuse image features. This results in the inability to effectively render the color information of the security X-ray image. Furthermore, the fusion of security X-ray images in related technologies relies on a large amount of image data and labeled data, which is difficult to obtain, leading to high costs for security X-ray fusion.

[0064] Please see Figure 2 , Figure 2 This is a flowchart illustrating a training method for an image fusion model according to an embodiment of this application.

[0065] In one embodiment, the training method for the image fusion model includes:

[0066] S202: Obtain a first sample image and a second sample image, wherein the first sample image includes the target object to be fused and the second sample image includes the target background associated with the target object.

[0067] For example, sample images are acquired, including a first sample image and a second sample image. The first sample image contains the target object to be fused, and the second sample image contains the target background associated with the target object. By fusing the first sample image and the second sample image, the fused image simultaneously contains the target object and the target background.

[0068] In one specific embodiment, the training method of the image fusion model in this embodiment can be applied to the field of security inspection X-ray image fusion. The first sample image is an image of contraband, and the second sample image is an image of a package. Both the contraband image and the package image are X-ray scans. The contraband image contains image information indicating that the target is a contraband, and the package image contains image information about the package's background. Through the method in this embodiment, the fused image simultaneously contains both the aforementioned contraband information and package information.

[0069] S204: Determine a first loss function based on the first initial network in the initial image fusion model. The first initial network is used to extract the first sample features of the first sample image.

[0070] For example, in this embodiment, the initial image fusion model is an untrained image fusion model. The initial image fusion model includes a first initial network for extracting first sample features from the first sample image, wherein the first initial network is a convolutional neural network.

[0071] For example, a first loss function is established based on a first initial network, which is used to determine the data loss of the first sample image when it is iterated in the first initial network.

[0072] S206: Determine the second loss function based on the second initial network in the initial image fusion model. The second initial network is used to extract the second sample features of the second sample image.

[0073] For example, the initial image fusion model in this embodiment is further provided with a second initial network, which is used to extract the second sample features of the second sample image, wherein the second initial network is a convolutional neural network.

[0074] For example, a second loss function is established based on a second initial network, which is used to determine the data loss of the second sample image when it is iterated in the second initial network.

[0075] S208: Determine the third loss function based on the third initial network in the initial image fusion model. The third initial network is used to fuse the features of the first sample and the features of the second sample.

[0076] For example, the initial image fusion model in this embodiment includes a third initial network, which is used to fuse the first sample features of the first sample image and the second sample features of the second sample image to obtain a fusion result. The third initial network is a convolutional neural network.

[0077] For example, a third loss function is established based on a third initial network. The third loss function is used to determine the data loss when the fused convolutional features are iterated in the third initial network.

[0078] For example, the third initial network is connected to the outputs of the forward propagation channels of both the first and second initial networks to simultaneously acquire the first sample features and the second sample features. After acquiring the first and second sample features, the third initial network first fuses the two features, and then processes the fused convolutional features to generate a fused image.

[0079] S210: The initial image fusion model is trained based on the first loss function, the second loss function, and the third loss function to obtain the target image fusion model.

[0080] For example, after obtaining the first loss function, the second loss function, and the third loss function, the initial image fusion model is trained by combining the above loss functions, thereby adjusting the parameters in the initial image fusion model to obtain the final target image fusion model.

[0081] In this embodiment, the first sample features and the second sample features are fused in the initial image fusion model to ensure that the image information of both the first and second sample images is retained in the fused image. At the same time, the first loss function generated during the feature extraction process of the first sample image, the second loss function generated during the feature extraction process of the second sample image, and the third loss function generated during the feature fusion process are combined to extract all loss data in the image fusion process. This allows the initial image fusion model to adjust the loss, enabling the trained image fusion model to accurately fuse images. This solves the technical problem of low image quality after fusion of security inspection X-ray images in related technologies, reduces data loss in the fused image, and improves the quality of the fused image.

[0082] In another embodiment, determining the first loss function based on the first initial network in the initial image fusion model includes:

[0083] Step 1: Based on the first initial network, obtain the features of the first sample;

[0084] Step 2: Reconstruct the image based on the features of the first sample to obtain the first reconstructed image;

[0085] Step 3: Determine the first loss function based on the first sample image and the first reconstructed image.

[0086] For example, a first sample image is obtained based on a first initial network, and convolutional feature extraction is performed on the first sample image to obtain first sample features. The first sample features are used as input to a third initial network for feature fusion. After obtaining the first sample features, the first sample features are reconstructed through the forward propagation channel of the first initial network to obtain a first reconstructed image. After generating the first reconstructed image, a first loss function is determined based on the first reconstructed image and the first sample image.

[0087] Specifically, the forward propagation channel of the first initial network includes an encoding channel and a decoding channel. The encoding channel is used to downsample the first sample image and extract the first sample features, while the decoding channel is used to upsample the first sample features to restore the first sample features to the reconstructed image.

[0088] Specifically, a first loss function is determined based on the first reconstructed image and the first sample image. This first loss function is used to adjust the loss of the output of the third initial network. In one specific embodiment, the first loss function is calculated as follows:

[0089]

[0090] Where f is the first sample image, This is the first reconstructed image.

[0091] In this embodiment, after extracting the first sample features from the first sample image, a first reconstructed image is obtained based on the first sample features. Then, a first loss function is established based on the loss between the first sample image and the first reconstructed image. This function can be directly determined without complex deduction and design, thereby improving the efficiency of image fusion. Furthermore, the first loss function established based on the method in this embodiment can accurately reflect the data loss during the forward propagation of the first initial network, thus improving the accuracy of loss correction for the fused image.

[0092] In another embodiment, determining the second loss function based on the second initial network in the initial image fusion model includes:

[0093] Step 1: Obtain the features of the second sample based on the second initial network;

[0094] Step 2: Reconstruct the image based on the features of the second sample to obtain the second reconstructed image;

[0095] Step 3: Determine the second loss function based on the second sample image and the second reconstructed image.

[0096] For example, a second sample image is obtained based on a second initial network, and convolutional feature extraction is performed on the second sample image to obtain second sample features. The second sample features are used as input to a third initial network for feature fusion. After obtaining the second sample features, the second sample features are reconstructed through the forward propagation channel of the second initial network to obtain a second reconstructed image. After generating the second reconstructed image, a second loss function is determined based on the second reconstructed image and the second sample image.

[0097] Specifically, the forward propagation channel of the second initial network includes an encoding channel and a decoding channel. The encoding channel is used to downsample the second sample image to extract the second sample features, and the decoding channel is used to upsample the second sample features to restore the second sample features to the reconstructed image.

[0098] Specifically, a second loss function is determined based on the second reconstructed image and the second sample image. This second loss function is used to adjust the loss of the output of the third initial network. In one specific embodiment, the second loss function is calculated as follows:

[0099]

[0100] Where b is the second sample image, This is the second reconstructed image.

[0101] In this embodiment, after extracting the second sample features from the second sample image, a second reconstructed image is obtained based on the second sample features. Then, a second loss function is established based on the loss between the second sample image and the second reconstructed image. This function can be directly determined without complex deduction and design, thereby improving the efficiency of image fusion. Simultaneously, the second loss function established based on the method in this embodiment can accurately reflect the data loss during the forward propagation of the second initial network, thus improving the accuracy of loss correction for the fused image.

[0102] In another embodiment, the above two embodiments will be described in detail.

[0103] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of an initial image fusion model according to an embodiment of this application.

[0104] Specifically, such as Figure 3As shown, the forward propagation channels of the first and second initial networks include an encoding channel and a decoding channel. The encoding channel performs downsampling operations such as convolution and pooling on the image to be fused to extract deep convolutional features. The decoding channel performs upsampling operations such as deconvolution and unpooling on the deep convolutional features to restore the reconstructed image from the deep convolutional features output by the encoding channel. The restored reconstructed image can be used for subsequent extraction of the loss function for the fused image. The forward propagation channels of the third initial network include a fusion channel and a decoding channel. The fusion channel performs normalized fusion of the first and second sample features to obtain fused convolutional features. The decoding channel performs upsampling operations such as deconvolution and unpooling on the fused convolutional features to reconstruct the fused convolutional features into a fused image.

[0105] Specifically, the third initial network is connected to the output of the encoding channels of both the first and second initial networks to extract the deepest convolutional features.

[0106] Specifically, the encoding channels of the first and second initial networks share network weights to ensure that the first and second sample features received by the third initial network are obtained based on the same convolution and pooling operations, thus enabling the third initial network to directly fuse the first and second sample features. Furthermore, the decoding channels of the first and second initial networks share network weights to ensure that the reconstructed images are obtained based on the same deconvolution and unpooling operations. Even further, the decoding channels of the third initial network share network weights with the decoding channels of the first and second initial networks to ensure that the decoding process of the fused image and the reconstructed image is identical.

[0107] Specifically, the encoding and decoding channels of the first initial network, the encoding and decoding channels of the second initial network, and the third initial network also include dense network modules. Unlike the short-circuit connections between typical convolutional layers (i.e., two consecutive convolutional layers are interconnected), the convolutional layers in the dense network modules are densely connected (i.e., each convolutional layer is simultaneously connected to all preceding convolutional layers). For example, if a dense network module includes a first, second, and third convolutional layer, then the second convolutional layer is connected to the first convolutional layer, and the third convolutional layer is simultaneously connected to both the first and second convolutional layers. Through these dense connections, feature reuse in the forward propagation channels can be achieved. For instance, in the aforementioned dense network module, the convolutional features output by the first convolutional layer are simultaneously used as inputs to the second and third convolutional layers, thereby improving the utilization rate of convolutional features in the channels. This can improve the network's computational performance while reducing network parameters and computational costs.

[0108] Specifically, the first loss function is used to calculate the loss between the first sample image and the first reconstructed image, the second loss function is used to calculate the loss between the second sample image and the second reconstructed image, and the third loss function is used to calculate the data loss generated during the convolutional feature fusion process. By adjusting the parameters of the third initial network using the first, second, and third loss functions, the image fusion model can be adjusted. A detailed analysis follows:

[0109] First, during the process of generating the reconstructed image from the first sample image, due to the sequential encoding and decoding of the first sample image, there is inevitably some loss of image data. Assume the first sample image is in 4x4 format, and its pixel matrix is ​​as follows:

[0110]

[0111] Encoding is performed through an encoding channel, and the parameters involved in the encoding are as follows:

[0112]

[0113] Where c is the convolution kernel and s is the convolution stride. The pixel matrix, after encoding, yields the feature matrix:

[0114]

[0115] As can be seen, image data is compressed during encoding, resulting in data loss. In the subsequent decoding process, since the lost data cannot be recovered, the feature matrix must be padded with values ​​to obtain the pixel matrix of the reconstructed image. For example, the reconstructed image can be obtained by padding with zero values.

[0116]

[0117] Therefore, the loss matrix between the first sample image and the reconstructed image is obtained as follows:

[0118]

[0119] Specifically, the aforementioned data losses occur during the generation of the first reconstructed image from the first sample image and the generation of the second reconstructed image from the second sample image, corresponding to the first loss function and the second loss function, respectively. Since the sub-networks of the entire image fusion model share weights, both losses are generated simultaneously during the generation of the fused image from the first and second sample images; these losses are the sum of the first and second loss functions. Additionally, the third initial network incurs further data loss during feature fusion, namely the third loss function. Therefore, the parameters of the third initial network can be adjusted simultaneously using the first, second, and third loss functions to obtain the desired fused image.

[0120] In another embodiment, determining the third loss function based on the third initial network in the initial image fusion model includes:

[0121] Step 1: Based on the third initial network, obtain the overlapping regions in the first sample image and the second sample image;

[0122] Step 2: Compare the overlapping regions with the first sample image to obtain the first similarity function;

[0123] Step 3: Compare the overlapping regions with the second sample image to obtain the second similarity function;

[0124] Step 4: Determine the third loss function based on the first and second similarity functions.

[0125] For example, the first sample image and the second sample image are fused using a third initial network to obtain a fused image. After image fusion, since the images in the overlapping regions have local structural similarity with the first and second sample images, the overlapping regions in the fused image can be compared with the first and second sample images respectively to obtain a first similarity function and a second similarity function. The third loss function of the image fusion model is then calculated using these first and second similarity functions. Understandably, the first and second similarity functions reflect the degree of loss of the overlapping regions compared to the first and second sample images, respectively.

[0126] For example, after determining the first similarity function and the second similarity function, their weights can be set respectively, and the third loss function can be determined using the first similarity function, the second similarity function, and their respective weights. The weights of the first and second similarity functions in the third loss function can be adjusted accordingly.

[0127] For example, since the image outside the overlapping region of the fused image is similar to the second sample image, the loss data of the fused image and the second sample image outside the overlapping region can also be combined when calculating the third loss function.

[0128] In one specific embodiment, the similarity function is determined as follows:

[0129]

[0130] Where X and Y represent the two images being compared for similarity, SSLM is the structural similarity test result of the windows in the image, MN is the total number of windows, and x ij ,y ij These are local windows of the two images. Based on the above method for determining the similarity function, the first similarity function MSSLM(fb,f) and the second similarity function MSSLM(fb,b) are obtained, where fb, f, and b are the fused image, the first sample image, and the second sample image, respectively.

[0131] Specifically, after determining the first similarity function and the second similarity function, their weights are determined respectively, resulting in the loss function between the fused image and the first sample image and the second sample image:

[0132] Loss fusion (fb,f,b)=1-(w·MSSLM(fb,f)+(1-w)·(fb,b))

[0133] Where w and 1-w are the weights of the first similarity function and the second similarity function, respectively, and preferably set to 0.5 and 0.5, respectively.

[0134] Specifically, the aforementioned loss function represents the loss between the entire fused image and the first and second sample images. However, when actually calculating the third loss function, only the loss data corresponding to the overlapping regions in the aforementioned loss function should be retained. Based on this, the loss function between the image within the fused region and the first and second sample images is obtained:

[0135] Loss gen 1 = + ⊙ fusion (fb,f,b)

[0136] Among them, M + This is a mask image of the first sample image, used to obtain the overlapping region.

[0137] Specifically, when calculating the third loss function, the loss function between the image outside the fusion region and the second sample image can also be combined:

[0138] Loss gen 2=(1-M + )⊙||,f||

[0139] Specifically, the final third loss function is obtained through the above calculations.

[0140] Loss gen =oss gen 1+oss gen 2

[0141] This embodiment utilizes local similarity to calculate the similarity loss between overlapping regions in the fused image and the first and second sample images. This can retain more feature information in the first and second sample images while improving the correlation between the fused image and the first and second sample images, thereby improving the quality of the fused image.

[0142] In another embodiment, training the initial image fusion model based on the first loss function, the second loss function, and the third loss function includes:

[0143] Step 1: Perform a weighted sum of the first loss function, the second loss function, and the third loss function to determine the target loss function;

[0144] Step 2: Train the initial image fusion model based on the target loss function.

[0145] For example, in this embodiment, the first loss function is used to determine the data loss of the first sample image when it is iterated in the first initial network, the second loss function is used to determine the data loss of the second sample image when it is iterated in the second initial network, and the third loss function is used to determine the data loss of the fused convolutional features when it is iterated in the third initial network.

[0146] For example, after obtaining the first loss function, the second loss function, and the third loss function, their respective weights are determined. Then, the first, second, and third loss functions are weighted and summed to obtain the target loss function of the initial image fusion model. By adjusting the weights of each loss function, the proportion of each loss function in the target loss function is adjusted to optimize the target loss function. After obtaining the target loss function, the parameters of the initial image fusion model are adjusted based on the target loss function to obtain the target image fusion model.

[0147] In one specific embodiment, the weights of the first loss function, the second loss function, and the third loss function are all set to 1, resulting in the target loss function:

[0148] Loss = Loss recon 1+Lossrecon 2+oss gen 2

[0149] This embodiment calculates the target loss function by weighted summation of the first loss function, the second loss function, and the third loss function, thereby improving the accuracy of the target loss function and consequently improving the accuracy of the target image fusion model parameters and the accuracy of the fused image.

[0150] In another embodiment, fusing the first sample features and the second sample features includes:

[0151] Step 1: Based on the features of the first sample and the features of the second sample, determine the overlapping region of the first sample image and the second sample image;

[0152] Step 2: Normalize the overlapping regions to generate the fusion result;

[0153] Step 3: Reconstruct the image from the fusion result to obtain the fused sample image.

[0154] For example, the third initial network is connected to both the first and second initial networks to receive first sample features and second sample features. Based on the first and second sample features, the overlapping region of the first and second sample images is determined, which is the region to be fused; the overlapping region is normalized, i.e., locally fused, to generate a fused image.

[0155] Specifically, the first sample image includes the target region of the target object, and the second sample image includes the background region. Based on the features of the first sample image, the foreground region of the first sample image can be determined, and then the region to be fused can be determined based on the foreground region; or, a mask image of the first sample image can be determined based on the features of the first sample image, which reflects the foreground region of the first sample image, and the fusion of local regions of the image can be controlled by the mask image.

[0156] This embodiment analyzes the features of the first sample and the features of the second sample to obtain the fusion region of the first sample image and the second sample image. The fusion region is then normalized, and the image information of the first sample image and the image information of the second sample image corresponding to the fusion region are fused together, which ensures the accuracy of the fusion region and thus improves the quality of the fused image.

[0157] In another embodiment, normalizing the overlapping regions includes:

[0158] Step 1: Determine the mask image of the first sample image based on the features of the first sample;

[0159] Step 2: Determine the normalization parameters based on the mask image, the features of the first sample, and the features of the second sample;

[0160] Step 3: Normalize the overlapping regions based on the normalization parameters.

[0161] For example, the first sample image includes the target region of the target object, and the second sample image includes the background region. The foreground region of the target object in the first sample image is determined based on the features of the first sample image, and a mask image of the first sample image is determined based on this foreground region. The mask image is used to extract the target region from the first sample image. For example, pixels corresponding to the foreground region in the mask image are set to 1, and the remaining pixels are set to 0. The corresponding target region can be obtained by multiplying the pixels of the mask image and the first sample image.

[0162] For example, after acquiring the mask image, normalization parameters for image fusion are determined based on the mask image and the first and second sample features. Specifically, the mask image is used to determine the fusion region between the first and second sample images, and the normalization parameters are determined based on the convolutional features corresponding to the images in the first and second sample images within the fusion region. These normalization parameters are computational parameters used in the image fusion process.

[0163] In one specific embodiment, the first sample image is an image of contraband, the second sample image is an image of a package, and the foreground area in the first sample image is the area of ​​contraband.

[0164] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of an image fusion module according to an embodiment of this application.

[0165] Specifically, the convolution features of the first sample image f and the second sample image b are respectively the first sample feature F f and the second sample feature F b For the first sample feature F f Analysis is performed to determine its mask image; first sample feature F f and the second sample feature F b The parameter γ is obtained after convolution operation by the image fusion module. fb and parameter β fb The second sample feature F b The parameter γ is obtained through convolution operation in the image fusion module. b and parameter β b Combining the above parameters and the mask image, the normalized parameters γ and β are obtained, and the specific calculation method is as follows:

[0166] γ=M + ⊙ fb +(1-+ )⊙ b

[0167] β=M + ⊙ fb +(1- + )⊙ b

[0168] Among them, M + The mask image for the first sample image, 1-M + Let be the complementary mask image of the first sample image, and ⊙ denotes the dot product operation between matrices.

[0169] Specifically, after obtaining the above normalization parameters, the second sample feature F is adjusted using the above normalization parameters. b Normalization is then performed to obtain the fused image. The calculation process for normalization is as follows:

[0170]

[0171] Where μ(b) and σ(b) are the second sample features F, respectively. b The mean and variance of the sample. The logical process of this normalization is as follows: first, subtract the second sample feature F... b The color features of the second sample image b are removed by dividing the mean by the standard deviation. Then, the color features of the fusion region are adaptively changed based on the normalization parameters γ and β, where the normalization parameters γ and β include mask image region information (i.e., fusion region information) and fusion region color information.

[0172] Preferably, before parameter extraction, the first sample features F are processed by a pooling layer (e.g., an average pooling layer). f Second sample feature F b The mask image and its complementary mask image are sampled to compress their data volume and improve the iteration speed.

[0173] Preferably, before normalizing the second sample image, the second sample features F are first processed. b BatchNorm processing is performed to ensure that the data dimension of the features input to the normalized nodes of the image fusion model remains consistent.

[0174] The method in this embodiment ensures that the image features of the image fusion region corresponding to the mask image are jointly determined by the first sample image and the second sample image, while the image features outside the image fusion region are determined by the second sample image. By integrating the first sample image into the second sample image, the background information of the second sample image is preserved, thereby improving the quality of the fused image.

[0175] In another embodiment, after obtaining the target image fusion model, the method further includes:

[0176] Step 1: Obtain a first image to be fused and a second image to be fused. The first image to be fused includes the fusion target, and the second image to be fused includes the fusion background associated with the fusion target.

[0177] Step 2: Based on the target image fusion model, perform image fusion on the first image to be fused and the second image to be fused to obtain the target fused image.

[0178] For example, after obtaining the target image fusion model, a first image to be fused and a second image to be fused are input into the target image fusion model for image fusion. The first image to be fused includes the fusion target, and the second image to be fused includes a fusion background associated with the fusion target. The target image fusion model performs feature extraction and feature fusion on the first and second images to be fused, thereby generating the target fused image.

[0179] This embodiment performs image fusion on the first image to be fused and the second image to be fused based on the target image fusion model, so that the target fused image contains the image information of both the first image to be fused and the second image to be fused, thereby improving the quality of the target fused image.

[0180] It should be noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0181] This embodiment also provides an image fusion apparatus for implementing the above embodiments and preferred embodiments; details already described will not be repeated. The terms "module," "unit," "subunit," etc., used below refer to combinations of software and / or hardware that perform a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0182] Figure 5 This is a structural block diagram of the training device for the image fusion model in this embodiment, as shown below. Figure 5 As shown, the device includes:

[0183] The acquisition module 10 is used to acquire a first sample image and a second sample image. The first sample image includes the target object to be fused, and the second sample image includes the target background associated with the target object.

[0184] The first determining module 20 is used to determine a first loss function based on the first initial network in the initial image fusion model, wherein the first initial network is used to extract the first sample features of the first sample image.

[0185] The first determining module 20 is also used to obtain the features of the first sample based on the first initial network;

[0186] The first sample features are used to reconstruct the image, resulting in the first reconstructed image;

[0187] Based on the first sample image and the first reconstructed image, determine the first loss function;

[0188] The second determining module 30 is used to determine a second loss function based on the second initial network in the initial image fusion model. The second initial network is used to extract the second sample features of the second sample image.

[0189] The second determining module 30 is also used to obtain second sample features based on the second initial network;

[0190] The second sample features are used to reconstruct the image, resulting in the second reconstructed image.

[0191] Based on the second sample image and the second reconstructed image, determine the second loss function;

[0192] The third determining module 40 is used to determine the third loss function based on the third initial network in the initial image fusion model. The third initial network is used to fuse the features of the first sample and the features of the second sample.

[0193] The third determining module 40 is also used to obtain overlapping regions in the first sample image and the second sample image based on the third initial network.

[0194] The overlapping regions are compared with the first sample image to obtain the first similarity function;

[0195] The overlapping regions are compared with the second sample image to obtain the second similarity function;

[0196] Based on the first similarity function and the second similarity function, the third loss function is determined;

[0197] The third determining module 40 is also used to determine the overlapping region of the first sample image and the second sample image based on the first sample features and the second sample features.

[0198] The overlapping regions are normalized to generate the fusion result;

[0199] Image reconstruction is performed on the fusion result to obtain the fused sample image;

[0200] The third determining module 40 is also used to determine the mask image of the first sample image based on the features of the first sample;

[0201] Based on the mask image, the features of the first sample, and the features of the second sample, the normalization parameters are determined.

[0202] The overlapping regions are normalized based on the normalization parameters.

[0203] Training module 50 is used to train the initial image fusion model based on the first loss function, the second loss function and the third loss function to obtain the target image fusion model;

[0204] The training module 50 is also used to perform a weighted summation of the first loss function, the second loss function, and the third loss function to determine the target loss function;

[0205] The initial image fusion model is trained based on the objective loss function;

[0206] The training device for the image fusion model also includes a fusion module;

[0207] The fusion module is used to acquire a first image to be fused and a second image to be fused. The first image to be fused includes a fusion target, and the second image to be fused includes a fusion background associated with the fusion target.

[0208] Based on the target image fusion model, the first image to be fused and the second image to be fused are fused to obtain the target fused image.

[0209] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0210] This embodiment also provides an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0211] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0212] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0213] S1: Obtain a first sample image and a second sample image. The first sample image includes the target object to be fused, and the second sample image includes the target background associated with the target object.

[0214] S2: Determine the first loss function based on the first initial network in the initial image fusion model. The first initial network is used to extract the first sample features of the first sample image.

[0215] S3: Determine the second loss function based on the second initial network in the initial image fusion model. The second initial network is used to extract the second sample features of the second sample image.

[0216] S4: Determine the third loss function based on the third initial network in the initial image fusion model. The third initial network is used to fuse the features of the first sample and the features of the second sample.

[0217] S5: The initial image fusion model is trained based on the first loss function, the second loss function, and the third loss function to obtain the target image fusion model.

[0218] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated in this embodiment.

[0219] Furthermore, in conjunction with the image fusion model training method provided in the above embodiments, this embodiment can also provide a storage medium for implementation. This storage medium stores a computer program; when executed by a processor, the computer program implements any of the image fusion model training methods in the above embodiments.

[0220] It should be understood that the specific embodiments described herein are merely illustrative of the application and not intended to limit it. All other embodiments derived by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0221] Obviously, the accompanying drawings are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar situations based on these drawings without any creative effort. Furthermore, it is understood that although the work done in this development process may be complex and lengthy, for those skilled in the art, certain design, manufacturing, or production modifications made based on the technical content disclosed in this application are merely conventional technical means and should not be considered as insufficient disclosure of this application.

[0222] The term "embodiment" in this application refers to a specific feature, structure, or characteristic described in connection with an embodiment that may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily imply the same embodiment, nor does it imply that it is mutually exclusive with or independent of other embodiments. It will be clearly or implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0223] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the appended claims.

Claims

1. A method for training an image fusion model, characterized in that, include: Acquire a first sample image and a second sample image, wherein the first sample image includes the target object to be fused, and the second sample image includes a target background associated with the target object; Based on the first initial network in the initial image fusion model, the first sample features of the first sample image are obtained; the first sample features are used to reconstruct the image to obtain the first reconstructed image; and a first loss function is determined based on the first sample image and the first reconstructed image. Based on the second initial network in the initial image fusion model, the second sample features of the second sample image are obtained; the second sample features are used to reconstruct the image to obtain the second reconstructed image; and the second loss function is determined based on the second sample image and the second reconstructed image. Based on the third initial network in the initial image fusion model, overlapping regions in the first sample image and the second sample image are obtained; the overlapping regions are compared with the first sample image to obtain a first similarity function; the overlapping regions are compared with the second sample image to obtain a second similarity function; based on the first similarity function, the second similarity function, and the mask image of the first sample image, a loss function is obtained between the image within the overlapping region and the first and second sample images; based on the loss function between the image within the overlapping region and the first and second sample images, and the loss function between the image outside the overlapping region and the second sample image, a third loss function is determined, and the third initial network is used to fuse the features of the first sample and the features of the second sample. The initial image fusion model is trained based on the first loss function, the second loss function, and the third loss function to obtain the target image fusion model.

2. The method of claim 1, wherein, The training of the initial image fusion model based on the first loss function, the second loss function, and the third loss function includes: The first loss function, the second loss function, and the third loss function are weighted and summed to determine the target loss function; The initial image fusion model is trained based on the target loss function.

3. The method of claim 1, wherein the image fusion model is trained using a convolutional neural network. The fusion of the first sample features and the second sample features includes: Based on the first sample features and the second sample features, the overlapping region of the first sample image and the second sample image is determined; The overlapping regions are normalized to generate a fusion result; The fusion result is then reconstructed to obtain a fused sample image.

4. The method of claim 3, wherein, The normalization process for the overlapping region includes: Based on the features of the first sample, a mask image of the first sample image is determined; Based on the mask image, the first sample features, and the second sample features, normalization parameters are determined. Based on the normalization parameters, the overlapping regions are normalized.

5. The method of claim 1, wherein, After obtaining the target image fusion model, the process also includes: Acquire a first image to be fused and a second image to be fused, wherein the first image to be fused includes a fusion target and the second image to be fused includes a fusion background associated with the fusion target; Based on the target image fusion model, the first image to be fused and the second image to be fused are fused to obtain the target fused image.

6. An image fusion model training apparatus, characterized in that, include: An acquisition module is used to acquire a first sample image and a second sample image, wherein the first sample image includes the target object to be fused and the second sample image includes a target background associated with the target object; The first determining module is used to obtain first sample features of the first sample image based on the first initial network in the initial image fusion model; perform image reconstruction on the first sample features to obtain a first reconstructed image; and determine a first loss function based on the first sample image and the first reconstructed image. The second determining module is used to obtain the second sample features of the second sample image based on the second initial network in the initial image fusion model; The second sample features are used to reconstruct the image, resulting in a second reconstructed image; a second loss function is determined based on the second sample image and the second reconstructed image. The third determining module is used to obtain overlapping regions in the first sample image and the second sample image based on the third initial network in the initial image fusion model; compare the overlapping regions with the first sample image to obtain a first similarity function; compare the overlapping regions with the second sample image to obtain a second similarity function; obtain a loss function between the image within the overlapping region and the first sample image and the second sample image based on the first similarity function, the second similarity function, and the mask image of the first sample image; and determine a third loss function based on the loss function between the image within the overlapping region and the first sample image and the second sample image, and the loss function between the image outside the overlapping region and the second sample image. The third initial network is used to fuse the features of the first sample image and the features of the second sample image. The training module is used to train the initial image fusion model based on the first loss function, the second loss function, and the third loss function to obtain the target image fusion model. 7.An electronic device comprising a memory and a processor, the electronic device characterized by, The memory stores a computer program, and the processor is configured to run the computer program to perform the training method of the image fusion model according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the training method for the image fusion model according to any one of claims 1 to 5.

Images