Multi-view industrial image fusion method based on cross attention mechanism

Abstract

The invention provides a multi-view industrial image fusion method based on a cross attention mechanism, and relates to the technical field of image processing, and the method comprises the steps: inputting two industrial images of different views into a first encoder and a second encoder respectively, and correspondingly obtaining a first feature image and a second feature image; inputting the first feature image and the second feature image into a decoder to obtain a first fusion feature and a second fusion feature; fusing the first fusion feature and the second fusion feature to generate a fused image; performing weighted summation on pixel reconstruction loss values and structure loss values of the fused image, the first feature image and the second feature image to generate a total loss function; adjusting parameters in the encoder and the decoder according to the total loss function; and inputting two industrial images with different visual angles into the adjusted encoder and decoder to obtain a target fusion image. The technical effect of the invention is that the requirement of a complex industrial scene for high-quality image fusion is met.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a multi-view industrial image fusion method, device, equipment and storage medium based on a cross-attention mechanism. Background Art

[0002] In recent years, with the continuous development of industrial automation and intelligent manufacturing technology, machine vision has been widely used in the industrial field. However, due to the complexity and variability of the industrial environment, traditional image fusion methods often fail to fully utilize the complementary information of images from different perspectives, and the quality and accuracy of the generated fused images are limited, which is difficult to meet the needs of practical applications.

[0003] In order to solve the above problems, the attention mechanism is usually used to improve the fusion effect. Specifically, by learning the correlation and importance between image regions, the weights of the feature maps are adaptively adjusted to improve the quality and semantic consistency of the fused image. This image fusion method based on the attention mechanism improves the fusion effect to a certain extent and can better handle complex industrial scenes. However, most of the above image fusion methods use unidirectional or symmetrical attention calculations, lacking the full utilization of the interactive information between images from different perspectives, resulting in insufficient detail retention and structural consistency of the fused image, which cannot meet the needs of complex industrial scenes for high-quality image fusion. Summary of the invention

[0004] The present application provides a multi-view industrial image fusion method, apparatus, device and storage medium based on a cross-attention mechanism, which are used to meet the needs of complex industrial scenarios for high-quality image fusion.

[0005] In a first aspect, the present application provides a multi-view industrial image fusion method based on a cross-attention mechanism, the method comprising: inputting two normalized industrial images of different viewpoints into a first encoder and a second encoder respectively, and obtaining a first feature image and a second feature image correspondingly, wherein the first encoder and the second encoder are weight-sharing encoders; inputting the first feature image and the second feature image into a decoder to obtain a first fusion feature and a second fusion feature; fusing the first fusion feature and the second fusion feature to generate a target fusion feature, and mapping the target fusion feature to the image space through a convolutional layer to generate a fusion image; calculating a pixel reconstruction loss value and a structural loss value of the fusion image and the first feature image and the second feature image, and weightedly summing the pixel reconstruction loss value and the structural loss value to generate a total loss function; adjusting the parameters in the first encoder, the second encoder and the decoder according to the total loss function, and obtaining a first target encoder, a second target encoder and a target decoder correspondingly; inputting the two industrial images of different viewpoints into the first target encoder and the second target encoder respectively, and obtaining a first target feature image and a second target feature image correspondingly, and inputting the first target feature image and the second target feature image into the target decoder to obtain a target fusion image.

[0006] By adopting the above technical solution, the weight-sharing encoder is used to process industrial images of different perspectives, which effectively reduces the number of model parameters and improves the computational efficiency. The decoder used in the method combines the self-attention and cross-attention mechanisms, which can fully capture the internal features of the image and the interactive information between images of different perspectives, thereby enhancing the effect of feature extraction and fusion. The target fusion features are generated by designing a feature fusion formula, and the convolution layer is used to map them to the image space, thereby achieving high-quality image fusion. The calculation of pixel reconstruction loss and structural loss is also introduced, and the total loss function is generated by weighted summation. This multi-objective optimization strategy not only ensures the pixel-level accuracy of the fused image, but also maintains the overall structure of the image, thereby improving the visual quality and semantic consistency of the fused image. The model parameters are optimized by the back-propagation algorithm, and the target encoder and decoder obtained can better adapt to complex industrial scenes. The target fused image finally generated has significant improvements in detail retention and structural consistency, meeting the needs of industrial applications for high-quality image fusion.

[0007] Optionally, the inputting the first feature image and the second feature image into the decoder to obtain the first fusion feature and the second fusion feature includes: inputting the first feature image and the second feature image into the self-attention module in the decoder respectively, and performing feature enhancement on the first feature image and the second feature image through the self-attention mechanism to obtain the first enhanced feature image and the second enhanced feature image; inputting the first enhanced feature image and the second enhanced feature image into the cross-attention module in the decoder, and performing feature interactive fusion on the first enhanced feature image and the second enhanced feature image through the cross-attention mechanism to obtain the first fusion feature and the second fusion feature.

[0008] By adopting the above technical solution, the self-attention module performs feature enhancement on the first feature image and the second feature image respectively to generate the first enhanced feature image and the second enhanced feature image, which can effectively capture the long-range dependency relationship within a single image and improve the expressiveness of the features. Subsequently, the cross-attention module performs feature interactive fusion on the two enhanced feature images to obtain the first fusion feature and the second fusion feature. This process makes full use of the complementary information between images from different perspectives and enhances the comprehensiveness and robustness of the features. This design combining self-attention and cross-attention not only improves the model's ability to understand the internal structure of a single image, but also enhances its ability to capture the relationship between multi-perspective images, thereby achieving a more comprehensive and accurate feature fusion while retaining the unique information of each perspective.

[0009] Optionally, fusing the first fusion feature and the second fusion feature to generate a target fusion feature includes: substituting the first fusion feature and the second fusion feature into a feature fusion formula to generate a target fusion feature; wherein the feature fusion formula is: ; In the formula, is the target fusion feature, is the weight coefficient, is the first fusion feature, It is the second fusion feature.

[0010] By adopting the above technical solution, the feature fusion formula is introduced to generate the target fusion feature, and the adaptive fusion of the first fusion feature and the second fusion feature is achieved. This weighted fusion method allows the model to dynamically adjust the weight coefficient according to the characteristics and importance of images from different perspectives, so as to more flexibly balance the contribution of the two features during the fusion process. This design can not only make full use of the complementary information of the two perspectives, but also adaptively adjust the fusion strategy according to the needs of the specific scene. By adjusting the weight coefficient, a balance can be achieved between retaining the unique information of each perspective and enhancing the common features, so as to generate more comprehensive and accurate target fusion features. This adaptive fusion mechanism is particularly suitable for processing complex industrial scenes, where the importance of different perspectives may change with the scene. Therefore, this method can generate higher quality fusion features, thereby improving the visual quality and semantic consistency of the final fused image, and better meet the diverse needs of image fusion in industrial applications.

[0011] Optionally, the calculating the pixel reconstruction loss value of the fused image and the first feature image and the second feature image includes: obtaining the width and height of the fused image, the pixel value of the fused image at each position, the pixel value of the first feature image at each position, and the pixel value of the second feature image at each position; substituting the width and height of the fused image, the pixel value of the fused image at each position, the pixel value of the first feature image at each position, and the pixel value of the second feature image at each position into a pixel reconstruction loss formula to calculate the pixel reconstruction loss value of the fused image and the first feature image and the second feature image; wherein the pixel reconstruction loss formula is: ; In the formula, is the pixel reconstruction loss value, is the width of the fused image, is the height of the fused image, To fuse the image at position The pixel value at The first feature image is at position The pixel value at The second feature image is at position The pixel value at .

[0012] By adopting the above technical solution, the width and height of the fused image, as well as the pixel values ​​of the fused image, the first feature image and the second feature image at each position are obtained, and then the loss value is calculated by the pixel reconstruction loss formula. This design takes into account all pixels of the entire image, and comprehensively evaluates the fusion effect by calculating the mean square error between the pixel value of the fused image and the average value of the pixel value of the original feature image. By dividing the error value by the total number of pixels in the image, a normalized loss value is obtained, so that images of different sizes can be compared fairly. This pixel-level evaluation method can capture subtle image differences and help maintain the consistency of the fused image with the original image at the pixel level. At the same time, it also provides an effective feedback signal for model optimization, which enables the fusion result to better fuse the features of multiple perspectives while retaining the original image information. This sophisticated loss calculation mechanism helps to improve the quality and accuracy of the fused image, and is particularly suitable for processing industrial image fusion tasks that require high precision, thereby better meeting the strict requirements for image fusion in industrial applications.

[0013] Optionally, calculating the structural loss value of the fused image and the first feature image and the second feature image includes: acquiring the width and height of the fused image, the edge detection result of the fused image, the edge detection result of the first feature image, and the edge detection result of the second feature image; substituting the width and height of the fused image, the edge detection result of the fused image, the edge detection result of the first feature image, and the edge detection result of the second feature image into a structural loss formula to calculate the structural loss value of the fused image and the first feature image and the second feature image; the structural loss formula is: ; In the formula, is the structural loss value, is the width of the fused image, is the height of the fused image, The edge detection result of the fused image is at position The value at The edge detection result of the first feature image is at position The value at The edge detection result of the second feature image is at position The value at .

[0014] By adopting the above technical solution, the width and height of the fused image, as well as the edge detection results of the fused image, the first feature image, and the second feature image are obtained, and then the loss value is calculated by the structural loss formula. This design focuses on comparing the edge information of the image, and comprehensively evaluates the degree of retention of structural information during the fusion process by calculating the difference between the edge detection result of the fused image and the average value of the edge detection result of the original feature image. By dividing the error value by the total number of pixels in the image, a normalized loss value is obtained, so that images of different sizes can be compared fairly. This edge-based evaluation method can effectively capture the structural features of the image and help maintain the consistency of the fused image with the original image at the structural level. At the same time, it provides a feedback signal for model optimization that focuses on the image structure, so that the fusion result can better fuse the features of multiple perspectives while retaining the structure of the original image.

[0015] Optionally, performing weighted summation of the pixel reconstruction loss value and the structural loss value to generate a total loss function includes: substituting the pixel reconstruction loss value and the structural loss value into a loss function calculation formula to generate a total loss function; wherein the loss function is: ; In the formula, is the total loss function, is the first preset parameter, is the pixel reconstruction loss value, is the second preset parameter, is the structural loss value.

[0016] By adopting the above technical solution and introducing the calculation mechanism of the total loss function, the weighted fusion of the pixel reconstruction loss value and the structural loss value is realized. The method substitutes the pixel reconstruction loss value and the structural loss value into the loss function calculation formula. By adjusting the values ​​of the first preset parameter and the second preset parameter, the relative importance of pixel-level reconstruction and structural consistency in the overall optimization goal can be flexibly controlled. This multi-objective optimization strategy can fully consider the overall structure of the image while ensuring the pixel-level accuracy of the fused image, thereby achieving a more comprehensive and balanced image fusion effect.

[0017] Optionally, adjusting the parameters in the first encoder, the second encoder and the decoder according to the total loss function to obtain a first target encoder, a second target encoder and a target decoder accordingly, includes: obtaining the gradient value of the total loss function with respect to each parameter in the first encoder, the second encoder and the decoder; iteratively optimizing the parameters in the first encoder, the second encoder and the decoder according to the gradient value using a back propagation algorithm to obtain trained parameters of the first encoder, the second encoder and the decoder; assigning the trained first encoder parameters to the first target encoder, assigning the trained second encoder parameters to the second target encoder, and assigning the trained decoder parameters to the target decoder to obtain the trained first target encoder, the second target encoder and the target decoder.

[0018] By adopting the above technical solution, the gradient values ​​of the total loss function with respect to each parameter in the first encoder, the second encoder and the decoder are obtained, and these gradient values ​​reflect the influence of each parameter on the overall performance. Then, the model parameters are iteratively optimized according to these gradient values ​​using the back propagation algorithm. This process can effectively adjust the model parameters to minimize the total loss function. Through multiple rounds of iterations, the model gradually learns better feature extraction and fusion strategies, thereby improving the overall performance. Finally, the trained parameters are assigned to the first target encoder, the second target encoder and the target decoder respectively, and the trained model is obtained. This optimization method can simultaneously consider pixel reconstruction and structural consistency, so that the model can better capture and fuse key features when processing industrial images from different perspectives. Through this end-to-end training process, the model can adaptively learn complex industrial scene features and improve the understanding and fusion capabilities of multi-view images. This optimization strategy not only improves the generalization ability of the model, but also enhances its robustness when processing various industrial scenes. Finally, the trained model can generate higher quality fused images, while retaining detail information while maintaining good structural consistency, so as to better meet the strict requirements for image fusion in industrial applications.

[0019] In the second aspect, the present application provides a multi-view industrial image fusion device based on a cross-attention mechanism, the device comprising: a first input module, a second input module, a fusion module, a calculation module, an adjustment module and a generation module; wherein the first input module is used to input two normalized industrial images of different viewpoints into a first encoder and a second encoder respectively, and obtain a first feature image and a second feature image correspondingly, and the first encoder and the second encoder are weight-sharing encoders; the second input module is used to input the first feature image and the second feature image into a decoder to obtain a first fusion feature and a second fusion feature; the fusion module is used to fuse the first fusion feature and the second fusion feature to generate a target fusion feature, and map the target fusion feature to the image through a convolutional layer. space, generating a fused image; the calculation module is used to calculate the pixel reconstruction loss value and the structure loss value of the fused image and the first feature image and the second feature image, and weighted sum the pixel reconstruction loss value and the structure loss value to generate a total loss function; the adjustment module is used to adjust the parameters in the first encoder, the second encoder and the decoder according to the total loss function, and correspondingly obtain the first target encoder, the second target encoder and the target decoder; the generation module is used to input the two industrial images of different perspectives into the first target encoder and the second target encoder respectively, and correspondingly obtain the first target feature image and the second target feature image, and input the first target feature image and the second target feature image into the target decoder to obtain a target fused image.

[0020] In a third aspect, the present application provides an electronic device, adopting the following technical solution: including a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes a computer program such as any of the above-mentioned multi-view industrial image fusion methods based on the cross-attention mechanism.

[0021] In a fourth aspect, the present application provides a computer-readable storage medium, which adopts the following technical solution: storing a computer program that can be loaded by a processor and execute any of the above-mentioned multi-view industrial image fusion methods based on the cross-attention mechanism.

[0022] In summary, the present application includes at least one of the following beneficial technical effects: By using a weight-sharing encoder to process industrial images of different perspectives, the number of model parameters is effectively reduced and the computational efficiency is improved. The decoder used in the method combines the self-attention and cross-attention mechanisms, which can fully capture the internal features of the image and the interactive information between images of different perspectives, thereby enhancing the effect of feature extraction and fusion. The target fusion features are generated by designing a feature fusion formula, and the convolutional layer is used to map them to the image space, thereby achieving high-quality image fusion. The calculation of pixel reconstruction loss and structural loss is also introduced, and the total loss function is generated by weighted summation. This multi-objective optimization strategy not only ensures the pixel-level accuracy of the fused image, but also maintains the overall structure of the image, thereby improving the visual quality and semantic consistency of the fused image. The model parameters are optimized by the back-propagation algorithm, and the target encoder and decoder obtained can better adapt to complex industrial scenes. The target fused image finally generated has significant improvements in detail retention and structural consistency, meeting the needs of industrial applications for high-quality image fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a flowchart of a multi-view industrial image fusion method based on a cross attention mechanism provided in an embodiment of the present application; Figure 2 It is a structural schematic diagram of a multi-view industrial image fusion device based on a cross attention mechanism provided in an embodiment of the present application; Figure 3 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application.

[0024] Description of reference numerals: 1000, electronic device; 1001, processor; 1002, communication bus; 1003, user interface; 1004, network interface; 1005, memory. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.

[0026] In the description of the embodiments of the present application, words such as "illustrative", "for example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "illustrative", "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "illustrative", "for example" or "for example" is intended to present related concepts in a concrete way.

[0027] Figure 1 is a flow chart of a multi-view industrial image fusion based on a cross attention mechanism provided by an embodiment of the present application. The present application discloses a multi-view industrial image fusion method based on a cross attention mechanism, such as Figure 1 As shown, the method includes S101-S106.

[0028] S101, inputting two normalized industrial images from different viewing angles into a first encoder and a second encoder respectively, and obtaining a first feature image and a second feature image correspondingly, wherein the first encoder and the second encoder are weight-sharing encoders.

[0029] When implementing the image fusion method of the present invention, it is first necessary to obtain two industrial images of different viewing angles as input. In order to eliminate the differences in image size, brightness, etc. and facilitate subsequent feature extraction and fusion processing, the two images are normalized. Normalization can unify the size of the image to a specified size and scale the pixel value to the range of [0, 1]. The two normalized images are respectively denoted as and .

[0030] Then, the normalized image and The encoder can use ViT (Vision Transformer) to extract features from the image after normalization. and Before inputting into the ViT encoder, the image first needs to be divided into fixed-size blocks. For example, if the input image is of size 384×384, it can be divided into 16×16 blocks, each of which is 24×24 in size. Then, each block is flattened into a vector and position embedding is added to preserve the spatial position information of the block. Next, these vector sequences are input into the ViT encoder.

[0031] The ViT encoder consists of multiple Transformer blocks, each of which includes two sublayers: multi-head self-attention and feed-forward network. In the self-attention mechanism, each block interacts with all other blocks and calculates the attention weights between them to capture the dependencies between blocks. Through multi-head self-attention, the model can learn different feature representations in different subspaces. After processing multiple Transformer blocks, the ViT encoder can extract high-level image features.

[0032] The first encoder and the second encoder use the same ViT structure and share weight parameters. For the first encoder, the divided image block sequence Input it, pass through the self-attention mechanism and feedforward neural network of multiple Transformer blocks, and get the first feature image Similarly, for the second encoder, the image block sequence Input it, and go through the exact same Transformer block to get the second feature image Both feature images are high-level feature representations extracted by the ViT encoder.

[0033] S102, inputting the first feature image and the second feature image into a decoder to obtain a first fusion feature and a second fusion feature.

[0034] In one example, the role of the decoder is to convert the feature image extracted by the encoder back to the original image space, while fusing the feature information from different perspectives. In this scheme, the decoder adopts a structure that combines the self-attention module and the cross-attention module. First, the first feature image is input into the self-attention module of the decoder. The self-attention module is similar to the self-attention mechanism in the encoder. It models the dependency between features by calculating the attention weights between different positions in the feature image. After being processed by the self-attention module, the representation ability of the first feature image can be enhanced and the key information therein can be highlighted. The enhanced first feature image is denoted as .

[0035] Similarly, the second feature image is input into another self-attention module of the decoder, and after the same processing, the enhanced second feature image is obtained. The role of the self-attention module is to perform feature enhancement within the feature image of each view, so that the subsequent fusion process can more effectively extract and utilize the complementary information between views.

[0036] Next, the enhanced feature image and The cross attention module is used to establish the association between features from different perspectives and to achieve interactive fusion of features by calculating the attention weights between feature images. Specifically, we first calculate Each position in The attention weights between all positions in , and then the Perform weighted summation to obtain the first fusion feature image Similarly, calculate Each position in The attention weights between all positions in Perform weighted summation to obtain the fused second feature image This process can be seen as the mutual reference and complementation of the features of the two perspectives in space, so that the fused feature image can take into account the advantages of different perspectives and contain more comprehensive and accurate information.

[0037] Based on the above embodiment, as an optional implementation, in S102, the first feature image and the second feature image are input into the decoder to obtain the first fusion feature and the second fusion feature, which specifically includes S201-S202: S201, inputting the first feature image and the second feature image into the self-attention module in the decoder respectively, and performing feature enhancement on the first feature image and the second feature image through the self-attention mechanism to obtain a first enhanced feature image and a second enhanced feature image.

[0038] In one example, the first feature image and the second feature image are respectively input into the self-attention module in the decoder. The self-attention module performs weighted fusion on each position in the feature image through the self-attention mechanism to generate an enhanced feature representation. For the first feature image, the self-attention module first calculates the attention weights between each position and other positions, which reflect the correlation and importance between different positions. Then, by multiplying these attention weights with the feature vectors of the corresponding positions and summing them, the self-attention module obtains the enhanced first feature image. Similarly, for the second feature image, the self-attention module obtains the enhanced second feature image through the same process.

[0039] S202, inputting the first enhanced feature image and the second enhanced feature image into the cross-attention module in the decoder, and performing feature interactive fusion on the first enhanced feature image and the second enhanced feature image through the cross-attention mechanism to obtain a first fused feature and a second fused feature.

[0040] In one example, the cross-attention module receives two inputs: a first enhanced feature image and a second enhanced feature image. Similar to the self-attention module, the cross-attention module also uses the attention mechanism to achieve feature interaction and fusion, but the difference is that the attention weight of the cross-attention module is calculated between two different feature images. For the first enhanced feature image, the cross-attention module regards it as a "query" and the second enhanced feature image as a "key" and "value". By calculating the attention weight between each position in the first enhanced feature image and all positions in the second enhanced feature image, the information of the second enhanced feature image is fused into the first enhanced feature image to obtain the first fused feature. Similarly, for the second enhanced feature image, the cross-attention module regards it as a "query" and the first enhanced feature image as a "key" and "value". By calculating the attention weight between each position in the second enhanced feature image and all positions in the first enhanced feature image, the information of the first enhanced feature image is fused into the second enhanced feature image to obtain the second fused feature.

[0041] S103, fusing the first fusion feature and the second fusion feature to generate a target fusion feature, and mapping the target fusion feature to the image space through a convolution layer to generate a fused image.

[0042] In one example, the first fusion feature and the second fusion feature are added element by element to obtain the target fusion feature. This operation can be regarded as a feature-level fusion strategy, which superimposes the fusion features of the two perspectives equally so that the fusion feature contains the comprehensive information from the two perspectives. The advantages of additive fusion are that it is simple and efficient, and it can preserve and All features in the image will not introduce additional information loss.

[0043] Next, in order to transform the target fusion feature into the final fusion image, it is necessary to map it from the feature space back to the image space. This step is usually achieved using a convolutional layer. Specifically, a mapping network consisting of multiple convolutional layers is designed, which takes the target fusion feature as input and generates a fusion image of the same size as the original input image after a series of convolution operations.

[0044] In the design of the mapping network, a residual connection structure is used, that is, the input features are directly added to the output features through a shortcut connection. This residual structure can help the network better learn the identity mapping and alleviate the difficulty of deep network training. At the same time, batch normalization and activation functions (such as ReLU) are inserted between convolutional layers to accelerate network convergence and improve feature representation capabilities.

[0045] For example, assuming that the size of the target fusion feature is 48×48×512, a mapping network with 3 convolutional layers can be designed. The first convolutional layer reduces the feature dimension from 512 to 256 and introduces nonlinearity through the ReLU activation function. The second convolutional layer reduces the feature dimension from 256 to 128, also using ReLU activation. The last convolutional layer reduces the feature dimension from 128 to 3, obtaining an output with the same number of channels as the original image. Through this progressive feature dimensionality reduction and nonlinear transformation, the mapping network can gradually transform the target fusion feature H into a visually clear and natural fusion image.

[0046] Based on the above embodiment, as an optional implementation, in S103, fusing the first fusion feature and the second fusion feature to generate the target fusion feature specifically includes: Substitute the first fusion feature and the second fusion feature into the feature fusion formula to generate the target fusion feature; wherein the feature fusion formula is: ; In the formula, is the target fusion feature, is the weight coefficient, is the first fusion feature, It is the second fusion feature.

[0047] Specifically, the first fusion feature and the second fusion feature are weighted and summed to obtain a target fusion feature that integrates the information of the two perspectives. The value range of the weight coefficient is [0, 1], which controls the relative importance of the first fusion feature and the second fusion feature in the final fusion result.

[0048] The selection of weight coefficients can be determined according to specific application scenarios and requirements. For example, if it is known in advance that the quality or information content of the first-view image is higher than that of the second-view image, the weight coefficient can be set to a larger value to increase the weight of the first-view feature in the fusion result. Conversely, if the quality or information content of the second-view image is higher, the weight coefficient can be set to a smaller value. In practical applications, the optimal weight coefficient can also be automatically selected from a candidate set through methods such as cross-validation or grid search to obtain the best fusion performance.

[0049] S104, calculating the pixel reconstruction loss value and the structure loss value of the fused image and the first feature image and the second feature image, and performing weighted summation on the pixel reconstruction loss value and the structure loss value to generate a total loss function.

[0050] Calculating the pixel reconstruction loss value of the fused image and the first feature image and the second feature image specifically includes: obtaining the width and height of the fused image, the pixel value of the fused image at each position, the pixel value of the first feature image at each position, and the pixel value of the second feature image at each position; Substitute the width and height of the fused image, the pixel values ​​of the fused image at each position, the pixel values ​​of the first feature image at each position, and the pixel values ​​of the second feature image at each position into the pixel reconstruction loss formula to calculate the pixel reconstruction loss values ​​of the fused image, the first feature image, and the second feature image; wherein, The pixel reconstruction loss formula is: ; In the formula, is the pixel reconstruction loss value, is the width of the fused image, is the height of the fused image, To fuse the image at position The pixel value at The first feature image is at position The pixel value at The second feature image is at position The pixel value at .

[0051] In one example, assume that the size of the fused image is W×H, where W is the width and H is the height. Use two nested loops to iterate over each pixel position of the fused image. , where i ranges from [1, W] and j ranges from [1, H]. For each position (i, j), we read the pixel values ​​of the fused image, the first feature image, and the second feature image at that position, respectively, and record them as , and These pixel values ​​are usually a three-dimensional vector representing the RGB color value at that location.

[0052] After obtaining the required information, the width W, height H, and pixel values ​​of the fused image at each position are calculated. , the pixel value of the first feature image at each position and the pixel value of the second feature image at each position Substitute into the pixel reconstruction loss formula to calculate the pixel reconstruction loss value between the fused image and the input image. The pixel reconstruction loss formula is defined as: in, represents the L2 norm, i.e., the Euclidean distance. The physical meaning of this formula is that the pixel value of the fused image at each position (i, j) is first calculated. The average value of the input image pixel value The difference between them is then squared and summed for all the locations, and finally divided by the total number of pixels W×H to get the average pixel reconstruction loss. The value range of is [0, +∞). The smaller the value, the closer the fused image is to the input image at the pixel level, and the higher the fusion quality.

[0053] For example, assuming the size of the fused image is 1024×768, the sizes of the first feature image and the second feature image are the same as the fused image. For position (100, 200), The RGB value is (0.8, 0.5, 0.2), The RGB value is (0.9, 0.6, 0.1), The RGB value of is (0.7, 0.4, 0.3). According to the pixel reconstruction loss formula, the loss value at this position is = , and so on, summing the loss values ​​at each position and dividing it by the total number of pixels 1024×768, we get the final pixel reconstruction loss value.

[0054] Calculating the structural loss value of the fused image, the first feature image, and the second feature image includes: Obtain the width and height of the fused image, the edge detection result of the fused image, the edge detection result of the first feature image, and the edge detection result of the second feature image; substitute the width and height of the fused image, the edge detection result of the fused image, the edge detection result of the first feature image, and the edge detection result of the second feature image into the structural loss formula to calculate the structural loss value of the fused image, the first feature image, and the second feature image; The structural loss formula is: ; In the formula, is the structural loss value, is the width of the fused image, is the height of the fused image, The edge detection result of the fused image is at position The value at The edge detection result of the first feature image is at position The value at The edge detection result of the second feature image is at position The value at .

[0055] In one example, in order to calculate the structural loss, it is first necessary to obtain the width and height of the fused image, the edge detection result of the fused image, the edge detection result of the first feature image, and the edge detection result of the second feature image. The width and height of the fused image can be obtained by accessing its size attribute. The edge detection result requires the use of a special edge detection algorithm, such as the Sobel operator, the Canny operator, etc., to extract the edges of the fused image, the first feature image, and the second feature image respectively. The edge detection algorithm can highlight the structural contours and texture details in the image, and provide important feature information for the calculation of the structural loss.

[0056] Assuming the size of the fused image is W×H, apply the edge detection algorithm to it and get the edge detection result , where S(·) represents the edge detection operation. The size and The same, but the pixel value is changed from the original RGB value to the gray value representing the edge strength. Similarly, edge detection is performed on the first feature image and the second feature image to obtain their edge detection results. and .

[0057] After obtaining the required information, the width W, height H, and edge detection results of the fused image are , the edge detection result of the first feature image And the edge detection result of the second feature image Substitute into the structural loss formula to calculate the structural loss value between the fused image and the input image The structural loss formula is defined as: ; in, Represents the value of the fused image edge detection result at position (i, j), and Respectively represent the values ​​of the edge detection results of the first feature image and the second feature image at position (i, j). The physical meaning of this formula is to first calculate the edge strength value of the fused image at each position (i, j). The mean edge intensity of the input image The difference between them is then squared and summed for all positions, and finally divided by the total number of pixels W×H to get the average structure loss. The value range of is [0, +∞). The smaller the value, the closer the fused image is to the input image in terms of structure, and the higher the fusion quality.

[0058] For example, assuming the size of the fused image is 1024×768, at position (100, 200), the value of S(I_fusion)(100, 200) is 0.8, The value of is 0.9, The value is 0.7. According to the structural loss formula, the loss value at this location = =0; and so on, the loss value of each position is summed and divided by the total number of pixels 1024×768 to obtain the final structural loss value.

[0059] The pixel reconstruction loss value and the structure loss value are weighted and summed to generate the total loss function, including: Substitute the pixel reconstruction loss value and the structure loss value into the loss function calculation formula to generate the total loss function; where the loss function is: ; In the formula, is the total loss function, is the first preset parameter, is the pixel reconstruction loss value, is the second preset parameter, is the structural loss value.

[0060] In one example, the pixel reconstruction loss value is calculated by weighted summation. and structural loss value Combined into a total loss function Specifically, we will and Substitute the loss function calculation formula to generate the total loss function: in, and are two preset weight parameters used to adjust the proportion of pixel reconstruction loss and structure loss in the total loss function. and The value of can flexibly control the model's emphasis on pixel-level details and structural semantic information, thereby generating a more balanced and high-quality fusion result.

[0061] For example, during training, the pixel reconstruction loss is calculated and structural loss value .Will Set to 1.0, Set to 0.8, the value of the total loss function = 0.05 + 0.016 = 0.066.

[0062] This total loss value combines the pixel reconstruction loss and the structural loss, reflecting the reconstruction quality of the fused image at multiple levels. As the optimization goal in the model training process, the model parameters are adjusted by minimizing the total loss function to generate a fusion result that is closer to the real image.

[0063] S105, adjusting parameters in the first encoder, the second encoder, and the decoder according to the total loss function, and obtaining a first target encoder, a second target encoder, and a target decoder accordingly.

[0064] In one example, a back propagation algorithm is used to perform gradient descent optimization on the parameters in the first encoder, the second encoder, and the decoder using a total loss function. First, a batch of input images are The current encoder and decoder are fed to generate the corresponding fused image. Then, the pixel reconstruction loss and structural loss are calculated based on the fused image and the input image, and the value of the total loss function is calculated based on the preset weight coefficient.

[0065] Next, use automatic differentiation tools (such as PyTorch or TensorFlow) to calculate the gradient of the total loss function for each parameter in the first encoder, the second encoder, and the decoder. These gradients indicate the importance of each parameter in reducing the total loss function. The larger the gradient value of the parameter, the more effective it is to adjust the parameter to reduce the loss function. Based on the calculated gradient, use an optimizer (such as Adam or SGD) to update each parameter and adjust them in the direction of the negative gradient to minimize the total loss function.

[0066] Through multiple rounds of iterative optimization, the parameters of the model are continuously updated, so that the value of the total loss function gradually decreases. In this process, the first encoder and the second encoder learn how to extract the key features of the input image, and the decoder learns how to fuse these features and map them back to high-quality images. After sufficient training, the first target encoder, the second target encoder, and the target decoder with optimized performance are finally obtained.

[0067] Based on the above embodiment, as an optional implementation, in S105, the parameters in the first encoder, the second encoder and the decoder are adjusted according to the total loss function, and the first target encoder, the second target encoder and the target decoder are obtained correspondingly, specifically including S501-S503: S501, obtaining the gradient value of the total loss function with respect to each parameter in the first encoder, the second encoder and the decoder.

[0068] S502 Iteratively optimizes the parameters in the first encoder, the second encoder, and the decoder using a back propagation algorithm according to the gradient value to obtain the trained parameters of the first encoder, the second encoder, and the decoder.

[0069] S503 assigns the trained first encoder parameters to the first target encoder, assigns the trained second encoder parameters to the second target encoder, and assigns the trained decoder parameters to the target decoder to obtain the trained first target encoder, second target encoder, and target decoder.

[0070] In one example, the gradient value of the total loss function with respect to each parameter in the first encoder, the second encoder, and the decoder is first obtained. The gradient value represents the rate of change of the total loss function in the parameter space, indicating the direction and magnitude of adjusting the parameter to minimize the value of the total loss function. The gradient value of each parameter can be calculated by automatic differentiation tools or manual derivation to obtain a gradient vector.

[0071] After obtaining the gradient value, the back propagation algorithm is used to iteratively optimize the parameters in the first encoder, the second encoder, and the decoder. The back propagation algorithm is a commonly used neural network training method. By backpropagating the gradient of the loss function from the output layer to the input layer, the parameters of each layer are adjusted to make the prediction results of the model closer to the true value. In each iteration, according to the current parameter value and gradient value, the parameters are updated along the negative gradient direction to gradually reduce the value of the total loss function. Common parameter update rules include stochastic gradient descent (SGD), Adam, RMSprop, etc. By setting the appropriate learning rate and momentum factor, the step size and stability of parameter adjustment can be controlled.

[0072] After multiple rounds of iterative optimization, the trained first encoder, second encoder and decoder parameters are obtained. These parameters have been adjusted according to the total loss function to generate higher quality and more realistic fused images. In order to apply the optimized parameters to the actual image fusion task, the trained first encoder parameters are assigned to the first target encoder, the trained second encoder parameters are assigned to the second target encoder, and the trained decoder parameters are assigned to the target decoder. In this way, the trained first target encoder, second target encoder and target decoder are obtained, which have stronger feature extraction and image reconstruction capabilities and can be applied to the subsequent image fusion process.

[0073] S106, inputting two industrial images from different perspectives into a first target encoder and a second target encoder respectively, obtaining a first target feature image and a second target feature image correspondingly, inputting the first target feature image and the second target feature image into a target decoder, and obtaining a target fusion image.

[0074] In one example, after completing the parameter optimization of the first encoder, the second encoder, and the decoder, a first target encoder, a second target encoder, and a target decoder with superior performance are obtained.

[0075] Given two industrial images from different perspectives and , first input them into the first target encoder and the second target encoder respectively. The first target encoder Perform feature extraction, and extract the The high-level feature representation of the first target feature image is obtained Similarly, the second target encoder is Perform feature extraction to obtain the second target feature image .

[0076] Next, the first target feature image and the second target feature image Input to the target decoder. The target decoder first uses the self-attention module to and Perform feature enhancement respectively to obtain enhanced feature images and Then, through the cross-attention module, the target decoder converts and Deep integration makes Each feature in information, and obtain the first fusion feature Similarly, Each feature in The information of the second fusion feature is obtained .

[0077] Finally, the target decoder converts the first fused feature and the second fusion feature Perform element-by-element addition to obtain the target fusion feature H'. Through the mapping network, the target decoder maps the target fusion feature H' from the feature space back to the image space to generate the final target fusion image.

[0078] Based on the above method, the present application also discloses a multi-view industrial image fusion device based on a cross attention mechanism, such as Figure 2 As shown, Figure 2 : is a structural schematic diagram of a multi-view industrial image fusion device based on a cross attention mechanism provided in an embodiment of the present application, the device includes: a first input module, a second input module, a fusion module, a calculation module, an adjustment module and a generation module; wherein, The first input module is used to input the two normalized industrial images of different viewing angles into the first encoder and the second encoder respectively, and obtain the first feature image and the second feature image accordingly, and the first encoder and the second encoder are weight-sharing encoders; the second input module is used to input the first feature image and the second feature image into the decoder to obtain the first fusion feature and the second fusion feature; the fusion module is used to fuse the first fusion feature and the second fusion feature to generate a target fusion feature, and map the target fusion feature to the image space through a convolution layer to generate a fusion image; the calculation module is used to calculate the pixel reconstruction loss value and the structure loss value of the fusion image and the first feature image and the second feature image, and perform weighted summation of the pixel reconstruction loss value and the structure loss value to generate a total loss function; the adjustment module is used to adjust the parameters in the first encoder, the second encoder and the decoder according to the total loss function, and obtain the first target encoder, the second target encoder and the target decoder accordingly; the generation module is used to input the two industrial images of different viewing angles into the first target encoder and the second target encoder respectively, and obtain the first target feature image and the second target feature image accordingly, and input the first target feature image and the second target feature image into the target decoder to obtain the target fusion image.

[0079] It should be noted that: when the device provided in the above embodiment realizes its function, only the division of the above functional modules is used as an example. In actual application, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0080] See also Figure 3 , is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 3 As shown, the electronic device 1000 may include: at least one processor 1001 , at least one network interface 1004 , a user interface 1003 , a memory 1005 , and at least one communication bus 1002 .

[0081] The communication bus 1002 is used to realize the connection and communication between these components.

[0082] The user interface 1003 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 1003 may also include a standard wired interface and a wireless interface.

[0083] The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0084] Among them, the processor 1001 may include one or more processing cores. The processor 1001 uses various interfaces and lines to connect various parts in the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 1005, and calling data stored in the memory 1005. Optionally, the processor 1001 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 1001 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 1001, and it can be implemented separately through a chip.

[0085] Among them, the memory 1005 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 1005 includes a non-transitory computer-readable storage medium. The memory 1005 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 1005 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 1005 may optionally also be at least one storage device located away from the aforementioned processor 1001. As Figure 3 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and an application program of a multi-view industrial image fusion method based on a cross-attention mechanism.

[0086] exist Figure 3In the electronic device 1000 shown, the user interface 1003 is mainly used to provide an input interface for the user and obtain data input by the user; and the processor 1001 can be used to call an application program stored in the memory 1005 for a multi-view industrial image fusion method based on a cross-attention mechanism. When executed by one or more processors, the electronic device executes one or more of the methods described in the above embodiments.

[0087] An electronic device readable storage medium stores instructions, which, when executed by one or more processors, enable the electronic device to execute one or more of the methods described in the above embodiments.

[0088] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for the present application.

[0089] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0090] In the several embodiments provided in the present application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0091] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0092] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0093] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned memory includes: various media that can store program codes, such as USB flash drives, mobile hard drives, magnetic disks or optical disks.

[0094] The above is only an exemplary embodiment of the present disclosure, and the scope of the present disclosure cannot be limited thereto. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the specification and practicing the disclosure here, those skilled in the art will easily think of other embodiments of the present disclosure. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the technical field not recorded in the present disclosure. The description and examples are only regarded as exemplary, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A multi-view industrial image fusion method based on cross-attention mechanism, characterized in that: The method comprises: Inputting the two normalized industrial images of different viewing angles into a first encoder and a second encoder respectively, and obtaining a first feature image and a second feature image correspondingly, wherein the first encoder and the second encoder are weight-sharing encoders; Inputting the first feature image and the second feature image into a decoder to obtain a first fused feature and a second fused feature; Fusing the first fusion feature and the second fusion feature to generate a target fusion feature, and mapping the target fusion feature to an image space through a convolution layer to generate a fused image; Calculating a pixel reconstruction loss value and a structure loss value of the fused image, the first feature image, and the second feature image, and performing a weighted summation of the pixel reconstruction loss value and the structure loss value to generate a total loss function; Adjusting parameters in the first encoder, the second encoder, and the decoder according to the total loss function to obtain a first target encoder, a second target encoder, and a target decoder accordingly; The two industrial images of different viewing angles are respectively input into the first target encoder and the second target encoder to obtain a first target feature image and a second target feature image respectively, and the first target feature image and the second target feature image are input into the target decoder to obtain a target fusion image.

2. The multi-view industrial image fusion method based on the cross attention mechanism according to claim 1 is characterized in that: The step of inputting the first feature image and the second feature image into a decoder to obtain a first fusion feature and a second fusion feature comprises: Inputting the first feature image and the second feature image into the self-attention module in the decoder respectively, and performing feature enhancement on the first feature image and the second feature image through the self-attention mechanism to obtain a first enhanced feature image and a second enhanced feature image; The first enhanced feature image and the second enhanced feature image are input into the cross-attention module in the decoder, and the first enhanced feature image and the second enhanced feature image are interactively fused through the cross-attention mechanism to obtain the first fused feature and the second fused feature.

3. The multi-view industrial image fusion method based on cross attention mechanism according to claim 1 is characterized in that: The fusing the first fusion feature and the second fusion feature to generate a target fusion feature includes: Substitute the first fusion feature and the second fusion feature into the feature fusion formula to generate the target fusion feature; wherein, The feature fusion formula is: ; In the formula, is the target fusion feature, is the weight coefficient, is the first fusion feature, It is the second fusion feature.

4. The multi-view industrial image fusion method based on cross attention mechanism according to claim 1, characterized in that: The calculating the pixel reconstruction loss value of the fused image, the first feature image and the second feature image comprises: Acquire the width and height of the fused image, the pixel value of the fused image at each position, the pixel value of the first feature image at each position, and the pixel value of the second feature image at each position; Substituting the width and height of the fused image, the pixel values ​​of the fused image at each position, the pixel values ​​of the first feature image at each position, and the pixel values ​​of the second feature image at each position into the pixel reconstruction loss formula, the pixel reconstruction loss values ​​of the fused image, the first feature image, and the second feature image are calculated; wherein, The pixel reconstruction loss formula is: ; In the formula, is the pixel reconstruction loss value, is the width of the fused image, is the height of the fused image, To fuse the image at position The pixel value at The first feature image is at position The pixel value at The second feature image is at position The pixel value at .

5. The multi-view industrial image fusion method based on cross attention mechanism according to claim 1, characterized in that: The calculating the structural loss value of the fused image, the first feature image and the second feature image comprises: Acquire the width and height of the fused image, the edge detection result of the fused image, the edge detection result of the first feature image, and the edge detection result of the second feature image; Substituting the width and height of the fused image, the edge detection result of the fused image, the edge detection result of the first feature image, and the edge detection result of the second feature image into a structural loss formula, and calculating the structural loss value of the fused image, the first feature image, and the second feature image; The structural loss formula is: ; In the formula, is the structural loss value, is the width of the fused image, is the height of the fused image, The edge detection result of the fused image is at position The value at The edge detection result of the first feature image is at position The value at The edge detection result of the second feature image is at position The value at .

6. The multi-view industrial image fusion method based on cross attention mechanism according to claim 1, characterized in that: The weighted summing of the pixel reconstruction loss value and the structure loss value to generate a total loss function includes: Substituting the pixel reconstruction loss value and the structure loss value into the loss function calculation formula to generate a total loss function; wherein, The loss function is: ; In the formula, is the total loss function, is the first preset parameter, is the pixel reconstruction loss value, is the second preset parameter, is the structural loss value.

7. The multi-view industrial image fusion method based on cross attention mechanism according to claim 1, characterized in that: The adjusting the parameters in the first encoder, the second encoder and the decoder according to the total loss function to obtain a first target encoder, a second target encoder and a target decoder accordingly comprises: Obtaining the gradient value of the total loss function with respect to each parameter in the first encoder, the second encoder, and the decoder; Iteratively optimizing the parameters of the first encoder, the second encoder, and the decoder using a back propagation algorithm according to the gradient value to obtain trained parameters of the first encoder, the second encoder, and the decoder; The trained first encoder parameters are assigned to the first target encoder, the trained second encoder parameters are assigned to the second target encoder, and the trained decoder parameters are assigned to the target decoder to obtain the trained first target encoder, second target encoder and target decoder.

8. A multi-view industrial image fusion device based on a cross attention mechanism, characterized in that: The device comprises: a first input module, a second input module, a fusion module, a calculation module, an adjustment module and a generation module; wherein, The first input module is used to input the two normalized industrial images of different viewing angles into the first encoder and the second encoder respectively, to obtain the first feature image and the second feature image correspondingly, and the first encoder and the second encoder are weight-sharing encoders; The second input module is used to input the first feature image and the second feature image into a decoder to obtain a first fusion feature and a second fusion feature; The fusion module is used to fuse the first fusion feature and the second fusion feature to generate a target fusion feature, and map the target fusion feature to the image space through a convolution layer to generate a fused image; The calculation module is used to calculate the pixel reconstruction loss value and the structure loss value of the fused image, the first feature image and the second feature image, and perform weighted summation of the pixel reconstruction loss value and the structure loss value to generate a total loss function; The adjustment module is used to adjust the parameters in the first encoder, the second encoder and the decoder according to the total loss function, and obtain a first target encoder, a second target encoder and a target decoder accordingly; The generation module is used to input the two industrial images of different perspectives into the first target encoder and the second target encoder respectively, to obtain a first target feature image and a second target feature image correspondingly, and to input the first target feature image and the second target feature image into the target decoder to obtain a target fusion image.

9. An electronic device, characterized in that: It includes a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: A computer program is stored which can be loaded by a processor and execute the method according to any one of claims 1 to 7.

Images