Multi-view image fusion method and device, computer program product and memory

Abstract

The embodiment of the invention discloses a multi-view image fusion method and device, a computer program product and a memory. The multi-view image fusion method comprises the following steps: acquiring multiple frames of original images; the multiple frames of original images are images shot from multiple visual angles at the same moment; sequentially performing feature extraction on the multiple frames of original images by adopting a neural network to obtain multiple pieces of feature coding data; each piece of feature coding data corresponds to each frame of original image; performing cross comparison on the plurality of feature coding data by adopting a cross-channel space attention mechanism to obtain a registration matrix; the registration matrix comprises a plurality of groups of relation matrixes, and the relation matrixes reflect the transformation relation between every two frames of original images with matched feature point pairs; sequentially performing affine transformation on the multiple frames of original images according to the corresponding relation matrixes to obtain multiple frames of aligned images; each aligned image is in one-to-one correspondence with each frame of original image; and inputting the multi-frame aligned images into a fusion network for fusion processing to obtain a multi-view fusion image.

Description

TECHNICAL FIELD

[0001] The present application relates to, but is not limited to, a multi-view image fusion method and device, a computer program product and a memory. BACKGROUND

[0002] Image registration refers to the process of aligning and matching multiple images with overlapping regions acquired by different image acquisition systems, at different times, or from different shooting angles. That is, a spatial transformation is found, and the multiple images are corrected to a unified coordinate system by performing the spatial transformation, so that the points in the multiple images corresponding to the same position in the overlapping region are one-to-one, thereby achieving the purpose of image information fusion.

[0003] Image registration has been widely used in computer vision, medical image processing, material mechanics, remote sensing and other fields, and has been the focus of image processing research for a long time. SUMMARY

[0004] Therefore, the embodiments of the present application provide a multi-view image fusion method and device, a computer program product and a memory, which can realize strong robustness of multi-view image fusion.

[0005] The technical scheme of the embodiments of the present application is implemented as follows:

[0006] The embodiments of the present application provide a multi-view image fusion method, which comprises: acquiring multiple frames of original images; wherein the multiple frames of original images are images taken from multiple angles at the same time; using a neural network to sequentially extract features from the multiple frames of original images to obtain multiple feature encoding data; wherein each feature encoding data corresponds to each frame of original image; using a cross-channel spatial attention mechanism to cross-compare the multiple feature encoding data to obtain a registration matrix; the registration matrix comprises multiple relationship matrices, and each relationship matrix reflects the transformation relationship between each two frames of original images with matching feature point pairs; sequentially performing affine transformation on the multiple frames of original images according to the corresponding relationship matrices to obtain multiple frames of aligned images; wherein each aligned image corresponds to each frame of original image one-to-one; inputting the multiple frames of aligned images into a fusion network for fusion processing to obtain a multi-view fusion image.

[0007] In the above scheme, the neural network is used to extract features from multiple frames of the original images in sequence to obtain multiple feature coding data, including: extracting features from multiple frames of the original images in sequence to obtain multiple feature images; performing feature coding processing on multiple feature images in sequence to obtain multiple feature coding data; wherein each feature coding data corresponds one-to-one to each feature image; aggregating multiple feature coding data along the channel dimension to obtain integrated coding data; wherein the integrated coding data serves as input data of the cross-channel spatial attention mechanism, and the integrated coding data contains multiple feature coding data.

[0008] In the above scheme, the cross-channel spatial attention mechanism is used to cross-compare the multiple feature encoding data to obtain a registration matrix, including: obtaining a preset initialization query vector and an initialization key vector; pre-training is performed based on the initialization query vector, the initialization key vector and the public data set to obtain an updated target query vector and a target key vector; according to the target key vector, the target query vector, and the integrated encoding data, a channel weight distribution is obtained; according to the target key vector, the target query vector, and the integrated encoding data, a spatial weight distribution is obtained; according to the channel weight distribution and the spatial weight distribution, an attention weight distribution is obtained; and the attention weight distribution is multiplied by the integrated encoding data to obtain a registration matrix.

[0009] In the above scheme, the channel weight distribution is calculated based on the target key vector, the target query vector, and the integrated coding data, including: performing a convolution operation on the integrated coding data to obtain an output result; performing a dot product operation on the target query vector, the target key vector and the output result to obtain an inter-channel weight matrix; and performing a normalization operation on the inter-channel weight matrix to obtain a channel weight distribution.

[0010] In the above scheme, the spatial weight distribution is calculated based on the target key vector, the target query vector, and the integrated coding data, including: performing a global average pooling operation on the integrated coding data to obtain an output vector; performing a dot product operation on the target query vector, the target key vector and the output vector to obtain a spatial weight matrix; and performing a normalization operation on the spatial weight matrix to obtain a spatial weight distribution.

[0011] In the above solution, affine transformation is sequentially performed on multiple frames of the original images according to the corresponding relationship matrix to obtain multiple frames of aligned images, including: multiplying each of the original images by the registration matrix to obtain the aligned image.

[0012] In the scheme, the fusion network comprises: a first convolution module, a second convolution module and a third convolution module connected in sequence; the first convolution module comprises: a first convolution layer, a first batch normalization layer and a first full connection layer connected in sequence; wherein the input channel number of the first convolution layer is equal to the total frame number of the original image, and the output channel number of the first convolution layer is 48-96; the input and output channel numbers of the first batch normalization layer and the first full connection layer are both 48-96; the second convolution module comprises: a second convolution layer, a second batch normalization layer and a second full connection layer connected in sequence; wherein the input channel number of the second convolution layer is 48-96, and the output channel number is 12-36; the input and output channel numbers of the second batch normalization layer and the second full connection layer are both 12-36; the third convolution module comprises: a third convolution layer, a third batch normalization layer and a third full connection layer connected in sequence; wherein the input channel number of the third convolution layer is 12-36, and the output channel number is 1; the input and output channel numbers of the third batch normalization layer and the third full connection layer are both 1.

[0013] The embodiment of the present application also provides a multi-view image fusion device, the device comprising: an acquisition module configured to acquire a plurality of original images; wherein the plurality of original images are images captured from multiple views at the same time; an analysis module configured to extract features from the plurality of original images in sequence by using a neural network to obtain a plurality of feature encoding data; wherein each feature encoding data corresponds to each original image; a comparison module configured to cross-compare the plurality of feature encoding data by using a cross-channel spatial attention mechanism to obtain a registration matrix; the registration matrix comprises a plurality of relationship matrices, and each relationship matrix reflects the transformation relationship between two original images with a matching feature point pair; a transformation module configured to perform affine transformation on the plurality of original images in sequence according to the corresponding relationship matrices to obtain a plurality of aligned images; wherein each aligned image corresponds to each original image in a one-to-one manner; and a fusion module configured to input the aligned images into a fusion network for fusion processing to obtain a multi-view fusion image.

[0014] The embodiment of the present application also provides a computer program product comprising a memory and a processor, the memory storing a computer program capable of running on the processor, and the processor implements the steps in the method of the above scheme when executing the program.

[0015] The embodiment of the present application also provides a computer readable memory having a computer program stored thereon, and the computer program is executed by a processor to implement the steps in the method of the above scheme.

[0016] It can be seen that the embodiment of the application provides a multi-view image fusion method, device, computer program product and memory. The multi-view image fusion method comprises: adopting a cross-channel spatial attention mechanism to cross compare a plurality of feature encoding data to obtain a registration matrix; the registration matrix comprises a plurality of relationship matrices, and the relationship matrix reflects the transformation relationship between each two frames of original images with a matching feature point pair.

[0017] It can be understood that the cross-channel spatial attention mechanism is adopted to process the feature encoding data corresponding to the plurality of frames of original images, so that the most relevant feature information in each channel (each frame of original image) and the implicit feature information focused on the important region in each frame of original image are better obtained, and the accuracy and stability of image registration are improved, and misalignment and deviation registration are avoided. At the same time, the cross-channel spatial attention mechanism can better learn and represent important feature information, extract important feature points by dynamically adjusting the weight between channels, so that the image fusion method of the application can adapt to different image data and tasks, and the robustness of the image registration process is improved. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 Flowchart of the multi-view image fusion method provided by the embodiment of the application Figure 1 ;

[0019] Figure 2 Flowchart of the multi-view image fusion method provided by the embodiment of the application Figure 2 ;

[0020] Figure 3 Flowchart of the multi-view image fusion method provided by the embodiment of the application Figure 3 ;

[0021] Figure 4 Flowchart of the multi-view image fusion method provided by the embodiment of the application Figure 4 ;

[0022] Figure 5 Flowchart of the multi-view image fusion method provided by the embodiment of the application Figure 5 ;

[0023] Figure 6 Flowchart of the multi-view image fusion method provided by the embodiment of the application Figure 6 ;

[0024] Figure 7 Structure diagram of the fusion network in the multi-view image fusion method provided by the embodiment of the application;

[0025] Figure 8 Structure diagram of the multi-view image fusion device provided by the embodiment of the application. DETAILED DESCRIPTION

[0026] In order to make the purposes, technical solutions and advantages of the present application clearer, the technical solutions of the present application are further described in detail below in conjunction with the drawings and embodiments. The described embodiments should not be regarded as limitations to the present application. All other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0027] In the following description, “some embodiments” are described, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0028] If similar descriptions of “first / second” appear in the application file, the following description is added. In the following description, the terms “first / second / third” referred to only distinguish similar objects, and do not represent a specific order of the objects. It can be understood that “first / second / third” can be interchanged in a specific order or sequence as allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0030] It should be noted that the multi-view camera system includes multiple cameras distributed at different positions of the device, which can provide environment images of different views. The multi-view camera system can be applied in the device to help the device operator perceive the environment in which the device is located.

[0031] However, the existing multi-view camera system has some problems. For example, the cameras at different positions are difficult to provide high-quality environment images respectively, and the different view environment images taken by each camera cannot be automatically aligned.

[0032] For another example, in a complex environment, multiple different view environment images can cause the device operator to have difficulty in focusing attention on the key area. Especially when the multi-view camera system is applied in a special device such as an excavator, since the excavator needs to complete the engineering work in a complex environment, and the current multi-view camera system cannot meet the visual requirements in the operation of the excavator, which leads to an increase in the difficulty of operating the excavator.

[0033] On this basis, multi-view image fusion methods appear and are used to solve the above problems. The key of multi-view image fusion method is image registration (image alignment). At present, image registration can be realized based on deep learning method, and image registration and alignment can also be realized by using descriptor labeling method. However, the deep learning method lacks interpretability and cannot be practically applied to complex engineering operations. The descriptor labeling method also lacks the process of dynamically selecting different view images, thereby causing the image registration and image fusion to lack robustness.

[0034] Based on this, the embodiment of the present application provides a multi-view image fusion method, which can be applied to an unmanned remote driving system to realize strong robustness of multi-view image fusion, improve the perception ability of the system operator to the environment around the system, and improve the precision and efficiency of system control.

[0035] Figure 1 The implementation flowchart of the multi-view image fusion method provided by the embodiment of the present application is shown in FIG. 1, which comprises steps S101-S105. Figure 1

[0036] S101, acquiring a plurality of original images; wherein the plurality of original images are images captured from multiple perspectives at the same time.

[0037] In the embodiment of the present application, a plurality of original images P i (i = 0, 1, 2,... n) of different perspectives are acquired from image acquisition systems at different positions. Wherein n is the perspective ordinal number (i.e. the original image ordinal number). The plurality of original images need to be images captured at the same time, that is, the acquired original images are required to be located in the same time frame.

[0038] For example, the original image can be a frame in a video, that is, the original image can be a video frame. The original image can also be a static image. The original image can be acquired by an image acquisition system. The image acquisition system can include a camera of a excavator, a camera of a smart phone, a camera of a tablet computer, a camera of a personal computer, a lens of a digital camera, a monitoring camera or a network camera, etc.; or the camera can be a camera for capturing images of visible light or infrared light.

[0039] In the embodiment of the present application, the original image can be an initial image directly acquired by the image acquisition system or a frame in a video directly captured by the image acquisition system (hereinafter referred to as an initial image), or an image obtained after pre-processing the initial image. The pre-processing can eliminate irrelevant information or noise information in the initial image to obtain the original image, so as to avoid the influence of data quality, data imbalance, etc. of the initial image on subsequent image processing.

[0040] ​S102, sequentially extracting features of the multiple original images by using a neural network to obtain multiple feature coding data; wherein each feature coding data corresponds to each original image.

[0041] In the embodiment of the present application, the feature points of the original image are extracted by using the existing neural network, and the feature coding processing is performed to obtain the feature coding data. The feature coding data contains the feature point information of the original image. At the same time, each original image of multiple different views contains independent and complementary image information, and each feature coding data also contains independent and complementary feature point information of the corresponding original image.

[0042] In the embodiment of the present application, the neural network includes but is not limited to VGG (Visual Geometry Group) network, GoogLeNet (GoogleInception Net) network, ResNet (Residual Network) network and other commonly used networks. The neural network can be a pre-trained network or an untrained network, which can be determined by the person skilled in the art according to the actual situation, and the present application does not limit it.

[0043] S103, cross comparing the multiple feature coding data by using a cross-channel spatial attention mechanism to obtain a registration matrix; the registration matrix includes multiple groups of relationship matrices, and the relationship matrix reflects the transformation relationship between each two original images with matching feature point pairs.

[0044] In the embodiment of the present application, the cross-channel spatial attention mechanism is used to process the feature coding data to obtain the registration matrix. That is, the cross-channel spatial attention mechanism is used to perform point-to-point matching on the feature points in the multiple original images to obtain the matching feature point pairs between each two original images. According to the matching result, the registration matrix can be calculated.

[0045] In the embodiment of the present application, the registration matrix includes multiple groups of relationship matrices, which reflect the transformation relationship between the two original images with matching feature point pairs in the multiple original images. That is, the registration matrix reflects the registration feature grid relationship. According to the transformation relationship (registration feature grid relationship) reflected by the registration matrix, each original image is transformed to map the matching feature points on the original images from different views to a common coordinate system, so as to ensure that each original image is spatially aligned. The image alignment process is also called image registration process.

[0046] It should be understood that the cross channel spatial attention mechanism is an important mechanism in the spatio-temporal attention module, which can selectively enhance or suppress the feature responses of different channels by learning the relationship between channels. Specifically, it calculates the importance weights of each channel and then applies these weights to each channel to obtain a registration matrix. In this way, the network can pay more attention to the features that are more important for the registration task and reduce the attention to unimportant features, thereby improving the accuracy of image registration, i.e., avoiding image misalignment or bias registration.

[0047] In addition, the cross channel spatial attention mechanism can automatically learn important features in the image, help the model better adapt to different input data, and provide more accurate registration results in processing complex scenes and changing backgrounds, making it achieve good results on different datasets and tasks.

[0048] That is, the cross channel spatial attention mechanism can extract important feature points by dynamically adjusting the weights between channels. In the embodiment of the present application, the feature encoding data corresponding to multiple frames of original images is processed by the cross channel spatial attention mechanism, which can better obtain the most relevant feature information in each channel (each frame of original image) and the implicit feature information that focuses on important positions in space, which is conducive to improving the accuracy and stability of image registration and avoiding misalignment and bias registration. At the same time, using the cross channel spatial attention mechanism can better learn and represent important features and adapt to different image data and tasks to obtain different registration matrices and improve the robustness of the image registration process.

[0049] S104, affine transformation is performed on the multiple frames of original images according to the corresponding relationship matrix to obtain multiple frames of aligned images; wherein each aligned image corresponds to each frame of original image one by one.

[0050] In the embodiment of the present application, the original images corresponding to the relationship matrix are subjected to affine transformation to obtain the aligned images according to the mutual correspondence relationship of the relationship matrix in the registration matrix. After affine transformation of the multiple frames of original images, an aligned multi-view image set is obtained; the aligned multi-view image set includes multiple frames of aligned images. That is, the cross channel spatial attention mechanism is used to obtain the registration matrix, find the matching feature point pairs between the original images, and solve the affine transformation parameters through the matching feature point pairs, and the image alignment (image registration) is performed through the transformation model determined by the affine transformation parameters. The image registration is based on affine transformation, which is a combination of linear transformation and translation transformation in geometry, and the transformation types include rotation, shearing, scaling and translation, etc. The combination of transformation types obtains the affine transformation model.

[0051] In the embodiments of the present application, the affine transformation model is determined by using the registration matrix obtained by the cross-channel spatial attention mechanism to align two or more original images, so as to facilitate subsequent integration or fusion of matching feature points representing the same object between multiple original images.

[0052] S105, inputting the multiple aligned images into a fusion network for fusion processing to obtain a multi-view fusion image.

[0053] It should be understood that image fusion is a process of fusing two or more images obtained in the same scene (i.e., in the same range) into one image after registration and alignment.

[0054] In the embodiments of the present application, the multiple aligned images are input into the trained convolutional neural network to obtain a multi-view fusion image. It can be understood that for feature points representing different objects in the aligned images, the corresponding feature point pixel values are retained; for feature points representing the same object in the aligned images, the feature point pixel values in the corresponding multiple aligned images are taken respectively, and a weight ratio is calculated to obtain a new pixel value, and finally a fused multi-view fusion image is obtained.

[0055] In the embodiments of the present application, the cross-channel spatial attention mechanism is used to process the feature encoding data corresponding to the multiple original images, so as to better obtain the most relevant feature information in each channel (each original image) and the implicit feature information focusing on the important regions in each original image in space, thereby improving the accuracy and stability of image registration and avoiding misalignment and deviation registration. At the same time, the cross-channel spatial attention mechanism can better learn and represent important feature information, extract important feature points by dynamically adjusting the weights between channels, so that the image fusion method of the present application can adapt to different image data and tasks, and thus improve the robustness of the image registration process.

[0056] In some embodiments of the present application, the cross-channel spatial attention mechanism can be implemented by Figure 2 S201-S203 shown in the figure can be implemented Figure 1 S102 shown in the figure will be described below in combination with each step.

[0057] S201, sequentially extracting features from multiple original images to obtain multiple feature images.

[0058] In the embodiments of the present application, the existing feature extraction network is used to extract feature points in each original image to obtain a feature image. The feature image can reflect the original image information to a certain extent. At the same time, each original image of multiple different views contains independent and complementary image information.

[0059] In the embodiments of the present application, the feature extraction network includes but is not limited to a VGG (Visual Geometry Group) network, a GoogLeNet (Google Inception Net) network, a ResNet (Residual Network) network and other commonly used feature extraction networks. The feature extraction network can be a pre-trained feature extraction network or an untrained feature extraction network. The skilled person can determine according to the actual situation, and the present application does not limit this.

[0060] For example, a pre-trained neural network, ResNet 50 network, is used as a feature extraction network to extract features of the original images of multiple views to obtain feature images M i (i = 0, 1, 2,... n). Wherein, n is the view number.

[0061] S202, sequentially performing feature encoding processing on the multiple feature images to obtain multiple feature encoding data; wherein each feature encoding data corresponds to each feature image.

[0062] In the embodiments of the present application, feature encoding is a key technology in the field of computer graphics, which is used to encode feature points of feature images, so as to save storage and transmission space in the process of image processing, and facilitate detection, tracking and identification of target feature points.

[0063] For example, the last pooling layer of the neural network, ResNet 50 network, is used as the network output, and ResNet 50 network is used to encode the feature images M i of each view to obtain feature encoding data N i (i = 0, 1, 2,... n). Wherein, n is the view number.

[0064] S203, aggregating the multiple feature encoding data along the channel dimension to obtain integrated encoding data; wherein the integrated encoding data is input data of the cross-channel spatial attention mechanism, and the integrated encoding data contains multiple feature encoding data.

[0065] In the embodiments of the present application, the integrated encoding data is a simple aggregation of the multiple feature encoding data, so as to be input data of the subsequent cross-channel spatial attention mechanism. Each feature encoding data in the integrated encoding data can reflect the image information of the corresponding original image.

[0066] Specifically, the dimension of the first feature encoding data N0is expanded to (1, p, q), and the other feature encoding data N iThe first feature encoding data N0 is concatenated with the (i = 1, 2,..., n) to obtain integrated encoding data M in the form of (n + 1, p, q). Wherein, n + 1 represents the dimension; n represents the view order number, and the original image of each view corresponds to a channel, that is, n represents the channel order number; p and q represent the feature vector size of the integrated encoding data; p represents the height of the feature vector; and q represents the width of the feature vector.

[0067] In some embodiments of the present application, the training of the module can be implemented by Figure 3 S301-S304 shown can be implemented by Figure 1 S103 shown will be described in combination with each step.

[0068] S301, obtaining a preset initialization query vector and an initialization key vector.

[0069] S302, pre-training according to the initialization query vector, the initialization key vector and a public data set to obtain an updated target query vector and a target key vector.

[0070] S303, obtaining a channel weight distribution according to the target key vector, the target query vector and the integrated encoding data.

[0071] S304, obtaining a spatial weight distribution according to the target key vector, the target query vector and the integrated encoding data.

[0072] S305, obtaining an attention weight distribution according to the channel weight distribution and the spatial weight distribution.

[0073] S306, multiplying the attention weight distribution and the integrated encoding data to obtain a registration matrix.

[0074] In the embodiments of the present application, the cross-channel spatial attention mechanism is first used to train the module, and the target key vector K and the target query vector Q obtained after training are obtained. The target key vector K and the target query vector Q are used to calculate the relationship between the channels and the space, and the dimensions of the two vectors are usually the same as the channel number n + 1.

[0075] In the embodiments of the present application, the channel weight distribution A c reflects the information difference of each feature point between different channels (that is, different original images). The spatial weight distribution A s reflects the position of each feature point in each frame of original image and the spatial relationship between surrounding feature points. The attention weight distribution A c is obtained by multiplying the channel weight distribution A s and the spatial weight distribution A c , which integrates the channel weight distribution A sThe reflected feature point information. Specifically, the channel weight distribution can be multiplied by the spatial weight distribution to obtain an attention weight distribution A.

[0076] In the embodiments of the present application, the calculation formula of the attention weight distribution is as follows:

[0077] A=A c *A s (1);

[0078] In the above formula (1), A is an attention weight distribution; A s is a spatial weight distribution; and A c is a channel weight distribution.

[0079] In the embodiments of the present application, the attention weight distribution A is finally multiplied by the integrated encoding data M to obtain a registration matrix O, the registration matrix O including a plurality of groups of relationship matrices, the relationship matrices being capable of matching the relationship between the feature point pairs of each two frames of original images, that is, the registration matrix O reflects the registration feature network relationship.

[0080] In the embodiments of the present application, the calculation formula of the registration matrix is as follows:

[0081] O=A*M (2);

[0082] In the above formula (2), O is a registration matrix; A is an attention weight distribution; and M is integrated encoding data.

[0083] In some embodiments of the present application, the registration matrix O can be obtained by Figure 4 S401-S403 shown in Figure 3 S301 shown below will be described in conjunction with each step.

[0084] S401, performing a convolution operation on the integrated encoding data to obtain an output result.

[0085] In the embodiments of the present application, the integrated encoding data M is transformed through a convolution operation to obtain an output result F n , that is, a new encoding data is generated by a convolution operation. The output result F n represents the correlation within each channel. The specific formula is as follows:

[0086] F n =Conv(M) (3);

[0087] In the above formula (3), F n is an output result; Conv is a convolution operation; and M is integrated encoding data.

[0088] For example, the integrated encoding data M is processed by using a convolutional neural network (CNN). A convolutional layer is used to perform a convolution operation to extract channel information, that is, to extract the information difference between different channels (that is, different original images) of each feature point.

[0089] S402, dot product operation is performed on the target query vector, the target key vector and the output result to obtain an inter-channel weight matrix.

[0090] In the embodiment of the present application, the specific formula for calculating the inter-channel weight matrix is as follows:

[0091] Z n = Q·K·F n (4);

[0092] In the above formula (4), Z n is the inter-channel weight matrix; Q is the target query vector; K is the target key vector; and F n is the output result.

[0093] S403, normalization operation is performed on the inter-channel weight matrix to obtain a channel weight distribution.

[0094] In the embodiment of the present application, the inter-channel weight matrix is normalized (softmax) to ensure that the inter-channel weight matrix meets the definition of the probability distribution, and the channel weight distribution is obtained. The specific formula is as follows:

[0095] A c = softmax(Z n ) (5);

[0096] In the above formula (5), A c is the channel weight distribution; softmax is the normalization operation; and Z n is the inter-channel weight matrix.

[0097] In some embodiments of the present application, S501-S503 shown in Figure 5 may be used to implement S302 shown in Figure 3 , which will be described in detail in combination with each step.

[0098] S501, global average pooling operation is performed on the integrated encoding data to obtain an output vector.

[0099] In the embodiment of the present application, for spatial attention, first, the integrated encoding data M is subjected to global average pooling (GAPool) operation to obtain the spatial average value of each channel, and an output vector F sFor example, the convolutional neural network (CNN) is used to process the integrated encoding data, and a global average pooling operation is performed by using a pooling layer to extract spatial information, that is, to extract the position of each feature point and the spatial relationship between surrounding feature points. The specific formula is as follows:

[0100] F s =GAPool(M) (6);

[0101] In the above formula (6), F s is an output vector; GAPool is a global average pooling operation; and M is integrated encoding data.

[0102] S502, dot product operation is performed on the target query vector, the target key vector and the output vector to obtain a weight matrix between spaces.

[0103] In the embodiment of the present application, the specific formula for calculating the weight matrix between spaces is as follows:

[0104] Z s =Q·K·F s (7);

[0105] In the above formula (7), Z s is a weight matrix between spaces; Q is a target query vector; K is a target key vector; and F s is an output result.

[0106] S503, normalization operation is performed on the weight matrix between spaces to obtain a spatial weight distribution.

[0107] In the embodiment of the present application, the weight matrix between spaces is normalized (softmax) to ensure that the weight matrix between spaces meets the definition of probability distribution, and the spatial weight distribution is obtained. The specific formula is as follows:

[0108] A s =softmax(Z s ) (8);

[0109] In the above formula (8), A s is a spatial weight distribution; softmax is a normalization operation; and Z s is a weight matrix between spaces.

[0110] In the embodiment of the present application, after the above operation is performed by using the cross-channel attention mechanism to obtain the registration matrix, affine transformation is performed on each frame of original image to obtain a plurality of aligned images.

[0111] In some embodiments of the present application, S104 shown in Figure 6 may be implemented by S601 shown in Figure 1 The following will be described in combination with each step.

[0112] S601, multiply each original image with the registration matrix to obtain an aligned image.

[0113] In the embodiments of the present application, for each original image, the respective matching feature points are transformed according to the registration matrix to ensure that they are aligned in space. For each original image of each view, a new image is obtained through affine transformation. The specific formula is as follows:

[0114]

[0115] In the above formula (9), P i is the original image of each view; is the aligned image; O is the registration matrix; i is the image (including the original image and the aligned image) sequence number; n is the view sequence number (i.e. the original image sequence number). In the embodiments of the present application, n+1 is the total number of views, which also represents the total number of original image frames.

[0116] In the embodiments of the present application, for each original image P i of each view, a new image (i.e. the aligned image ) is obtained through affine transformation (reflected by the registration matrix), so as to map the feature points on the original images from different views to a common coordinate system, thereby realizing the registration and alignment of the images.

[0117] In the embodiments of the present application, the plurality of aligned images are sent into the trained convolutional neural network for fusion processing, so as to obtain a multi-view fusion image. The specific steps can be explained by the following S701-S705.

[0118] S701, construct a fusion network, such as Figure 7As shown, the fusion network comprises: a first convolution module 110, a second convolution module 120 and a third convolution module 130 connected in sequence; the first convolution module 110 comprises: a first convolution layer Conv1, a first batch normalization layer Batchnorm1 and a first full connection layer Softmax1 connected in sequence; wherein the input channel number of the first convolution layer Conv1 is equal to the total frame number of the original image, and the output channel number is 48-96; the input and output channel numbers of the first batch normalization layer Batchnorm1 and the first full connection layer Softmax1 are all 48-96; the second convolution module 120 comprises: a second convolution layer Conv2, a second batch normalization layer Batchnorm2 and a second full connection layer Softmax2 connected in sequence; wherein the input channel number of the second convolution layer Conv2 is 48-96, and the output channel number is 12-36; the input and output channel numbers of the second batch normalization layer Batchnorm2 and the second full connection layer Softmax2 are all 12-36; the third convolution module 130 comprises: a third convolution layer Conv3, a third batch normalization layer Batchnorm3 and a third full connection layer Softmax3 connected in sequence; wherein the input channel number of the third convolution layer Conv3 is 12-36, and the output channel number is 1; the input and output channel numbers of the third batch normalization layer Batchnorm3 and the third full connection layer Softmax3 are all 1.

[0119] In the embodiment of the application, in the first convolution module 110, the output channel number of the first convolution layer Conv1 is the same as the input channel number of the first batch normalization layer Batchnorm1; the output channel number of the first batch normalization layer Batchnorm1 is the same as the input channel number of the first full connection layer Softmax1. The second convolution module 120 and the third convolution module 130 are set in the same way.

[0120] For example, the fusion network comprises a convolutional neural network. The original image is obtained through n+1 views, and the original image has n+1 frames, that is, the input channel number of the first convolution layer is n+1.

[0121] S702, construct a fusion network loss function.

[0122] In the embodiment of the application, the specific formula of the loss function is as follows:

[0123] L(I F ,I input )=SSIM(I F ,I input ) (10)

[0124]

[0125] In the above formula (10) and the above formula (11), x is an input image 1 when the fusion network is trained; y is an input image 2 when the fusion network is trained; μ represents a mean value; σ represents a mean square error; C1 and C2 represent hyperparameters of a stability metric, C1 is 0.5, and C2 is 1; I F represents a multi-view fusion image; I input represents an input image.

[0126] S703, training the fusion network using a public data set.

[0127] In the embodiment of the application, the training parameters are as follows: a training batch is 100, a learning rate is 0.01, a training set is 5000 pairs, a test set is 100 pairs, and an optimizer uses an SGD (stochastic gradient descent) optimizer. The fusion network selects and inputs training images (an input image 1 and an input image 2) from a public data set to obtain a fusion image I F and calculates a loss function. The SGD optimizer adaptively adjusts network parameters of the first convolutional module, the second convolutional module, and the third convolutional module according to a convergence condition of the loss function, to obtain the trained fusion network.

[0128] S704, expanding a dimension of the first frame-aligned image to (1, p, q), concatenating other frame-aligned images with the first frame-aligned image to obtain a multi-channel image in the form of (n+1, p, q).

[0129] S705, inputting the multi-channel image as an input image into the trained fusion network to obtain a multi-view fusion image.

[0130] In summary, the embodiment of the application uses a cross-channel spatial attention mechanism to obtain aligned feature points (a registration matrix) of original images, to avoid misalignment and deviation registration. An affine transformation is used to achieve high-quality image alignment. A fusion network is used to integrate information of the aligned images, to maximize the retention of main features of the original images, to ensure clear outlines and rich textures of the features. The image fusion method of the application achieves strong robustness of image registration and fusion. That is, the image fusion method of the application can be applied to complex and variable engineering operations, and has good effects, and is easy to implement in engineering operations. Using the image fusion method of the application, the environmental perception ability of operators can be improved, operation precision and efficiency can be enhanced, and safety in engineering operations can be improved.

[0131] The embodiment of the application also provides a multi-view image fusion device, as shown in Figure 8As shown, the multi-view image fusion device 200 includes: an acquisition module 210 configured to acquire a plurality of frames of original images; the plurality of frames of original images are images taken from multiple perspectives at the same time; an analysis module 220 configured to sequentially perform feature extraction on the plurality of frames of original images using a neural network to obtain a plurality of feature encoding data; wherein each feature encoding data corresponds to each frame of original image; a comparison module 230 configured to cross-compare the plurality of feature encoding data using a cross-channel spatial attention mechanism to obtain a registration matrix; the registration matrix includes a plurality of relationship matrices, and the relationship matrix reflects the transformation relationship between the corresponding feature points in each two frames of original images with matching feature point pairs; a transformation module 240 configured to sequentially perform affine transformation on the plurality of frames of original images according to the corresponding relationship matrix to obtain a plurality of frames of aligned images; wherein each of the aligned images corresponds to each of the original images one by one; and a fusion module 250 configured to input the aligned images into a fusion network for fusion processing to obtain a multi-view fusion image.

[0132] In some embodiments of the present application, the analysis module 220 is further configured to sequentially perform feature extraction on the plurality of frames of original images to obtain a plurality of feature images; sequentially perform feature encoding processing on the plurality of feature images to obtain a plurality of feature encoding data; wherein each of the feature encoding data corresponds to each of the feature images one by one; and aggregate the plurality of feature encoding data along the channel dimension to obtain integrated encoding data; wherein the integrated encoding data is input data of the cross-channel spatial attention mechanism, and the integrated encoding data contains the plurality of feature encoding data.

[0133] In some embodiments of the present application, the processing module is pre-trained using a public data set, and the target key vector K and the target query vector Q can be obtained after training. The target key vector K and the target query vector Q are used to calculate the relationship between the channels and the space. The dimensions of the two vectors are usually the same as the number of channels n+1.

[0134] In some embodiments of the present application, the comparison module 230 is further configured to obtain a preset initialization query vector and an initialization key vector; pre-train the initialization query vector, the initialization key vector, and the public data set to obtain an updated target query vector and a target key vector; obtain a channel weight distribution according to the target key vector, the target query vector, and the integrated encoding data; obtain a spatial weight distribution according to the target key vector, the target query vector, and the integrated encoding data; obtain an attention weight distribution according to the channel weight distribution and the spatial weight distribution; and multiply the attention weight distribution and the integrated encoding data to obtain the registration matrix.

[0135] In some embodiments of the present application, the comparison module 230 is further configured to perform a convolution operation on the integrated encoding data to obtain an output result; perform a dot product operation on the target query vector, the target key vector and the output result to obtain an inter-channel weight matrix; and perform a normalization operation on the inter-channel weight matrix to obtain a channel weight distribution.

[0136] In some embodiments of the present application, the comparison module 230 is further configured to perform a global average pooling operation on the integrated encoding data to obtain an output vector; perform a dot product operation on the target query vector, the target key vector and the output vector to obtain an inter-space weight matrix; and perform a normalization operation on the inter-space weight matrix to obtain a space weight distribution.

[0137] In some embodiments of the present application, the transformation module 240 is further configured to multiply each original image by a registration matrix to obtain an aligned image.

[0138] The above device embodiments are similar to the descriptions of the above method embodiments, and have similar beneficial effects to the method embodiments. In some embodiments, the device provided by the embodiments of the present application has functions or includes modules that can be used to execute the methods described in the above method embodiments. For technical details not disclosed in the device embodiments of the present application, please refer to the description of the method embodiments of the present application.

[0139] It should be noted that, in the embodiments of the present application, if the image fusion method described above is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods of the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk, and various media that can store program codes. Thus, the embodiments of the present application are not limited to any specific hardware, software or firmware, or any combination of hardware, software and firmware.

[0140] The embodiments of the present application also provide a computer program product, which includes a memory and a processor, the memory stores a computer program capable of running on the processor, and the processor implements part or all of the steps in the above method when executing the program.

[0141] The embodiments of the present application further provide a computer readable memory, which stores a computer program, and the computer program is executed by a processor to implement some or all steps of the above method. The computer readable memory can be transitory or non-transitory.

[0142] The above descriptions of the device, memory and computer program product embodiments are similar to the descriptions of the above method embodiments, and have similar beneficial effects as the method embodiments. For technical details not disclosed in the device, memory and computer program product embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

[0143] It should be noted that the multi-view image fusion method provided by the embodiments of the present application can be applied to excavators or other special vehicles, and is suitable for remote driving of special vehicles such as excavators. The present application is not limited here. The embodiments of the multi-view image fusion method provided by the present application and the embodiments of the multi-view image fusion device belong to the same concept. In the technical solutions described in each embodiment, each technical feature can be combined arbitrarily without conflict.

[0144] The above embodiments only exemplarily illustrate the principles and effects of the present application, and are not used to limit the present application. Any person skilled in the art can modify or change the above embodiments without departing from the spirit and scope of the present application. Therefore, all equivalent modifications or changes completed by those skilled in the art without departing from the spirit and technical thought of the present application should be covered by the protection scope of the present application.

Claims

1. A multi-view image fusion method, characterized by, The method comprises: acquiring a plurality of original images; wherein the plurality of original images are images captured from multiple perspectives at the same time; using a neural network to sequentially extract features from the plurality of original images to obtain a plurality of feature encoding data; wherein each feature encoding data corresponds to each original image; using a cross-channel spatial attention mechanism to cross-compare the plurality of feature encoding data to obtain a registration matrix; the registration matrix comprises a plurality of relationship matrices, and each relationship matrix reflects the transformation relationship between each two original images with matching feature point pairs; sequentially performing affine transformation on the plurality of original images according to the corresponding relationship matrices to obtain a plurality of aligned images; wherein each aligned image corresponds to each original image one by one; inputting the plurality of aligned images into a fusion network for fusion processing to obtain a multi-perspective fusion image.

2. The multi-view image fusion method of claim 1, wherein, The method comprises: sequentially extracting features from the plurality of original images to obtain a plurality of feature images; sequentially performing feature encoding processing on the plurality of feature images to obtain a plurality of feature encoding data; wherein each feature encoding data corresponds to each feature image one by one; aggregating the plurality of feature encoding data along the channel dimension to obtain integrated encoding data; wherein the integrated encoding data is used as input data of the cross-channel spatial attention mechanism, and the integrated encoding data contains the plurality of feature encoding data.

3. The multi-view image fusion method of claim 2, wherein, The method comprises: obtaining a preset initial query vector and an initial key vector; pre-training according to the initial query vector, the initial key vector, and a public data set to obtain an updated target query vector and a target key vector; obtaining a channel weight distribution according to the target key vector, the target query vector, and the integrated encoding data; obtaining a spatial weight distribution according to the target key vector, the target query vector, and the integrated encoding data; obtaining an attention weight distribution according to the channel weight distribution and the spatial weight distribution; multiplying the attention weight distribution and the integrated encoding data to obtain a registration matrix.

4. The multi-view image fusion method of claim 3, wherein, The method comprises: performing convolution operation on the integrated encoding data to obtain an output result; performing dot product operation on the target query vector, the target key vector, and the output result to obtain an inter-channel weight matrix; performing normalization operation on the inter-channel weight matrix to obtain a channel weight distribution.

5. The multi-view image fusion method of claim 3, wherein, The method comprises: performing global average pooling operation on the integrated encoding data to obtain an output vector; performing dot product operation on the target query vector, the target key vector, and the output vector to obtain an inter-channel weight matrix; The weight matrix between the spaces is normalized to obtain a spatial weight distribution.

6. The multi-view image fusion method of claim 1, wherein, The multiple frames of original images are sequentially subjected to affine transformation according to the corresponding relationship matrices to obtain multiple frames of aligned images, including: Each original image is multiplied by the registration matrix to obtain an aligned image.

7. The multi-view image fusion method of claim 1, wherein, The fusion network comprises a first convolution module, a second convolution module and a third convolution module connected in sequence. The first convolution module comprises a first convolution layer, a first batch normalization layer and a first full connection layer connected in sequence; wherein the input channel number of the first convolution layer is equal to the total number of frames of the original images, and the output channel number of the first convolution layer is 48-96; the input and output channel numbers of the first batch normalization layer and the first full connection layer are both 48-96; The second convolution module comprises a second convolution layer, a second batch normalization layer and a second full connection layer connected in sequence; wherein the input channel number of the second convolution layer is 48-96, and the output channel number is 12-36; the input and output channel numbers of the second batch normalization layer and the second full connection layer are both 12-36; The third convolution module comprises a third convolution layer, a third batch normalization layer and a third full connection layer connected in sequence; wherein the input channel number of the third convolution layer is 12-36, and the output channel number is 1; the input and output channel numbers of the third batch normalization layer and the third full connection layer are both 1.

8. A multi-view image fusion apparatus characterized by comprising: The device comprises: An acquisition module configured to acquire multiple frames of original images; wherein the multiple frames of original images are images captured from multiple perspectives at the same time; An analysis module configured to sequentially extract features from the multiple frames of original images using a neural network to obtain multiple feature encoding data; wherein each feature encoding data corresponds to each frame of original image; A comparison module configured to cross-compare the multiple feature encoding data using a cross-channel spatial attention mechanism to obtain a registration matrix; the registration matrix comprises multiple relationship matrices, and the relationship matrix reflects the transformation relationship between each two frames of original images with matching feature point pairs; A transformation module configured to sequentially perform affine transformation on the multiple frames of original images according to the corresponding relationship matrices to obtain multiple frames of aligned images; wherein each aligned image corresponds to each frame of original image one-to-one; A fusion module configured to input the aligned images into a fusion network for fusion processing to obtain a multi-perspective fusion image.

9. A computer program product comprising a memory and a processor, said memory storing a computer program operable on the processor, characterized in that, The processor executes the program to realize the steps in the method of any one of claims 1-7.

10. A computer readable memory having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps in the method of any one of claims 1-7.

Images