Thermal infrared image stereo matching method and device based on Transformer

Abstract

The present invention discloses a thermal infrared image stereo matching method and device based on Transformer, belonging to the field of computer vision technology. The method comprises obtaining a left view and a right view after correction of a thermal infrared image; extracting features from the left and right views using a UNet feature extraction network to obtain a multi-scale feature map; processing the multi-scale feature map based on a Transformer feature conversion module to obtain a conversion feature map of the left and right views; and generating a disparity map based on the conversion feature map of the left and right views. Based on a stereo matching algorithm based on deep learning, the present invention adopts a Transformer feature conversion module to perform feature conversion in view of the characteristics of sparse texture or even large-area textureless thermal infrared images, thereby solving the difficulties of thermal infrared image stereo matching and achieving a good thermal infrared stereo matching effect.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a thermal infrared image stereo matching method and device based on Transformer. Background Art

[0002] Stereo matching is a hot issue in the field of computer vision. The technology is mainly used in three-dimensional modeling and perception, robot navigation, autonomous driving, intelligent transportation and other aspects.

[0003] At present, the method of stereo matching based on deep learning mainly trains convolutional neural networks (CNN) and estimates the disparity values ​​of the corresponding pixels of the left and right views by weighted sum of pixel similarity. First, the features of the left and right views are extracted respectively under the condition of the corrected left and right views, and then the feature maps of the left and right views are used to construct a four-dimensional cost volume. The obtained cost volume is aggregated by three-dimensional convolution to obtain an aggregated cost volume, and finally the aggregated cost volume is used to generate a disparity map through disparity regression.

[0004] Stereo matching based on deep learning is based on visible light images. Visible light images are sensitive to light, and the quality of information that can be transmitted by images will be greatly affected in complex scenes and environments, such as haze and low light. Thermal infrared images reflect the surface temperature distribution in the scene, so they are not sensitive to light, have good cloud and fog penetration, and special camouflage recognition capabilities. Therefore, it is very meaningful to study stereo matching algorithms for thermal infrared images.

[0005] Since thermal infrared images have sparse textures and even large areas without textures compared to visible light images, traditional stereo matching algorithms for visible light images are not applicable to thermal infrared matching. The current stereo matching algorithms based on visible light basically consist of four parts: feature extraction, cost volume construction, cost aggregation, and disparity regression. Regarding the problem of how to extract features of left and right views in thermal infrared image stereo matching, the feature extraction part of the traditional visible light stereo matching algorithm usually uses convolution operations to extract features of different resolution sizes and then fuses them in different ways, such as: UNet, FPN and other feature extractors. However, the features extracted by these methods have only limited receptive fields and cannot model long-range relationships and spatial dependencies. In addition, the extracted features contain less position information and semantic information, and cannot distinguish textureless areas, which leads to the technical problem of poor stereo matching effect of thermal infrared images. Feature extraction is the key to extracting image feature information, but for the characteristics of thermal infrared images with few textures, the traditional feature extraction network cannot extract enough features, and the feature receptive field obtained by the traditional feature extraction method is limited.

[0006] In the related technology, the invention patent application with application number 202110635322.3 discloses a neural network construction method for multimodal image visibility detection. The multi-scale deformable convolutional neural network adjusted by the attention model can effectively extract the atmospheric characteristics of fog with different concentrations under different receptive fields; and the fusion of bimodal image features can achieve modal complementarity, provide richer and more effective atmospheric feature information, and significantly improve the accuracy and robustness of visibility detection under small sample conditions. However, it uses the traditional stereo matching method, which directly uses the left and right views to calculate the disparity map, and does not require model training, but the disparity map error is large.

[0007] The invention patent application with application number 201910647474.8 discloses a multi-camera stereo matching method and device, which trains a feature extractor based on image features, and obtains scene depth information based on the trained image feature extractor, overcoming the problems of sparse features in traditional RGB matching and high computational complexity of feature extraction and matching. However, its essence is structured light stereo matching. Stereo matching based on deep learning and traditional stereo matching use the feature points of the object itself, while structured light uses feature points actively provided by the light source or direct codewords. Due to the variety of photographed objects, each binocular stereo matching faces different images and requires re-extraction of feature points; while structured light projects the same pattern, and the feature points are fixed and do not need to change according to changes in the scene. However, it is basically unusable in outdoor environments because it is easily affected by strong natural light outdoors, causing the projected coded light to be submerged.

[0008] In summary, the current deep learning-based stereo matching algorithms are all for visible light images. The current stereo matching methods are not universal and are difficult to apply to stereo matching of thermal infrared images. In addition, the features extracted by the convolutional network in the current stereo matching feature extractor have only a limited receptive field and cannot model long-range relationships and spatial dependencies. In addition, the extracted features contain less position information and semantic information and cannot distinguish textureless areas. Summary of the invention

[0009] The technical problem to be solved by the present invention is how to achieve more accurate matching of sparse textures in thermal infrared images.

[0010] The present invention solves the above technical problems through the following technical means:

[0011] On the one hand, the present invention proposes a thermal infrared image stereo matching method based on Transformer, the method comprising the following steps:

[0012] Obtaining a left view and a right view after the thermal infrared image correction;

[0013] Using a UNet feature extraction network to extract features from the left view and the right view to obtain a multi-scale feature map, wherein the multi-scale feature map includes multi-scale feature maps corresponding to the left view and the right view respectively;

[0014] Processing the multi-scale feature map based on a Transformer feature conversion module to obtain a conversion feature map of the left view and a conversion feature map of the right view;

[0015] A disparity map is generated based on the conversion feature map of the left view and the conversion feature map of the right view.

[0016] Based on the traditional stereo matching algorithm, the present invention adopts the Transformer feature conversion module to perform feature conversion in view of the characteristics of thermal infrared images that have sparse texture or even large areas without texture, so as to solve the difficulty of thermal infrared image stereo matching and achieve good thermal infrared stereo matching effect.

[0017] Furthermore, the Transformer feature conversion module includes a position encoding module and an attention module, the multi-scale feature map is used as the input of the position encoding module, the output of the position encoding module is used as the input of the attention module, and the position encoding module is used to add the multi-scale feature map to the feature map after the multi-scale feature map is subjected to two-dimensional convolution;

[0018] The attention module includes a self-attention layer and a mutual attention layer connected in sequence.

[0019] Furthermore, the formula for implementing the attention mechanism of the attention module is as follows:

[0020] Attention(Q,K,V)=φ(Q)φ(K) T V

[0021] Among them, Attention() is the attention function, Q is the input vector Query of the attention layer, K is the input vector Key of the attention layer, V is the input vector Value of the attention layer, φ(·)=elu(·)+1.

[0022] Further, generating a disparity map based on the conversion feature map of the left view and the conversion feature map of the right view includes:

[0023] Constructing a four-dimensional cost volume based on the conversion feature map of the left view and the conversion feature map of the right view;

[0024] Performing cost aggregation on the four-dimensional cost volume through three-dimensional convolution to obtain aggregated cost volumes of different scales;

[0025] The disparity regression equation is used to estimate the aggregated cost volumes of different scales to obtain the disparity map.

[0026] Furthermore, the formula of the parallax regression equation is expressed as follows:

[0027]

[0028] Among them, c d is the prediction cost, σ() is the softmax function, d is the disparity value, To predict the disparity value, D max is the maximum parallax.

[0029] In addition, the present invention also proposes a thermal infrared image stereo matching device based on Transformer, the device comprising:

[0030] An acquisition module, used for acquiring a left view and a right view after the thermal infrared image correction;

[0031] A feature extraction module, configured to extract features from the left view and the right view using a UNet feature extraction network to obtain a multi-scale feature map, wherein the multi-scale feature map includes multi-scale feature maps corresponding to the left view and the right view respectively;

[0032] A Transformer feature conversion module, used to process the multi-scale feature map based on the Transformer feature conversion module to obtain a conversion feature map of the left view and a conversion feature map of the right view;

[0033] A disparity map generation module is used to generate a disparity map based on the conversion feature map of the left view and the conversion feature map of the right view.

[0034] Furthermore, the Transformer feature conversion module includes a position encoding module and an attention module, the multi-scale feature map is used as the input of the position encoding module, the output of the position encoding module is used as the input of the attention module, and the position encoding module is used to add the multi-scale feature map to the feature map after the multi-scale feature map is subjected to two-dimensional convolution;

[0035] The attention module includes a self-attention layer and a mutual attention layer connected in sequence.

[0036] Furthermore, the formula for implementing the attention mechanism of the attention module is as follows:

[0037] Attention(Q,K,V)=φ(Q)φ(K) T V

[0038] Among them, Attention() is the attention function, Q is the input vector Query of the attention layer, K is the input vector Key of the attention layer, V is the input vector Value of the attention layer, φ(·)=elu(·)+1.

[0039] Furthermore, the disparity map generation module includes:

[0040] A construction unit, configured to construct a four-dimensional cost volume based on the conversion feature map of the left view and the conversion feature map of the right view;

[0041] An aggregation unit, used for performing cost aggregation on the four-dimensional cost volume through three-dimensional convolution to obtain aggregated cost volumes of different scales;

[0042] The estimation unit is used to estimate the aggregated cost volumes of different scales by using a disparity regression equation to obtain the disparity map.

[0043] Furthermore, the formula of the parallax regression equation is expressed as follows:

[0044]

[0045] Among them, c d is the prediction cost, σ() is the softmax function, d is the disparity value, To predict the disparity value, D max is the maximum parallax.

[0046] The advantages of the present invention are:

[0047] (1) Based on the traditional stereo matching algorithm, the present invention adopts the Transformer feature conversion module to perform feature conversion in view of the characteristics of thermal infrared images that have sparse textures or even large areas without textures, so as to solve the difficulties of thermal infrared image stereo matching and achieve good thermal infrared stereo matching effects.

[0048] (2) The present invention uses the position coding in the Transformer feature conversion module to obtain more position information and semantic information based on the original feature information, and uses the self-attention layer and mutual attention layer of the Transformer feature conversion module to obtain the global receptive field. The difficulty of stereo matching of thermal infrared images is well solved through position coding, self-attention and mutual attention.

[0049] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1is a flow chart of a thermal infrared image stereo matching method based on Transformer in the first embodiment of the present invention;

[0051] Figure 2 is an overall flow chart of the thermal infrared image stereo matching method based on Transformer in the first embodiment of the present invention;

[0052] Figure 3 is a structural diagram of a matching network model in a first embodiment of the present invention;

[0053] Figure 4 is a structural diagram of the attention module in the first embodiment of the present invention;

[0054] Figure 5 is a schematic diagram comparing the results of matching the left and right views using the method of the present invention and the reference method;

[0055] Figure 6 It is a structural diagram of a thermal infrared image stereo matching device based on Transformer in the second embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in combination with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] like Figure 1 As shown, an embodiment of the present invention proposes a thermal infrared image stereo matching method based on Transformer, and the method includes the following steps:

[0058] S10, obtaining a left view and a right view after the thermal infrared image correction;

[0059] It should be noted that the thermal infrared image can be acquired by using a thermal infrared sensor, and the thermal infrared image can be corrected by using the Bouguet epipolar correction method.

[0060] S20, using a UNet feature extraction network to extract features from the left view and the right view to obtain a multi-scale feature map, wherein the multi-scale feature map includes multi-scale feature maps corresponding to the left view and the right view respectively;

[0061] It should be noted that the left and right views are sent to the UNet feature extraction network for transmission, and the left and right views pass through the same feature extraction module to obtain four feature maps of different resolutions.

[0062] S30, processing the multi-scale feature map based on a Transformer feature conversion module to obtain a conversion feature map of the left view and a conversion feature map of the right view;

[0063] S40: Generate a disparity map based on the conversion feature map of the left view and the conversion feature map of the right view.

[0064] Based on the traditional stereo matching algorithm, this embodiment adopts the Transformer feature conversion module to perform feature conversion in view of the characteristics of thermal infrared images that have sparse texture or even large areas without texture, so as to solve the difficulty of thermal infrared image stereo matching and achieve good thermal infrared stereo matching effect.

[0065] In one embodiment, the Transformer feature conversion module includes a position encoding module and an attention module, the multi-scale feature map is used as the input of the position encoding module, the output of the position encoding module is used as the input of the attention module, and the position encoding module is used to add the multi-scale feature map to the feature map after the multi-scale feature map is subjected to two-dimensional convolution;

[0066] The attention module includes a self-attention layer and a mutual attention layer connected in sequence.

[0067] It should be noted that the feature map extracted by the feature extraction module is sent to the Transformer feature conversion module, first through the position encoding module, and the multi-scale feature map extracts features through the two-dimensional convolution layer to obtain the feature extraction map. The feature extraction map is added and fused with the multi-scale feature map. The fused features contain more position information and semantic information. The fused features are then flattened and sent to the attention module with self-attention and mutual attention, and the conversion features with global receptive field are obtained through the attention module.

[0068] This embodiment introduces a position encoding module to use two-dimensional convolution to add more position information and semantic information on the basis of the original multi-scale feature map. The features obtained after the 3*3 convolution operation of the multi-scale feature map contain more position information and semantic information. The convolved features are added to the original multi-scale feature map to increase the feature information of the image.

[0069] By introducing self-attention layers and mutual attention layers, the receptive field of the original features is increased to model long-range relationships and spatial dependencies. Self-attention layers and mutual attention layers are used to convert image features into contextual information to obtain the global receptive field to solve the problem of sparse texture.

[0070] In one embodiment, the attention module implements the attention mechanism as follows:

[0071] Attention(Q,K,V)=φ(Q)φ(K) T V

[0072] Among them, Attention() is the attention function, Q is the input vector Query of the attention layer, K is the input vector Key of the attention layer, V is the input vector Value of the attention layer, φ(·)=elu(·)+1.

[0073] It should be noted that the attention operation selects relevant information by measuring the similarity between Q and each K, and the output vector is obtained by the weighted sum of the similarity score values. In self-attention, Q, K, and V are vectors corresponding to the same feature map, while in mutual attention, Q, K, and V are vectors corresponding to different left and right feature maps.

[0074] In one embodiment, if Figure 2 As shown, the step S30 includes the following steps:

[0075] Constructing a four-dimensional cost volume based on the conversion feature map of the left view and the conversion feature map of the right view;

[0076] It should be noted that the conversion features are sent to the cost volume construction module, and the conversion feature maps of the left and right views are used to construct a four-dimensional cost volume by splicing the channel, height, width and disparity, where conversion features of different scales construct four-dimensional cost volumes of different scales.

[0077] Performing cost aggregation on the four-dimensional cost volume through three-dimensional convolution to obtain aggregated cost volumes of different scales;

[0078] It should be noted that the four-dimensional cost volume is sent to the cost aggregation module. First, the four-dimensional cost volume of the smallest scale is sent to the three-dimensional convolution to obtain the minimum scale aggregation cost. In order to further fuse the cost volumes of different scales, except for the cost volume of the smallest scale, the cost volumes of the other three scales are spliced ​​with the cost volume sampled on the previous scale, and then sent to the three-dimensional convolution to obtain the aggregation cost volumes of different scales.

[0079] The disparity regression equation is used to estimate the aggregated cost volumes of different scales to obtain the disparity map.

[0080] It should be noted that the aggregated cost body obtains the disparity map through the disparity regression module, and the disparity regression method is used to estimate the continuous disparity map. The softmax operation is used to obtain the possibility of each disparity value corresponding to the aggregated cost body. The predicted disparity value is obtained by multiplying each disparity value by its corresponding possibility and summing them up.

[0081] In one embodiment, the prediction cost c is obtained by the softmax operation. d To calculate the probability of each disparity value d, predict the disparity value The disparity regression equation is obtained by multiplying each disparity value by its corresponding probability and summing them up. The formula of the disparity regression equation is as follows:

[0082]

[0083] Among them, c d is the prediction cost, σ() is the softmax function, d is the disparity value, To predict the disparity value, D max is the maximum parallax.

[0084] In one embodiment, the matching network model structure for implementing the above-mentioned Transformer-based thermal infrared image stereo matching method is as follows: Figure 3 As shown, the specific structural process is as follows:

[0085] (1) Feature extraction network, used to extract features from the input image to be matched:

[0086] The images to be matched are the left and right rectified images of thermal infrared. First, the UNet feature extraction network is used to extract the features of the left and right views, and feature maps with sizes of 1 / 8, 1 / 16, 1 / 32, and 1 / 64 of the original images are extracted respectively. The features of the left view are recorded as L1~L4, and the features of the right view are recorded as R1~R4.

[0087] Among them, the UNet feature extraction network includes three convolutional layers, a maximum pooling layer, four residual blocks, a pyramid pooling layer, a second splicing layer and two convolutional layers. Its feature extraction process is:

[0088] The left view and the right view are spliced ​​along the channel through the first splicing layer and sent to the feature extractor. After three convolutional layers, the sizes of the convolutional layers are 3*3*16, 3*3*16, and 3*3*32 respectively. Each convolutional layer is followed by batch normalization and activation function ReLu to obtain features of 1 / 2 the size of the original image; the obtained features are sent to the maximum pooling layer function MaxPool to obtain features of 1 / 4 the size of the original image; the obtained feature map is sent to four residual blocks in turn to obtain four features of 1 / 8, 1 / 16, 1 / 32, and 1 / 64 of the original image size respectively; in order to obtain richer features, the minimum scale feature is sent to the pyramid pooling layer to obtain new features, and the features of the remaining three scales are spliced ​​with the features sampled from the previous scale through the second splicing layer, and then L1~L4 and R1~R4 are obtained through 3*3 convolution and 1*1 convolution, and each convolutional layer is followed by batch normalization and activation function ReLu.

[0089] (2) Transformer feature conversion module, which is used to convert the feature map output by the feature extraction network to obtain a converted feature map:

[0090] The multi-scale feature maps L1~L4 and R1~R4 are sent to the Transformer feature conversion module to obtain the conversion features. The Transformer feature conversion module includes a position encoding module and an attention module. The attention module includes a self-attention layer and a mutual attention layer. Specifically, Figure 3 The lower part of the structure.

[0091] The features L1~L4 and R1~R4 extracted by UNet are sent to the Transformer feature conversion module, and the left and right features of different scales are sent to the position encoding. The features obtained by 3*3 convolution of the left and right features are added and fused with the original features to obtain the encoded left and right features. The encoded left features are recorded as PL1~PL4, and the encoded right features are recorded as PR1~PR4. PL1~PL4 and PR1~PR4 are flattened and sent to the self-attention layer and the mutual attention layer n times. The number of times the features of different scales pass through the self-attention layer and the mutual attention layer is 1, 2, 4, and 8 in order from small to large according to the feature scale. PL1~PL4 and PR1~PR4 pass through the self-attention layer and the mutual attention layer n times to obtain the converted left and right features. The converted left features are recorded as TL1~TL2, and the converted right features are recorded as TR1~TR4. The attention module is as follows Figure 4 As shown, the attention operation selects relevant information by measuring the similarity between Q and each K, and the output vector is obtained by the weighted sum of the similarity scores V. The three input vectors Q, K and V here use the similarity metric function sim(Q,K) = φ(Q)φ(K) T , where φ(·)=elu(·)+1, elu is the activation function. In the self-attention layer, Q, K, V are vectors corresponding to the same (left or right) feature map, and in the mutual attention layer, Q, K, V are vectors corresponding to different (left and right) feature maps.

[0092] (3) Cost construction module:

[0093] The converted left features TL1~TL2 and the converted right features TR1~TR4 are sent to the cost volume construction module to construct a four-dimensional cost volume. The cost volume is composed of the channel, height, width and disparity of the left and right features of the same scale, thereby obtaining four cost volumes of different scales, which are respectively denoted as C1~C4.

[0094] (4) Cost aggregation module:

[0095] The four cost volumes C1~C4 of different scales are sent to the cost aggregation module to obtain four aggregated cost volumes of different scales, and the four aggregated cost volumes of different scales are recorded as AC1~AC4. The cost volume C4 of the smallest scale is sent to the three-dimensional convolution to obtain the aggregated cost volume AC4. The cost volume C4 is concatenated with C3 after trilinear interpolation upsampling, three-dimensional convolution and activation function ReLu to obtain the aggregated cost volume AC3. The cost volume C3 is concatenated with C2 after trilinear interpolation upsampling, three-dimensional convolution and activation function ReLu to obtain the aggregated cost volume AC2. The cost volume C2 is concatenated with C1 after trilinear interpolation upsampling, three-dimensional convolution and activation function ReLu to obtain the aggregated cost volume, and then sent to the three-dimensional convolution to obtain the aggregated cost volume. The obtained aggregated cost volume is subjected to bilinear interpolation operation to obtain the aggregated cost volume AC1 with the same scale as the original image.

[0096] (5) Disparity regression module:

[0097] The four aggregated cost volumes AC1 to AC4 are sent to the disparity regression module. The three aggregated cost volumes AC2 to AC4 are obtained by trilinear interpolation to obtain the cost volume with the same scale as the original image. The four aggregated cost volumes of different scales are sent to the softmax function to predict the possibility of each disparity value, and the predicted disparity value is obtained by multiplying each disparity value by the sum of its corresponding probabilities.

[0098] Because the feature receptive field obtained by the feature extraction network is limited and the features contain less image information, the Transformer feature conversion module is set up based on Transformer, including a position encoding module to add more position information and semantic information, and self-attention and mutual attention modules to obtain the global receptive field.

[0099] like Figure 5 As shown, the comparison between the disparity map obtained by the method of this embodiment and the final disparity map of the benchmark network (HSM) is shown. Figure 5 -(a) is the left view of the thermal infrared image, Figure 5 -(b) is the right view of the thermal infrared image, Figure 5 -(c) is the baseline network disparity map, Figure 5 -(d) is the disparity map obtained by the method of this embodiment. This implementation selected 3 pairs of thermal infrared samples from Figure 5It can be found that the method of the embodiment of the present invention matches more accurately and has less noise. The comparative technical solution only uses the traditional convolutional neural network to extract features, which can handle some image matching with rich textures, but it is difficult to accurately match parts with sparse textures. The method of the embodiment of the present invention benefits from the Transformer feature conversion module, and can accurately match the disparity on these samples with sparse textures.

[0100] In addition, if Figure 6 As shown, the embodiment of the present invention also proposes a thermal infrared image stereo matching device based on Transformer, and the device includes:

[0101] An acquisition module 10 is used to acquire a left view and a right view after the thermal infrared image correction;

[0102] A feature extraction module 20, configured to extract features from the left view and the right view using a UNet feature extraction network to obtain a multi-scale feature map, wherein the multi-scale feature map includes multi-scale feature maps corresponding to the left view and the right view respectively;

[0103] A Transformer feature conversion module 30, configured to process the multi-scale feature map based on a Transformer feature conversion module to obtain a conversion feature map of the left view and a conversion feature map of the right view;

[0104] The disparity map generation module 40 is used to generate a disparity map based on the conversion feature map of the left view and the conversion feature map of the right view.

[0105] In one embodiment, the Transformer feature conversion module 20 includes a position encoding module and an attention module, the multi-scale feature map is used as the input of the position encoding module, the output of the position encoding module is used as the input of the attention module, and the position encoding module is used to add the multi-scale feature map to the feature map after the multi-scale feature map is subjected to two-dimensional convolution;

[0106] The attention module includes a self-attention layer and a mutual attention layer connected in sequence.

[0107] In one embodiment, the attention module implements the attention mechanism as follows:

[0108] Attention(Q,K,V)=φ(Q)φ(K) T V

[0109] Among them, Attention() is the attention function, Q is the input vector Query of the attention layer, K is the input vector Key of the attention layer, V is the input vector Value of the attention layer, φ(·)=elu(·)+1.

[0110] In one embodiment, the disparity map generation module 40 includes:

[0111] A construction unit, configured to construct a four-dimensional cost volume based on the conversion feature map of the left view and the conversion feature map of the right view;

[0112] An aggregation unit, used for performing cost aggregation on the four-dimensional cost volume through three-dimensional convolution to obtain aggregated cost volumes of different scales;

[0113] The estimation unit is used to estimate the aggregated cost volumes of different scales by using a disparity regression equation to obtain the disparity map.

[0114] In one embodiment, the parallax regression equation is expressed as follows:

[0115]

[0116] Among them, c d is the prediction cost, σ() is the softmax function, d is the disparity value, To predict the disparity value, D max is the maximum parallax.

[0117] It should be noted that other embodiments or implementation methods of the thermal infrared image stereo matching device based on Transformer described in the present invention can refer to the above-mentioned method embodiments, which will not be repeated here.

[0118] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0119] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0120] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0121] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0122] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A thermal infrared image stereo matching method based on Transformer, characterized in that: The method comprises: Obtaining a left view and a right view after the thermal infrared image correction; Using a UNet feature extraction network to extract features from the left view and the right view to obtain a multi-scale feature map, wherein the multi-scale feature map includes multi-scale feature maps corresponding to the left view and the right view respectively; The multi-scale feature map is processed based on a Transformer feature conversion module to obtain a conversion feature map of the left view and a conversion feature map of the right view, wherein the Transformer feature conversion module includes a position encoding module and an attention module, wherein the multi-scale feature map is used as an input of the position encoding module, and an output of the position encoding module is used as an input of the attention module, wherein the position encoding module is used to add the multi-scale feature map to a feature map after two-dimensional convolution of the multi-scale feature map; and the attention module includes a self-attention layer and a mutual attention layer connected in sequence; Based on the conversion feature map of the left view and the conversion feature map of the right view, a disparity map is generated, including: constructing a four-dimensional cost volume based on the conversion feature map of the left view and the conversion feature map of the right view; performing cost aggregation on the four-dimensional cost volume through three-dimensional convolution to obtain aggregated cost volumes of different scales; using a disparity regression equation, estimating the aggregated cost volumes of different scales to obtain the disparity map, and the formula of the disparity regression equation is expressed as follows: Among them, c d is the prediction cost, σ() is the softmax function, d is the disparity value, To predict the disparity value, D max is the maximum parallax.

2. The thermal infrared image stereo matching method based on Transformer according to claim 1, characterized in that: The formula for implementing the attention mechanism of the attention module is as follows: Attention(Q,K,V)=φ(Q)φ(K) T V Among them, Attention() is the attention function, Q is the input vector Query of the attention layer, K is the input vector Key of the attention layer, V is the input vector Value of the attention layer, and φ(·)=elu(·)+1.

3. A thermal infrared image stereo matching device based on Transformer, characterized in that: The device comprises: An acquisition module, used for acquiring a left view and a right view after the thermal infrared image correction; A feature extraction module, configured to extract features from the left view and the right view using a UNet feature extraction network to obtain a multi-scale feature map, wherein the multi-scale feature map includes multi-scale feature maps corresponding to the left view and the right view respectively; A Transformer feature conversion module, used to process the multi-scale feature map based on the Transformer feature conversion module to obtain a conversion feature map of the left view and a conversion feature map of the right view, wherein the Transformer feature conversion module includes a position encoding module and an attention module, wherein the multi-scale feature map is used as an input of the position encoding module, and the output of the position encoding module is used as an input of the attention module, wherein the position encoding module is used to add the multi-scale feature map to the feature map after the multi-scale feature map is subjected to two-dimensional convolution; and the attention module includes a self-attention layer and a mutual attention layer connected in sequence; A disparity map generation module, configured to generate a disparity map based on the conversion feature map of the left view and the conversion feature map of the right view; The disparity map generation module comprises: A construction unit, configured to construct a four-dimensional cost volume based on the conversion feature map of the left view and the conversion feature map of the right view; An aggregation unit, used for performing cost aggregation on the four-dimensional cost volume through three-dimensional convolution to obtain aggregated cost volumes of different scales; The estimation unit is used to estimate the aggregated cost volume of different scales by using a disparity regression equation to obtain the disparity map. The formula of the disparity regression equation is as follows: Among them, c d is the prediction cost, σ() is the softmax function, d is the disparity value, To predict the disparity value, D max is the maximum parallax.

4. The thermal infrared image stereo matching device based on Transformer according to claim 3, characterized in that: The formula for implementing the attention mechanism of the attention module is as follows: Attention(Q,K,V)=φ(Q)φ(K) T V Among them, Attention() is the attention function, Q is the input vector Query of the attention layer, K is the input vector Key of the attention layer, V is the input vector Value of the attention layer, and φ(·)=elu(·)+1.

Images