Three-dimensional scene reconstruction method based on feature fusion of cross-channel attention mechanism

Abstract

The present invention relates to the field of robot visual perception technology, specifically a three-dimensional scene reconstruction method based on cross-channel attention mechanism feature fusion, including: S1, collecting multiple binocular stereo vision images to be three-dimensionally reconstructed after distortion correction and stereo correction, and constructing a training set; S2, constructing a three-dimensional scene reconstruction network; S3, selecting a binocular stereo vision image to input into the three-dimensional scene reconstruction network, and obtaining the predicted disparity of each different stage; S4, constructing a loss function, calculating the loss between the predicted disparity and the real disparity of each different stage, and looping from S3 to S4 until the loss function converges; then updating the weight of the three-dimensional scene reconstruction network; S5, testing the trained three-dimensional scene reconstruction network, and obtaining the three-dimensional depth information of the binocular stereo vision three-dimensional reconstruction scene. The present invention greatly improves the efficiency of context feature fusion and improves the binocular stereo vision disparity estimation performance.

Description

Technical Field

[0001] The present invention relates to the field of robot visual perception technology, and in particular to a three-dimensional scene reconstruction method based on feature fusion of a cross-channel attention mechanism. Background Art

[0002] With the continuous advancement of intelligent manufacturing technology, robots, as important production units of smart factories, are playing an increasingly important role in promoting the transformation and upgrading of the manufacturing industry. Three-dimensional environmental perception technology is the basis for robots to achieve intelligence. Due to the uniqueness and importance of the human binocular imaging system, three-dimensional environmental perception technology based on binocular stereo vision has become the core of computer vision research. Binocular stereo vision simulates the observation principle of the human binocular vision system, uses binocular cameras to stereo match the acquired binocular images to obtain accurate parallax of pixels in the image, and then obtains accurate depth information of the surrounding environment by using camera parameters. It has the advantages of rich information, high stability and low price. Equipped with a binocular vision perception system, robots have the ability to observe the working scene in real time, which greatly enhances their intelligence and can complete various processing tasks in complex unstructured working environments more intelligently. Robot vision perception-related technologies based on binocular stereo vision have been widely studied and applied in the field of robot intelligent manufacturing.

[0003] However, when using binocular stereo vision robots for visual perception, parallax will occur between the two eyes. There is an urgent need for a three-dimensional scene reconstruction method to accurately predict and repair the parallax between the two eyes. Summary of the invention

[0004] The present invention provides a three-dimensional scene reconstruction method based on cross-channel attention mechanism feature fusion to solve the technical problems mentioned in the background technology.

[0005] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0006] The present invention provides a three-dimensional scene reconstruction method based on cross-channel attention mechanism feature fusion, comprising the following steps:

[0007] S1, collect multiple binocular stereo images to be reconstructed after distortion correction and stereo correction, and construct a training set;

[0008] S2. Construct a 3D scene reconstruction network, which includes a feature extraction network based on a cross-channel attention mechanism, a multi-scale hybrid matching cost construction module, a matching cost aggregation network, and a multi-stage disparity regression prediction module connected in sequence;

[0009] S3, select a binocular stereo vision image from the training set and input it into the 3D scene reconstruction network to obtain the predicted disparity of the matching cost at each different stage;

[0010] S4, constructing a loss function, calculating the loss between the predicted disparity and the real disparity of the matching cost at each different stage, and looping from S3 to S4 until the loss function converges; then using the back propagation algorithm to update the weights of the 3D scene reconstruction network to obtain the trained 3D scene reconstruction network;

[0011] S5. Testing the trained 3D scene reconstruction network to obtain 3D depth information of the binocular stereoscopic vision 3D reconstruction scene.

[0012] Furthermore, the S3 specifically includes the following steps:

[0013] S31. Select a binocular stereo vision image from the training set and input it into the feature extraction network to obtain the final left image global feature And the final right image global features ;

[0014] S32, the final left image global features And the final right image global features Input into the multi-scale hybrid matching cost construction module to obtain the similarity between the left and right images, and output the matching cost volume vector containing the similarity measure ;

[0015] S33, matching cost volume vector The input is sent to the cost aggregation network module embedded in the multi-stage context feature fusion module to aggregate the matching cost features and obtain the matching cost features after convolution at different stages.

[0016] S34, inputting the matching cost features after convolution at different stages into a multi-stage disparity regression prediction module to obtain the predicted disparity of the matching cost at each different stage.

[0017] Furthermore, the S31 specifically includes the following steps:

[0018] S311, select a binocular stereo vision image from the training set, respectively denoted as left image X L and the right image X R , then the left image X L and the right image X R They are input into the image patch embedding layer respectively, and the left image patch embedding features are output respectively. and right image patch embedding features ;

[0019] S312, then embed the left image patch into the feature and right image patch embedding features They are respectively input into the multi-stage cascade visual transformer feature extraction module embedded in the multi-stage context feature fusion module to realize the multi-stage context global multi-path feature fusion and obtain the final left image global feature And the final right image global features .

[0020] Further, the multi-stage cascade visual transformer feature extraction module in S312 is composed of multiple visual transformers; each visual transformer includes a layer of cross-covariance-based attention layer, a layer of local patch interaction layer and a layer of MLP feature extraction layer connected in sequence;

[0021] In a cross-covariance based attention layer, the input features are First, features are extracted through three layers of parallel linear feature extraction layers, expressed by the formula as follows:

[0022] ;

[0023] ;

[0024] ;

[0025] in, It is the linear feature extraction layer; is the feature normalization function; , and All are the intermediate values ​​of the self-attention mechanism of the visual transformer, representing three different feature matrices;

[0026] Then construct the attention feature , and combined with the output of the three-layer parallel linear feature extraction layer to obtain the output Y of the attention layer based on cross covariance; attention feature And the output Y is as follows:

[0027] ;

[0028] ;

[0029] in, The function is a random dropout layer, which is used to reduce the risk of overfitting of the model and enhance the generalization performance of the model; Represents a matrix multiplication operation; represents the normalized exponential function;

[0030] A local patch interaction layer includes two cascaded separable convolutional layers, each of which is cascaded with a normalization layer BN and an activation function layer GeLU; it realizes the interaction of boundary features between local patches and enhances the implicit communication of diagonal scattering attention between local patch blocks of the cross covariance attention layer;

[0031] An MLP feature extraction layer includes two cascaded linear feature extraction layers. Each linear feature extraction layer is cascaded with an activation function layer GeLU, a Dropout function, and a neural network normalization layer LayerNorm to realize MLP-based visual transformer feature extraction, thereby obtaining the left image transformer feature or the right image transformer feature.

[0032] Furthermore, the S32 specifically includes the following steps:

[0033] S321, first construct a group-related matching cost vector ;

[0034] ;

[0035] in, and is the final left image global feature And the final right image global features In the feature channel The dimensions are divided equally into The features obtained by the group, , the number of channels for each set of features is , The function represents the multiplication of corresponding eigenvalues; Indicates Parallax plane, , i represents the scale; D represents the disparity dimension; Indicates the number of feature channels where the current left and right images need to be multiplied by feature values; b indicates the feature dimension of the batch;

[0036] S322, construct matching cost vector based on splicing correlation ;

[0037] ;

[0038] in, and is the final left image global feature And the final right image global features In the feature channel The dimensions are divided equally into The features obtained by the group, , the number of channels for each set of features is ; represents the connection function;

[0039] S323, then concatenate and fuse the matching cost vectors constructed in S321 and S322 to obtain a matching cost body vector containing a similarity measure , the formula is as follows:

[0040] .

[0041] Furthermore, the S33 specifically includes the following steps:

[0042] S331, first use two layers of three-dimensional convolutional layers Matching cost volume vector for input Perform three-dimensional convolution feature aggregation to obtain updated matching cost features ;

[0043] ;

[0044] S332, define the number of stages ;

[0045] S333, using two layers of three-dimensional convolutional layers For the updated matching cost feature Or input features for feature aggregation to obtain the first matching cost feature after convolution :

[0046] ;

[0047] S334, also uses two layers of three-dimensional convolutional layers For the updated matching cost feature Perform feature aggregation to obtain the second matching cost feature after convolution ;

[0048] ;

[0049] S335, using two layers of deconvolutional layers And two skip-connected 3D convolutional layers , and through the cross-channel attention mechanism model , perform feature aggregation on the matching cost features obtained in S333 and S334 to obtain the fourth matching cost feature after convolution , the process is as follows:

[0050] ;

[0051] ;

[0052] in, represents the third matching cost feature;

[0053] S336, the number of stages , and the convolutional matching cost feature Defined as input features,

[0054] And loop from S333 to S335 until the stage number It is equal to the total number of stages L; thus, the matching cost features after convolution at different stages are obtained.

[0055] Furthermore, the implementation of S34 is as follows:

[0056] The matching cost features after convolution at different stages are respectively input into the multi-stage disparity regression prediction module. The multi-stage disparity regression prediction module is composed of multiple disparity regression and prediction modules. The matching cost features after convolution at different stages are respectively transmitted to the corresponding different disparity regression and prediction modules, so as to obtain the predicted disparity of the matching cost at each different stage.

[0057] Furthermore, the predicted disparity of each matching cost at different stages in S34 is The calculation process of the predicted disparity of the stage matching cost is as follows:

[0058] S341, firstly convolute the fourth matching cost feature Input into two layers of 3D convolutional layers to get the matching cost ;

[0059] S342, then use the upsampling layer to match the cost Sampling is performed to obtain upsampled features ;

[0060] S343, Final Utilization Function upsampling features Parallax dimension Perform normalization and then upsample the normalized features Perform linear regression and get Predicted disparity of stage matching cost .

[0061] Furthermore, the loss function in S4 is specifically:

[0062] ;

[0063] in, 2 is the loss function, The predicted disparity output for prediction The corresponding The training weights of the estimated results of each stage; is true parallax; smooth Loss Function The definition is as follows:

[0064] ;

[0065] in, It represents the predicted result value corresponding to the pixel in the image.

[0066] Furthermore, the implementation of S5 is as follows:

[0067] First, based on the known intrinsic baseline of the binocular camera And focal length , get the corrected binocular stereo image and , and then input into the trained 3D scene reconstruction network to output the predicted disparity Then, based on the relationship between disparity and depth, the three-dimensional depth information of the binocular stereoscopic vision three-dimensional reconstruction scene is obtained. , the relationship between parallax and depth is as follows:

[0068] .

[0069] Beneficial effects of the present invention:

[0070] 1. The present invention proposes a three-dimensional scene reconstruction method based on feature fusion of cross-channel attention mechanism. By utilizing the cross-channel attention mechanism of context multi-path, efficient fusion of context features is achieved, which effectively improves the feature representation ability when matching left and right images in the process of binocular stereo vision three-dimensional reconstruction.

[0071] 2. The present invention proposes a matching cost aggregation network based on the cross-channel attention mechanism, which improves the aggregation speed of multi-scale matching cost context features and realizes multi-stage high-precision prediction of disparity.

[0072] 3. The present invention also provides a feature extraction network based on a cross-channel attention mechanism, which solves the problem of low feature fusion efficiency in the multi-stage context feature fusion process in the prior art, thereby improving the binocular stereo vision disparity estimation performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 is a flow chart of the present invention;

[0074] Figure 2 It is a structural block diagram of the present invention;

[0075] Figure 3exemplified images input in the embodiments of the present invention, wherein (a) is the left image and (b) is the right image;

[0076] Figure 4 is a disparity map obtained in an embodiment of the present invention;

[0077] Figure 5 It is a structural diagram of the feature extraction network based on the cross-channel attention mechanism in the present invention;

[0078] Figure 6 It is a structural diagram of the matching cost aggregation network in the present invention. DETAILED DESCRIPTION

[0079] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Preferred embodiments of the present invention are provided in the drawings. However, the present invention can be implemented in many other different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive.

[0080] The terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0081] Reference Figure 1 and Figure 2 , the embodiment of the present application provides a three-dimensional scene reconstruction method based on cross-channel attention mechanism feature fusion, comprising the following steps:

[0082] S1, collect multiple binocular stereo images to be reconstructed after distortion correction and stereo correction, and construct a training set;

[0083] S2. Construct a 3D scene reconstruction network, which includes a feature extraction network based on a cross-channel attention mechanism, a multi-scale hybrid matching cost construction module, a matching cost aggregation network, and a multi-stage disparity regression prediction module connected in sequence;

[0084] S3, select a binocular stereo vision image from the training set and input it into the 3D scene reconstruction network to obtain the predicted disparity of the matching cost at each different stage;

[0085] S4, constructing a loss function, calculating the loss between the predicted disparity and the real disparity of the matching cost at each different stage, and looping from S3 to S4 until the loss function converges; then using the back propagation algorithm to update the weights of the 3D scene reconstruction network to obtain the trained 3D scene reconstruction network;

[0086] S5. Testing the trained 3D scene reconstruction network to obtain 3D depth information of the binocular stereoscopic vision 3D reconstruction scene.

[0087] In some embodiments, S3 specifically includes the following steps:

[0088] S31. Select a binocular stereo vision image from the training set and input it into the feature extraction network to obtain the final left image global feature And the final right image global features ; The image feature dimensions of the two are , in this example, The value is 2. The value is 128;

[0089] S32, the final left image global features And the final right image global features Input into the multi-scale hybrid matching cost construction module to obtain the similarity between the left and right images, and output the matching cost volume vector containing the similarity measure ; The feature dimension is ,in is the parallax dimension. In this example, The value is 192;

[0090] S33, matching cost volume vector The input is sent to the cost aggregation network module embedded in the multi-stage context feature fusion module to aggregate the matching cost features and obtain the matching cost features after convolution at different stages.

[0091] S34, inputting the matching cost features after convolution at different stages into a multi-stage disparity regression prediction module to obtain the predicted disparity of the matching cost at each different stage.

[0092] In some embodiments, reference Figure 5 , the S31 specifically includes the following steps:

[0093] S311, select a binocular stereo vision image from the training set, respectively denoted as left image X L and the right image X R In this embodiment, Figure 3 Taking (a) and (b) in the figure as an example, the dimension of the input image is ,in is the batch size of images fed into the network for training, and are the width and height of the image respectively, and then the left image X L and the right image X R They are respectively input into the Image Patch Embedding layer, and the left image patch embedding features are output respectively. and right image patch embedding features The characteristic dimensions of the two are ;

[0094] S312, then embed the left image patch into the feature and right image patch embedding features They are respectively input into the multi-stage cascade visual transformer feature extraction module embedded in the multi-stage context feature fusion module to realize the multi-stage context global multi-path feature fusion and obtain the final left image global feature And the final right image global features .

[0095] In some embodiments, the multi-stage context feature fusion module in S312 uses a cross-channel attention mechanism to implement multi-stage context global multi-path feature fusion. The cross-channel attention mechanism (CCAM) is defined as follows:

[0096] ;

[0097] ;

[0098] ;

[0099] Among them, g represents the conformance representation of global adaptive pooling sampling; The function represents the adaptive global average pooling sampling function. If the input sampling data is a two-dimensional image feature, The function uses the AdaptiveAvgPool2d function. If the input sampling data is a three-dimensional matching cost feature, The function uses AdaptiveAvgPool3d function; The function uses a 1-dimensional convolution function. The function is represented as an activation function normalized to between 0 and 1; Represents the result of a one-dimensional convolution function; S represents of Function output;

[0100] Next, use The function implements the cross-channel attention feature robust fusion of multi-stage contextual global multi-path features and outputs the fused features (including the final left image global features And the final right image global features ), defined as follows:

[0101] ;

[0102] ;

[0103] Where k represents the current stage number; represents the Softmax function output of S; e represents the exponential function.

[0104] In some embodiments, reference Figure 5 , the S312 specifically includes the following steps:

[0105] S3121, first define the number of stages ; Then define the left image patch embedding feature and right image patch embedding features Embed features for two input patches;

[0106] S3122, the two input patch embedding features or the left and right image fusion features obtained in the previous stage are respectively input into the first In the visual transformer, we get Left image transformer features and Right image transformer features ;

[0107] Embed the left image patch into features And all the current left image transformer features form a set of data, embedding the right image patch into the feature And all the current right image transformer features form another set of data, and the two sets of data are input into the In the feature fusion function CCAM, we get Left image fusion features and Right image fusion features ;

[0108] S3123, number of stages , and loop from S3122 to S3123 until , where N is the total number of stages for visual transformer feature extraction, that is, the current number of stages has reached the final stage, and the final left image global feature is obtained And the final right image global features .

[0109] In some embodiments, the multi-stage cascaded vision transformer feature extraction module in S312 is composed of multiple vision transformers;

[0110] Each visual transformer consists of a cross-covariance-based attention layer, a local patch interaction layer, and an MLP feature extraction layer connected sequentially;

[0111] In a cross-covariance-based attention layer, first, the input feature size is reshaped to change the feature dimension size to , where M , then enter the feature Feature extraction is performed through three layers of parallel linear feature extraction layers, and the formula is expressed as follows:

[0112] ;

[0113] ;

[0114] ;

[0115] in, It is the linear feature extraction layer; is the feature normalization function; , and All are the intermediate values ​​of the self-attention mechanism of the visual transformer, representing three different feature matrices;

[0116] Then construct the attention feature , and combined with the output of the three-layer parallel linear feature extraction layer to obtain the output Y of the attention layer based on cross covariance; attention feature And the output Y is as follows:

[0117] ;

[0118] ;

[0119] in, The function is a random dropout layer, which is used to reduce the risk of overfitting of the model and enhance the generalization performance of the model; Represents a matrix multiplication operation; represents the normalized exponential function;

[0120] A local patch interaction layer consists of two cascaded separable convolutional layers (the convolution kernel size of each separable convolutional layer is ; The stride of each convolution layer is 1), and each separable convolution layer is cascaded with a normalization layer BN and an activation function layer GeLU; the interaction of boundary features between local patches is realized, and the implicit communication of diagonal scattering attention between local patch blocks of the cross covariance attention layer is enhanced;

[0121] An MLP feature extraction layer includes two cascaded linear feature extraction layers. Each linear feature extraction layer is cascaded with an activation function layer GeLU, a Dropout function, and a neural network normalization layer LayerNorm to realize MLP-based visual transformer feature extraction, thereby obtaining the left image transformer feature or the right image transformer feature.

[0122] Next, by integrating the constructed multi-stage context feature fusion module based on the cross-channel attention mechanism, multi-stage context global feature extraction is realized, as described below:

[0123] Let the multi-stage context feature fusion module function based on the cross-channel attention mechanism be ; The number of stages that need to be multi-stage feature fusion is , defined as follows:

[0124] ;

[0125] in, , , in this example The value is 4. Current stage Features , by fusing all the previously extracted features , and obtain the fused features (include and ), will be used as the Phase input.

[0126] In some embodiments, the S32 specifically includes the following steps:

[0127] S321, first construct a group-related matching cost vector ;

[0128] ;

[0129] in, and is the final left image global feature And the final right image global features In the feature channel The dimensions are divided equally into The features obtained by the group, , the number of channels for each set of features is , The function represents the multiplication of corresponding eigenvalues; Indicates Parallax plane, , i represents the scale, and D represents the parallax dimension; in this example, and The values ​​are 32 and 4 respectively. The matching cost vector constructed The feature vector dimension is ; and Represented as the coordinates of the pixel in the image, ; Indicates the number of feature channels where the current left image and right image need to be multiplied by feature values; Represents feature channel Dimensions are divided equally The first group; b represents the characteristic dimension of the batch;

[0130] S322, construct matching cost vector based on splicing correlation ;

[0131] ;

[0132] in, and is the final left image global feature And the final right image global features In the feature channel The dimensions are divided equally into The features obtained by the group, , the number of channels for each set of features is ; In this example, and The values ​​are 16 and 8 respectively. The matching cost vector constructed based on splicing correlation The feature vector dimension is ; Represents feature channel The dimensions are divided equally into The first Group; represents the connection function;

[0133] S323, then concatenate and fuse the matching cost vectors constructed in S321 and S322 to obtain a matching cost body vector containing a similarity measure ;

[0134]

[0135] Matching cost volume vector after splicing The feature dimension is ,in In this example, The value is 2. The value is 64.

[0136] In some embodiments, reference Figure 6 , the S33 specifically includes the following steps:

[0137] S331, first use two layers of three-dimensional convolutional layers (The convolution kernel size is ; The convolution layer step size is 1; each convolution layer is cascaded with a normalization layer BN3d and an activation function layer ReLU) to match the input cost volume vector Perform three-dimensional convolution feature aggregation to obtain updated matching cost features ;

[0138] ;

[0139] S332, define the number of stages ;

[0140] S333, using two layers of three-dimensional convolutional layers (The convolution kernel size is ; The step size of the first convolution layer is 2, and the step size of the remaining layers is 1; each convolution layer is cascaded with a normalization layer BN3d and an activation function layer ReLU) to update the matching cost features Or input features for feature aggregation to obtain the first matching cost feature after convolution :

[0141] ;

[0142] S334, also uses two layers of three-dimensional convolutional layers (The convolution kernel size is ; The step size of the first convolution layer is 2, and the step size of the remaining layers is 1; each convolution layer is cascaded with a normalization layer BN3d and an activation function layer ReLU) to update the matching cost features Perform feature aggregation to obtain the second matching cost feature after convolution ;

[0143] ;

[0144] S335, using two layers of deconvolutional layers (The convolution kernel size is ; The convolution layer stride is 2; each convolution layer is cascaded with a normalization layer BN3d and an activation function layer ReLU) and two skip-connected 3D convolution layers (The convolution kernel size is ; The convolution layer step size is 1; each convolution layer is cascaded with a normalization layer BN3d and an activation function layer ReLU), and the cross-channel attention mechanism model , perform feature aggregation on the matching cost features obtained in S333 and S334 to obtain the fourth matching cost feature after convolution , the process is as follows:

[0145] ;

[0146] ;

[0147] in, represents the third matching cost feature;

[0148] S336, the number of stages , and the convolutional matching cost feature Defined as input features,

[0149] And loop from S333 to S335 until the stage number Equal to the total number of stages L. In this example, the total number of stages The value is 4; thus, the matching cost features after convolution at different stages are obtained.

[0150] In some embodiments, the implementation of S34 is as follows:

[0151] The matching cost features after convolution at different stages are respectively input into the multi-stage disparity regression prediction module, which is composed of multiple disparity regression and prediction modules. The matching cost features after convolution at different stages are respectively transmitted to the corresponding different disparity regression and prediction modules, so as to obtain the predicted disparity of the matching cost at each different stage.

[0152] In some embodiments, the predicted disparity of each different stage matching cost in S34 is The calculation process of the predicted disparity of the stage matching cost is as follows:

[0153] S341, firstly convolute the fourth matching cost feature Input to two layers of 3D convolutional layers (the convolution kernel size is ; The convolution layer step size is 1; After each convolution layer, the normalization layer BN3d and the activation function layer ReLU are cascaded to obtain the matching cost Its characteristic dimension is ;

[0154] S342, then use the upsampling layer to match the cost Sampling is performed to obtain upsampled features Its characteristic dimension is ;

[0155] S343, Final Utilization Function upsampling features Parallax dimension Perform normalization and then upsample the normalized features Perform linear regression and obtain Predicted disparity of stage matching cost ;

[0156] ;

[0157] in, The dimension of the disparity feature vector is In this example, The value range is ;

[0158] In some embodiments, the loss function in S4 is specifically:

[0159] ;

[0160] in, 2 is the loss function, The predicted disparity output for prediction The corresponding The training weights of the disparity estimation results at each stage; is true parallax; smooth Loss Function The definition is as follows:

[0161] ;

[0162] in, It represents the predicted result value corresponding to the pixel in the image.

[0163] In some embodiments, the implementation of S5 is as follows:

[0164] First, based on the known intrinsic baseline of the binocular camera And focal length , get the corrected binocular stereo image and , and then input into the trained 3D scene reconstruction network to output the predicted disparity Then, based on the relationship between disparity and depth, the three-dimensional depth information of the binocular stereoscopic vision three-dimensional reconstruction scene is obtained. , the relationship between parallax and depth is as follows:

[0165] .

[0166] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered within the protection scope of the present invention. In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the ability of ordinary technicians in the field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A 3D scene reconstruction method based on feature fusion of cross-channel attention mechanism, characterized in that: The steps include: S1, collect multiple binocular stereo images to be reconstructed after distortion correction and stereo correction, and construct a training set; S2. Construct a 3D scene reconstruction network, which includes a feature extraction network based on a cross-channel attention mechanism, a multi-scale hybrid matching cost construction module, a matching cost aggregation network, and a multi-stage disparity regression prediction module connected in sequence; S3, select a binocular stereo vision image from the training set and input it into the 3D scene reconstruction network to obtain the predicted disparity of the matching cost at each different stage; S4, constructing a loss function, calculating the loss between the predicted disparity and the real disparity of the matching cost at each different stage, and looping from S3 to S4 until the loss function converges; then using the back propagation algorithm to update the weights of the 3D scene reconstruction network to obtain the trained 3D scene reconstruction network; S5, testing the trained 3D scene reconstruction network to obtain 3D depth information of the binocular stereoscopic 3D reconstruction scene; The S3 specifically includes the following steps: S31. Select a binocular stereo vision image from the training set and input it into the feature extraction network to obtain the final left image global feature And the final right image global features ; S32, the final left image global features And the final right image global features Input into the multi-scale hybrid matching cost construction module to obtain the similarity between the left and right images, and output the matching cost volume vector containing the similarity measure ; S33, matching cost volume vector The input is sent to the cost aggregation network module embedded in the multi-stage context feature fusion module to aggregate the matching cost features and obtain the matching cost features after convolution at different stages. S34, inputting the matching cost features after convolution at different stages into a multi-stage disparity regression prediction module to obtain the predicted disparity of the matching cost at each different stage.

2. The three-dimensional scene reconstruction method according to claim 1, characterized in that: The S31 specifically includes the following steps: S311, select a binocular stereo vision image from the training set, respectively denoted as left image X L and the right image X R , then the left image X L and the right image X R They are input into the image patch embedding layer respectively, and the left image patch embedding features are output respectively. and right image patch embedding features ; S312, then embed the left image patch into the feature and right image patch embedding features They are respectively input into the multi-stage cascade visual transformer feature extraction module embedded in the multi-stage context feature fusion module to realize the multi-stage context global multi-path feature fusion and obtain the final left image global feature And the final right image global features .

3. The three-dimensional scene reconstruction method according to claim 2, characterized in that: The multi-stage cascade visual transformer feature extraction module in S312 is composed of multiple visual transformers; each visual transformer includes a layer of cross-covariance-based attention layer, a layer of local patch interaction layer and a layer of MLP feature extraction layer connected in sequence; In a cross-covariance based attention layer, the input features are First, three layers of parallel linear feature extraction layers are used to extract features, which are expressed by the formula as follows: ； ； ； in, It is the linear feature extraction layer; is the feature normalization function; , and All are the intermediate values ​​of the self-attention mechanism of the visual transformer, representing three different feature matrices; Then construct the attention feature , and combined with the output of the three-layer parallel linear feature extraction layer to obtain the output Y of the attention layer based on cross covariance; attention feature And the output Y is as follows: ； ； in, The function is a random dropout layer, which is used to reduce the risk of overfitting of the model and enhance the generalization performance of the model; Represents a matrix multiplication operation; represents the normalized exponential function; A local patch interaction layer includes two cascaded separable convolutional layers, each of which is cascaded with a normalization layer BN and an activation function layer GeLU; it realizes the interaction of boundary features between local patches and enhances the implicit communication of diagonal scattering attention between local patch blocks of the cross covariance attention layer; An MLP feature extraction layer includes two cascaded linear feature extraction layers. Each linear feature extraction layer is cascaded with an activation function layer GeLU, a Dropout function, and a neural network normalization layer LayerNorm to realize MLP-based visual transformer feature extraction, thereby obtaining the left image transformer feature or the right image transformer feature.

4. The three-dimensional scene reconstruction method according to claim 3, characterized in that: The S32 specifically includes the following steps: S321, first construct a group-related matching cost vector ; ； in, and is the final left image global feature And the final right image global features In the feature channel The dimensions are divided equally into The features obtained by the group, , the number of channels for each set of features is , The function represents the multiplication of corresponding eigenvalues; Indicates Parallax plane, , i represents the scale; D represents the disparity dimension; Indicates the number of feature channels where the current left image and right image need to be multiplied by feature values; b indicates the feature dimension of the batch; S322, construct matching cost vector based on splicing correlation ; ； in, and is the final left image global feature And the final right image global features In the feature channel The dimensions are divided equally into The features obtained by the group, , the number of channels for each set of features is ; represents the connection function; S323, then concatenate and fuse the matching cost vectors constructed in S321 and S322 to obtain a matching cost body vector containing a similarity measure , the formula is as follows: 。 5. The three-dimensional scene reconstruction method according to claim 4, characterized in that: The S33 specifically includes the following steps: S331, first use two layers of three-dimensional convolutional layers Matching cost volume vector for input Perform three-dimensional convolution feature aggregation to obtain updated matching cost features ; ； S332, define the number of stages ; S333, using two layers of three-dimensional convolutional layers For the updated matching cost feature Or input features for feature aggregation to obtain the first matching cost feature after convolution : ； S334, also uses two layers of three-dimensional convolutional layers For the updated matching cost feature Perform feature aggregation to obtain the second matching cost feature after convolution ; ； S335, using two layers of deconvolutional layers And two skip-connected 3D convolutional layers , and through the cross-channel attention mechanism model , perform feature aggregation on the matching cost features obtained in S333 and S334 to obtain the fourth matching cost feature after convolution , the process is as follows: ； ； in, Indicates The third matching cost feature in the stage; S336, the number of stages , and the convolutional matching cost feature Defined as input features, And loop from S333 to S335 until the stage number It is equal to the total number of stages L; thus, the matching cost features after convolution at different stages are obtained.

6. The three-dimensional scene reconstruction method according to claim 5, characterized in that: The implementation of S34 is as follows: The matching cost features after convolution at different stages are respectively input into the multi-stage disparity regression prediction module. The multi-stage disparity regression prediction module is composed of multiple disparity regression and prediction modules. The matching cost features after convolution at different stages are respectively transmitted to the corresponding different disparity regression and prediction modules, so as to obtain the predicted disparity of the matching cost at each different stage.

7. The three-dimensional scene reconstruction method according to claim 6, characterized in that: The predicted disparity of each matching cost at different stages in S34 The calculation process of the predicted disparity of the stage matching cost is as follows: S341, firstly convolute the fourth matching cost feature Input into two layers of 3D convolutional layers to get the matching cost ; S342, then use the upsampling layer to match the cost Sampling is performed to obtain upsampled features ; S343, Final Utilization Function upsampling features Parallax dimension Perform normalization and then upsample the normalized features Perform linear regression and obtain Predicted disparity of stage matching cost .

8. The three-dimensional scene reconstruction method according to claim 7, characterized in that: The loss function in S4 is specifically: ； in, 2 is the loss function, The predicted disparity output for prediction The corresponding The training weights of the disparity estimation results at each stage; is true parallax; smooth Loss Function The definition is as follows: ； in, It represents the predicted result value corresponding to the pixel in the image.

9. The three-dimensional scene reconstruction method according to claim 8, characterized in that: The implementation of S5 is as follows: First, based on the known intrinsic baseline of the binocular camera And focal length , get the corrected binocular stereo image and , and then input into the trained 3D scene reconstruction network to output the predicted disparity Then, based on the relationship between disparity and depth, the three-dimensional depth information of the binocular stereoscopic vision three-dimensional reconstruction scene is obtained. , the relationship between parallax and depth is as follows: 。

Images