Depth map stitching method based on salient region matching

Abstract

To provide a depth map stitching method based on prominent region matching with excellent robustness, accuracy, and effectiveness in many scenes, which can achieve stitching without a disparity artifact of a depth map by a network model assisting depth map stitching using an RGB color image, and significantly improve the quality of a depth map stitching result image.SOLUTION: The depth map stitching method includes the steps of: constructing a depth map dataset; dividing the depth map dataset into a training dataset and a test dataset including a depth map and a color image corresponding to the depth map; inputting the test dataset into a depth map stitching sub-network to obtain initial depth map stitching results; and inputting the initial depth map stitching results into a depth value correction sub-network to obtain final depth map stitching results.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to the technical field of image stitching, and in particular to a depth map stitching method based on salient region matching. [Background technology]

[0002] Depth map stitching is conducive to the development of virtual reality technology and has great applied research value, and can be widely applied in fields such as robotics, virtual reality, machine manufacturing, 3D measurement, biomedicine, and archaeological exploration.

[0003] In recent years, image stitching algorithms for RGB color images have become more mature, and corresponding image stitching methods have been disclosed in, for example, Patent Document 1 and Patent Document 2. However, the field of depth map stitching is still lacking. Because depth maps have low resolution and sparse features, making feature point extraction and matching difficult, applying the above-mentioned stitching algorithms based on RGB color images to depth map stitching can result in large disparity artifacts and depth value mismatch problems for identical objects, which have a significant impact on subsequent visual tasks. In addition, due to the sparse characteristics of depth maps, the depth values ​​of identical objects do not match after coarse stitching of depth maps. To address this problem, traditional methods use point-to-point depth compensation, but there is a problem with depth value skipping. Therefore, this application innovatively proposes a depth compensation network based on salient region matching to significantly improve stitching efficiency. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Chinese Patent Application Publication No. 115131204 [Patent Document 2] Chinese Patent Application Publication No. 111062873 Summary of the Invention [Problem to be solved by the invention]

[0005] In view of the above-mentioned technical problem that disparity artifacts occur when applying traditional stitching algorithms to depth map stitching, a depth map stitching method based on salient region matching is provided. The present invention mainly includes the steps of: constructing a depth map dataset; dividing the depth map dataset into a training dataset and a test dataset including depth maps and corresponding color images; inputting the test dataset into a depth map stitching sub-network to obtain an initial depth map stitching result; and inputting the initial depth map stitching result into a depth value correction sub-network to obtain a final depth map stitching result. This effectively solves the problem that disparity artifacts occur or depth values ​​do not match when applying RGB color image stitching algorithms to depth map stitching, and suitably corrects the depth value information of images, thereby significantly improving the image quality of the depth map stitching result. [Means for solving the problem]

[0006] The technical means of the present invention are as follows.

[0007] According to one embodiment of the present invention, a depth map stitching method based on salient region matching includes the steps of: acquiring M image pairs, each of which includes two depth maps that have overlapping regions but are not identical; and constructing a depth map dataset using a dense depth estimation network to acquire color images corresponding to the depth maps, where M is a positive integer; dividing the depth map dataset into a training dataset and a test dataset; Select any one of the image pairs from the test dataset as the test target image pair, and input the test target image pair and the color image corresponding to the test target image pair into the depth map stitching subnet to obtain an initial depth map stitching result; Select any one of the color images corresponding to the test target image pair as the reference image, and input the initial depth map stitching result and the reference image into the depth value correction subnet to obtain a final depth map stitching result.

[0008] Furthermore, the depth map stitching subnet sets parameters to obtain an initial stitching subnet, and constructs it by training the initial stitching subnet with the training dataset to obtain the depth map stitching subnet.

[0009] Furthermore, training the initial stitching subnet with the training dataset to obtain the depth map stitching subnet includes N image pairs in the training dataset, the color images corresponding to the N image pairs, where 0 < N < M and N is an integer, inputting the training dataset into the initial stitching subnet to obtain N output images, calculating the first loss values of the N output images and their corresponding actual target images respectively, where the first loss value is a weighted sum value of the content loss value and the seam loss value, performing error backpropagation on the N first loss values to update the weights of the initial stitching subnet, and obtaining the depth map stitching subnet.

[0010] Furthermore, the depth map stitching subnet It includes a homography matrix estimation stage and a warping alignment stage, and the homography matrix estimation stage is calculated by the following formula:

Number

Number

[0011] Furthermore, the depth value correction sub-network constructs a PGN saliency detection sub-network model, and constructs an ARTR image region matching sub-network model.

[0012] Furthermore, constructing the PGN saliency detection sub-network model includes N image pairs of the training dataset, the color images corresponding to the N image pairs, where 0 < N < M and N is an integer, constructing an initial saliency detection sub-network model, obtaining a mask image corresponding to the color images corresponding to the N image pairs, inputting the color images corresponding to the N image pairs into the initial saliency detection sub-network model for training to obtain 2N output masks, calculating the second loss values between the 2N mask images and their corresponding output masks respectively, performing error backpropagation based on the second loss values to update the weights of the initial saliency detection sub-network model, and obtaining the PGN saliency detection sub-network model.

[0013] Furthermore, the initial saliency detection sub-network model includes an encoder structure, a cross-modal graft structure, and a decoder structure, which are sequentially arranged, the encoder structure including a Swin Transformer and a Resnet-1, and the decoder including a Swin decoder, a graft feature decoder, and a Resnet decoder; The second loss value is calculated by the following formula:

number

[0014] Furthermore, constructing the ARTR image region matching sub-network model includes: Constructing an initial image region matching sub-network model; Inputting the color images corresponding to the N image pairs into the PGN saliency detection sub-network model to obtain 2N reference masks, and selecting N reference masks from the 2N reference masks, wherein the N reference masks correspond one-to-one to the N image pairs; obtaining N actual matching regions corresponding to the N image pairs; Inputting the training data set into an initial stitching sub-network to obtain N stitched images; Inputting the N stitched images and the N reference masks into the initial image region matching sub-network model to obtain N output matching regions; Calculating third loss values ​​for the N actual matching regions and the N output matching regions, respectively, and performing backpropagation using the third loss values ​​to update the weights of the initial image region matching sub-network model to obtain the ARTR image region matching sub-network model.

[0015] Furthermore, the third loss value is calculated by the following formula:

number

[0016] Compared with the prior art, the present invention has the following advantages:

[0017] 1. The depth map stitching method based on salient region matching provided in this invention uses color images to assist depth map stitching, thereby solving the problem that the lack of feature points in the depth map leads to inaccurate homography matrix estimation and serious disparity artifacts.

[0018] 2. The depth map stitching method based on salient region matching provided in the present invention is based on ARTR's image region matching, and solves the discontinuous visual perception problem caused by depth value jumps caused by traditional point-to-point matching.

[0019] 3. The depth map stitching method based on salient region matching provided by the present invention is based on PGN saliency detection to obtain the foreground region of the reference image, and then performs matching and depth value correction on the foreground region, thereby solving the problem that matching the entire image results in very large time overhead.

[0020] 4. The depth map stitching method based on salient region matching provided in the present invention solves the problem of loss not converging and poor training results by setting different learning rates for different epochs in the neural network. [Brief explanation of the drawings]

[0021] In order to more clearly explain the technical means in the embodiments of the present invention or the prior art, the accompanying drawings necessary for the description of the embodiments or the prior art will be briefly introduced below. It goes without saying that the following accompanying drawings are some embodiments of the present invention, and those skilled in the art can further obtain other accompanying drawings from these accompanying drawings without any creative effort.

[0022] [Figure 1] 1 is a flowchart of a depth map stitching method based on salient region matching provided in the present invention; [Figure 2] 1 is another flowchart of the depth map stitching method based on salient region matching provided in the present invention; [Figure 3] FIG. 1 is a configuration diagram of a network framework of the depth map stitching method based on salient region matching provided in the present invention. [Figure 4] FIG. 1 is a configuration diagram of a depth map stitching sub-network framework of the depth map stitching method based on salient region matching provided in the present invention; [Figure 5] FIG. 1 is a diagram illustrating the configuration of a PGN saliency detection sub-network model of the depth map stitching method based on salient region matching provided in the present invention; [Figure 6]FIG. 1 is a diagram illustrating the configuration of the ARTR image region matching sub-network model of the depth map stitching method based on salient region matching provided in the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0023] In order to clarify the purpose, technical means and advantages of the embodiments of the present invention, the technical means of the embodiments of the present invention will be described clearly and completely below with reference to the drawings in the embodiments of the present invention, but it goes without saying that the described embodiments are not all embodiments but only some embodiments of the present invention. Any other embodiments that a person skilled in the art can obtain based on the embodiments of the present invention without any creative effort shall be included in the protection scope of the present invention.

[0024] It should be noted that terms such as "first" and "second" in the present specification, claims, and drawings are used to distinguish between similar objects, not to describe a particular order or priority. It is understood that the embodiments of the present invention described herein can be performed in orders other than those illustrated herein, and therefore the data employed in this manner may be interchanged where appropriate. Furthermore, the terms "comprise" and "have," as well as any variations thereof, refer to a non-exclusive inclusion. For example, a process, method, system, product, or apparatus comprising a series of steps or units need not be limited to those steps or units explicitly recited, but may also include other steps or units that are not explicitly recited or that make the process, method, product, or apparatus unique.

[0025] A specific embodiment of the depth map stitching method based on salient region matching provided by the present invention will be described with reference to Figure 1, which is a flowchart of the depth map stitching method based on salient region matching provided by the present invention, Figure 2, which is another flowchart of the depth map stitching method based on salient region matching provided by the present invention, Figure 3, which is a configuration diagram of the network framework of the depth map stitching method based on salient region matching provided by the present invention, and Figure 4, which is a configuration diagram of the depth map stitching sub-network framework of the depth map stitching method based on salient region matching provided by the present invention.

[0026] The depth map stitching method based on salient region matching of the present invention includes: Constructing a depth map dataset includes acquiring M image pairs, each of which includes two depth maps that have overlapping regions but are not identical, and acquiring color images corresponding to the depth maps using a dense depth estimation network, where M is a positive integer; dividing the depth map dataset into a training dataset and a test dataset; Selecting one image pair from the test dataset as a test image pair, and inputting the test image pair and a color image corresponding to the test image pair into a depth map stitching sub-network to obtain an initial depth map stitching result; The method includes a step of selecting one of the color images corresponding to the test image pair as a reference image, and inputting the initial depth map stitching result and the reference image into a depth value correction sub-network to obtain a final depth map stitching result.

[0027] It is to be understood that the image stitching dataset UDIS-D may be used directly to obtain the M image pairs, the color images may be RGB images, and the depth map dataset may be referred to as a depth image dataset.

[0028] In some selectable embodiments, continuing to refer to FIGS. 3 and 4, the depth map stitching sub-network sets parameters to obtain an initial stitching sub-network, and is constructed by training the initial stitching sub-network with a training dataset to obtain a depth map stitching sub-network.

[0029] Training the initial stitching sub-network with a training dataset to obtain a depth map stitching sub-network includes N image pairs in the training dataset, color images corresponding to the N image pairs, where 0 < N < M and N is an integer, inputting the training dataset into the initial stitching sub-network to obtain N output images, calculating a first loss value between the N output images and their corresponding actual target images, where the first loss value is a weighted sum value of a content loss value and a seam loss value, and performing error backpropagation based on the N first loss values to update the weights of the initial stitching sub-network to obtain a depth map stitching sub-network.

[0030] It should be understood that the initial stitching sub-network is an unlabeled network. When the training dataset is input into the initial stitching sub-network and each time an image pair and its corresponding color image are input, the initial stitching sub-network outputs a corresponding output image, and the color image serves to assist in depth map stitching during the training process.

[0031] In some selectable embodiments, continuing to refer to FIG. 4, the depth map stitching sub-network includes a homography matrix estimation stage and a warping alignment stage. The homography matrix estimation stage is calculated by the following formula,

Equation

number

[0032] The homography matrix estimation stage can be seen to include two branches, each containing a five-layer network, that share weight information and are used to process different depth maps in an image pair. As shown in Figure 4, the first through fourth layers each consist of two convolutional layers and one pooling layer, while the fifth layer is a simple regression network consisting of three convolutional layers and two fully connected layers. The warping alignment stage includes a low-resolution transformation and a high-resolution optimization, each consisting of a four-layer network.

[0033] In some alternative embodiments, referring to FIG. 5, which is a configuration diagram of a PGN saliency detection sub-network model of the Transformer-based depth map stitching method provided in the present invention, and FIG. 6, which is a configuration diagram of an ARTR image region matching sub-network model of the Transformer-based depth map stitching method provided in the present invention, the depth value correction sub-network is: We constructed a PGN saliency detection subnetwork model, It is constructed by constructing an ARTR image region matching sub-network model.

[0034] In some alternative embodiments, and continuing to refer to FIGS. 5 and 6, constructing a PGN saliency detection sub-network model includes: The training dataset includes N image pairs and color images corresponding to the N image pairs, where 0 < N < M, N is an integer, and Construct an initial saliency detection sub-network model, and Obtain mask images corresponding to the color images corresponding to the N image pairs, and Input the color images corresponding to the N image pairs into the initial saliency detection sub-network model for training to obtain 2N output masks, and Calculate the second loss values of the 2N mask images and their corresponding output masks respectively, perform error backpropagation based on the second loss values to update the weights of the initial saliency detection sub-network model, and obtain the PGN saliency detection sub-network model.

[0035] In some selectable embodiments, continuing to refer to FIGS. 5 and 6, the initial saliency detection sub-network model includes an encoder structure, a cross-modal graft structure, and a decoder structure provided in sequence. The encoder structure includes a Swin Transformer and a Resnet-1, and the decoder includes a Swin decoder, a graft feature decoder, and a Resnet decoder.

[0036] It can be understood that the Swin Transformer processes low-resolution images to obtain accurate semantic information, the Resnet-18 processes high-resolution images to obtain rich detailed information, and the cross-modal graft structure combines the information of the second layer of the Swin Transformer and the fifth layer of the Resnet-18 to process high-resolution images.

[0037] The second loss value is calculated by the following formula

Equation

[0038] In some alternative embodiments, and continuing to refer to FIGS. 5 and 6, constructing the ARTR image region matching sub-network model includes: Constructing an initial image region matching sub-network model; Input color images corresponding to the N image pairs into a PGN saliency detection sub-network model to obtain 2N reference masks, and select N reference masks from the 2N reference masks, and the N reference masks correspond one-to-one to the N image pairs; Obtaining N actual matching regions corresponding to the N image pairs; Inputting the training dataset into an initial stitching sub-network to obtain N stitched images; Inputting the N stitched images and N reference masks into an initial image region matching sub-network model to obtain N output matching regions; Calculating third loss values ​​of the N actual matching regions and the N output matching regions respectively, and performing error backpropagation using the third loss values ​​to update the weights of the initial image region matching sub-network model to obtain an ARTR image region matching sub-network model.

[0039] It can be seen that the ARTR image region matching sub-network model includes a feature extractor based on Resnet-50, a Transformer structure, and a multi-layer sensor, of which the Transformer consists of a six-layer encoder and a six-layer decoder structure, and the multi-layer sensor consists of three fully connected layers.

[0040] In some alternative embodiments, and continuing to refer to FIGS. 5 and 6, the third loss value is calculated as follows:

number

[0041] In addition, the depth map stitching sub-network, PGN saliency detection sub-network model and ARTR image region matching sub-network model used in this invention all use the Tensorflow framework to train the network, and use the Adam optimizer as the optimization algorithm. sess.run() is called to calculate the final output value and the corresponding loss, and then the backward algorithm is called to calculate the gradient for each layer, and the parameters are updated using the Adam optimizer. Finally, the network training weights are recorded and saved, the batch size is set to 16, the learning rate is set to 0.001, and 3000 epochs are run on an RTX2080 graphics card.

[0042] The depth map stitching method based on salient region matching provided in the present invention includes the steps of: constructing a depth map dataset; dividing the depth map dataset into a training dataset and a test dataset including depth maps and corresponding color images; inputting the test dataset into a depth map stitching sub-network to obtain an initial depth map stitching result; and inputting the initial depth map stitching result into a depth value correction sub-network to obtain a final depth map stitching result. By using a network model to assist depth map stitching with RGB color images, disparity artifact-free stitching of depth maps is realized, which is helpful for processing the image stitching result image in the next step. By using Transformer for image region matching, the problem of depth value jumps occurring and poor visual perception content effects caused by point-to-point matching of images and depth value correction is solved. Furthermore, based on the introduction of PGN saliency detection theory, the time cost of region matching and depth value correction is reduced, and the problem of mismatched depth values ​​of the same object is effectively solved. The quality of the depth map stitching result image can be greatly improved, and it has excellent robustness, accuracy and effectiveness in many scenes.

[0043] The above embodiment numbers of the present invention are for illustrative purposes only and do not represent the superiority or inferiority of the embodiments.

[0044] In the above-described embodiments of the present invention, the description of each embodiment focuses on different points, and for the parts not described in detail in one embodiment, reference may be made to the relevant descriptions of other embodiments.

[0045] Finally, it should be noted that the above embodiments are merely for explaining the technical means of the present invention and are not intended to limit the same, and the present invention has been described in detail with reference to the above embodiments, but it is possible to modify the technical means described in the above embodiments or to replace part or all of the technical features with equivalents, and it is obvious to those skilled in the art that such modifications and replacements do not deviate from the scope of the technical means of the embodiments of the present invention.

[0046] (Addendum) (Appendix 1) constructing a depth map dataset, including acquiring M image pairs each including two non-identical depth maps with overlapping regions, and acquiring color images corresponding to the depth maps with a dense depth estimation network, where M is a positive integer; dividing the depth map dataset into a training dataset and a test dataset; Selecting one of the image pairs from the test dataset as a test image pair, and inputting the test image pair and the color image corresponding to the test image pair into a depth map stitching sub-network to obtain an initial depth map stitching result; a step of selecting one of the color images corresponding to the test image pair as a reference image, and inputting the initial depth map stitching result and the reference image into a depth value correction sub-network to obtain a final depth map stitching result.

[0047] (Appendix 2) The depth map stitching sub-network comprises: Set the parameters to obtain the initial stitching subnetwork, The method for depth map stitching based on salient region matching according to Appendix 1, characterized in that it is constructed by training the initial stitching sub-network with the training dataset to obtain the depth map stitching sub-network.

[0048] (Appendix 3) Training the initial stitching sub-network with the training dataset to obtain the depth map stitching sub-network includes: The training dataset includes N image pairs and the color images corresponding to the N image pairs, where 0 < N < M and N is an integer, Inputting the training dataset into the initial stitching sub-network to obtain N output images, Calculating the first loss value between the N output images and their corresponding actual target images, where the first loss value is a weighted sum value of the content loss value and the seam loss value, Performing error backpropagation based on the N first loss values to update the weights of the initial stitching sub-network to obtain the depth map stitching sub-network. The method for depth map stitching based on salient region matching according to Appendix 2 is characterized by including the above steps.

[0049] (Appendix 4) The depth map stitching sub-network includes: A homography matrix estimation stage and a warping alignment stage. The homography matrix estimation stage is calculated by the following formula: [Equation] Here, L PW is the first loss value between the output image and its corresponding actual target image, [Equation] is pixel multiplication operation, E is the identity matrix with all elements equal to 1, H() is the homography matrix warping operation, and IA rgb and I B rgb is the color image corresponding to the test target image pair, and is a depth map stitching method based on salient region matching according to Appendix 2

[0050] (Appendix 5) The depth value correction sub-network constructs a PGN saliency detection sub-network model and constructs an ARTR image region matching sub-network model, and is a depth map stitching method based on salient region matching according to Appendix 1

[0051] (Appendix 6) Constructing the PGN saliency detection sub-network model includes N image pairs in the training dataset, the color images corresponding to the N image pairs, where 0 < N < M and N is an integer constructing an initial saliency detection sub-network model obtaining a mask image corresponding to the color image corresponding to the N image pairs inputting the color images corresponding to the N image pairs into the initial saliency detection sub-network model for training to obtain 2N output masks calculating the second loss values of the 2N mask images and their corresponding output masks respectively, performing error backpropagation using the second loss values to update the weights of the initial saliency detection sub-network model, and obtaining the PGN saliency detection sub-network model, and is a depth map stitching method based on salient region matching according to Appendix 5

[0052] (Appendix 7) The initial saliency detection sub-network model includes an encoder structure, a cross-modal graft structure, and a decoder structure, which are sequentially arranged, the encoder structure including a Swin Transformer and a Resnet-1, and the decoder including a Swin decoder, a graft feature decoder, and a Resnet decoder; The second loss value is calculated by the following formula:

number

[0053] (Appendix 8) Constructing the ARTR image region matching sub-network model includes: Constructing an initial image region matching sub-network model; inputting the color images corresponding to the N image pairs into the PGN saliency detection sub-network model to obtain 2N reference masks, and selecting N reference masks from the 2N reference masks, wherein the N reference masks correspond one-to-one to the N image pairs; obtaining N actual matching regions corresponding to the N image pairs; Inputting the training data set into an initial stitching sub-network to obtain N stitched images; Inputting the N stitched images and the N reference masks into the initial image region matching sub-network model to obtain N output matching regions; 6. The depth map stitching method based on salient region matching described in Appendix 5, characterized in that it includes: calculating third loss values ​​for the N actual matching regions and the N output matching regions respectively; and performing error backpropagation using the third loss values ​​to update the weights of the initial image region matching sub-network model to obtain the ARTR image region matching sub-network model.

[0054] (Appendix 9) The third loss value is calculated by the following formula:

number

Claims

[Claim 1] constructing a depth map dataset by obtaining M image pairs, each of which includes two non-identical color images with overlapping regions and two corresponding depth maps, wherein the color image pairs are input to a dense depth estimation network to generate depth map (D A , D B ) pairs, where M is a positive integer; dividing the depth map dataset into a training dataset and a test dataset; selecting one of the depth map (D A , D B ) pairs from the test dataset as a test image pair, and inputting the test image pair and a color image (I A , I B ) pair corresponding to the test image pair into a depth map stitching sub-network to obtain an initial depth map stitching result; selecting one of the depth map (D A , D B ) pairs as a reference image, and inputting the initial depth map stitching result and the reference image into a depth value correction sub-network to obtain a final depth map stitching result; The depth map stitching sub-network comprises: Set the parameters to obtain the initial stitching subnetwork, training the initial stitching sub-network on the training dataset to obtain the depth map stitching sub-network; The depth map stitching sub-network comprises: The method includes a homography matrix estimation step and a warping alignment step, and the homography matrix estimation step is calculated using the following equation: [Equation 1] where L PW is the first loss value between the output image and its corresponding actual target image; [Equation 2] is a pixel multiplication operation, E is an all-ones identity matrix, H() is a homography matrix warping operation, I A rgb and I B rgb are the color images corresponding to the test image pair, The depth value correction sub-network comprises: Construct a PGN saliency detection sub-network model; It is constructed by constructing an ARTR image region matching sub-network model; Constructing the PGN saliency detection sub-network model includes: the training dataset includes N depth map pairs and N color images corresponding to the depth map pairs, where 0<N<M, and N is an integer; Constructing an initial saliency detection sub-network model; obtaining mask images corresponding to the color images corresponding to the N depth map pairs; training the initial saliency detection sub-network model using the color images corresponding to the N depth map pairs to obtain 2N output masks; calculating second loss values ​​for the 2N mask images and the corresponding output masks, respectively, and performing backpropagation using the second loss values ​​to update weights of the initial saliency detection sub-network model to obtain the PGN saliency detection sub-network model; the initial saliency detection sub-network model includes an encoder structure, a cross-modal graft structure, and a decoder structure, which are arranged in sequence, the encoder structure including a Swin Transformer and a Resnet-1, and the decoder structure including a Swin decoder, a graft feature decoder, and a Resnet decoder; The second loss value is calculated by the following formula: [Equation 3] where l bce is the binary cross-entropy loss, G xy is the GroundTruth label of the pixel point (x, y), P xy is the predicted probability of the predicted salient image, L AG is the second loss value of the mask image and the corresponding output mask, CAM ij is the cross-attention matrix, H is the height of the image, W is the width of the image, α is the dimension of the image, β is a hyperparameter used to adjust the weight ω, RP is the salient image of the second layer of the Swin Transformer, and SP is the salient image of the fifth layer of the Resnet-1. Constructing the ARTR image region matching sub-network model includes: Constructing an initial image region matching sub-network model; inputting the color images corresponding to the N depth map pairs into the PGN saliency detection sub-network model to obtain 2N reference masks, and selecting N reference masks from the 2N reference masks, wherein the N reference masks correspond one-to-one to the N depth map pairs; obtaining N actual matching regions corresponding to the N depth map pairs; inputting the training data set into an initial stitching sub-network to obtain N stitched images; Inputting the N stitched images and the N reference masks into the initial image region matching sub-network model to obtain N output matching regions; calculating third loss values ​​of the N actual matching regions and the N output matching regions, respectively, and performing backpropagation using the third loss values ​​to update the weights of the initial image region matching sub-network model to obtain the ARTR image region matching sub-network model; The third loss value is calculated by the following formula: [Equation 4] Here, L corr is the matching estimation error, L cycle is the matching mutual consistency error, α is the search area, a′ is the matching area, R is the stitched image obtained by the initial stitching sub-network, F Φ represents the transformer operation, and D B is the depth map.

Images