Depth map stitching method based on prominent region matching

Abstract

To provide a depth map stitching method based on prominent region matching with excellent robustness, accuracy, and effectiveness in many scenes, which can achieve stitching without a disparity artifact of a depth map by a network model assisting depth map stitching using an RGB color image, and significantly improve the quality of a depth map stitching result image.SOLUTION: The depth map stitching method includes the steps of: constructing a depth map dataset; dividing the depth map dataset into a training dataset and a test dataset including a depth map and a color image corresponding to the depth map; inputting the test dataset into a depth map stitching sub-network to obtain initial depth map stitching results; and inputting the initial depth map stitching results into a depth value correction sub-network to obtain final depth map stitching results.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to the technical field of image stitching, and in particular to a depth map stitching method based on salient region matching. [Background technology]

[0002] Depth map stitching is conducive to the development of virtual reality technology, has extremely high applied research value, and can be widely applied in fields such as robotics, virtual reality, machine manufacturing, three-dimensional measurement, biomedicine, and archaeological exploration.

[0003] In recent years, image stitching algorithms for RGB color images have become mature, for example, patent document 1, patent document 2, etc. disclose corresponding image stitching methods, but the depth map stitching field is still blank. Depth maps have low resolution and sparse features, making it difficult to extract and match feature points, so when the above stitching algorithm based on RGB color images is applied to depth map stitching, large disparity artifacts occur and the problem of depth values ​​of the same object not matching occurs, which has a great impact on subsequent visual tasks. In addition, due to the characteristics of the sparse features of the depth map, the phenomenon of depth values ​​of the same object not matching occurs after the depth map is roughly stitched. In the conventional method, point-to-point depth correction is performed, but there is a problem of depth value skip. Therefore, this application innovatively proposes a depth correction network based on salient region matching to greatly improve stitching efficiency. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] China Patent Application Publication No. 115131204 [Patent Document 2] China Patent Application Publication No. 111062873 Summary of the Invention [Problem to be solved by the invention]

[0005] In view of the above-mentioned technical problem that there is disparity artifact when traditional stitching algorithm is applied to depth map stitching, a depth map stitching method based on salient region matching is provided. The present invention mainly includes the steps of constructing a depth map dataset, dividing the depth map dataset into a training dataset and a test dataset including depth map and corresponding color image, inputting the test dataset into a depth map stitching sub-network to obtain an initial depth map stitching result, and inputting the initial depth map stitching result into a depth value correction sub-network to obtain a final depth map stitching result, thereby effectively solving the problem that there is disparity artifact or depth value does not match when RGB color image stitching algorithm is applied to depth map stitching, suitably correcting the depth value information of image, and significantly improving the image quality of depth map stitching result. [Means for solving the problem]

[0006] The technical means of the present invention are as follows.

[0007] According to one embodiment of the present invention, a method for depth map stitching based on salient region matching includes the steps of constructing a depth map dataset, including obtaining M image pairs each including two depth maps that have overlapping regions but are not identical, and obtaining color images corresponding to the depth maps using a dense depth estimation network, where M is a positive integer; dividing the depth map dataset into a training dataset and a test dataset; Select any one of the image pairs from the test dataset as the test target image pair, and input the test target image pair and the color image corresponding to the test target image pair into the depth map stitching sub-network to obtain an initial depth map stitching result; Select any one of the color images corresponding to the test target image pair as the reference image, and input the initial depth map stitching result and the reference image into the depth value correction sub-network to obtain a final depth map stitching result.

[0008] Furthermore, the depth map stitching sub-network sets parameters to obtain an initial stitching sub-network, and constructs it by training the initial stitching sub-network with the training dataset to obtain the depth map stitching sub-network.

[0009] Furthermore, training the initial stitching sub-network with the training dataset to obtain the depth map stitching sub-network includes N image pairs in the training dataset, the color images corresponding to the N image pairs, where 0 < N < M and N is an integer, inputting the training dataset into the initial stitching sub-network to obtain N output images, calculating the first loss values of the N output images and their corresponding actual target images respectively, where the first loss value is a weighted sum value of the content loss value and the seam loss value, performing error backpropagation based on the N first loss values to update the weights of the initial stitching sub-network to obtain the depth map stitching sub-network.

[0010] Furthermore, the depth map stitching sub-network It includes a homography matrix estimation stage and a warping alignment stage. The homography matrix estimation stage is calculated by the following formula:

Number

Number

[0011] Furthermore, the depth value correction sub-network constructs a PGN saliency detection sub-network model, and constructs an ARTR image region matching sub-network model.

[0012] Furthermore, constructing the PGN saliency detection sub-network model includes N image pairs of the training dataset, the color images corresponding to the N image pairs, where 0 < N < M and N is an integer, constructing an initial saliency detection sub-network model, obtaining a mask image corresponding to the color images corresponding to the N image pairs, inputting the color images corresponding to the N image pairs into the initial saliency detection sub-network model for training to obtain 2N output masks, calculating the second loss values between the 2N mask images and their corresponding output masks respectively, and performing error backpropagation according to the second loss values to update the weights of the initial saliency detection sub-network model to obtain the PGN saliency detection sub-network model.

[0013] Further, the initial saliency detection sub-network model includes an encoder structure, a cross-modal graft structure, and a decoder structure, which are arranged in sequence, the encoder structure includes a Swin Transformer and a Resnet-1, and the decoder includes a Swin decoder, a graft feature decoder, and a Resnet decoder; The second loss value is calculated by the following formula:

number

[0014] Furthermore, constructing the ARTR image region matching sub-network model includes: Constructing an initial image region matching sub-network model; inputting the color images corresponding to the N image pairs into the PGN saliency detection sub-network model to obtain 2N reference masks, and selecting N reference masks from the 2N reference masks, the N reference masks corresponding one-to-one to the N image pairs; obtaining N actual matching regions corresponding to the N image pairs; Inputting the training data set into an initial stitching sub-network to obtain N stitched images; Inputting the N stitched images and the N reference masks into the initial image region matching sub-network model to obtain N output matching regions; Calculating third loss values ​​of the N actual matching areas and the N output matching areas respectively, and performing error backpropagation according to the third loss values ​​to update the weights of the initial image region matching sub-network model to obtain the ARTR image region matching sub-network model.

[0015] Furthermore, the third loss value is calculated by the following formula:

number

[0016] Compared with the prior art, the present invention has the following advantages:

[0017] 1. The depth map stitching method based on salient region matching provided in the present invention uses a color image to assist depth map stitching, thereby solving the problem that the lack of feature points in the depth map leads to inaccurate homography matrix estimation and serious disparity artifacts.

[0018] 2. The depth map stitching method based on salient region matching provided in the present invention is based on the image region matching of ARTR, and solves the discontinuous visual perception problem caused by depth value jumps caused by traditional point-to-point matching.

[0019] 3. The depth map stitching method based on salient region matching provided in the present invention is based on PGN saliency detection, obtains the foreground region of the reference image, and performs matching and depth value correction on the foreground region, thereby solving the problem that matching the entire image causes very large time overhead.

[0020] 4. The depth map stitching method based on salient region matching provided in the present invention solves the problem of loss not converging and poor training results by setting different learning rates for different epochs in the neural network. [Brief description of the drawings]

[0021] In order to more clearly explain the technical means in the embodiments of the present invention or the prior art, the following will briefly introduce the accompanying drawings necessary for the description of the embodiments or the prior art. It goes without saying that the following accompanying drawings are some embodiments of the present invention, and those skilled in the art can further obtain other accompanying drawings from these accompanying drawings without creative efforts.

[0022] [Figure 1] 1 is a flow chart of a depth map stitching method based on salient region matching provided in the present invention. [Diagram 2] 4 is another flowchart of the depth map stitching method based on salient region matching provided in the present invention. [Diagram 3] FIG. 2 is a configuration diagram of the network framework of the depth map stitching method based on salient region matching provided in the present invention. [Figure 4] FIG. 2 is a configuration diagram of a depth map stitching sub-network framework of the depth map stitching method based on salient region matching provided in the present invention; [Diagram 5] FIG. 1 is a configuration diagram of a PGN saliency detection sub-network model of the depth map stitching method based on salient region matching provided in the present invention; [Figure 6]FIG. 2 is a configuration diagram of the ARTR image region matching sub-network model of the depth map stitching method based on salient region matching provided in the present invention; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] In order to make clearer the purpose, technical means and advantages of the embodiments of the present invention, the technical means of the embodiments of the present invention will be described clearly and completely below with reference to the drawings in the embodiments of the present invention, but it goes without saying that the described embodiments are not all the embodiments but only some of the embodiments of the present invention. Any other embodiments that a person skilled in the art can obtain based on the embodiments of the present invention without any creative effort shall be included in the protection scope of the present invention.

[0024] It should be noted that terms such as "first" and "second" in the present specification, claims and drawings are used to distinguish between similar objects, and are not used to describe a particular order or priority. It is understood that the embodiments of the present invention described herein can be performed according to orders other than those illustrated herein, and therefore the data thus employed may be interchanged where appropriate. Moreover, the terms "comprise" and "have", as well as any variations thereof, refer to a non-exclusive inclusion. For example, a process, method, system, product or apparatus that includes a series of steps or units need not be limited to those steps or units that are expressly recited, but may further include other steps or units that are not expressly recited or that make the process, method, product or apparatus unique.

[0025] A specific embodiment of the depth map stitching method based on salient region matching provided by the present invention will be described with reference to Figure 1, which is a flowchart of the depth map stitching method based on salient region matching provided by the present invention, Figure 2, which is another flowchart of the depth map stitching method based on salient region matching provided by the present invention, Figure 3, which is a configuration diagram of the network framework of the depth map stitching method based on salient region matching provided by the present invention, and Figure 4, which is a configuration diagram of the depth map stitching sub-network framework of the depth map stitching method based on salient region matching provided by the present invention.

[0026] The depth map stitching method based on salient region matching of the present invention includes: Constructing a depth map dataset includes obtaining M image pairs, each of which includes two depth maps that have overlapping regions but are not identical, and obtaining color images corresponding to the depth maps with a dense depth estimation network, where M is a positive integer; splitting the depth map dataset into a training dataset and a test dataset; Selecting an image pair from the test dataset as a test image pair, and inputting the test image pair and a color image corresponding to the test image pair into a depth map stitching sub-network to obtain an initial depth map stitching result; The method includes a step of selecting one of the color images corresponding to the test image pair as a reference image, and inputting the initial depth map stitching result and the reference image into a depth value correction sub-network to obtain a final depth map stitching result.

[0027] It can be understood that the image stitching dataset UDIS-D may be directly used to obtain the M image pairs, the color images may be RGB images, and the depth map dataset may be referred to as a depth image dataset.

[0028] In some selectable embodiments, continuing to refer to FIGS. 3 and 4, the depth map stitching sub-network sets parameters to obtain an initial stitching sub-network, and constructs by training the initial stitching sub-network with a training data set to obtain a depth map stitching sub-network.

[0029] Training the initial stitching sub-network with a training data set to obtain a depth map stitching sub-network includes N image pairs in the training data set, including color images corresponding to the N image pairs, where 0 < N < M and N is an integer, inputting the training data set into the initial stitching sub-network to obtain N output images, calculating a first loss value between the N output images and their corresponding actual target images, where the first loss value is a weighted sum value of a content loss value and a seam loss value, and performing error backpropagation based on the N first loss values to update the weights of the initial stitching sub-network to obtain a depth map stitching sub-network.

[0030] It should be understood that the initial stitching sub-network is an unlabeled network. When the training data set is input into the initial stitching sub-network and each time an image pair and its corresponding color image are input, the initial stitching sub-network correspondingly outputs one output image, and the color image serves to assist in depth map stitching during the training process.

[0031] In some selectable embodiments, continuing to refer to FIG. 4, the depth map stitching sub-network includes a homography matrix estimation stage and a warping alignment stage. The homography matrix estimation stage is calculated by the following formula

Equation

number

[0032] It can be seen that the homography matrix estimation stage includes two branches, each of which includes a five-layer network, which share weight information and are used to process different depth maps in an image pair respectively. As shown in Figure 4, the first to fourth layer networks each consist of two convolution layers and one pooling layer, and the fifth layer network is a simple regression network, consisting of three convolution layers and two fully connected layers. The warping alignment stage includes low-resolution deformation and high-resolution optimization, each of which consists of a four-layer network.

[0033] In some alternative embodiments, referring to FIG. 5, which is a configuration diagram of a PGN saliency detection sub-network model of the Transformer-based depth map stitching method provided in the present invention, and FIG. 6, which is a configuration diagram of an ARTR image region matching sub-network model of the Transformer-based depth map stitching method provided in the present invention, the depth value correction sub-network is: We constructed a PGN saliency detection subnetwork model, It is constructed by constructing an ARTR image region matching sub-network model.

[0034] In some alternative embodiments, and continuing to refer to FIGS. 5 and 6, constructing a PGN saliency detection sub-network model includes: The training dataset includes N pairs of images and color images corresponding to the N pairs of images, where 0 < N < M, N is an integer, and constructing an initial saliency detection sub-network model; obtaining a mask image corresponding to the color image corresponding to the N pairs of images; inputting the color images corresponding to the N pairs of images into the initial saliency detection sub-network model for training to obtain 2N output masks; calculating the second loss values of the 2N mask images and their corresponding output masks respectively, performing error backpropagation according to the second loss values to update the weights of the initial saliency detection sub-network model, and obtaining a PGN saliency detection sub-network model.

[0035] In some alternative embodiments, continuing to refer to FIGS. 5 and 6, the initial saliency detection sub-network model includes an encoder structure, a cross-modal graft structure, and a decoder structure provided in sequence. The encoder structure includes a Swin Transformer and a Resnet-18, and the decoder includes a Swin decoder, a graft feature decoder, and a Resnet decoder.

[0036] It is understandable that the Swin Transformer processes low-resolution images to obtain accurate semantic information, the Resnet-18 processes high-resolution images to obtain rich detailed information, and the cross-modal graft structure combines the information of the second layer of the Swin Transformer and the fifth layer of the Resnet-18 to process high-resolution images.

[0037] The second loss value is calculated by the following formula:

Equation

[0038] In some alternative embodiments, and continuing to refer to FIGS. 5 and 6, constructing the ARTR image region matching sub-network model includes: Constructing an initial image region matching sub-network model; Inputting color images corresponding to the N image pairs into a PGN saliency detection sub-network model to obtain 2N reference masks, and selecting N reference masks from the 2N reference masks, the N reference masks corresponding to the N image pairs one-to-one; Obtaining N actual matching regions corresponding to the N image pairs; Inputting the training dataset into an initial stitching sub-network to obtain N stitched images; Input the N stitched images and the N reference masks into an initial image region matching sub-network model to obtain N output matching regions; Calculating third loss values ​​of the N actual matching areas and the N output matching areas respectively, and performing error backpropagation according to the third loss values ​​to update the weights of the initial image region matching sub-network model to obtain an ARTR image region matching sub-network model.

[0039] It can be understood that the ARTR image region matching sub-network model includes a feature extractor based on Resnet-50, a Transformer structure, and a multi-layer sensor, in which the Transformer consists of a 6-layer encoder and a 6-layer decoder structure, and the multi-layer sensor consists of three fully connected layers.

[0040] In some alternative embodiments, and continuing to refer to FIGS. 5 and 6, the third loss value is calculated according to the following formula:

number

[0041] In addition, the depth map stitching sub-network, PGN saliency detection sub-network model and ARTR image region matching sub-network model used in the present invention all use the Tensorflow framework to train the network, use the Adam optimizer as the optimization algorithm, call sess.run() to calculate the final output value and the corresponding loss, then call the backward algorithm to calculate the gradient for each layer, update the parameters through the Adam optimizer, and finally record and save the network training weights, set the batch size to 16, set the learning rate to 0.001, and run 3000 epochs on the RTX2080 graphics card.

[0042] The depth map stitching method based on salient region matching provided in the present invention includes the steps of constructing a depth map dataset, dividing the depth map dataset into a training dataset and a test dataset including depth maps and corresponding color images, inputting the test dataset into a depth map stitching sub-network to obtain an initial depth map stitching result, and inputting the initial depth map stitching result into a depth value correction sub-network to obtain a final depth map stitching result, thereby using a network model to assist depth map stitching with RGB color images to realize disparity artifact-free stitching of depth maps, which is helpful for processing the image stitching result image in the next step. By using Transformer for image region matching, the problem of depth value jumps occurring and poor visual sensing content effects caused by point-to-point matching of traditional images to correct depth values ​​is solved; and based on the introduction of PGN saliency detection theory, the time cost of region matching and depth value correction is reduced, the problem of depth values ​​of the same objects not matching is effectively solved, and the quality of the depth map stitching result image can be greatly improved, with excellent robustness, accuracy and effectiveness in many scenes.

[0043] The above embodiment numbers of the present invention are merely for the purpose of explanation and do not indicate superiority or inferiority of the embodiments.

[0044] In the above-described embodiments of the present invention, the description of each embodiment focuses on different points, and for the parts not described in detail in one embodiment, reference may be made to the relevant descriptions of other embodiments.

[0045] Finally, the following should be noted: The above embodiments are merely for explaining the technical means of the present invention, and are not intended to limit the same, and the present invention has been described in detail with reference to the above embodiments. However, the technical means described in the above embodiments can be modified or equivalently replaced in part or all of the technical features, and it is obvious to those skilled in the art that the essence of the corresponding technical means does not depart from the scope of the technical means of each embodiment of the present invention through such modifications or replacements.

[0046] (Additional Note) (Appendix 1) Constructing a depth map dataset, comprising: acquiring M image pairs, each of which includes two non-identical depth maps having overlapping regions; and acquiring color images corresponding to the depth maps with a dense depth estimation network, where M is a positive integer; dividing the depth map dataset into a training dataset and a test dataset; Selecting one of the image pairs from the test dataset as a test image pair, and inputting the test image pair and the color image corresponding to the test image pair into a depth map stitching sub-network to obtain an initial depth map stitching result; selecting one of the color images corresponding to the test image pair as a reference image, and inputting the initial depth map stitching result and the reference image into a depth value correction sub-network to obtain a final depth map stitching result.

[0047] (Appendix 2) The depth map stitching sub-network comprises: Set the parameters to obtain the initial stitching sub-network, The method for depth map stitching based on salient region matching according to Supplementary Note 1, characterized in that it is constructed by training the initial stitching sub-network with the training dataset to obtain the depth map stitching sub-network.

[0048] (Supplementary Note 3) Training the initial stitching sub-network with the training dataset to obtain the depth map stitching sub-network includes: The training dataset includes N pairs of the images and the color images corresponding to the N pairs of the images, where 0 < N < M and N is an integer, Inputting the training dataset into the initial stitching sub-network to obtain N output images, Calculating a first loss value between the N output images and their corresponding actual target images, where the first loss value is a weighted sum value of a content loss value and a seam loss value, Performing error backpropagation based on the N first loss values to update the weights of the initial stitching sub-network to obtain the depth map stitching sub-network. The method for depth map stitching based on salient region matching according to Supplementary Note 2 is characterized by including the above steps.

[0049] (Supplementary Note 4) The depth map stitching sub-network includes: A homography matrix estimation stage and a warping alignment stage. The homography matrix estimation stage is calculated by the following formula:

Equation

Equation

[0050] (Appended note 5) The depth value correction sub-network constructs a PGN saliency detection sub-network model and constructs an ARTR image region matching sub-network model, and is the depth map stitching method based on salient region matching according to appended note 1.

[0051] (Appended note 6) The construction of the PGN saliency detection sub-network model includes the training dataset containing N image pairs, the color images corresponding to the N image pairs, where 0 < N < M and N is an integer constructing an initial saliency detection sub-network model obtaining a mask image corresponding to the color image corresponding to the N image pairs inputting the color images corresponding to the N image pairs into the initial saliency detection sub-network model for training to obtain 2N output masks calculating the second loss values of the 2N mask images and their corresponding output masks respectively, performing error backpropagation using the second loss values to update the weights of the initial saliency detection sub-network model, and obtaining the PGN saliency detection sub-network model, and is the depth map stitching method based on salient region matching according to appended note 5.

[0052] (Appended note 7) The initial saliency detection sub-network model includes an encoder structure, a cross-modal graft structure, and a decoder structure, which are arranged in sequence, the encoder structure includes a Swin Transformer and a Resnet-1, and the decoder includes a Swin decoder, a graft feature decoder, and a Resnet decoder; The second loss value is calculated by the following formula:

number

[0053] (Appendix 8) Constructing the ARTR image region matching sub-network model includes: Constructing an initial image region matching sub-network model; inputting the color images corresponding to the N image pairs into the PGN saliency detection sub-network model to obtain 2N reference masks, and selecting N reference masks from the 2N reference masks, the N reference masks corresponding one-to-one to the N image pairs; obtaining N actual matching regions corresponding to the N image pairs; Inputting the training data set into an initial stitching sub-network to obtain N stitched images; Inputting the N stitched images and the N reference masks into the initial image region matching sub-network model to obtain N output matching regions; 6. The depth map stitching method based on salient region matching described in Appendix 5, characterized in that it includes: respectively calculating a third loss value for the N actual matching regions and the N output matching regions; and performing error backpropagation according to the third loss value to update the weights of the initial image region matching sub-network model to obtain the ARTR image region matching sub-network model.

[0054] (Appendix 9) The third loss value is calculated by the following formula:

number

Claims

1. Constructing a depth map dataset, comprising: acquiring M image pairs, each of which includes two non-identical depth maps having overlapping regions; and acquiring color images corresponding to the depth maps with a dense depth estimation network, where M is a positive integer; dividing the depth map dataset into a training dataset and a test dataset; Selecting one of the image pairs from the test dataset as a test image pair, and inputting the test image pair and the color image corresponding to the test image pair into a depth map stitching sub-network to obtain an initial depth map stitching result; selecting one of the color images corresponding to the test image pair as a reference image, and inputting the initial depth map stitching result and the reference image into a depth value correction sub-network to obtain a final depth map stitching result.

2. The depth map stitching sub-network includes: Set the parameters to obtain the initial stitching sub-network, The depth map stitching method based on salient region matching according to claim 1 , characterized in that the depth map stitching sub-network is constructed by training the initial stitching sub-network on the training dataset to obtain the depth map stitching sub-network.

3. Training the initial stitching sub-network on the training dataset to obtain the depth map stitching sub-network includes: the training data set includes N of the image pairs, N of the color images corresponding to the image pairs, 0<N<M, N being an integer; inputting the training data set into the initial stitching sub-network to obtain N output images; Calculating first loss values ​​for the N output images and their corresponding real target images, the first loss values ​​being a weighted sum of a content loss value and a seam loss value; and performing backpropagation using the N first loss values ​​to update weights of the initial stitching sub-network to obtain the depth map stitching sub-network.

4. The depth map stitching sub-network includes: The method includes a homography matrix estimation step and a warping alignment step, and the homography matrix estimation step is calculated by the following equation: [0010] Here, L ＰＷ is the first loss value of the output image and its corresponding real target image, [0025] is a pixel multiplication operation, E is an identity matrix with all ones, H() is a homography matrix warping operation, and I Ａ ｒｇｂ and I Ｂ ｒｇｂ The method for depth map stitching based on salient region matching according to claim 2 , wherein: i i i is the color image corresponding to the test image pair.

5. The depth value correction sub-network comprises: Construct a PGN saliency detection sub-network model; The depth map stitching method based on salient region matching as claimed in claim 1, characterized in that the method is constructed by constructing an ARTR image region matching sub-network model.

6. Constructing the PGN saliency detection sub-network model includes: the training data set includes N image pairs, N color images corresponding to the image pairs, 0<N<M, N being an integer; Constructing an initial saliency detection sub-network model; obtaining a mask image corresponding to the color images corresponding to the N image pairs; training the initial saliency detection sub-network model using the color images corresponding to the N image pairs to obtain 2N output masks; 6. The depth map stitching method based on salient region matching according to claim 5, further comprising: calculating second loss values ​​of the 2N mask images and the corresponding output masks, respectively; and performing backpropagation according to the second loss values ​​to update weights of the initial saliency detection sub-network model to obtain the PGN saliency detection sub-network model.

7. The initial saliency detection sub-network model includes an encoder structure, a cross-modal graft structure, and a decoder structure, which are arranged in sequence, the encoder structure includes a Swin Transformer and a Resnet-1, and the decoder includes a Swin decoder, a graft feature decoder, and a Resnet decoder; The second loss value is calculated by the following formula: [0030] Here, l ｂｃｅ is the binary cross entropy loss, and G ｘｙ is the GroundTruth label of the pixel point (x, y), and P ｘｙ is the predicted probability of the predicted salient image, and L ＡＧ is the second loss value of the mask image and the corresponding output mask, and CAM ｉｊ is a cross-attention matrix, H is the height of the image, W is the width of the image, α is the dimension of the image, β is a hyper-parameter used to adjust the weight ω, RP is a salient image of the second layer of the Swin Transformer, and SP is a salient image of the fifth layer of the Resnet-1.

8. Constructing the ARTR image region matching sub-network model includes: Constructing an initial image region matching sub-network model; inputting the color images corresponding to the N image pairs into the PGN saliency detection sub-network model to obtain 2N reference masks, and selecting N reference masks from the 2N reference masks, the N reference masks corresponding one-to-one to the N image pairs; obtaining N actual matching regions corresponding to the N image pairs; inputting the training data set into an initial stitching sub-network to obtain N stitched images; Inputting the N stitched images and the N reference masks into the initial image region matching sub-network model to obtain N output matching regions; 6. The depth map stitching method based on salient region matching as claimed in claim 5, further comprising: respectively calculating a third loss value for the N actual matching regions and the N output matching regions; and performing backpropagation to update weights of the initial image region matching sub-network model according to the third loss value to obtain the ARTR image region matching sub-network model.

9. The third loss value is calculated by the following formula: [0045] Here, L ｃｏｒｒ is the matching estimation error, and L ｃｙｃｌｅ is the mutual consistency error of matching, α is the search area, a′ is the matching area, R is the stitched image obtained by the initial stitching sub-network, F Φ represents a transformer operation, and D Ｂ The method for depth map stitching based on salient region matching according to claim 8 , wherein:

Images