Training Method and Apparatus for Visual Depth Estimation Model Based on Dynamic Uncertainty

Abstract

The present disclosure relates to a training method of a visual depth estimation model based on dynamic uncertainty, the method comprising: obtaining a training set, the training set containing multiple groups of image pairs corresponding to time series frames in a dynamic scene; inputting the training set into a visual depth estimation model to be trained for training iteration, the visual depth estimation model comprising: a depth prediction network, a pose prediction network and a semantic segmentation network; during the training, the dynamic and static object states corresponding to each pixel point in a to-be-reconstructed image are determined by combining a geometric consistency self-discovered mask and a semantic segmentation mask; the to-be-reconstructed image is a selected one of the image pairs; and a target loss function for a dynamic object region is determined according to the dynamic and static object states and the reconstruction processing of the image pairs. During the training iteration, the target loss function is reduced as an optimization objective, and the trained depth prediction network is used for monocular visual depth estimation, and the inference accuracy in a dynamic scene is improved.

Description

Technical Field

[0001] This disclosure relates to the field of machine vision technology, and in particular to a training method and apparatus for a visual depth estimation model based on dynamic uncertainty. Background Technology

[0002] In the field of computer vision, vision-based depth estimation (especially monocular depth estimation) has become increasingly important because it can understand the three-dimensional geometry of a scene from two-dimensional observations and recover scene depth information. Classical depth estimation algorithms or training methods generally require obtaining ground-proof depth values ​​as training labels for supervised training. However, obtaining ground-proof depth values ​​is difficult and the labeling cost is high. A new trend in the industry is to use self-supervised deep learning for depth estimation, which eliminates the need to obtain ground-proof depth values.

[0003] In realizing the concept disclosed herein, the inventors discovered at least the following technical problems in related technologies: In self-supervised deep learning algorithms or schemes, training data is primarily in static scenes. However, in real-world scenarios, most tasks involve tracking or estimating the depth of dynamic targets in dynamic scenes. Because dynamic points violate epipolar geometry constraints, the reconstruction process of dynamic regions is abnormal, leading to a deterioration in the self-supervised learning effect of depth estimation. Therefore, most related technologies rely on detection methods to determine the regions where dynamic points are located and remove these dynamic points, using the image data after removing dynamic points for training. While this reduces the damage caused by noise from dynamic points during training, the lack of sufficient regularization in these dynamic regions also results in poor inference results for dynamic regions in the trained model. Therefore, improving the depth prediction performance of depth estimation models for images containing pixels of dynamic objects is a pressing technical problem that needs to be solved. Summary of the Invention

[0004] To address or at least partially address the aforementioned technical problems, embodiments of this disclosure provide a training method and apparatus for a visual depth estimation model based on dynamic uncertainty.

[0005] In a first aspect, embodiments of this disclosure provide a training method for a visual depth estimation model based on dynamic uncertainty. The method includes: acquiring a training set containing multiple image pairs corresponding to time-series frames in a dynamic scene; inputting the training set into a visual depth estimation model to be trained for training iterations, the visual depth estimation model comprising: a depth prediction network, a pose prediction network, and a semantic segmentation network; during training, determining the dynamic and static object states corresponding to each pixel in the image to be reconstructed by combining a geometric consistency self-discovery mask and a semantic segmentation mask; the image to be reconstructed being one of the selected images from the image pairs; determining a target loss function for the dynamic object region based on the dynamic and static object states combined with the reconstruction processing of the image pairs; wherein, during the training iterations, reducing the target loss function is the optimization objective, and the trained depth prediction network is used for monocular visual depth estimation.

[0006] In some embodiments, the training set is input into the visual depth estimation model to be trained for training iteration, including: inputting each image pair in the training set into a depth prediction network, a pose prediction network, and a semantic segmentation network for depth prediction processing, pose prediction processing, and semantic segmentation processing, respectively, to obtain predicted depth, relative pose, and semantic labels corresponding to the semantic segmentation regions; determining the target probability matrix of each pixel in the image to be reconstructed belonging to a dynamic object based on the predicted depth, the relative pose, and the semantic labels; determining the photometric reprojection loss function obtained by reconstructing the image to be reconstructed from another image viewpoint of the image pair based on the target probability matrix; using the photometric reprojection loss function as the target loss function; and simultaneously training and iterating the depth prediction network and the pose prediction network with the optimization objective of reducing the photometric reprojection loss function; or, simultaneously training and iterating the depth prediction network, the pose prediction network, and the semantic segmentation network.

[0007] In some embodiments, determining the target probability matrix of each pixel in the image to be reconstructed belonging to a dynamic object based on the predicted depth, relative pose, and semantic labels includes: determining the depth prediction difference of the same pixel in the two images of the image pair based on the relative pose and predicted depth of the image pair; determining a first probability matrix of each pixel in the image to be reconstructed belonging to a dynamic object based on the depth prediction difference; determining a second probability matrix of each pixel belonging to a dynamic object based on the semantic labels corresponding to the semantic segmentation regions and the preset dynamic and static attributes corresponding to the semantic labels; and integrating the first probability matrix and the second probability matrix to obtain the target probability matrix.

[0008] In some embodiments, the first probability matrix is ​​formed by arranging the first probability values ​​of each pixel in the image to be reconstructed as belonging to a dynamic object according to the pixel's position; the second probability matrix is ​​formed by arranging the second probability values ​​of each pixel in the image to be reconstructed as belonging to a dynamic object according to the pixel's position. Integrating the first and second probability matrices to obtain the target probability matrix includes: performing a weighted sum operation between the elements at each position in the first probability matrix and the corresponding elements at the corresponding positions in the second probability matrix to obtain the element values ​​of the target probability matrix at the corresponding positions; wherein the preset weights corresponding to the first and second probability matrices are preset based on the segmentation accuracy of the semantic segmentation network; the higher the segmentation accuracy of the semantic segmentation network, the higher the preset weights of the corresponding second probability matrix.

[0009] In some embodiments, the expressions for the elements in the target probability matrix corresponding to the states of the above-mentioned dynamic and static objects are as follows:

[0010]

[0011] Where U represents the probability value corresponding to each pixel in the target probability matrix; f(U) segment U self-mask This indicates that the probability value of each pixel in the target probability matrix is ​​related to U. segment and U self-mask All are related; U self-mask This represents the probability value corresponding to the geometric consistency self-discovery mask, which corresponds to an element in the first probability matrix; U segment represents the probability value corresponding to the semantic segmentation mask, which corresponds to the element in the second probability matrix; (1-β) represents the preset weight corresponding to the first probability matrix, and β represents the preset weight corresponding to the second probability matrix.

[0012] In some embodiments, the target loss function is obtained by performing binary masking on the target probability matrix corresponding to the state of the dynamic and static objects and then combining it with the reconstruction processing of the image pairs.

[0013] The process of performing binary masking on the target probability matrix is ​​represented as follows:

[0014]

[0015] Where M represents the first mask result of the binary mask, used to determine the loss corresponding to the case where the pixel is more likely to be a dynamic object; N represents the second mask result of the binary mask, used to determine the loss corresponding to the case where the pixel is less likely to be a dynamic object; γ represents the threshold of the binary mask.

[0016] The objective loss function described above is expressed as follows:

[0017] L up =M⊙L p +θN⊙L p ,

[0018] Among them, L up L represents the target loss function, which is the photometric reprojection loss function; p θ represents the basic photometric reprojection loss, used to measure the photometric consistency between the source image of the image to be reconstructed and the reconstructed image obtained by depth estimation and relative pose transformation from another image viewpoint; θ represents the preset weight coefficient.

[0019] In some embodiments, the method further includes: constructing a geometric consistency loss function based on the relative pose output by the pose prediction network and the reprojection constraints in the geometric relationship; performing a weighted sum operation on the target loss function for the dynamic object region and the geometric consistency loss function to obtain a comprehensive loss function; wherein, during the training iteration, reducing the comprehensive loss function is also simultaneously optimized.

[0020] Secondly, embodiments of this disclosure provide a training apparatus for a visual depth estimation model based on dynamic uncertainty. The apparatus includes a data acquisition module and a training module. The data acquisition module acquires a training set containing multiple image pairs corresponding to time-series frames in a dynamic scene. The training module inputs the training set into a visual depth estimation model to be trained for iterative training. The visual depth estimation model includes a depth prediction network, a pose prediction network, and a semantic segmentation network. During training, the dynamic and static object states corresponding to each pixel in the image to be reconstructed are determined by combining a geometric consistency self-discovery mask and a semantic segmentation mask. The image to be reconstructed is one of the selected images from the image pairs. A target loss function for the dynamic object region is determined based on the dynamic and static object states combined with the reconstruction processing of the image pairs. During the training iterations, reducing the target loss function is the optimization objective. The trained depth prediction network is used for monocular visual depth estimation.

[0021] Thirdly, embodiments of this disclosure provide an electronic device. The electronic device includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, communication interface, and memory communicate with each other via the communication bus; the memory stores computer programs; and the processor, when executing the program stored in the memory, implements the training method for the visual depth estimation model based on dynamic uncertainty as described above.

[0022] Fourthly, embodiments of this disclosure provide a computer-readable storage medium. The computer-readable storage medium stores a computer program that, when executed by a processor, implements the training method for a visual depth estimation model based on dynamic uncertainty as described above.

[0023] The technical solutions provided in the embodiments of this disclosure have at least some or all of the following advantages:

[0024] By setting up a depth prediction network, a pose prediction network, and a semantic segmentation network, the dynamic and static object states corresponding to each pixel in the image to be reconstructed are comprehensively determined during training by combining a geometric consistency self-discovery mask and a semantic segmentation mask. This achieves a refined distinction and consideration of the dynamic and static probabilities of each pixel in the image to be reconstructed. Combining the geometric consistency self-discovery mask and the semantic segmentation mask has the comprehensive advantage of integrating advantages and compensating for the shortcomings of each. By adding a semantic segmentation mask on top of the geometric consistency self-discovery mask implemented by combining depth prediction and pose prediction, it helps to solve the problem of large edge noise when inferring dynamic and static pixels using geometric consistency, thus improving the recognition accuracy of edge parts. At the same time, since semantic segmentation itself has limitations, namely, it strongly depends on the dynamic or static attributes corresponding to the recognized semantic labels, and in some cases, the dynamic and static properties of objects in an image are not absolute, combining the geometric consistency self-discovery mask on top of the semantic segmentation mask also helps to improve the recognition accuracy in cases where certain semantic attributes change. Overall, by constructing a target loss function for dynamic object regions and using it as the optimization objective of training iterations, the inference accuracy of the finally trained depth prediction network in dynamic scenes can be improved. Attached Figure Description

[0025] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0026] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0027] Figure 1 A system architecture diagram illustrating a training method for a visual depth estimation model based on dynamic uncertainty, according to embodiments of the present disclosure, is shown schematically.

[0028] Figure 2 A flowchart illustrating a training method for a visual depth estimation model based on dynamic uncertainty according to an embodiment of the present disclosure is shown.

[0029] Figure 3 A detailed implementation flowchart of step S220 of an embodiment of the present disclosure is shown schematically;

[0030] Figure 4 A detailed implementation flowchart of step S320 of an embodiment of the present disclosure is shown schematically;

[0031] Figure 5 The illustrations schematically show the results of (a) an initial image after self-masking processing based on (b1) a conventional method and (b2) the results after processing based on (b2) a combination of geometric consistency self-discovery mask and semantic segmentation mask provided by the embodiments of the present disclosure.

[0032] Figure 6 A schematic block diagram of a training apparatus for a visual depth estimation model based on dynamic uncertainty according to an embodiment of the present disclosure is shown; and

[0033] Figure 7 A schematic block diagram of an electronic device provided in an embodiment of the present disclosure is shown. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0035] The first exemplary embodiment of this disclosure provides a method for training a visual depth estimation model based on dynamic uncertainty.

[0036] Figure 1 A system architecture diagram illustrating a training method for a visual depth estimation model based on dynamic uncertainty, according to embodiments of the present disclosure, is shown schematically. Figure 2 A flowchart illustrating a training method for a visual depth estimation model based on dynamic uncertainty according to an embodiment of the present disclosure is shown.

[0037] Combination Figure 1 and Figure 2 As shown, the visual depth estimation model to be trained includes: a depth prediction network (Depth net) 110, a pose prediction network (Pose net) 120, and a semantic segmentation network (Seg net) 130. The training method for the visual depth estimation model based on dynamic uncertainty includes the following steps: S210 and S220.

[0038] In step S210, a training set is obtained, which contains multiple sets of image pairs corresponding to time-series frames in a dynamic scene.

[0039] For example in Figure 1 The example provides a pair of images: the first image I s Second image I t The first and second images are two time-series frames of a dynamic scene. This dynamic scene can be a scene corresponding to video data captured by a vehicle's camera while the vehicle is in motion, or a scene corresponding to video data captured by roadside equipment. This video data contains both moving and stationary objects. The video data is divided into multiple time-series frames according to different moments, and two adjacent time-series frames form an image pair. In the two images of this image pair, the moving objects are called dynamic objects, such as moving vehicles or pedestrians. If pixels in a certain area of ​​the image correspond to a dynamic object, that area is called a dynamic region. Conversely, in the image pair, stationary objects are called static objects, and if pixels in a certain area correspond to a static object, that area is called a static region.

[0040] In the embodiments of this disclosure, during the training iterations of the visual depth estimation model, dynamic regions in the training data are not removed. Instead, the dynamic and static object states corresponding to each pixel in the image to be reconstructed are determined by combining a geometric consistency self-discovery mask and a semantic segmentation mask. Based on this, the dynamic uncertainty corresponding to each pixel is obtained, i.e., the probability matrix of whether each pixel is a dynamic object. The target loss function corresponding to the dynamic region is then constructed based on this dynamic uncertainty. This achieves a fine distinction and consideration of the dynamic and static probabilities of each pixel during training. Furthermore, the organic combination of the geometric consistency self-discovery mask and the semantic segmentation mask has the comprehensive advantage of integrating advantages and compensating for the shortcomings of each, thereby improving the inference accuracy of the final trained depth prediction network in dynamic scenes.

[0041] In step S220, the training set is input into the visual depth estimation model to be trained for training iterations. The visual depth estimation model includes a depth prediction network, a pose prediction network, and a semantic segmentation network. During training, the dynamic and static object states corresponding to each pixel in the image to be reconstructed are determined by combining a geometric consistency self-discovery mask and a semantic segmentation mask. The image to be reconstructed is one of the images selected from the image pair. The target loss function for the dynamic object region is determined based on the dynamic and static object states combined with the reconstruction processing of the image pair. During the training iterations, the optimization objective is to reduce the target loss function. The trained depth prediction network is used for monocular visual depth estimation.

[0042] Figure 3 A detailed implementation flowchart of step S220 of an embodiment of the present disclosure is shown schematically.

[0043] In some embodiments, refer to Figure 1 and Figure 3 As shown, in step S220 above, the above training set is input into the visual depth estimation model to be trained for training iteration, including the following steps: S310, S320, S330 and S340.

[0044] In step S310, each pair of images in the above training set (referred to as image pair {I}) is... s I t (As an example) The inputs are fed into the depth prediction network (Depth net) 110, the pose prediction network (Pose net) 120, and the semantic segmentation network (Seg net) 130 for depth prediction, pose prediction, and semantic segmentation, respectively, to obtain the predicted depths Ds and Dt, and the relative pose P. st Semantic tags Seg corresponding to semantic segmentation regions s .

[0045] For example, image pairs {I s I t The image is input into a depth prediction network (Depth net) 110 for depth prediction processing to obtain the first image I. s The corresponding predicted depth Ds, the second image I t The corresponding predicted depth Dt; the image pair {I s I t After concatenation, the images are input into the Pose Net 120 for pose prediction processing to obtain the first image I. s With the second image I t The relative pose P between st ;Pair the image with {I s I t The input is fed into the semantic segmentation network (Seg net) 130 for semantic segmentation processing to obtain the first image I. s Second image I t Each semantic segmentation result, that is, the entire image is divided into multiple semantic segmentation regions, and each semantic segmentation region has a corresponding semantic label. In the embodiments of this disclosure, since the target loss function is subsequently constructed for the image to be reconstructed, that is, one image is randomly selected from the image pair as the image to be reconstructed, in this step S310, semantic segmentation can also be performed only on the image to be reconstructed in the image pair, with the image to be reconstructed as the first image I. s As an example, the image to be reconstructed, I, is obtained. sThe corresponding semantic segmentation region and its corresponding semantic label Seg s .

[0046] Reference Figure 1 As shown, with the first image I s As an example of the image to be reconstructed, some semantic segmentation regions in the image are labeled as: car; some are labeled as: tree; some are labeled as: ground; some are labeled as: sign; some are labeled as: lawn; and some are labeled as: telephone pole.

[0047] In step S320, based on the predicted depth, the relative pose, and the semantic label, the target probability matrix of each pixel in the image to be reconstructed belonging to a dynamic object is determined.

[0048] In some embodiments, refer to Figure 1 As shown, for a set of input images {I s I t Based on this image, after processing by the depth prediction network (Depth net) 110, the pose prediction network (Pose net) 120, and the semantic segmentation network (Seg net) 130 respectively, the corresponding predicted depths Ds and Dt, and the relative pose P can be obtained. st Semantic tags Seg corresponding to semantic segmentation regions s Based on the predicted depths Ds and Dt and the relative pose P, stGeometric consistency self-discovery masking is performed to obtain the dynamic and static object states corresponding to each pixel in the image to be reconstructed under one consideration dimension, i.e., the dynamic uncertainty of each pixel. In this embodiment, a first probability matrix is ​​used to represent the dynamic uncertainty of pixels in the image processed by geometric consistency self-discovery masking. Simultaneously, semantic segmentation masking is performed based on the semantic labels corresponding to the semantic segmentation regions mentioned above to obtain the dynamic and static object states corresponding to each pixel in the image to be reconstructed under another consideration dimension, corresponding to the dynamic uncertainty of each pixel. In this embodiment, a second probability matrix is ​​used to represent the dynamic uncertainty of pixels in the image processed by semantic segmentation masking. By comprehensively considering the above two dimensions, combining geometric consistency self-discovery masking and semantic segmentation masking, the dynamic and static object states (i.e., the dynamic uncertainty of each pixel) corresponding to each pixel in the image to be reconstructed are comprehensively determined. Based on the dynamic and static object states and the reconstruction processing of image pairs, the target loss function for dynamic object regions is determined. This allows for fine-grained differentiation and consideration of the dynamic and static probabilities of each pixel during training, and combines the advantages of geometric consistency self-discovery masking and semantic segmentation masking while complementing their weaknesses, thus improving the inference accuracy of the trained deep prediction network in dynamic scenes.

[0049] Figure 4 A detailed implementation flowchart of step S320 of an embodiment of the present disclosure is shown schematically.

[0050] In some embodiments, combined with Figure 1 and Figure 4 As shown, in step S320 above, the target probability matrix of each pixel in the image to be reconstructed belonging to the dynamic object is determined based on the predicted depth, relative pose and semantic label, including the following steps: S410, S420, S430 and S440.

[0051] In step S410, the depth prediction difference of the same pixel in the two images of the above image pair is determined based on the relative pose and predicted depth of the image pair.

[0052] In step S420, based on the aforementioned depth prediction differences, a first probability matrix is ​​determined for each pixel in the image to be reconstructed to belong to a dynamic object.

[0053] In steps S410 and S420 above, the predicted depths established by the depth prediction network and the pose prediction network are inconsistent. This is essentially a depth map reconstruction process, which reconstructs the target image I using the estimated inter-frame relative pose. t Source Image I from the Viewpoint s Reconstructing a depth map, normally, involves using the target image I t Source image I obtained from viewpoint prediction sReconstructing the depth map and the source image I s Depth maps should ideally be consistent. However, since this assumption is based on static scenes, if dynamic objects exist in the image, the predicted depth and reconstructed depth for these objects will differ. This discrepancy can be calculated using the formula: (difference between predicted and reconstructed depths) / (product of predicted and reconstructed depths). Therefore, for static objects, the smaller the difference between predicted and reconstructed depths, the smaller the depth inconsistency; while for dynamic objects, the larger the difference, the greater the depth inconsistency. This depth inconsistency allows for pixel-level dynamic feature identification, meaning it can determine the probability that a pixel belongs to a dynamic object based on this depth prediction difference. Generally, the greater the depth inconsistency, the higher the probability that the corresponding pixel belongs to a dynamic object. The probability value ranges from 0 to 1, and extreme values ​​can be used.

[0054] Steps S410 to S420 above correspond to the process of obtaining the dynamic uncertainty of pixels in the image based on geometric consistency auto-mask processing. In this process, the value corresponding to the dynamic uncertainty of the auto-mask is the element in the first probability matrix mentioned above. Simultaneously, a threshold filtering mask is applied, filtering out pixels below a threshold and focusing on pixels with dynamic features (pixels with a higher probability of belonging to dynamic objects). The first probability matrix can also be described as the dynamic uncertainty map of the auto-mask.

[0055] In step S430, based on the semantic labels corresponding to the semantic segmentation regions and the preset dynamic and static attributes corresponding to the semantic labels, a second probability matrix is ​​determined for each pixel to belong to a dynamic object.

[0056] Since each semantic tag has a pre-set corresponding static or dynamic attribute (i.e., preset static and dynamic attributes), for example, vehicles are generally assigned dynamic attributes, people are assigned dynamic attributes, roads are assigned static attributes, signs are assigned static attributes, etc., the corresponding attribute of the semantic tag of each pixel can be determined according to the preset static and dynamic attributes of each semantic tag, and probability values ​​can be assigned according to the corresponding attributes. For example, a value of 1 can be assigned to the dynamic attribute and a value of 0 can be assigned to the static attribute; or, a value of 0.9 can be assigned to the dynamic attribute and a value of 0.1 can be assigned to the static attribute; this is not a limitation. In this way, a second probability matrix is ​​obtained in which each pixel belongs to a dynamic object from the perspective of semantic segmentation. The probability values ​​at each position can be described as a dynamic uncertainty graph or as a second probability matrix.

[0057] In some embodiments, when assigning values, the differences between the edge position and the internal position of the segmented region are also considered to assign different probabilities. For example, for the same semantic segmentation region, if the region corresponds to dynamic attributes, the probability assigned to the edge position of the dynamic region is less than the probability assigned to the internal position. For example, the edge pixel of the dynamic region is assigned a value of 0.7, while the internal pixel of the dynamic region is assigned a value of 0.9. This processing helps the trained deep prediction network to more clearly distinguish different object boundaries in dynamic scenes.

[0058] The dynamic uncertainty is assigned to each pixel in the segmented image based on its location within the region. Uncertainty is high at the object's edges and low near the center. When using semantic segmentation, the uncertainty of whether a pixel represents a dynamic feature differs between the segmentation boundary and the internal segmentation region; higher uncertainty indicates a lower probability of it being a dynamic feature. The concept of a dynamic uncertainty map is introduced for two reasons: firstly, to make visualization clearer, and secondly, to illustrate that for dynamic objects, the dynamic uncertainty of edge pixels is inconsistent with that of internal pixels, and thus involves noise.

[0059] Step S430 above corresponds to the process of obtaining the dynamic uncertainty of pixels in the image based on semantic segmentation mask processing. In this process, the value of the dynamic uncertainty of the semantic segmentation mask is the element in the second probability matrix. At the same time, after threshold filtering mask processing, the focus is on pixels with dynamic features.

[0060] In step S440, the target probability matrix is ​​obtained by integrating the first probability matrix and the second probability matrix.

[0061] By adding a semantic segmentation mask to the geometrically consistent self-discovery mask achieved by combining depth prediction and pose prediction, the problem of large edge noise when inferring dynamic and static pixels using geometric consistency is addressed, thus improving the accuracy of edge recognition. Simultaneously, since semantic segmentation itself has limitations—strongly relying on the dynamic or static attributes corresponding to the recognized semantic labels—and in reality, the dynamic and static states of objects in an image are not absolute in some cases, combining the geometrically consistent self-discovery mask with the semantic segmentation mask also helps improve the recognition accuracy in situations where certain semantic attributes change. Overall, by constructing a target loss function for dynamic object regions and using it as the optimization objective for training iterations, the inference accuracy of the finally trained deep prediction network in dynamic scenes is improved.

[0062] In some embodiments, the first probability matrix is ​​formed by arranging the first probability values ​​of each pixel in the image to be reconstructed as belonging to a dynamic object according to the position of the pixel; the second probability matrix is ​​formed by arranging the second probability values ​​of each pixel in the image to be reconstructed as belonging to a dynamic object according to the position of the pixel.

[0063] In step S440 above, the target probability matrix is ​​obtained by integrating the first probability matrix and the second probability matrix, including: performing a weighted sum operation on the elements at each position of the first probability matrix and the corresponding elements at the corresponding positions of the second probability matrix to obtain the values ​​of the elements at the corresponding positions of the target probability matrix; wherein, the preset weights corresponding to the first probability matrix and the second probability matrix are preset according to the segmentation accuracy of the semantic segmentation network, and the higher the segmentation accuracy of the semantic segmentation network, the higher the preset weight of the corresponding second probability matrix.

[0064] In some embodiments, the semantic segmentation network itself has a certain segmentation accuracy and does not participate in subsequent training iterations. For example, in subsequent step S340, only the depth prediction network and the pose prediction network are trained and iterated synchronously. In other embodiments, the semantic segmentation network is also trained and iterated synchronously, which corresponds to the scheme in subsequent step S340 where the depth prediction network, the pose prediction network, and the semantic segmentation network are trained and iterated synchronously.

[0065] Regardless of the scenario, the preset weights when integrating the first probability matrix and the second probability matrix can be adjusted based on the segmentation accuracy of the introduced semantic segmentation network itself: if the segmentation accuracy of the semantic segmentation network is high, the weights should be set higher; if the segmentation accuracy of the semantic segmentation network is low, the weights should be set lower.

[0066] Based on the steps S410 to S440 above, by adding a semantic segmentation mask to the geometric consistency self-discovery mask achieved by combining depth prediction and pose prediction, it helps to solve the problem of large edge noise when using geometric consistency to infer the corresponding dynamic and static pixels, and improves the recognition accuracy of edge parts. At the same time, since semantic segmentation itself has limitations, namely, it strongly depends on the dynamic or static attributes corresponding to the recognized semantic labels, in fact, the dynamic and static properties of objects in the image are not absolute in some cases. Therefore, by combining the geometric consistency self-discovery mask with the semantic segmentation mask, it also helps to improve the recognition accuracy in certain cases where semantic attributes change.

[0067] In some embodiments, the expressions for the elements in the target probability matrix corresponding to the states of the above-mentioned dynamic and static objects are as follows:

[0068]

[0069] Where U represents the probability value corresponding to each pixel in the target probability matrix; f(U) segment U self-mask This indicates that the probability value of each pixel in the target probability matrix is ​​related to U. segment and U self-mask All are related; U self-mask This represents the probability value corresponding to the geometric consistency self-discovery mask, which corresponds to an element in the first probability matrix; U segment represents the probability value corresponding to the semantic segmentation mask, which corresponds to the element in the second probability matrix; (1-β) represents the preset weight corresponding to the first probability matrix, and β represents the preset weight corresponding to the second probability matrix.

[0070] In step S330, based on the target probability matrix, the photometric reprojection loss function obtained by reconstructing the image to be reconstructed under another image viewpoint of the image pair is determined; the photometric reprojection loss function is used as the target loss function.

[0071] In some embodiments, the target loss function is obtained by performing binary masking on the target probability matrix corresponding to the states of the static and dynamic objects and then combining it with the reconstruction processing of image pairs.

[0072] The process of performing binary masking on the target probability matrix is ​​represented as follows:

[0073]

[0074] Where M represents the first mask result of the binary mask, used to determine the loss corresponding to the case where the pixel is more likely to be a dynamic object; N represents the second mask result of the binary mask, used to determine the loss corresponding to the case where the pixel is less likely to be a dynamic object; and γ represents the threshold of the binary mask.

[0075] The masking results M and N of the binary mask are used to extract dynamic and static features, respectively. γ is the threshold of the binary mask. To avoid interference from ambiguous features, different values ​​can be set for the thresholds of M and N. Dynamic features with higher probabilities are identified as dynamic features, those with lower probabilities are identified as static features, and those with ambiguous probabilities are not added to the loss function to reduce errors.

[0076] The objective loss function described above is expressed as follows:

[0077] L up =M⊙L p +θN⊙L p (4)

[0078] Among them, L up L represents the target loss function, which is the photometric reprojection loss function;p The base photometric reprojection loss measures the photometric consistency between the source image and the reconstructed image obtained through depth estimation and relative pose transformation from another image's perspective. θ represents the preset weight coefficients, which, as hyperparameters, can be adaptively adjusted according to different scenes to optimize the training results. For example, grayscale images can be used for the base photometric reprojection loss calculation.

[0079] With the second image I t As an example of the image to be reconstructed, the expression for the corresponding basic photometric reprojection loss is as follows:

[0080]

[0081] Among them, L p (D s ) represents the image I to be reconstructed t The corresponding basic photometric reprojection loss; SSIM represents the structural similarity index; I t Indicates the luminosity value of the target image; I s→t (D s ) is based on the first image I s Predicted depth D from the perspective s and relative camera pose T t→s The photometric values ​​corresponding to the reconstructed target image obtained after reprojection; ||I t -I s→t (D s )||1 represents I t -I s→t (D s The corresponding 1-norm represents the sum of the absolute values ​​of the vector elements; α is a preset coefficient that can be adjusted according to the actual scenario; for example, in this embodiment, α is set to 0.85.

[0082] In step S340, with the optimization objective of reducing the photometric reprojection loss function, the depth prediction network and the pose prediction network are simultaneously trained and iterated; or, the depth prediction network, the pose prediction network, and the semantic segmentation network are simultaneously trained and iterated.

[0083] In some embodiments, the semantic segmentation network is introduced only during the stage of constructing the dynamic uncertainty of pixels in the image. The semantic segmentation network may not participate in subsequent training iterations, thus reducing the training workload. In other embodiments, the semantic segmentation network, depth prediction network, and pose prediction network can also be trained and iterated simultaneously.

[0084] Based on the above steps S310 to S340, the dynamic and static probabilities of each pixel can be finely distinguished and considered during training. By combining the advantages of geometric consistency self-discovery mask and semantic segmentation mask and complementing each other's shortcomings, the inference accuracy of the trained deep prediction network in dynamic scenes can be improved.

[0085] In summary, in the embodiments including steps S210 to S220, by setting up a depth prediction network, a pose prediction network, and a semantic segmentation network, the dynamic and static object states corresponding to each pixel in the image to be reconstructed are comprehensively determined during training by combining a geometric consistency self-discovery mask and a semantic segmentation mask. This achieves a refined distinction and consideration of the dynamic and static probabilities of each pixel in the image to be reconstructed. Furthermore, combining the geometric consistency self-discovery mask with the semantic segmentation mask offers the comprehensive advantage of integrating advantages and compensating for the shortcomings of each. By adding a semantic segmentation mask on top of the geometric consistency self-discovery mask achieved through combining depth prediction and pose prediction, it helps to solve the problem of utilizing geometric consistency... This addresses the issue of significant edge noise corresponding to both dynamic and static pixels, improving the accuracy of edge recognition. Furthermore, since semantic segmentation itself has limitations—it heavily relies on the dynamic or static attributes corresponding to the recognized semantic labels—the dynamic and static properties of objects in an image are not always absolute. Therefore, combining a geometrically consistent self-discovery mask with a semantic segmentation mask can also help improve recognition accuracy in situations where certain semantic attributes change. Overall, by constructing a target loss function for dynamic object regions and using it as the optimization objective for training iterations, the inference accuracy of the final trained deep prediction network in dynamic scenes can be improved.

[0086] In some embodiments, the method further includes: constructing a geometric consistency loss function based on the relative pose output by the pose prediction network and the reprojection constraints in the geometric relationship; and performing a weighted sum operation on the target loss function for the dynamic object region and the geometric consistency loss function to obtain a comprehensive loss function.

[0087] During the aforementioned training iterations, the optimization objective is also to reduce the overall loss function.

[0088] Figure 5 The illustrations schematically show the results of (a) an initial image processed by a conventional method based on (b1) a self-masking process according to an embodiment of the present disclosure, and (b2) the results of processing by combining a geometric consistency self-discovery mask and a semantic segmentation mask provided by an embodiment of the present disclosure.

[0089] Reference Figure 5As shown in (a) and (b1), the results of self-masking processing based on traditional methods exhibit numerous burrs at the object edges, indicating the presence of noise in many edge regions. (Refer to...) Figure 5 As shown in (a) and (b2), the object recognition boundary is clear and the monocular depth can be well estimated in the result of processing by combining the geometric consistency self-discovery mask and the semantic segmentation mask provided by the disclosed embodiment, which proves the advantages of the method provided by the disclosed embodiment.

[0090] A second exemplary embodiment of this disclosure provides a training apparatus for a visual depth estimation model based on dynamic uncertainty.

[0091] Figure 6 A schematic block diagram of a training apparatus for a visual depth estimation model based on dynamic uncertainty according to an embodiment of the present disclosure is shown.

[0092] Reference Figure 6 As shown in the embodiments of this disclosure, the training device 600 for a visual depth estimation model based on dynamic uncertainty includes: a data acquisition module 610 and a training module 620.

[0093] The aforementioned data acquisition module 610 is used to acquire a training set, which contains multiple image pairs corresponding to time-series frames in a dynamic scene.

[0094] The training module 620 is used to input the training set into the visual depth estimation model to be trained for training iterations. The visual depth estimation model includes a depth prediction network, a pose prediction network, and a semantic segmentation network. During training, the dynamic and static object states corresponding to each pixel in the image to be reconstructed are determined by combining the geometric consistency self-discovery mask and the semantic segmentation mask. The image to be reconstructed is one of the images selected from the image pair. Based on the dynamic and static object states and the reconstruction processing of the image pair, a target loss function is determined for the dynamic object region. During the training iterations, reducing the target loss function is the optimization objective. The trained depth prediction network is used for monocular visual depth estimation.

[0095] In some embodiments, the training module is further configured to: construct a geometric consistency loss function based on the relative pose output by the pose prediction network and the reprojection constraints in the geometric relationship; and perform a weighted sum operation on the target loss function for the dynamic object region and the geometric consistency loss function to obtain a comprehensive loss function. During the training iterations, reducing the comprehensive loss function is also simultaneously optimized.

[0096] The training device provided in this embodiment, by setting up a depth prediction network, a pose prediction network, and a semantic segmentation network, determines the dynamic and static object states corresponding to each pixel in the image to be reconstructed during training by combining a geometric consistency self-discovery mask and a semantic segmentation mask. This achieves refined differentiation and consideration of the dynamic and static probabilities of each pixel in the image to be reconstructed. Furthermore, combining the geometric consistency self-discovery mask and the semantic segmentation mask offers the comprehensive advantage of integrating their strengths and compensating for their respective weaknesses. By adding a semantic segmentation mask to the geometric consistency self-discovery mask achieved through depth prediction and pose prediction, it helps solve the problem of inferring dynamic and static images using geometric consistency. To address the issue of high edge noise corresponding to prime points, this method improves the accuracy of edge recognition. Simultaneously, since semantic segmentation itself has limitations—it heavily relies on the dynamic or static attributes corresponding to the recognized semantic labels—the dynamics and statics of objects in an image are not absolute in some cases. Therefore, combining a geometrically consistent self-discovery mask with the semantic segmentation mask also helps improve recognition accuracy in situations where certain semantic attributes change. Overall, by constructing a target loss function for dynamic object regions and using it as the optimization objective for training iterations, the inference accuracy of the final trained deep prediction network in dynamic scenes can be improved.

[0097] For more details of this embodiment, please refer to the relevant description of the first embodiment, which will not be repeated here.

[0098] Any plurality of the functional modules included in the training device 600 may be combined into one module, or any one of the modules may be split into multiple modules. Alternatively, at least a portion of the functionality of one or more of these modules may be combined with at least a portion of the functionality of other modules and implemented in one module. At least one of the functional modules included in the training device 600 may be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any appropriate combination of any of these three implementation methods. Alternatively, at least one of the functional modules included in the training device 600 may be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0099] A third exemplary embodiment of this disclosure provides an electronic device.

[0100] Figure 7 A schematic block diagram of an electronic device provided in an embodiment of the present disclosure is shown.

[0101] Reference Figure 7 As shown, the electronic device 700 provided in this embodiment includes a processor 701, a communication interface 702, a memory 703, and a communication bus 704. The processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704. The memory 703 is used to store computer programs. When the processor 701 executes the program stored in the memory, it implements the training method of the visual depth estimation model based on dynamic uncertainty as described above.

[0102] A fourth exemplary embodiment of this disclosure also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program that, when executed by a processor, implements the training method for a visual depth estimation model based on dynamic uncertainty as described above.

[0103] The computer-readable storage medium may be included in the device or apparatus described in the above embodiments; or it may exist independently and not assembled into the device or apparatus. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0104] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0105] It should be noted that the collection, gathering, updating, analysis, processing, use, transmission, and storage of user personal information involved in the technical solutions provided in this disclosure comply with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.

[0106] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0107] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for training a visual depth estimation model based on dynamic uncertainty, characterized in that, The method comprises the following steps: acquiring a training set containing multiple groups of image pairs corresponding to time-series frames in a dynamic scene; inputting the training set into a visual depth estimation model to be trained for training iteration, wherein the visual depth estimation model comprises a depth prediction network, a pose prediction network and a semantic segmentation network; during the training, the dynamic and static object states corresponding to each pixel point in a to-be-reconstructed image are determined by combining a geometric consistency self-discovered mask and a semantic segmentation mask; the to-be-reconstructed image is one selected from the image pairs; determining a target loss function for a dynamic object region according to the dynamic and static object states and reconstruction processing of the image pairs; wherein, during the training iteration, the target loss function is taken as an optimization objective to reduce the target loss function, and the trained depth prediction network is used for monocular visual depth estimation; wherein, the expression of an element in a target probability matrix corresponding to the dynamic and static object states is as follows: ， wherein, denotes a probability value corresponding to each pixel point in the target probability matrix; denotes that the probability values of each pixel point in the target probability matrix are related to and ; denotes a probability value corresponding to the geometric consistency self-discovery mask, corresponding to an element in the first probability matrix; denotes a probability value corresponding to the semantic segmentation mask, corresponding to an element in the second probability matrix; denotes a preset weight corresponding to the first probability matrix, denotes a preset weight corresponding to the second probability matrix; the target loss function is obtained by performing binary mask processing on the target probability matrix corresponding to the dynamic and static object states and combining the reconstruction processing of the image pairs; the process of performing binary mask processing on the target probability matrix is as follows: ， ， wherein, denotes a first mask result of a binary mask, used to determine a loss corresponding to a case that a pixel is more likely to be a dynamic object; denotes a second mask result of a binary mask, used to determine a loss corresponding to a case that a pixel is less likely to be a dynamic object; denotes a threshold value of a binary mask. the target loss function is as follows: ， wherein, represents a target loss function, which is a photometric reprojection loss function; represents a basic photometric reprojection loss, which is used to measure photometric consistency between a source image of a to-be-reconstructed image and a reconstructed image obtained by depth estimation and another image view angle relative pose transformation reconstruction; represents a preset weight coefficient.

2. The method of claim 1, wherein, inputting the training set into the visual depth estimation model to be trained for training iteration, comprising: inputting each group of image pairs in the training set into the depth prediction network, the pose prediction network and the semantic segmentation network respectively for depth prediction processing, pose prediction processing and semantic segmentation processing to obtain a predicted depth, a relative pose and a semantic label corresponding to a semantic segmentation region; determining a target probability matrix of each pixel point in the to-be-reconstructed image belonging to a dynamic object according to the predicted depth, the relative pose and the semantic label; determining a photometric re-projection loss function of the to-be-reconstructed image reconstructed under another image perspective of the image pairs according to the target probability matrix; the photometric re-projection loss function is taken as the target loss function; taking the photometric re-projection loss function as an optimization objective to reduce the photometric re-projection loss function, and performing synchronous training iteration on the depth prediction network and the pose prediction network; or performing synchronous training iteration on the depth prediction network, the pose prediction network and the semantic segmentation network.

3. The method of claim 2, wherein, determining a target probability matrix of each pixel point in the to-be-reconstructed image belonging to a dynamic object according to the predicted depth, the relative pose and the semantic label, comprising: determining a depth prediction difference of a same pixel point in the two images of the image pairs according to the relative pose and the predicted depth of the image pairs; determining a first probability matrix of each pixel point in the to-be-reconstructed image belonging to a dynamic object according to the depth prediction difference; determining a second probability matrix of each pixel point belonging to a dynamic object according to the semantic label corresponding to the semantic segmentation region and a preset dynamic and static attribute corresponding to the semantic label; integrating the first probability matrix and the second probability matrix to obtain the target probability matrix.

4. The method of claim 3, wherein, The first probability matrix is formed by arranging the first probability values of each pixel point in the to-be-reconstructed image belonging to a dynamic object according to the positions of the pixel points; and the second probability matrix is formed by arranging the second probability values of each pixel point in the to-be-reconstructed image belonging to a dynamic object according to the positions of the pixel points. The first probability matrix and the second probability matrix are integrated to obtain a target probability matrix, including: The elements at each position of the first probability matrix and the elements at the corresponding positions of the second probability matrix are subjected to weighted sum operation to obtain the element values of the target probability matrix at the corresponding positions; wherein the preset weights corresponding to the first probability matrix and the second probability matrix are preset according to the segmentation accuracy of the semantic segmentation network, and the higher the segmentation accuracy of the semantic segmentation network, the higher the preset weight corresponding to the second probability matrix.

5. The method of claim 1, wherein, Further comprising: A geometric consistency loss function is constructed according to the relative pose output by the pose prediction network and the re-projection constraint in the geometric relationship; The target loss function for the dynamic object region and the geometric consistency loss function are subjected to weighted sum operation to obtain a comprehensive loss function; During the training iteration, the comprehensive loss function is also simultaneously reduced as an optimization target. 6.A device for training a visual depth estimation model based on dynamic uncertainty, characterized in that, The data acquisition module is configured to acquire a training set, the training set containing multiple groups of image pairs corresponding to time-series frames in a dynamic scene; The training module is configured to input the training set to a visual depth estimation model to be trained for training iteration, the visual depth estimation model including a depth prediction network, a pose prediction network, and a semantic segmentation network; during the training, the dynamic and static object states corresponding to each pixel point in a to-be-reconstructed image are determined by combining the geometric consistency mask and the semantic segmentation mask; the to-be-reconstructed image is one of the image pairs; A target loss function for a dynamic object region is determined according to the dynamic and static object states and the reconstruction processing of the image pairs; During the training iteration, the target loss function is reduced as an optimization target, and the trained depth prediction network is used for monocular visual depth estimation; An expression of an element in the target probability matrix corresponding to the dynamic and static object states is as follows: The target loss function is obtained by performing binary mask processing on the target probability matrix corresponding to the dynamic and static object states and then combining the reconstruction processing of the image pairs; ， wherein, denotes a probability value corresponding to each pixel point in the target probability matrix; denotes that the probability values of each pixel point in the target probability matrix are related to and denotes a probability value corresponding to the geometric consistency self-discovery mask, corresponding to an element in the first probability matrix; denotes a probability value corresponding to the semantic segmentation mask, corresponding to an element in the second probability matrix; denotes a preset weight corresponding to the first probability matrix, denotes a preset weight corresponding to the second probability matrix;​ A process of performing binary mask processing on the target probability matrix is as follows: The target loss function is represented as follows: ， ， wherein, represents a first mask result of a binary mask, used to determine a loss corresponding to a case that a pixel is more likely to be a dynamic object; represents a second mask result of a binary mask, used to determine a loss corresponding to a case that a pixel is less likely to be a dynamic object; represents a threshold value of a binary mask; The device includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus. ， wherein, represents a target loss function, which is a photometric reprojection loss function; represents a basic photometric reprojection loss, which is used to measure photometric consistency between a source image of a to-be-reconstructed image and a reconstructed image obtained by depth estimation and another image view angle relative pose transformation reconstruction; represents a preset weight coefficient.

7. An electronic device, comprising: The memory is configured to store a computer program. The processor is configured to execute the program stored in the memory to implement the method of any one of claims 1-5. The computer program is executed by the processor to implement the method of any one of claims 1-5.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 1-5.

Images