Dynamic uncertainty-based visual depth estimation model training method and device

Abstract

The invention relates to a visual depth estimation model training method based on dynamic uncertainty, and the method comprises the steps: obtaining a training set which comprises a plurality of groups of image pairs corresponding to a time sequence frame in a dynamic scene; inputting the training set into a visual depth estimation model to be trained for training iteration, wherein the visual depth estimation model comprises a depth prediction network, a pose prediction network and a semantic segmentation network; during the training period, comprehensively determining a dynamic and static object state corresponding to each pixel point in the to-be-reconstructed image by combining the geometric consistency self-discovery mask and the semantic segmentation mask; the to-be-reconstructed image is an image selected from the image pair; and determining a target loss function aiming at the dynamic object region according to the dynamic and static object states in combination with reconstruction processing of the image pair. Wherein in the training iteration period, reduction of a target loss function is taken as an optimization target, the trained depth prediction network is used for monocular vision depth estimation, and the reasoning accuracy in a dynamic scene is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of machine vision technology, and in particular to a training method and device for a visual depth estimation model based on dynamic uncertainty. Background Art

[0002] In the field of computer vision, vision-based depth estimation (especially monocular depth estimation) has become increasingly important due to its ability to understand the three-dimensional geometry of a scene from two-dimensional observations and recover scene depth information. Classic depth estimation algorithms or training methods generally require obtaining ground truth depth values as training labels for supervised training. However, obtaining ground truth depth values is difficult and the labeling cost is high. A new trend in the industry is to use self-supervised deep learning to perform depth estimation, eliminating the need for ground truth depth values.

[0003] In the process of realizing the concept of the present disclosure, the inventors found that there are at least the following technical problems in the related art: in the algorithms or solutions based on self-supervised deep learning, the training data are basically in static scenes. However, in actual scenes, most of the tracking or depth estimation of dynamic targets in dynamic scenes is carried out. Since the dynamic points violate the epipolar geometry constraints, the reconstruction process of the dynamic area part is abnormal, resulting in a deterioration in the self-supervised learning effect of depth estimation. Therefore, the processing methods in the related art are mostly based on some detection methods to determine the area where the dynamic points are located and eliminate these dynamic points, and use the image data after eliminating the dynamic points for training. Although this can reduce the damage caused by some dynamic point noise during training, since these dynamic areas are not fully regularized during training, it also causes the problem of poor inference results of the trained model for dynamic areas. Therefore, how to improve the depth prediction effect of the depth estimation model for images containing dynamic object pixels is a technical problem that needs to be solved urgently. Summary of the Invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the embodiments of the present disclosure provide a training method and device for a visual depth estimation model based on dynamic uncertainty.

[0005] In the first aspect, an embodiment of the present disclosure provides a training method for a visual depth estimation model based on dynamic uncertainty. The above method includes: obtaining a training set, the above training set includes multiple groups of image pairs corresponding to time series frames in a dynamic scene; inputting the above training set into the visual depth estimation model to be trained for training iteration, the above visual depth estimation model includes: a depth prediction network, a pose prediction network and a semantic segmentation network; during training, the dynamic and static object states corresponding to each pixel in the image to be reconstructed are comprehensively determined by combining the geometric consistency self-discovery mask and the semantic segmentation mask; the above image to be reconstructed is an image selected from the above image pair; according to the above dynamic and static object states, the target loss function for the dynamic object area is determined in combination with the reconstruction processing of the image pair; wherein, during the above training iteration, the optimization goal is to reduce the above target loss function, and the trained depth prediction network is used for monocular visual depth estimation.

[0006] In some embodiments, the training set is input into the visual depth estimation model to be trained for training iterations, including: inputting each group of image pairs in the training set into the depth prediction network, the pose prediction network and the semantic segmentation network for depth prediction processing, pose prediction processing and semantic segmentation processing respectively, to obtain the predicted depth, relative pose and semantic labels corresponding to the semantic segmentation area; according to the predicted depth, the relative pose and the semantic labels, determining the target probability matrix of each pixel point of the image to be reconstructed belonging to a dynamic object; according to the target probability matrix, determining the photometric reprojection loss function obtained by reconstructing the image to be reconstructed from the perspective of another image in the image pair; the photometric reprojection loss function is used as the target loss function; with the reduction of the photometric reprojection loss function as the optimization goal, the depth prediction network and the pose prediction network are trained iteratively synchronously; or, the depth prediction network, the pose prediction network and the semantic segmentation network are trained iteratively synchronously.

[0007] In some embodiments, based on the above-mentioned predicted depth, relative pose and the above-mentioned semantic label, a target probability matrix is determined that each pixel point of the image to be reconstructed belongs to a dynamic object, including: determining the depth prediction difference corresponding to the same pixel point in the two images of the above-mentioned image pair based on the relative pose and predicted depth of the image pair; determining a first probability matrix that each pixel point in the image to be reconstructed belongs to a dynamic object based on the above-mentioned depth prediction difference; determining a second probability matrix that each pixel point belongs to a dynamic object based on the semantic label corresponding to the semantic segmentation area and the preset dynamic and static attributes corresponding to the semantic label; and integrating the above-mentioned first probability matrix and the above-mentioned second probability matrix to obtain a target probability matrix.

[0008] In some embodiments, the first probability matrix is composed of the first probability values of each pixel in the image to be reconstructed that belong to a dynamic object, arranged according to the position of the pixel; the second probability matrix is composed of the second probability values of each pixel in the image to be reconstructed that belong to a dynamic object, arranged according to the position of the pixel. According to the integration of the first probability matrix and the second probability matrix, a target probability matrix is obtained, including: performing weighted sum operations on the elements at each position on the first probability matrix and the elements at the corresponding position on the second probability matrix, to obtain the element values of the target probability matrix at the corresponding position; wherein the preset weights corresponding to the first probability matrix and the second probability matrix are preset according to the segmentation accuracy of the semantic segmentation network. The higher the segmentation accuracy of the semantic segmentation network, the higher the preset weight of the corresponding second probability matrix.

[0009] In some embodiments, the expression of the elements in the target probability matrix corresponding to the above-mentioned dynamic and static object states is as follows:

[0010]

[0011] Among them, U represents the probability value corresponding to each pixel in the target probability matrix; f(U segment ,U self-mask ) indicates that the probability value of each pixel in the target probability matrix is segment and U self-mask All related; U self-mask represents the probability value corresponding to the geometric consistency self-discovery mask, corresponding to the element in the first probability matrix; U segment represents the probability value corresponding to the semantic segmentation mask, corresponding to the element in the second probability matrix; (1-β) represents the preset weight corresponding to the first probability matrix, and β represents the preset weight corresponding to the second probability matrix.

[0012] In some embodiments, the target loss function is obtained by performing binary masking on the target probability matrix corresponding to the dynamic and static object states and then combining it with the reconstruction process of the image pair;

[0013] The process of binary masking the target probability matrix is expressed as follows:

[0014]

[0015] Where M represents the first mask result of the binary mask, which is used to determine the loss corresponding to the case where the probability of the pixel being a dynamic object is higher; N represents the second mask result of the binary mask, which is used to determine the loss corresponding to the case where the probability of the pixel being a dynamic object is lower; γ represents the threshold of the binary mask;

[0016] The above objective loss function is expressed as follows:

[0017] L up =M⊙L p +θN⊙L p ,

[0018] Among them, L up Represents the target loss function, which is the photometric reprojection loss function; L p represents the basic photometric reprojection loss, which is used to measure the photometric consistency between the source image to be reconstructed and the reconstructed image obtained by depth estimation and relative pose transformation from the perspective of another image; θ represents the preset weight coefficient.

[0019] In some embodiments, the above method also includes: constructing a geometric consistency loss function based on the relative pose output by the pose prediction network combined with the reprojection constraints in the geometric relationship; performing weighted sum operations on the above target loss function for the dynamic object area and the above geometric consistency loss function to obtain a comprehensive loss function; wherein, during the above training iteration, reducing the above comprehensive loss function is also an optimization goal.

[0020] In the second aspect, an embodiment of the present disclosure provides a training device for a visual depth estimation model based on dynamic uncertainty. The above-mentioned device includes: a data acquisition module and a training module. The above-mentioned data acquisition module is used to obtain a training set, and the above-mentioned training set includes multiple groups of image pairs corresponding to time series frames in dynamic scenes. The above-mentioned training module is used to input the above-mentioned training set into the visual depth estimation model to be trained for training iteration, and the above-mentioned visual depth estimation model includes: a depth prediction network, a pose prediction network and a semantic segmentation network; during the training, the dynamic and static object states corresponding to each pixel point in the image to be reconstructed are comprehensively determined by combining the geometric consistency self-discovery mask and the semantic segmentation mask; the above-mentioned image to be reconstructed is an image selected from the above-mentioned image pair; the target loss function for the dynamic object area is determined according to the above-mentioned dynamic and static object states combined with the reconstruction processing of the image pair; wherein, during the above-mentioned training iteration, the reduction of the above-mentioned target loss function is the optimization goal, and the trained depth prediction network is used for monocular visual depth estimation.

[0021] In a third aspect, embodiments of the present disclosure provide an electronic device. The electronic device includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus; the memory is configured to store a computer program; and the processor is configured to implement the above-described training method for a visual depth estimation model based on dynamic uncertainty when executing the program stored in the memory.

[0022] In a fourth aspect, embodiments of the present disclosure provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method for training a visual depth estimation model based on dynamic uncertainty.

[0023] The above technical solutions provided by the embodiments of the present disclosure have at least some or all of the following advantages:

[0024] By setting up a depth prediction network, a pose prediction network, and a semantic segmentation network, the dynamic and static object states corresponding to each pixel in the reconstructed image are comprehensively determined during training by combining geometric consistency self-discovery masks and semantic segmentation masks. This achieves a refined distinction and consideration of the dynamic and static probabilities of each pixel in the reconstructed image, and combines the geometric consistency self-discovery mask with the semantic segmentation mask, which has the comprehensive advantage of integrating the advantages and complementing their respective shortcomings. By adding a semantic segmentation mask to the geometric consistency self-discovery mask achieved by combining depth prediction and pose prediction, it helps to solve the problem of large edge noise corresponding to dynamic and static pixels using geometric consistency inference, and improves the recognition accuracy of edge parts. At the same time, due to the limitations of semantic segmentation itself, it is strongly dependent on the dynamic or static attributes corresponding to the recognized semantic labels. In fact, in some cases, the dynamic and static nature of objects in the image are not absolute. Therefore, by combining the geometric consistency self-discovery mask with the semantic segmentation mask, it also helps to improve the recognition accuracy in cases where certain semantic attributes change. Overall, by constructing a target loss function for dynamic object regions and using it as the optimization target for training iterations, the reasoning accuracy of the final trained depth prediction network in dynamic scenes is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0026] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0027] Figure 1 A system architecture diagram schematically illustrates a method for training a visual depth estimation model based on dynamic uncertainty according to an embodiment of the present disclosure;

[0028] Figure 2 The flowchart of the training method of the visual depth estimation model based on dynamic uncertainty according to an embodiment of the present disclosure is schematically shown;

[0029] Figure 3 The following schematically shows a detailed implementation flow chart of step S220 of an embodiment of the present disclosure;

[0030] Figure 4 The following schematically shows a detailed implementation flow chart of step S320 of an embodiment of the present disclosure;

[0031] Figure 5 Schematically illustrating the results of (a) an initial image after (b1) performing self-masking processing based on a traditional method and (b2) performing processing based on a combination of geometric consistency self-discovery masking and semantic segmentation masking provided by an embodiment of the present disclosure, respectively, according to an embodiment of the present disclosure;

[0032] Figure 6 Schematically shows a structural block diagram of a training device for a visual depth estimation model based on dynamic uncertainty according to an embodiment of the present disclosure; and

[0033] Figure 7 The structural block diagram of the electronic device provided by the embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0034] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0035] A first exemplary embodiment of the present disclosure provides a training method for a visual depth estimation model based on dynamic uncertainty.

[0036] Figure 1 A system architecture diagram schematically illustrates a method for training a visual depth estimation model based on dynamic uncertainty according to an embodiment of the present disclosure; Figure 2 A flowchart of a method for training a visual depth estimation model based on dynamic uncertainty according to an embodiment of the present disclosure is schematically shown.

[0037] Combine Figure 1 and Figure 2 As shown, the visual depth estimation model to be trained includes: a depth prediction network (Depth net) 110, a pose prediction network (Pose net) 120 and a semantic segmentation network (Seg net) 130. The training method of the visual depth estimation model based on dynamic uncertainty includes the following steps: S210 and S220.

[0038] In step S210 , a training set is obtained, where the training set includes multiple image pairs corresponding to time-series frames in a dynamic scene.

[0039] For example, Figure 1 A set of image pairs is exemplified in FIG: the first image I s and the second image I t , the first image and the second image are two time-series frames in a dynamic scene. The above-mentioned dynamic scene can be a scene corresponding to the video data captured by the vehicle camera during the driving process, or a scene corresponding to the video data captured by the roadside equipment. There are dynamically moving objects and stationary objects in the video data. The video data is divided into multiple time-series frames according to different moments, and the two time-series frames corresponding to the adjacent moments constitute a group of image pairs. In the two images of this image pair, the dynamically moving object is called a dynamic object, such as a moving vehicle, a walking pedestrian, etc. If the pixel points in a certain area in the image correspond to a dynamic object, then the area is called a dynamic area. Correspondingly, in the image pair, the stationary object is called a static object. If the pixel points in a certain area correspond to a static object, then the area is called a static area.

[0040] In the embodiment of the present disclosure, during the training iteration of the visual depth estimation model, the dynamic areas in the training data are not eliminated. Instead, the dynamic and static object states corresponding to each pixel in the image to be reconstructed are comprehensively determined by combining the geometric consistency self-discovery mask and the semantic segmentation mask. Based on this, the dynamic uncertainty corresponding to each pixel is obtained, that is, the probability matrix of whether each pixel is a dynamic object; and the target loss function corresponding to the dynamic area is constructed based on this dynamic uncertainty. This achieves a refined distinction and consideration of the dynamic and static probabilities of each pixel during training, and organically combines the geometric consistency self-discovery mask with the semantic segmentation mask, which has the comprehensive advantage of integrating advantages and supplementing their respective shortcomings, so that the reasoning accuracy of the depth prediction network obtained by training in dynamic scenes is improved.

[0041] In step S220, the training set is input into the visual depth estimation model to be trained for training iterations, wherein the visual depth estimation model comprises: a depth prediction network, a pose prediction network and a semantic segmentation network; during the training, the dynamic and static object states corresponding to each pixel in the image to be reconstructed are comprehensively determined by combining the geometric consistency self-discovery mask and the semantic segmentation mask; the image to be reconstructed is an image selected from the image pair; the target loss function for the dynamic object area is determined according to the dynamic and static object states combined with the reconstruction processing of the image pair; wherein, during the training iteration, reducing the target loss function is the optimization goal, and the trained depth prediction network is used for monocular visual depth estimation.

[0042] Figure 3 The detailed implementation flow chart of step S220 in one embodiment of the present disclosure is schematically shown.

[0043] In some embodiments, reference Figure 1 and Figure 3 As shown, in the above step S220, the above training set is input into the visual depth estimation model to be trained for training iteration, including the following steps: S310, S320, S330 and S340.

[0044] In step S310, each image pair in the training set (in the form of image pairs {I s , I t} as an example) is input to the depth prediction network (Depth net) 110, the pose prediction network (Pose net) 120 and the semantic segmentation network (Seg net) 130 for depth prediction processing, pose prediction processing and semantic segmentation processing respectively, and the predicted depth Ds and Dt, the relative pose P st Semantic label Seg corresponding to the semantic segmentation area s .

[0045] For example, the image pair {I s , I t} Input to the depth prediction network (Depth net) 110 for depth prediction processing to obtain the first image I s The corresponding predicted depth Ds, the second image I t The corresponding predicted depth Dt; the image pair {I s , I t After splicing, the image is input to the pose prediction network (Pose net) 120 for pose prediction processing to obtain the first image I s With the second image I t The relative pose P between st ; Match the image to {I s , I t} Input to the semantic segmentation network (Seg net) 130 for semantic segmentation processing to obtain the first image I s and the second image I t The semantic segmentation results of each image are that the entire image is divided into multiple semantic segmentation regions, and each semantic segmentation region has a corresponding semantic label. In the embodiment of the present disclosure, since the subsequent construction of the target loss function is for the image to be reconstructed, that is, any one image in the image pair is selected as the image to be reconstructed, in this step S310, semantic segmentation can also be performed only on the image to be reconstructed in the image pair, with the image to be reconstructed as the first image I s As an example, the image to be reconstructed I is obtained sThe corresponding semantic segmentation area and its corresponding semantic label Seg s .

[0046] Reference Figure 1 As shown, the first image I s As an example of an image to be reconstructed, some semantic segmentation areas in the image correspond to the semantic label: car; some semantic segmentation areas correspond to the semantic label: tree; some semantic segmentation areas correspond to the semantic label: ground; some semantic segmentation areas correspond to the semantic label: signboard; some semantic segmentation areas correspond to the semantic label: lawn; and some semantic segmentation areas correspond to the semantic label: telephone pole.

[0047] In step S320, a target probability matrix is determined for each pixel of the image to be reconstructed belonging to a dynamic object based on the predicted depth, the relative pose, and the semantic label.

[0048] In some embodiments, reference Figure 1 As shown, for a set of input image pairs {I s , I t}, according to the image, after being processed by the depth prediction network (Depth net) 110, the pose prediction network (Pose net) 120 and the semantic segmentation network (Seg net) 130, the corresponding predicted depth Ds and Dt, the relative pose P st Semantic label Seg corresponding to the semantic segmentation area s , according to the above predicted depth Ds, Dt and relative posture P stPerforming geometric consistency self-discovery mask processing, obtains the dynamic and static object state corresponding to each pixel in the image to be reconstructed under one consideration dimension, that is, the dynamic uncertainty corresponding to each pixel. In this embodiment, a first probability matrix is used to represent the dynamic uncertainty of the pixels in the image processed by the geometric consistency self-discovery mask. At the same time, semantic segmentation mask processing can also be performed according to the semantic labels corresponding to the above-mentioned semantic segmentation areas, to obtain the dynamic and static object state corresponding to each pixel in the image to be reconstructed under another consideration dimension, corresponding to the dynamic uncertainty of each pixel. In this embodiment, a second probability matrix is used to represent the dynamic uncertainty of the pixels in the image processed by the semantic segmentation mask. By comprehensively considering the above-mentioned two dimensions, combining the geometric consistency self-discovery mask and the semantic segmentation mask to comprehensively determine the dynamic and static object state corresponding to each pixel in the image to be reconstructed (that is, the dynamic uncertainty of each pixel), and determining the target loss function for the dynamic object area based on the above-mentioned dynamic and static object state combined with the reconstruction processing of the image pair, so that the dynamic and static probability of each pixel can be refined and considered during training, and the advantages of the geometric consistency self-discovery mask and the semantic segmentation mask are combined and their shortcomings are complemented, so that the reasoning accuracy of the trained deep prediction network in dynamic scenes is improved.

[0049] Figure 4 The detailed implementation flow chart of step S320 in one embodiment of the present disclosure is schematically shown.

[0050] In some embodiments, combined Figure 1 and Figure 4 As shown, in the above step S320, based on the above predicted depth, relative pose and the above semantic label, the target probability matrix of each pixel point of the image to be reconstructed belonging to the dynamic object is determined, including the following steps: S410, S420, S430 and S440.

[0051] In step S410, a depth prediction difference of a same pixel point in two images of the image pair is determined based on the relative pose and predicted depth of the image pair.

[0052] In step S420, a first probability matrix is determined for each pixel in the to-be-reconstructed image to belong to a dynamic object based on the depth prediction difference.

[0053] In the above steps S410 and S420, the predicted depths established by the depth prediction network and the pose prediction network are inconsistent. This is essentially a depth map reconstruction process. The target image I is reconstructed by estimating the relative pose between frames. t Source image under viewing angle I s Reconstructing the depth map, normally, is done by using the target image I t Source image I obtained by perspective prediction sReconstructed depth map and source image I s The depth map should remain consistent. However, since this assumption is based on static scenes, if there are dynamic objects in the image, there will be inconsistencies between the predicted depth and the reconstructed depth for the dynamic objects. This inconsistency can be calculated using a simple formula: (difference between predicted depth and reconstructed depth) / (product of predicted depth and reconstructed depth). Therefore, for static objects, the smaller the difference between the predicted depth and the reconstructed depth, the smaller the depth inconsistency. For dynamic objects, the larger the difference between the predicted depth and the reconstructed depth, the larger the corresponding depth inconsistency. Therefore, this depth inconsistency can be used to determine pixel-level dynamic features, that is, the probability of a pixel belonging to a dynamic object can be determined based on this depth prediction difference. Generally speaking, the greater the depth inconsistency, the higher the probability that the corresponding pixel belongs to a dynamic object. The probability value ranges from 0 to 1 and can take endpoint values.

[0054] Steps S410 to S420 correspond to the process of obtaining the dynamic uncertainty of pixels in the image using a geometrically consistent auto-discovery mask. During this process, the values corresponding to the dynamic uncertainty of the auto-discovery mask are the elements of the first probability matrix. This process is then subjected to a threshold filtering mask, which filters out pixels below the threshold and focuses on pixels with dynamic features (pixels with a high probability of belonging to a dynamic object). The first probability matrix can also be described as a dynamic uncertainty map for the auto-discovery mask.

[0055] In step S430 , a second probability matrix is determined for each pixel point belonging to a dynamic object based on the semantic label corresponding to the semantic segmentation region and the preset dynamic and static attributes corresponding to the semantic label.

[0056] Since each semantic label is pre-set with a corresponding static or dynamic attribute (that is, preset dynamic and static attributes), for example, vehicles are generally assigned dynamic attributes, people are assigned dynamic attributes, roads are assigned static attributes, signboards are assigned static attributes, etc., therefore, according to the preset dynamic and static attributes corresponding to each semantic label, the corresponding attributes of the semantic label of each pixel point can be determined, and probability values can be assigned according to the corresponding attributes, for example, a value of 1 is assigned to the dynamic attribute and a value of 0 is assigned to the static attribute; or, a value of 0.9 is assigned to the dynamic attribute and a value of 0.1 is assigned to the static attribute; this is not a limitation; in this way, a second probability matrix is obtained that each pixel point belongs to a dynamic object from the perspective of semantic segmentation, and the probability value at each position can be described as a dynamic uncertainty map, or as a second probability matrix.

[0057] In some embodiments, when assigning values, the difference between the edge position and the internal position of the segmented area is also considered to assign probabilities of different sizes. For example, for the same semantic segmentation area, if the area corresponds to a dynamic attribute, the probability assigned to the edge position of the dynamic area is smaller than the probability assigned to the internal position of the dynamic area. For example, the edge pixel of the dynamic area is assigned a value of 0.7, and the internal position pixel of the dynamic area is assigned a value of 0.9. This processing helps the trained depth prediction network to more clearly distinguish the boundaries of different objects in dynamic scenes.

[0058] Dynamic uncertainty is assigned to each pixel in the segmented image based on its regional location. Uncertainty is high at the edge of an object, and low near the center. When processing images using semantic segmentation, the uncertainty of whether a pixel is a dynamic feature differs between the segmentation boundary and the internal region. Greater uncertainty indicates a lower probability of a dynamic feature. The concept of a dynamic uncertainty map is introduced to enhance visualization and to demonstrate that for dynamic objects, the dynamic uncertainty of edge pixels is inconsistent with that of internal pixels, indicating the presence of noise.

[0059] The above step S430 corresponds to the process of obtaining the dynamic uncertainty of the pixel points in the image based on semantic segmentation mask processing. In this process, the value corresponding to the dynamic uncertainty of the semantic segmentation mask is the element in the second probability matrix. At the same time, after threshold filtering mask processing, it focuses on the pixel points of dynamic features.

[0060] In step S440, the first probability matrix and the second probability matrix are integrated to obtain a target probability matrix.

[0061] By adding a semantic segmentation mask on the basis of the geometric consistency self-discovery mask achieved by combining depth prediction and pose prediction, it helps to solve the problem of large edge noise corresponding to dynamic and static pixels inferred by geometric consistency, and improves the recognition accuracy of the edge part; at the same time, due to the limitations of semantic segmentation itself, it is strongly dependent on the dynamic or static attributes corresponding to the recognized semantic label. In fact, in some cases, the dynamic and static of objects in the image are not absolute. Therefore, by combining the geometric consistency self-discovery mask on the basis of the semantic segmentation mask, it also helps to improve the recognition accuracy in cases where certain semantic attributes change; overall, by constructing a target loss function for dynamic object areas and using it as the optimization target for training iterations, the reasoning accuracy of the depth prediction network finally trained in dynamic scenes can be improved.

[0062] In some embodiments, the above-mentioned first probability matrix is composed of the first probability values of each pixel point in the image to be reconstructed that belong to a dynamic object, arranged according to the position of the pixel point; the above-mentioned second probability matrix is composed of the second probability values of each pixel point in the image to be reconstructed that belong to a dynamic object, arranged according to the position of the pixel point.

[0063] In the above step S440, the target probability matrix is obtained by integrating the above first probability matrix and the above second probability matrix, including: performing weighted sum operations on the elements at each position on the above first probability matrix and the elements at the corresponding positions on the above second probability matrix to obtain the element values of the target probability matrix at the corresponding positions; wherein, the preset weights corresponding to the first probability matrix and the second probability matrix are preset according to the segmentation accuracy of the above semantic segmentation network. The higher the segmentation accuracy of the above semantic segmentation network, the higher the preset weight of the corresponding second probability matrix.

[0064] In some embodiments, the semantic segmentation network itself has a certain segmentation accuracy and the semantic segmentation network does not participate in subsequent training iterations. For example, in the subsequent step S340, only the depth prediction network and the pose prediction network are synchronously trained and iterated. In other embodiments, the semantic segmentation network is also trained and iterated synchronously, that is, the depth prediction network, the pose prediction network and the semantic segmentation network are synchronously trained and iterated corresponding to the subsequent step S340.

[0065] Regardless of the above scenario, the preset weights when integrating the first probability matrix and the second probability matrix can be adjusted according to the segmentation accuracy of the introduced semantic segmentation network itself: if the segmentation accuracy of the semantic segmentation network is higher, the weights are set higher; if the segmentation accuracy of the semantic segmentation network is lower, the weights are set lower.

[0066] Based on the above steps S410 to S440, by adding a semantic segmentation mask on the basis of the geometric consistency self-discovery mask achieved by combining depth prediction and pose prediction, it helps to solve the problem of large edge noise corresponding to dynamic and static pixels inferred by geometric consistency, and improves the recognition accuracy of the edge part; at the same time, since semantic segmentation itself has limitations, that is, it strongly depends on the dynamic attributes or static attributes corresponding to the recognized semantic labels. In fact, in some cases, the dynamic and static of objects in the image are not absolute. Therefore, by combining the geometric consistency self-discovery mask on the basis of the semantic segmentation mask, it also helps to improve the recognition accuracy in cases where certain semantic attributes change.

[0067] In some embodiments, the expression of the elements in the target probability matrix corresponding to the above-mentioned dynamic and static object states is as follows:

[0068]

[0069] Among them, U represents the probability value corresponding to each pixel in the target probability matrix; f(U segment ,U self-mask ) indicates that the probability value of each pixel in the target probability matrix is segment and U self-mask All related; U self-mask represents the probability value corresponding to the geometric consistency self-discovery mask, corresponding to the element in the first probability matrix; U segment represents the probability value corresponding to the semantic segmentation mask, corresponding to the element in the second probability matrix; (1-β) represents the preset weight corresponding to the first probability matrix, and β represents the preset weight corresponding to the second probability matrix.

[0070] In step S330, based on the target probability matrix, a photometric reprojection loss function obtained by reconstructing the image to be reconstructed from the perspective of another image in the image pair is determined; the photometric reprojection loss function is used as the target loss function.

[0071] In some embodiments, the target loss function is obtained by performing binary masking on the target probability matrix corresponding to the static and dynamic object states and then combining it with the reconstruction process of the image pair.

[0072] The process of binary masking the target probability matrix is expressed as follows:

[0073]

[0074] Among them, M represents the first mask result of the binary mask, which is used to determine the loss corresponding to the case where the possibility of the pixel being a dynamic object is higher; N represents the second mask result of the binary mask, which is used to determine the loss corresponding to the case where the possibility of the pixel being a dynamic object is lower; γ represents the threshold of the binary mask.

[0075] The mask results M and N of the binary mask are used to extract dynamic and static features respectively. γ is the threshold of the binary mask. In order to avoid the interference of fuzzy features, different values can be set for the threshold settings of M and N. Dynamic features with a higher probability are judged as dynamic features, and features with a lower probability are judged as static features. Features with more ambiguous probabilities are not added to the loss function to reduce errors.

[0076] The above objective loss function is expressed as follows:

[0077] L up =M⊙L p +θN⊙L p , (4)

[0078] Among them, L up Represents the target loss function, which is the photometric reprojection loss function; Lp represents the basic photometric reprojection loss, which measures the photometric consistency between the source image to be reconstructed and the reconstructed image obtained by depth estimation and relative pose transformation from another image's perspective. θ represents the preset weight coefficient, a hyperparameter that can be adaptively adjusted to optimize training results based on different scenarios. For example, grayscale images can be used to perform the basic photometric reprojection loss.

[0079] With the second image I t As an example of an image to be reconstructed, the corresponding basic photometric reprojection loss is expressed as follows:

[0080]

[0081] Among them, L p (D s ) represents the image to be reconstructed I t The corresponding basic photometric reprojection loss; SSIM represents the structural similarity index; I t Represents the luminosity value of the target image; I s→t (D s ) is based on the first image I s The predicted depth D under the perspective of s and relative camera pose T t→s The luminosity value corresponding to the reconstructed target image obtained after reprojection; ||I t -I s→t (D s )||1 represents I t -I s→t (D s ) corresponds to the 1-norm, which represents the sum of the absolute values of the vector elements; α is a preset coefficient that can be adjusted according to the actual scenario; for example, in this embodiment, α is set to 0.85.

[0082] In step S340, the depth prediction network and the pose prediction network are trained iteratively synchronously with the optimization goal of reducing the photometric reprojection loss function; or, the depth prediction network, the pose prediction network and the semantic segmentation network are trained iteratively synchronously.

[0083] In some embodiments, the semantic segmentation network is introduced only during the phase of constructing the dynamic uncertainty of pixels in the image. The semantic segmentation network may not be involved in subsequent training iterations, thus reducing the training workload. In other embodiments, the semantic segmentation network, depth prediction network, and pose prediction network may be trained and iterated simultaneously.

[0084] Based on the above steps S310 to S340, it is possible to make a fine distinction and consideration of the dynamic and static probabilities of each pixel during training, and combine the advantages of geometric consistency self-discovery masks and semantic segmentation masks to complement each other's shortcomings, so that the reasoning accuracy of the trained depth prediction network in dynamic scenes is improved.

[0085] In summary, in the embodiment including steps S210 to S220, by setting up a depth prediction network, a posture prediction network and a semantic segmentation network, the dynamic and static object states corresponding to each pixel point in the image to be reconstructed are comprehensively determined by combining the geometric consistency self-discovery mask and the semantic segmentation mask during training; the fine distinction and consideration of the dynamic and static probabilities of each pixel point in the image to be reconstructed are achieved, and the geometric consistency self-discovery mask and the semantic segmentation mask are combined, which has the comprehensive advantages of integrating advantages and making up for their respective shortcomings. By adding a semantic segmentation mask on the basis of realizing the geometric consistency self-discovery mask by combining depth prediction and posture prediction, it is helpful to solve the problem of using geometric consistency to detect the static and dynamic objects. The problem of large edge noise corresponding to dynamic and static pixels can be solved by semantic segmentation, which improves the recognition accuracy of edge parts. At the same time, due to the limitations of semantic segmentation itself, it is strongly dependent on the dynamic or static attributes corresponding to the recognized semantic labels. In fact, in some cases, the dynamic and static of objects in the image are not absolute. Therefore, by combining the geometric consistency self-discovery mask on the basis of semantic segmentation mask, it is also helpful to improve the recognition accuracy in cases where certain semantic attributes change. Overall, by constructing a target loss function for dynamic object areas and using it as the optimization target for training iterations, the reasoning accuracy of the deep prediction network finally trained in dynamic scenes can be improved.

[0086] In some embodiments, the above method also includes: constructing a geometric consistency loss function based on the relative posture output by the posture prediction network combined with the reprojection constraints in the geometric relationship; performing weighted sum operations on the above target loss function for the dynamic object area and the above geometric consistency loss function to obtain a comprehensive loss function.

[0087] During the above training iterations, the optimization goal is also to reduce the above comprehensive loss function.

[0088] Figure 5 Schematically shown are schematic diagrams of the results of (a) an initial image after self-masking processing based on (b1) a traditional method and (b2) a combination of geometric consistency self-discovery mask and semantic segmentation mask processing provided by an embodiment of the present disclosure for an embodiment of the present disclosure.

[0089] Reference Figure 5As shown in (a) and (b1), the results of the self-masking process based on the traditional method have many burrs on the edge of the object, indicating that there is a lot of noise in the edge area. Figure 5 As shown in (a) and (b2), the results of processing using the combination of geometric consistency self-discovery mask and semantic segmentation mask provided by the disclosed embodiment have clear boundaries for object recognition and can perform good estimation of monocular depth, demonstrating the advantages of the method provided by the disclosed embodiment.

[0090] A second exemplary embodiment of the present disclosure provides a training device for a visual depth estimation model based on dynamic uncertainty.

[0091] Figure 6 The structural block diagram of a training device for a visual depth estimation model based on dynamic uncertainty according to an embodiment of the present disclosure is schematically shown.

[0092] Reference Figure 6 As shown, the training device 600 of the dynamic uncertainty-based visual depth estimation model provided by the embodiment of the present disclosure includes: a data acquisition module 610 and a training module 620.

[0093] The data acquisition module 610 is used to acquire a training set, which includes multiple groups of image pairs corresponding to time-series frames in dynamic scenes.

[0094] The training module 620 is used to input the training set into a visual depth estimation model to be trained for training iterations. The visual depth estimation model includes a depth prediction network, a pose prediction network, and a semantic segmentation network. During training, the dynamic and static object states corresponding to each pixel in the image to be reconstructed are comprehensively determined by combining a geometric consistency self-discovery mask and a semantic segmentation mask; the image to be reconstructed is one image selected from the image pair. Based on the dynamic and static object states and the reconstruction processing of the image pair, a target loss function for the dynamic object region is determined. During the training iterations, the optimization goal is to reduce the target loss function. The trained depth prediction network is used for monocular visual depth estimation.

[0095] In some embodiments, the training module is further configured to: construct a geometric consistency loss function based on the relative pose output by the pose prediction network in combination with the reprojection constraints in the geometric relationship; and perform a weighted sum operation on the target loss function for the dynamic object region and the geometric consistency loss function to obtain a comprehensive loss function. During the training iterations, reducing the comprehensive loss function is also optimized.

[0096] The training device provided by this embodiment sets a depth prediction network, a posture prediction network and a semantic segmentation network. During the training period, the training device comprehensively determines the dynamic and static object states corresponding to each pixel point in the image to be reconstructed by combining the geometric consistency self-discovery mask and the semantic segmentation mask; realizes the refined distinction and consideration of the dynamic and static probabilities of each pixel point in the image to be reconstructed, and combines the geometric consistency self-discovery mask with the semantic segmentation mask, which has the comprehensive advantages of integrating advantages and making up for their respective shortcomings. By adding the semantic segmentation mask on the basis of realizing the geometric consistency self-discovery mask by combining depth prediction and posture prediction, it helps to solve the problem of inferring dynamic and static images by using geometric consistency. The problem of large edge noise corresponding to the pixel points is solved, and the recognition accuracy of the edge part is improved; at the same time, due to the limitations of semantic segmentation itself, it is strongly dependent on the dynamic or static attributes corresponding to the recognized semantic labels. In fact, in some cases, the dynamic and static of objects in the image are not absolute. Therefore, by combining the geometric consistency self-discovery mask on the basis of the semantic segmentation mask, it is also helpful to improve the recognition accuracy in cases where certain semantic attributes change; overall, by constructing a target loss function for the dynamic object area and using it as the optimization target for training iterations, the reasoning accuracy of the deep prediction network finally trained in dynamic scenes can be improved.

[0097] For more details of this embodiment, please refer to the relevant description of the first embodiment, which will not be repeated here.

[0098] Any number of the functional modules included in the above-mentioned training device 600 can be combined into one module, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. At least one of the functional modules included in the training device 600 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented in hardware or firmware by any other reasonable method of integrating or packaging the circuit, or can be implemented in any one of the three implementation methods of software, hardware and firmware, or in an appropriate combination of any of them. Alternatively, at least one of the functional modules included in the training device 600 can be at least partially implemented as a computer program module, which can perform the corresponding function when the computer program module is executed.

[0099] A third exemplary embodiment of the present disclosure provides an electronic device.

[0100] Figure 7 The structural block diagram of the electronic device provided by the embodiment of the present disclosure is schematically shown.

[0101] Reference Figure 7 As shown, the electronic device 700 provided by the embodiment of the present disclosure includes a processor 701, a communication interface 702, a memory 703 and a communication bus 704, wherein the processor 701, the communication interface 702 and the memory 703 communicate with each other through the communication bus 704; the memory 703 is used to store computer programs; the processor 701 is used to implement the training method of the visual depth estimation model based on dynamic uncertainty as described above when executing the program stored in the memory.

[0102] A fourth exemplary embodiment of the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned training method for a visual depth estimation model based on dynamic uncertainty.

[0103] The computer-readable storage medium may be included in the device or apparatus described in the above embodiments, or may exist independently without being incorporated into the device or apparatus. The computer-readable storage medium carries one or more programs, and when the one or more programs are executed, the method according to the embodiments of the present disclosure is implemented.

[0104] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, such as, but not limited to, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0105] It should be noted that the collection, collection, updating, analysis, processing, use, transmission, and storage of user personal information involved in the technical solutions provided by the embodiments of this disclosure all comply with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken with respect to user personal information to prevent unauthorized access to user personal information data and to maintain the security of user personal information, network security, and national security.

[0106] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0107] The foregoing description is intended only to provide specific embodiments of the present disclosure, intended to enable those skilled in the art to understand and implement the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not to be limited to the embodiments shown herein, but is to be construed in the broadest manner consistent with the principles and novel features claimed herein.

Claims

1. A training method for a visual depth estimation model based on dynamic uncertainty, characterized in that: include: Obtaining a training set, wherein the training set includes multiple groups of image pairs corresponding to time-series frames in a dynamic scene; Inputting the training set into a visual depth estimation model to be trained for training iterations, the visual depth estimation model comprising: a depth prediction network, a pose prediction network, and a semantic segmentation network; during training, comprehensively determining the state of a moving or static object corresponding to each pixel in an image to be reconstructed by combining a geometric consistency self-discovery mask and a semantic segmentation mask; the image to be reconstructed being one of the image pairs; Determining a target loss function for a dynamic object region based on the dynamic and static object states in combination with a reconstruction process of the image pair; During the training iteration, reducing the target loss function is optimized, and the trained depth prediction network is used for monocular visual depth estimation.

2. The method according to claim 1, characterized in that Inputting the training set into the visual depth estimation model to be trained for training iterations, including: Input each image pair in the training set into the depth prediction network, the pose prediction network and the semantic segmentation network for depth prediction processing, pose prediction processing and semantic segmentation processing respectively, to obtain the predicted depth, relative pose and semantic labels corresponding to the semantic segmentation area; Determining a target probability matrix for each pixel of the image to be reconstructed belonging to a dynamic object based on the predicted depth, the relative pose, and the semantic label; Determining, based on the target probability matrix, a photometric reprojection loss function obtained by reconstructing the image to be reconstructed from the perspective of another image in the image pair; the photometric reprojection loss function serving as the target loss function; With the optimization goal of reducing the photometric reprojection loss function, the depth prediction network and the pose prediction network are synchronously trained iteratively; or, the depth prediction network, the pose prediction network and the semantic segmentation network are synchronously trained iteratively.

3. The method according to claim 2, characterized in that Determining a target probability matrix of each pixel of the image to be reconstructed belonging to a dynamic object based on the predicted depth, relative pose, and semantic label, including: Determining a depth prediction difference between two images of the image pair corresponding to a same pixel point based on the relative pose and predicted depth of the image pair; Determining a first probability matrix of each pixel in the to-be-reconstructed image belonging to a dynamic object based on the depth prediction difference; Determine a second probability matrix for each pixel point belonging to a dynamic object based on the semantic label corresponding to the semantic segmentation area and the preset dynamic and static attributes corresponding to the semantic label; The first probability matrix and the second probability matrix are integrated to obtain a target probability matrix.

4. The method according to claim 3, characterized in that The first probability matrix is formed by arranging the first probability values of each pixel in the image to be reconstructed that it belongs to a dynamic object according to the positions of the pixel points; the second probability matrix is formed by arranging the second probability values of each pixel in the image to be reconstructed that it belongs to a dynamic object according to the positions of the pixel points; The target probability matrix is obtained by integrating the first probability matrix and the second probability matrix, including: The elements at each position on the first probability matrix are weighted and calculated with the elements at the corresponding positions on the second probability matrix to obtain the element values at the corresponding positions of the target probability matrix; wherein, the preset weights corresponding to the first probability matrix and the second probability matrix are preset according to the segmentation accuracy of the semantic segmentation network. The higher the segmentation accuracy of the semantic segmentation network, the higher the preset weight of the corresponding second probability matrix.

5. The method according to any one of claims 1 to 4, characterized in that The expression of the elements in the target probability matrix corresponding to the dynamic and static object states is as follows: Among them, U represents the probability value corresponding to each pixel in the target probability matrix; f(U segment ,U self-mask ) indicates that the probability value of each pixel in the target probability matrix is segment and U self-mask All related; U self-mask represents the probability value corresponding to the geometric consistency self-discovery mask, corresponding to the element in the first probability matrix; U segment represents the probability value corresponding to the semantic segmentation mask, corresponding to the element in the second probability matrix; (1-β) represents the preset weight corresponding to the first probability matrix, and β represents the preset weight corresponding to the second probability matrix.

6. The method according to claim 5, characterized in that The target loss function is obtained by performing binary masking on the target probability matrix corresponding to the dynamic and static object states and then combining it with the reconstruction processing of the image pair; The process of binary masking the target probability matrix is expressed as follows: Where M represents the first mask result of the binary mask, which is used to determine the loss corresponding to the case where the probability of the pixel being a dynamic object is higher; N represents the second mask result of the binary mask, which is used to determine the loss corresponding to the case where the probability of the pixel being a dynamic object is lower; γ represents the threshold of the binary mask; The objective loss function is expressed as follows: L up =M⊙L p +θN⊙L p , Among them, L up Represents the target loss function, which is the photometric reprojection loss function; L p represents the basic photometric reprojection loss, which is used to measure the photometric consistency between the source image to be reconstructed and the reconstructed image obtained by depth estimation and relative pose transformation from the perspective of another image; θ represents the preset weight coefficient.

7. The method according to claim 1, characterized in that Also includes: According to the relative pose output by the pose prediction network and the reprojection constraint in the geometric relationship, a geometric consistency loss function is constructed; Performing a weighted sum operation on the target loss function for the dynamic object area and the geometric consistency loss function to obtain a comprehensive loss function; During the training iteration, the optimization goal is also to reduce the comprehensive loss function.

8. A training device for a visual depth estimation model based on dynamic uncertainty, characterized in that: include: A data acquisition module is used to acquire a training set, wherein the training set includes multiple groups of image pairs corresponding to time-series frames in a dynamic scene; a training module configured to input the training set into a visual depth estimation model to be trained for training iterations, the visual depth estimation model comprising: a depth prediction network, a pose prediction network, and a semantic segmentation network; during training, comprehensively determining the state of a moving or static object corresponding to each pixel in an image to be reconstructed by combining a geometric consistency self-discovery mask and a semantic segmentation mask; the image to be reconstructed being one of the image pairs; Determining a target loss function for a dynamic object region based on the dynamic and static object states in combination with a reconstruction process of the image pair; During the training iteration, reducing the target loss function is optimized, and the trained depth prediction network is used for monocular visual depth estimation.

9. An electronic device, characterized in that: The processor, the communication interface, the memory and the communication bus are connected to each other via the communication bus. Memory for storing computer programs; A processor, configured to implement the method according to any one of claims 1 to 7 when executing a program stored in a memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images