Reinforcement learning method and device of depth estimation model, electronic equipment and readable storage medium

Abstract

The invention provides a reinforcement learning method and device for a depth estimation model, electronic equipment and a readable storage medium, and relates to the technical field of monocular depth estimation. On the basis of offline unsupervised training, an online reinforcement learning mechanism is adopted to carry out real-time fine tuning on a deployed depth estimation model, so that the robustness and stability of the depth estimation model in a dynamic scene are improved. In this way, the whole process does not depend on marked depth data, depth estimation is achieved only through the motion features of the monocular image sequence and the unsupervised indexes, and the problems that in the prior art, data cost is high, and dynamic scene adaptation is poor are effectively solved.

Description

Technical Field

[0001] This invention relates to the field of monocular depth estimation technology, and more specifically, to a reinforcement learning method, apparatus, electronic device, and readable storage medium for a depth estimation model. Background Technology

[0002] Monocular depth estimation is one of the core tasks in computer vision, aiming to recover the distance information from each pixel in a scene to the camera using only a single 2D image, thereby achieving a leap from 2D observation to 3D spatial understanding. This technology has wide application value in many high-level intelligent systems such as autonomous driving environmental perception, robot autonomous navigation, augmented reality (AR), 3D map reconstruction, and remote sensing terrain mapping.

[0003] In recent years, with the rapid development of deep learning technology, monocular depth estimation has gradually shifted from traditional methods based on handcrafted features to a technology path dominated by data-driven deep neural network models. Current mainstream methods can be mainly divided into three categories: traditional handcrafted feature methods, pure deep learning methods, and multi-sensor fusion methods. Among these, due to considerations of hardware cost and deployment flexibility, pure deep learning methods have become a research hotspot because they rely solely on monocular image input, especially supervised / semi-supervised / unsupervised deep regression models and deep generative models based on Generative Adversarial Networks (GANs).

[0004] The first type of typical method is a deep regression model based on supervised, semi-supervised, or unsupervised learning: Early work often employed Convolutional Neural Networks (CNNs) to construct end-to-end deep regression architectures, using paired monocular image-real depth map datasets (such as KITTI and NYU Depth V2) for training within a supervised learning framework. While these methods can achieve high accuracy, they heavily rely on expensive and difficult-to-obtain large-scale real depth annotation data (typically requiring LiDAR or structured light equipment for acquisition), limiting their generalization ability in open scenes.

[0005] To alleviate reliance on labeled data, researchers have proposed an unsupervised monocular depth estimation scheme. Previous unsupervised methods have shown that training a monocular depth estimation model using calibrated binocular images or monocular video sequences is sufficient to predict depth maps and poses. Current mainstream unsupervised methods use photometric loss constraints between images as supervision signals, employing CNNs to design models that extract image features and predict dense depth maps. However, due to the fixed receptive field of CNNs (the receptive field being the area mapped by each pixel in the feature map output by each layer in the original image), these models are more prone to getting trapped in local optima during optimization. In contrast, Transformer-based architectures utilize global attention mechanisms to capture a wider receptive field, compensating for the shortcomings of CNNs in modeling global context. However, the computational complexity of Transformers makes real-time deployment impractical. Semi-supervised methods, combining a small amount of labeled data with unsupervised constraints, improve the depth accuracy of key regions.

[0006] Therefore, the drawbacks of the three methods in the first category are as follows: supervised models rely on expensive labeled data, have poor generalization, and are prone to errors in dynamic scenes; semi-supervised models require strict control of the labeling ratio, have difficulty in modal fusion, and are prone to failure of dynamic constraints; unsupervised models rely on scene motion or binocular calibration, resulting in low absolute accuracy and poor detail. All three types of models share common problems such as the susceptibility of depth distribution logic to distortion and the lack of real-time dynamic adaptation.

[0007] The second representative technique is the deep generation model based on Generative Adversarial Networks (GANs). This type of method constructs an adversarial mechanism between a generator and a discriminator, enabling the generator to learn output results that better conform to the natural depth distribution, while the discriminator is responsible for distinguishing the spatial statistical characteristics of the generated depth map from the real depth map. It constrains the real distribution of the generated depth map through adversarial loss, and improves detail accuracy by combining pixel-level loss and edge loss. Although this can optimize the distribution and detail of monocular depth maps, it has significant adaptation defects in dynamic scenes (such as sudden changes in lighting, object movement / occlusion, and cross-scene switching), and is prone to mode collapse or outputting unreasonable depth distribution, leading to misjudgment by the perception system. Summary of the Invention

[0008] The purpose of this invention is to provide a reinforcement learning method, apparatus, electronic device, and readable storage medium for deep estimation models, so as to improve the problems existing in the prior art.

[0009] The embodiments of the present invention can be implemented as follows: In a first aspect, the present invention provides a reinforcement learning method for a deep estimation model, comprising: Acquire the real-time monocular video stream captured by the vehicle-mounted camera; For each preprocessed current video frame in the monocular video stream, the current video frame is input into the depth estimation model to obtain the current depth map; the depth estimation model is obtained by unsupervised training of a pre-built GAN model; Based on the current video frame, the previous video frame, and the current depth map, the current state vector and the current total reward are determined; the current state vector includes the error feature vector of the current video frame relative to the previous video frame and the dynamic feature vector corresponding to the dynamic region. The current state vector is input into the policy network and the value network respectively to obtain the current action vector and the current value estimate; When the number of consecutively processed video frames reaches N, the PPO loss is calculated based on the state vector, action vector, total reward, and value estimate corresponding to the N consecutive video frames. After updating the parameters of the value network and policy network through the Adam optimizer, the parameters of the depth estimation model are updated based on the updated policy network to complete one round of online reinforcement learning.

[0010] In an optional implementation, the step of determining the current state vector and the current total reward based on the current video frame, the previous video frame, and the current depth map includes: Calculate optical flow field data based on the current video frame and the previous video frame; During the generation of the current depth map, intermediate feature maps output by the intermediate convolutional layers of the encoder of the depth estimation model are obtained. Based on the optical flow field data and the intermediate feature map, the dynamic feature vector is determined; Based on the intrinsic parameter matrix of the vehicle camera and the previous depth map corresponding to the previous video frame, the previous depth map is projected onto the view of the current video frame to obtain the current reprojected image. Calculate the reprojection error based on the current reprojected image and the current depth map; Based on the current depth map, the edge smoothness is calculated; the error feature vector includes the reprojection error and the edge smoothness. The current state vector is obtained by concatenating the dynamic feature vector, the reprojection error, and the edge smoothness. Obtain the historical depth maps of the previous M video frames of the current video frame, and determine the inter-frame stability coefficient based on the M historical depth maps and the current depth map; The reprojection error, the edge smoothness, and the inter-frame stability coefficient are input into the reward function to obtain the current total reward.

[0011] In an optional implementation, the step of determining the dynamic feature vector based on the optical flow field data and the intermediate feature map includes: Based on the horizontal and vertical displacements corresponding to each pixel in the optical flow field data, the optical flow amplitude of each pixel is calculated. Each pixel whose optical flow amplitude is greater than a preset optical flow amplitude threshold is designated as a target pixel; the dynamic region includes all the target pixels. After extracting the pixel values ​​of all target pixels from the intermediate feature map, average pooling is performed to obtain the dynamic feature vector.

[0012] In an optional implementation, the reward function is calculated using the following formula:

[0013]

[0014] In the formula, This indicates that the current video frame is the [number]. One video frame, This represents the current total reward. , , All are weighting coefficients. , , These represent the reprojection error, the edge smoothness, and the inter-frame stability coefficient, respectively. , These represent the pixel height and pixel width of the current video frame, respectively. Represents pixels The variance of the M+1 pixel values ​​corresponding to the M historical depth maps and the current depth map.

[0015] In an optional implementation, the depth estimation model is trained in the following manner: Acquire multiple monocular image sequences and generate a dataset based on all monocular image sequences; Based on a preset batch size, several training samples in the dataset are divided into multiple data subsets; the training samples include standard image pairs and pseudo-depth maps; For each subset of data, the GAN model is trained unsupervised using the subset of data; wherein, the depth estimation model is the generator in the trained GAN model.

[0016] In an optional implementation, the step of generating a dataset based on all monocular image sequences includes: Each frame in each monocular image sequence is preprocessed and then grouped into several image pairs according to every two consecutive frames; each image pair includes a source frame and a target frame. For each image pair, optical flow field data from the source frame to the target frame is calculated; Calculate the optical flow amplitude of each pixel in the optical flow field data, and use linear interpolation to map each optical flow amplitude to an initial depth value within a set depth range to obtain an initial depth map; Based on the initial depth map and the camera intrinsic parameter matrix, the initial depth map is projected onto the view of the target frame to obtain an intermediate depth map; The intermediate depth map is compared with the initial depth map to determine the error region where the depth value difference exceeds the set depth value. If the error region is the entire map region, a new initial depth map is generated after obtaining a new set depth range. If the error region is a local region, the intermediate depth map is filtered. Each abruptly changed pixel in the filtered intermediate depth map is detected, and the depth value of each abruptly changed pixel is replaced with the median depth value of its neighborhood to obtain the pseudo-depth map corresponding to the image pair.

[0017] In an optional implementation, the standard image pair includes a source frame and a target frame, and the GAN model includes a generator and a discriminator; the step of unsupervised training of the GAN model using the data subset includes: For each training sample in the data subset, the source frame and target frame in the training sample are input into the generator sequentially to obtain the source depth map and the target depth map, respectively. Based on the pseudo depth map in the training sample and the source depth map, an interpolated depth map of the two is generated. The source frame in the training sample is stitched together with the pseudo depth map, the source depth map, and the interpolated depth map to obtain three stitched images. The three stitched images are then input into the discriminator in sequence to obtain the first confidence map, the second confidence map, and the third confidence map, respectively. Based on the source depth map and the camera intrinsic parameter matrix, the source depth map is projected onto the view of the target frame to obtain the target reprojection image; Based on the target depth map and the camera intrinsic parameter matrix, the target depth map is projected onto the view of the source frame to obtain the source reprojection image; The reprojection loss of the training sample is obtained by summing the differences in depth values ​​of all pixels between the pseudo-depth map in the training sample and the target reprojection image. The disparity consistency loss of the training sample is obtained by summing the depth value differences of all pixels between the target reprojection image and the source reprojection image. The mean values ​​of reprojection loss and disparity consistency loss for all training samples are calculated to obtain the total reprojection loss and the total disparity consistency loss. Input the first confidence map, second confidence map, and third confidence map corresponding to each training sample into the discriminative loss function to obtain the discriminative adversarial loss; The second confidence map corresponding to each training sample is input into the generation loss function to obtain the generative adversarial loss. The total reprojection loss, the total disparity consistency loss, and the generative adversarial loss are weighted and summed to obtain the total generative loss. The generator's parameters are updated using the total generation loss, or the discriminator and the generator's parameters are updated using the discriminant adversarial loss and the total generation loss, respectively.

[0018] In a second aspect, the present invention provides a reinforcement learning apparatus for a deep estimation model, comprising: The video acquisition module is used to acquire the monocular video stream captured in real time by the vehicle-mounted camera; The depth estimation module is used to input the current video frame into the depth estimation model for each preprocessed current video frame in the monocular video stream to obtain the current depth map; the depth estimation model is obtained by unsupervised training of a pre-built GAN model; The reinforcement learning module is used for: Based on the current video frame, the previous video frame, and the current depth map, the current state vector and the current total reward are determined; the current state vector includes the error feature vector of the current video frame relative to the previous video frame and the dynamic feature vector corresponding to the dynamic region. The current state vector is input into the policy network and the value network respectively to obtain the current action vector and the current value estimate; When the number of consecutively processed video frames reaches N, the PPO loss is calculated based on the state vector, action vector, total reward, and value estimate corresponding to the N consecutive video frames. After updating the parameters of the value network and policy network through the Adam optimizer, the parameters of the depth estimation model are updated based on the updated policy network to complete one round of online reinforcement learning.

[0019] Thirdly, the present invention provides an electronic device, comprising: a memory and a processor, wherein the memory stores a software program, and when the electronic device is running, the processor executes the software program to implement the reinforcement learning method for the deep estimation model as described in the first aspect above.

[0020] Fourthly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the reinforcement learning method for the depth estimation model described in the first aspect.

[0021] Compared with existing technologies, this invention provides a reinforcement learning method, apparatus, electronic device, and readable storage medium for a depth estimation model. It acquires a monocular video stream captured in real-time by an onboard camera. For each pre-processed current video frame in the monocular video stream, the current video frame is input into the depth estimation model to obtain the current depth map. Based on the current video frame, the previous video frame, and the current depth map, the current state vector and the current total reward are determined. The current state vector includes the error feature vector of the current video frame relative to the previous video frame and the dynamic feature vector corresponding to the dynamic region. The current state vector is input into the policy network and the value network respectively to obtain the current action vector and the current value estimate. When the number of consecutively processed video frames reaches N, the PPO loss is calculated based on the state vector, action vector, total reward, and value estimate corresponding to N consecutive video frames. After updating the parameters of the value network and policy network using the Adam optimizer, the parameters of the depth estimation model are updated based on the updated policy network to complete one round of online reinforcement learning. The depth estimation model of this invention is obtained by offline unsupervised training of a pre-built GAN model, thus eliminating the need for labeled data. Furthermore, during online reinforcement learning, dynamic feature vectors of dynamic regions are incorporated into the state vector, so that the final depth estimation model can accurately estimate depth even when facing dynamic scenes, avoiding perceptual misjudgment. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating a reinforcement learning method for a deep estimation model provided in an embodiment of the present invention.

[0024] Figure 2 This is a schematic diagram of the two-stage training architecture of the depth estimation model provided in an embodiment of the present invention.

[0025] Figure 3 This is a schematic diagram of the offline training process of the depth estimation model provided in an embodiment of the present invention.

[0026] Figure 4 This is a schematic diagram of the structure of the encoding end in the generator.

[0027] Figure 5 This is a schematic diagram of the decoding end in the generator.

[0028] Figure 6 This is a schematic diagram of the structure of a reinforcement learning device for a depth estimation model provided in an embodiment of the present invention.

[0029] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0031] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0032] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0033] Furthermore, the terms "first" and "second" are used only to distinguish descriptions and should not be interpreted as indicating or implying relative importance.

[0034] It should be noted that, where there is no conflict, the features in the embodiments of the present invention can be combined with each other.

[0035] Please see Figure 1 , Figure 1 This is a flowchart illustrating a reinforcement learning method for a deep estimation model provided in an embodiment of the present invention. The execution subject of this method can be a computing device such as a smartphone, personal laptop, personal computer, or server. The reinforcement learning method includes the following steps S201 to S205.

[0036] S201. Acquire the monocular video stream captured in real time by the vehicle-mounted camera.

[0037] In this embodiment, each frame of the monocular video stream needs to undergo standardized preprocessing operations, including noise reduction, size normalization, and pixel normalization (pixel values ​​scaled to the [0,1] range), to ensure the consistency and computability of the input data.

[0038] S202. For each preprocessed current video frame in the monocular video stream, input the current video frame into the depth estimation model to obtain the current depth map.

[0039] Among them, the depth estimation model is a generator obtained by offline unsupervised training of a pre-built GAN model. Its internal parameters have learned the basic mapping rules from two-dimensional images to depth maps through multi-constraint joint optimization in the offline stage.

[0040] S203. Based on the current video frame, the previous video frame, and the current depth map, determine the current state vector and the current total reward.

[0041] The current state vector characterizes the visual and error context encountered at the current inference moment. It consists of two parts: one is the error feature vector of the current video frame relative to the previous video frame, reflecting the degree of deviation in geometric consistency of the current depth prediction; the other is the dynamic feature vector corresponding to the dynamic regions of the current video frame relative to the previous video frame, used to characterize regions in the image with significant motion and their semantic feature responses. Together, these state vectors constitute the complete observation input of the reinforcement learning agent to the environmental state.

[0042] S204. Input the current state vector into the policy network and the value network respectively to obtain the current action vector and the current value estimate.

[0043] The policy network is responsible for outputting an action decision, i.e., the current action vector, given the current state. This action vector represents the direction and magnitude of the adjustment to the key parameters of the depth estimation model. The value network is used to evaluate the long-term expected return of the current state and output the current value estimate, providing a benchmark reference for subsequent policy updates.

[0044] S205. When the number of consecutively processed video frames reaches N, the PPO loss is calculated based on the state vector, action vector, total reward, and value estimate corresponding to the N consecutive video frames. After updating the parameters of the value network and policy network through the Adam optimizer, the parameters of the deep estimation model are updated based on the updated policy network to complete one round of online reinforcement learning.

[0045] It is understandable that each time the current action vector is obtained, the parameters of the depth estimation model are not updated directly based on the current action vector. Instead, the policy update phase is only initiated when the number of continuously processed video frames reaches a certain number N (for example, N can be 20 or 30).

[0046] In practice, a complete trajectory dataset is constructed using the state vectors, action vectors, total reward, and value estimate accumulated from processing N consecutive video frames. Based on this dataset, the Proximal Policy Optimization (PPO) loss is calculated. This PPO loss controls the step size range of policy updates, preventing training instability caused by drastic parameter changes. Subsequently, the PPO loss is backpropagated using the Adaptive Moment Estimation (Adam) optimizer to update the parameters of the value network and policy network of the PPO model. During this process, the updated policy network possesses stronger environmental adaptability, which can then be used to guide the parameter adjustment of the deep estimation model. Ultimately, this achieves a closed-loop online optimization of the deep estimation model, completing one round of online reinforcement learning.

[0047] The reinforcement learning method for the depth estimation model provided in this embodiment of the invention improves its robustness and stability in dynamic scenes by employing an online reinforcement learning mechanism on the basis of offline unsupervised training to fine-tune the deployed depth estimation model in real time. This approach does not rely on labeled depth data throughout the process, but achieves depth estimation solely through the motion features and unsupervised metrics of monocular image sequences, effectively solving the problems of high data costs and poor adaptability to dynamic scenes in existing technologies.

[0048] Please see Figure 2 This invention adopts a two-stage architecture of "learning deep mapping rules in the offline unsupervised training stage + optimizing and adapting to dynamic scenarios in the online reinforcement learning stage". The two stages will be introduced in turn below.

[0049] Next, we will introduce the offline unsupervised training phase. Please refer to [link / reference]. Figure 3 The offline training process of the depth estimation model may include the following steps S101~S103.

[0050] S101. Obtain multiple monocular image sequences and generate a dataset based on all monocular image sequences.

[0051] In this embodiment, the monocular image sequence can be obtained from continuous shooting by vehicle-mounted cameras or other mobile platforms under different environmental conditions. Each monocular image sequence can include 200-800 frames of images, and multiple monocular image sequences can cover static, slowly changing and some dynamic scenes to ensure that the model can be exposed to diverse visual inputs during subsequent training.

[0052] Optional, training parameter settings for unsupervised training may include: (1) Optimizer: Adam is used (where the learning rate of the generator and discriminator is 1e-4, , ); (2) Batch size: can be set to 8 (GPU parallel computing); (3) Training rounds: 200 rounds, the first 50 rounds are the warm-up stage and only the generator is trained, and the last 150 rounds train the generator and the discriminator together; (4) Termination condition: The training rounds are reached or the total loss generated after 100 consecutive iterations is reached. If the decrease is less than 0.001, stop training and save the pre-trained generator.

[0053] S102. Based on the preset batch size, divide several training samples in the dataset into multiple data subsets.

[0054] In this embodiment, to adapt to the batch training mechanism of deep neural networks, before each round of training, the training samples in the dataset need to be divided into multiple data subsets based on a preset batch size. Each data subset contains a fixed number of training samples, which can be used for one forward and backward propagation calculation, ensuring that the gradient updates during training are statistically representative and improving convergence stability. The training samples include standard image pairs and pseudo-depth maps.

[0055] S103. For each subset of data, use that subset of data to perform unsupervised training on the GAN model.

[0056] In this embodiment, during each training round, the GAN model is trained unsupervised for each data subset. During this process, the generator receives images from standard image pairs as input and attempts to predict depth maps that match their structure; the discriminator then combines the images with the generated depth maps to distinguish between real and fake images, pushing the generator to output results that better reflect the distribution of natural scenes. The entire training process does not rely on any externally labeled depth information, but only on the photometric consistency between images, disparity constraints, and adversarial mechanisms to optimize parameters. Finally, after training, the generator in the trained GAN model becomes the depth estimation model.

[0057] It should be understood that in the training process of unsupervised monocular depth estimation, the lack of true depth labels makes it difficult for the model to be optimized directly through pixel-level regression. To solve this problem, it is necessary to construct an alternative supervisory signal that can reflect the geometric structure of the scene and has reasonable distribution characteristics. In the process of generating samples based on monocular image sequences, this invention constructs a pseudo-depth map as an approximate depth representation, which becomes an important data foundation to support the training of Generative Adversarial Networks (GANs).

[0058] That is, the process of “generating a dataset based on all monocular image sequences” in step S101 above may include the following sub-steps S1011 to S1016.

[0059] S1011. After preprocessing each frame in each monocular image sequence, group them into several image pairs according to every two consecutive frames.

[0060] In this embodiment, the preprocessing process for each frame of a monocular image sequence may include the following operations: (1) Adopt Gaussian filtering is used for noise reduction. (1) In formula (1), The center coordinates of the filter kernel .

[0061] (2) Uniform size: The original image is scaled to a fixed resolution using bilinear interpolation. (2) In formula (2), The weights are bilinear interpolation weights, calculated from the offset between the target coordinates and the original coordinates.

[0062] (3) Pixel normalization: that is, the pixel value is normalized from... Mapped to The formula is: .

[0063] After preprocessing, each sequence needs to form an image pair of every two consecutive frames. This image pair contains a source frame and a target frame arranged in chronological order, representing different observation perspectives of the same scene at adjacent times.

[0064] Next, for each image pair, perform the following steps S1012~S1016 to generate a training sample based on each image pair, thus obtaining the dataset.

[0065] S1012. For each image pair, calculate the optical flow field data from the source frame to the target frame.

[0066] In this embodiment, the optical flow field data characterizes the two-dimensional displacement vector of each pixel in the image between adjacent frames, reflecting the relative motion of objects or the camera itself in the scene. Specifically, the optical flow field data includes the optical flow vector of each pixel, which includes pixel displacement in the horizontal direction and pixel displacement in the vertical direction. The method for calculating the optical flow field data is existing technology and will not be elaborated upon here.

[0067] S1013. Calculate the optical flow amplitude of each pixel in the optical flow field data, and use linear interpolation to map each optical flow amplitude to an initial depth value within a set depth range to obtain an initial depth map.

[0068] For each pixel in the optical flow field data Calculate the optical flow amplitude: (3) In formula (3), and They are pixels Pixel displacement in the horizontal direction and pixel displacement in the vertical direction.

[0069] Because in monocular vision, the parallax produced by near objects is usually greater than that of distant objects, the optical flow amplitude is statistically inversely proportional to the depth value. Therefore, when performing preliminary depth mapping, it should be done according to the principle of... Following the principle that "the larger the optical flow amplitude, the smaller the depth value," the optical flow amplitude corresponding to each pixel is mapped to an initial depth value within a set depth range. For example, assuming the set depth range is 0.1m to 100m, the initial depth value of the pixel with the largest optical flow amplitude (representing near objects) is mapped to 0.1m, the initial depth value of the pixel with the smallest optical flow amplitude (representing distant objects) is mapped to 100m, and the initial depth values ​​of the remaining pixels with optical flow amplitudes in the middle range are assigned using linear interpolation. For regions with missing optical flow (i.e., optical flow amplitude of 0), such as static backgrounds and occluded areas, filtering and smoothing are used to avoid depth jumps.

[0070] S1014. Based on the initial depth map and the camera intrinsic parameter matrix, project the initial depth map onto the view of the target frame to obtain the intermediate depth map.

[0071] In this embodiment, the initial depth map is geometrically projected using the camera intrinsic parameter matrix, transforming it from the source frame viewpoint to the target frame viewpoint to obtain an intermediate depth map. This process simulates the imaging logic of 3D spatial points being re-landed in another viewpoint through the camera projection equation, used to verify whether the current depth assumption is consistent with actual observations. This projection process may include the following two steps: (1) Based on the camera intrinsic parameter matrix and the pixel values ​​of each pixel in the initial depth map, each pixel is transformed into the camera coordinate system to obtain the 3D coordinates corresponding to each pixel; wherein, the 3D coordinates of each pixel are transformed into the camera coordinate system. The formula for transforming to 3D coordinates in the camera coordinate system is as follows: (4) (5) In formulas (4) and (5), Represents pixels In the initial depth map The depth value in the middle, Represents 3D coordinates; Represents the camera intrinsic parameter matrix. and For camera focal length, These are the coordinates of the image center.

[0072] (2) Based on the camera intrinsic parameter matrix, the 3D coordinates corresponding to each pixel are reprojected onto the view of the target frame to obtain an intermediate depth map; wherein, for the 3D coordinates corresponding to a pixel The formula for reprojection is: (6) In formula (6), Represents the 3D coordinates of a pixel. The new pixels obtained by reprojecting them onto the viewpoint of the target frame.

[0073] S1015. Compare the intermediate depth map with the initial depth map to determine the error area where the depth value difference exceeds the set depth value. If the error area is the entire map area, a new initial depth map is generated after obtaining a new set depth range. If the error area is a local area, the intermediate depth map is filtered.

[0074] In this embodiment, the intermediate depth map is compared pixel by pixel with the original initial depth map, and the region where the difference in depth value between the two exceeds a set threshold (e.g., 10m) is defined as the error region. If the error region covers the entire map, it means that the currently set depth range does not match the actual scene. At this time, the set depth range is adjusted and a new initial depth map is regenerated until an intermediate result with local deviation as the main factor is obtained.

[0075] When the error region is only a localized area, it indicates that most areas have reasonable depth assumptions, and only some noisy or discontinuous areas need to be repaired. In this case, the intermediate depth map is filtered to smooth out abnormal fluctuations and retain the main structural features.

[0076] S1016. Detect each abruptly changed pixel in the filtered intermediate depth map, and replace the depth value of each abruptly changed pixel with the median depth value of its neighborhood to obtain the pseudo-depth map corresponding to the image pair.

[0077] In this embodiment, each pixel in the filtered intermediate depth map that experiences a depth abrupt change is detected. These locations typically correspond to object edges or occlusion boundaries, and are prone to jagged artifacts. To improve this phenomenon, the depth value of each pixel with a sudden change is replaced with the median depth value of other pixels in its neighborhood. This suppresses the influence of outliers while maintaining clear boundaries, and finally outputs the pseudo-depth map corresponding to the image pair.

[0078] The GAN model consists of a generator and a discriminator. Here, we will first give a brief introduction to the network structure of the two.

[0079] I. The generator structure is described below: The generator employs a U-shaped "encoder-decoder" architecture. The encoder extracts semantic features from the image through convolutional layers, while the decoder restores the depth map resolution through transposed convolutions. (See also...) Figure 4 The encoding end consists of 6 encoding layers (Conv1-Conv6), each with a structure of "Conv+BN+LeakyReLU". Please refer to [link / reference]. Figure 5 The decoding end includes 5 decoding layers (Deconv1-Deconv5), and each layer has a structure of "TransConv+BN+ReLU".

[0080] II. The structure of the discriminator is described below: The discriminator employs a PatchGAN structure, focusing on a 64×64 local pixel block to determine whether the generated depth map conforms to the depth distribution patterns of the real scene. The discriminator's output is a 64×64×1 confidence map, in which the pixels... pixel values Let be a real number representing the confidence score of a local pixel block in conforming to the depth distribution pattern of the real scene. A larger value indicates a higher degree of "image-depth matching" for the corresponding local pixel block. The smaller the value, the more likely the corresponding local pixel block has a "depth logic contradiction".

[0081] During each batch of training in any round of training, the GAN model will be updated once based on a subset of data. Specifically, the implementation process of "using this subset of data to perform unsupervised training of the GAN model" in step S103 above may include the following sub-steps S1031 to S103c.

[0082] S1031. For each training sample in the data subset, the source frame and target frame in the training sample are input into the generator in sequence to obtain the source depth map and the target depth map respectively.

[0083] In this embodiment, steps S1031 to S1037 need to be executed for each training sample in a subset of data.

[0084] S1032. Based on the pseudo depth map and the source depth map in the training sample, generate the interpolated depth map of the two.

[0085] To further enhance the discriminator's ability to assess the authenticity of the generated results, interpolation samples are introduced to meet the requirements of the gradient penalty mechanism. Specifically, based on the pseudo-depth map and the source depth map in the training samples, an interpolated depth map is generated. Mathematically, this interpolated depth map represents a linear mixture of the two, serving as a smoothing constraint in the subsequent discriminator loss calculation.

[0086] S1033. The source frame in the training sample is stitched together with the pseudo depth map, the source depth map and the interpolated depth map to obtain three stitched images. The three stitched images are then input into the discriminator to obtain the first confidence image, the second confidence image and the third confidence image, respectively.

[0087] Subsequently, the source frames from the training samples are concatenated along the channel dimension with the pseudo-depth map, the source depth map, and the interpolated depth map, forming three distinct stitched images. Each stitched image contains a joint representation of the RGB image information and the corresponding depth map, serving as input to the discriminator. These three stitched images are sequentially input into the discriminator, and after multiple convolutional processing layers, the first confidence map, second confidence map, and third confidence map are output, respectively. These confidence maps reflect the discriminator's local judgment of the "realism" of different input combinations; a higher value indicates that the combination is closer to the true distribution.

[0088] S1034. Based on the source depth map and the camera intrinsic parameter matrix, project the source depth map onto the view of the target frame to obtain the target reprojection image.

[0089] S1035. Based on the target depth map and the camera intrinsic parameter matrix, project the target depth map onto the view of the source frame to obtain the source reprojection image.

[0090] To introduce geometric consistency constraints and further enhance the spatial rationality of the depth map, a geometric projection transformation is performed on the source depth map from the source frame viewpoint to the target frame viewpoint, while a geometric projection transformation is performed on the target depth map from the target frame viewpoint to the source frame viewpoint. The projection principles of steps S1034 and S1035 are consistent with the logic shown in the above formulas (4) to (6), and will not be elaborated here.

[0091] S1036. Calculate and sum the depth differences of all pixels between the pseudo-depth map and the target reprojection image in the training sample to obtain the reprojection loss of the training sample.

[0092] In this embodiment, the reprojection loss of a training sample measures whether the current target depth map can accurately reconstruct the true depth distribution under the target frame's viewpoint, and is one of the key supervision signals for unsupervised depth estimation. The formula for calculating the reprojection loss of a training sample is as follows: (7) In formula (7), This represents the reprojection loss of a training sample. Indicates the target frame (here) (an abbreviation of the English word "target") , These represent pixel height and pixel width, respectively. Represents a pixel In pseudo-depth map Reprojected image of the target The difference in pixel values. In It is an abbreviation of the word "camouflage," which means disguise. In It is an abbreviation of the word "projection".

[0093] S1037. Calculate and sum the depth differences of all pixels between the target reprojection image and the source reprojection image to obtain the disparity consistency loss of the training sample.

[0094] In this embodiment, the disparity consistency loss of a training sample is used to constrain the consistency of bidirectional projection and reduce the system bias caused by unidirectional estimation. The formula for calculating the disparity consistency loss of a training sample is as follows: (8) In formula (8), This represents the disparity consistency loss of the training samples. Represents a pixel Reprojected images of the target Reprojection image from source The difference in pixel values. In It is an abbreviation of the word "source".

[0095] S1038. Calculate the mean of the reprojection loss and disparity consistency loss for all training samples to obtain the total reprojection loss and the total disparity consistency loss.

[0096] In this embodiment, the mean values ​​of reprojection loss and disparity consistency loss for all training samples are calculated to obtain more statistically representative total reprojection loss and total disparity consistency loss.

[0097] S1039. Input the first confidence map, second confidence map and third confidence map corresponding to each training sample into the discriminative loss function to obtain the discriminative adversarial loss.

[0098] The formula for calculating the discriminant loss function is as follows:

[0099] In formula (9), The discriminative adversarial loss is calculated after forward propagation on a subset of data. Indicates the first confidence plot, Indicates the second confidence plot, This represents the third confidence plot; The weights are used for gradient penalty. Indicates the source frame in the data subset; This refers to the pseudo-depth map previously generated through constraints (used to replace the real depth map as a supervision signal in unlabeled scenarios). Represents the target depth map. Represents the source depth map With target depth map Interpolation depth map between the two.

[0100] The difference between the first two expectations in the discriminant loss function is the core of the adversarial loss, while the third expectation is the gradient penalty term, which is used to constrain the Lipschitz continuity of the discriminator to ensure that its gradient does not explode or vanish, thereby ensuring the stability of training.

[0101] S103a. Input the second confidence map corresponding to each training sample into the generation loss function to obtain the generation adversarial loss.

[0102] The formula for calculating the generation loss function is as follows: (10) In formula (10), The generative adversarial loss is calculated after forward propagation of a subset of data.

[0103] It is understandable that both the discriminant loss function and the generative loss function are existing technologies, so we will not go into detail here.

[0104] S103b: The total reprojection loss, total parallax consistency loss, and generative adversarial loss are weighted and summed to obtain the total generative loss.

[0105] In this embodiment, the total loss is generated. The calculation formula is: ,in, , These represent the total reprojection loss and the total parallax consistency loss, respectively. All are weighting coefficients.

[0106] S103c. Update the generator parameters using the total generation loss, or update the discriminator and generator parameters using the discriminant adversarial loss and the total generation loss respectively.

[0107] If the current epoch is one of the first 50 training epochs, then only the generated total loss is used. Update the generator parameters; if the current round is one of the last 150 training rounds, then apply the discriminative adversarial loss. and total loss generated The parameters of the discriminator and generator are updated.

[0108] The above describes the training process in the offline unsupervised training phase. Next, we will introduce the optimization process in the online reinforcement learning phase in detail.

[0109] For step S201 above, when acquiring the monocular video stream, it is necessary to preprocess each frame of the video stream in real time. This preprocessing process is the same as the preprocessing method in the offline unsupervised training stage described above, and will not be repeated here.

[0110] As can be seen from steps S202-S203 above, for each preprocessed video frame, it is necessary to input it into the depth estimation model before determining the corresponding state vector and total reward. When the cumulative number of processed consecutive video frames reaches N frames, a round of reinforcement learning can be initiated.

[0111] Next, the process of determining the current state vector and the current total reward in step S203 above may include the following sub-steps S2031 to S2039.

[0112] S2031. Calculate optical flow field data based on the current video frame and the previous video frame.

[0113] S2032. During the generation of the current depth map, obtain the intermediate feature map output by the intermediate convolutional layer of the encoder of the depth estimation model.

[0114] Optional, such as Figure 4The intermediate convolutional layer can be a second coding layer (Conv2). Choosing the intermediate feature map output by the second coding layer avoids noise interference from shallow features and does not lose key semantic information of dynamic regions. At the same time, the number of dimensions obtained by compression through average pooling can meet the efficiency requirements of online RL real-time inference.

[0115] S2033. Based on optical flow field data and intermediate feature maps, determine the dynamic feature vector.

[0116] Optionally, the implementation process of step S2033 may include steps a1 to a3: Step a1: Based on the horizontal and vertical displacements corresponding to each pixel in the optical flow field data, calculate the optical flow amplitude of each pixel. Step a2: Each pixel whose optical flow amplitude is greater than a preset optical flow amplitude threshold is designated as a target pixel; the dynamic region includes all target pixels; Step a3: After extracting the pixel values ​​of all target pixels from the intermediate feature map, perform average pooling to obtain the dynamic feature vector.

[0117] In this embodiment, a 128-dimensional dynamic feature vector can be obtained by average pooling. It can reflect the dynamic characteristics of the current video frame relative to the previous video frame. The dynamic area may be a pedestrian, electric vehicle, or bicycle passing by in front. For example, the preset optical flow amplitude threshold can be set to 5 or 7, which is not limited here.

[0118] S2034. Based on the intrinsic parameter matrix of the vehicle-mounted camera and the previous depth map corresponding to the previous video frame, project the previous depth map onto the view of the current video frame to obtain the current reprojected image.

[0119] The implementation principle of step S2034 is consistent with the projection principle of the above formulas (4) to (6).

[0120] S2035. Calculate the reprojection error based on the current reprojected image and the current depth map.

[0121] (11) In formula (11), This indicates the reprojection error. This indicates that the current video frame is the [number]. One video frame, Represents a pixel In the current depth map Compared with the current reprojected image The difference in pixel values.

[0122] S2036. Calculate edge smoothness based on the current depth map.

[0123] In this embodiment, the Sobel operator can be used to calculate the current depth map. The edge gradient is calculated, and then the edge smoothness is calculated using the following formula: (12) In formula (12), Indicates edge smoothness. Indicates the current depth map The edge area within.

[0124] S2037. Concatenate the dynamic feature vector, reprojection error, and edge smoothness to obtain the current state vector.

[0125] In this embodiment, the aforementioned error feature vector is the result of splicing reprojection error and edge smoothness: The current state vector is: The current state vector is a 130-dimensional vector.

[0126] S2038. Obtain the historical depth maps of the previous M video frames of the current video frame, and determine the inter-frame stability coefficient based on the M historical depth maps and the current depth map.

[0127] The formula for calculating the inter-frame stability coefficient is as follows: (13) In formula (13), This represents the inter-frame stability coefficient. Represents pixels The variance of the M+1 pixel values ​​corresponding to the M historical depth maps and the current depth map. For example, M can be 4, meaning the inter-frame stability coefficient reflects the pixel jump scale of the 5 consecutively generated depth maps.

[0128] S2039. Input the reprojection error, edge smoothness, and inter-frame stability coefficient into the reward function to obtain the current total reward.

[0129] The formula for calculating the reward function is as follows: (14) In formula (14), This indicates the current total reward. , , All are weighting coefficients.

[0130] Offline unsupervised training only optimizes single frames and does not focus on temporal stability. However, based on the above steps S2031~S2039, this invention solves the problem of poor adaptation to dynamic scenes by incorporating the dynamic feature vector of the dynamic region when calculating the state vector during the reinforcement learning stage. Furthermore, this invention can solve the problem of "deep jumps in consecutive frames under dynamic scenes" by incorporating the inter-frame stability coefficient into the reward calculation during the reinforcement learning stage.

[0131] Next, for step S204 above, the current state vector is... By inputting the policy network and value network of the PPO model respectively, the current action vector can be obtained. and current value estimate .

[0132] When the number of consecutively processed video frames reaches N, the trajectory data corresponding to the accumulated N consecutive video frames is obtained. The trajectory data includes the state vector, action vector, total reward, and value estimate corresponding to the N video frames.

[0133] Next, for step S205 above, PPO loss is divided into strategy loss and value loss. The implementation process of step S205 may include the following steps b1 to b3.

[0134] Step b1: Calculate the value loss based on the total reward and value estimate of N video frames.

[0135] The value loss is calculated using the mean squared error (MSE), and its formula is as follows: (15) (16) In formulas (15) and (16), Indicates a loss of value. Represents the th video frame in N video frames. One video frame, This indicates the result of calculating the first objective using the temporal difference (TD) method. State vectors The corresponding target value, The first value network output represents the value network output. A value estimate, Indicates the first Total reward; Represented as discount factor ( ), which represents the discount rate of future rewards.

[0136] Step b2: Calculate the policy loss based on the state vectors, total reward, and value estimation of N video frames.

[0137] In this embodiment, generalized dominance estimation needs to be performed first:

[0138] in, , For example, 0.99.

[0139] Next, a pruning objective function is used based on the calculated policy loss, and its calculation formula is as follows:

[0140] in, Indicates the probability ratio. , This represents the current policy of the policy network. This represents the old policy of the policy network. .

[0141] Step b3: Based on the value loss and policy loss, update the parameters of the value network and policy network using the Adam optimizer, and then update the parameters of the deep estimation model based on the updated policy network to complete one round of online reinforcement learning.

[0142] Among them, minimizing through the Adam optimizer (With a value weight of 0.5), the model parameters of the policy network and value network are then updated to ensure stable policy updates. Next, the depth estimation model is used to process the latest video frame (frame N+1). The state vector is then calculated and input into the policy network to obtain the latest action vector. At this point, the parameters of the depth estimation model can be updated based on this latest action vector.

[0143] It should be noted that the execution order of each step in the above method embodiments is not limited to that shown in the attached figures, and the execution order of each step shall be subject to the actual application situation.

[0144] In order to perform the corresponding steps in the above method embodiments and various possible implementations, an implementation of a reinforcement learning device for a deep estimation model is given below.

[0145] Please see Figure 6 , Figure 6 A schematic diagram of the structure of a reinforcement learning device for a depth estimation model provided in an embodiment of the present invention is shown. The reinforcement learning device 200 for the depth estimation model includes: a video acquisition module 210, a depth estimation module 220, and a reinforcement learning module 230.

[0146] The video acquisition module 210 is used to acquire the monocular video stream captured in real time by the vehicle-mounted camera; The depth estimation module is used to input the current video frame into the depth estimation model for each preprocessed current video frame in the monocular video stream to obtain the current depth map; the depth estimation model is obtained by unsupervised training of a pre-built GAN model; Reinforcement learning module 220 is used for: Based on the current video frame, the previous video frame, and the current depth map, determine the current state vector and the current total reward; the current state vector includes the error feature vector of the current video frame relative to the previous video frame and the dynamic feature vector corresponding to the dynamic region. Input the current state vector into the policy network and the value network respectively to obtain the current action vector and the current value estimate; When the number of consecutively processed video frames reaches N, the PPO loss is calculated based on the state vector, action vector, total reward, and value estimate corresponding to the N consecutive video frames. After updating the parameters of the value network and policy network through the Adam optimizer, the parameters of the deep estimation model are updated based on the updated policy network to complete one round of online reinforcement learning.

[0147] Optionally, the video acquisition module 210 can be used to implement the above step S201, the depth estimation module 220 can be used to implement the above step S202, and the reinforcement learning module 230 can be used to implement the above steps S203~S204 and their sub-steps.

[0148] Optionally, the reinforcement learning device 200 of the depth estimation model may also include an offline training module for implementing the above steps S101~S103 and their sub-steps.

[0149] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the reinforcement learning device 200 for the depth estimation model described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0150] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. The electronic device 300 includes a processor 310, a memory 320, and a bus 330, with the processor 310 connected to the memory 320 via the bus 330.

[0151] The memory 320 can be used to store software programs or firmware, for example, the software program or firmware corresponding to the reinforcement learning device 200 of the depth estimation model described above. The processor 310 executes various functional applications and data processing by running the software program stored in the memory 320 to implement the reinforcement learning method of the depth estimation model as provided in the embodiments of the present invention.

[0152] The memory 320 may be, but is not limited to, RAM (Random Access Memory), ROM (Read Only Memory), FLASH (Flash Memory), PROM (Programmable Read-Only Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electric Erasable Programmable Read-Only Memory), etc.

[0153] The processor 310 can be an integrated circuit chip with signal processing capabilities, capable of executing software programs, such as the software program corresponding to the reinforcement learning device 200 that executes the aforementioned depth estimation model. The processor 310 can be a general-purpose processor, including: CPU (Central Processing Unit), NP (Network Processor), SoC (System on Chip), etc.; it can also be: DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0154] Understandable. Figure 7 The structure shown is for illustrative purposes only; the electronic device 300 may also include components that are more advanced than those shown. Figure 7 The more or fewer components shown, or having the same Figure 7 The different configurations shown. Figure 7 The components shown can be implemented using hardware, software, or a combination thereof.

[0155] This invention also provides a computer-readable storage medium storing a computer program. When executed by a processor, this computer program implements the reinforcement learning method for the depth estimation model disclosed in the above embodiments. The computer-readable storage medium can be, but is not limited to, various media capable of storing program code, such as a USB flash drive, external hard drive, ROM, RAM, PROM, EPROM, EEPROM, FLASH disk, or optical disk.

[0156] This invention also provides a computer program product that, when executed by a processor, implements the reinforcement learning method for the deep estimation model disclosed in the above embodiments.

[0157] In summary, this invention provides a reinforcement learning method, apparatus, electronic device, and readable storage medium for a depth estimation model. It acquires a monocular video stream captured in real-time by an onboard camera. For each pre-processed current video frame in the monocular video stream, the current video frame is input into the depth estimation model to obtain the current depth map. Based on the current video frame, the previous video frame, and the current depth map, the current state vector and the current total reward are determined. The current state vector includes the error feature vector of the current video frame relative to the previous video frame and the dynamic feature vector corresponding to the dynamic region. The current state vector is input into the policy network and the value network respectively to obtain the current action vector and the current value estimate. When the number of consecutively processed video frames reaches N, the PPO loss is calculated based on the state vector, action vector, total reward, and value estimate corresponding to N consecutive video frames. After updating the parameters of the value network and policy network using the Adam optimizer, the parameters of the depth estimation model are updated based on the updated policy network to complete one round of online reinforcement learning. The depth estimation model of this invention is obtained by offline unsupervised training of a pre-built GAN model, thus eliminating the need for labeled data. Furthermore, during online reinforcement learning, dynamic feature vectors of dynamic regions are incorporated into the state vector, so that the final depth estimation model can accurately estimate depth even when facing dynamic scenes, avoiding perceptual misjudgment.

[0158] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A reinforcement learning method for a deep estimation model, characterized in that, include: Acquire the real-time monocular video stream captured by the vehicle-mounted camera; For each preprocessed current video frame in the monocular video stream, the current video frame is input into the depth estimation model to obtain the current depth map; The depth estimation model is obtained by unsupervised training of a pre-built GAN model; Based on the current video frame, the previous video frame, and the current depth map, the current state vector and the current total reward are determined; the current state vector includes the error feature vector of the current video frame relative to the previous video frame and the dynamic feature vector corresponding to the dynamic region. The current state vector is input into the policy network and the value network respectively to obtain the current action vector and the current value estimate; When the number of consecutively processed video frames reaches N, the PPO loss is calculated based on the state vector, action vector, total reward, and value estimate corresponding to the N consecutive video frames. After updating the parameters of the value network and policy network through the Adam optimizer, the parameters of the depth estimation model are updated based on the updated policy network to complete one round of online reinforcement learning.

2. The reinforcement learning method for the deep estimation model according to claim 1, characterized in that, The step of determining the current state vector and the current total reward based on the current video frame, the previous video frame, and the current depth map includes: Calculate optical flow field data based on the current video frame and the previous video frame; During the generation of the current depth map, intermediate feature maps output by the intermediate convolutional layers of the encoder of the depth estimation model are obtained. Based on the optical flow field data and the intermediate feature map, the dynamic feature vector is determined; Based on the intrinsic parameter matrix of the vehicle camera and the previous depth map corresponding to the previous video frame, the previous depth map is projected onto the view of the current video frame to obtain the current reprojected image. Calculate the reprojection error based on the current reprojected image and the current depth map; Based on the current depth map, the edge smoothness is calculated; the error feature vector includes the reprojection error and the edge smoothness. The current state vector is obtained by concatenating the dynamic feature vector, the reprojection error, and the edge smoothness. Obtain the historical depth maps of the previous M video frames of the current video frame, and determine the inter-frame stability coefficient based on the M historical depth maps and the current depth map; The reprojection error, the edge smoothness, and the inter-frame stability coefficient are input into the reward function to obtain the current total reward.

3. The reinforcement learning method for the depth estimation model according to claim 2, characterized in that, The step of determining the dynamic feature vector based on the optical flow field data and the intermediate feature map includes: Based on the horizontal and vertical displacements corresponding to each pixel in the optical flow field data, the optical flow amplitude of each pixel is calculated. Each pixel whose optical flow amplitude is greater than a preset optical flow amplitude threshold is designated as a target pixel; the dynamic region includes all the target pixels. After extracting the pixel values ​​of all target pixels from the intermediate feature map, average pooling is performed to obtain the dynamic feature vector.

4. The reinforcement learning method for the depth estimation model according to claim 2, characterized in that, The formula for calculating the reward function is as follows: In the formula, This indicates that the current video frame is the [number]. One video frame, This represents the current total reward. , , All are weighting coefficients. , , These represent the reprojection error, the edge smoothness, and the inter-frame stability coefficient, respectively. , These represent the pixel height and pixel width of the current video frame, respectively. Represents pixels The variance of the M+1 pixel values ​​corresponding to the M historical depth maps and the current depth map.

5. The reinforcement learning method for the deep estimation model according to claim 1, characterized in that, The depth estimation model is trained in the following way: Acquire multiple monocular image sequences and generate a dataset based on all monocular image sequences; Based on a preset batch size, several training samples in the dataset are divided into multiple data subsets; the training samples include standard image pairs and pseudo-depth maps; For each subset of data, the GAN model is trained unsupervised using the subset of data; wherein, the depth estimation model is the generator in the trained GAN model.

6. The reinforcement learning method for the depth estimation model according to claim 5, characterized in that, The step of generating a dataset based on all monocular image sequences includes: Each frame in each monocular image sequence is preprocessed and then grouped into several image pairs according to every two consecutive frames; each image pair includes a source frame and a target frame. For each image pair, optical flow field data from the source frame to the target frame is calculated; Calculate the optical flow amplitude of each pixel in the optical flow field data, and use linear interpolation to map each optical flow amplitude to an initial depth value within a set depth range to obtain an initial depth map; Based on the initial depth map and the camera intrinsic parameter matrix, the initial depth map is projected onto the view of the target frame to obtain an intermediate depth map; The intermediate depth map is compared with the initial depth map to determine the error region where the depth value difference exceeds the set depth value. If the error region is the entire map region, a new initial depth map is generated after obtaining a new set depth range. If the error region is a local region, the intermediate depth map is filtered. Each abruptly changed pixel in the filtered intermediate depth map is detected, and the depth value of each abruptly changed pixel is replaced with the median depth value of its neighborhood to obtain the pseudo-depth map corresponding to the image pair.

7. The reinforcement learning method for the deep estimation model according to claim 5, characterized in that, The standard image pair includes a source frame and a target frame, and the GAN model includes a generator and a discriminator; The step of performing unsupervised training of the GAN model using the data subset includes: For each training sample in the data subset, the source frame and target frame in the training sample are input into the generator sequentially to obtain the source depth map and the target depth map, respectively. Based on the pseudo depth map in the training sample and the source depth map, an interpolated depth map of the two is generated. The source frame in the training sample is stitched together with the pseudo depth map, the source depth map, and the interpolated depth map to obtain three stitched images. The three stitched images are then input into the discriminator in sequence to obtain the first confidence map, the second confidence map, and the third confidence map, respectively. Based on the source depth map and the camera intrinsic parameter matrix, the source depth map is projected onto the view of the target frame to obtain the target reprojection image; Based on the target depth map and the camera intrinsic parameter matrix, the target depth map is projected onto the view of the source frame to obtain the source reprojection image; The reprojection loss of the training sample is obtained by summing the differences in depth values ​​of all pixels between the pseudo-depth map in the training sample and the target reprojection image. The disparity consistency loss of the training sample is obtained by summing the depth value differences of all pixels between the target reprojection image and the source reprojection image. The mean values ​​of reprojection loss and disparity consistency loss for all training samples are calculated to obtain the total reprojection loss and the total disparity consistency loss. Input the first confidence map, second confidence map, and third confidence map corresponding to each training sample into the discriminative loss function to obtain the discriminative adversarial loss; The second confidence map corresponding to each training sample is input into the generation loss function to obtain the generative adversarial loss. The total reprojection loss, the total disparity consistency loss, and the generative adversarial loss are weighted and summed to obtain the total generative loss. The generator's parameters are updated using the total generation loss, or the discriminator and the generator's parameters are updated using the discriminant adversarial loss and the total generation loss, respectively.

8. A reinforcement learning device for a deep estimation model, characterized in that, include: The video acquisition module is used to acquire the monocular video stream captured in real time by the vehicle-mounted camera; The depth estimation module is used to input the current video frame into the depth estimation model for each preprocessed current video frame in the monocular video stream to obtain the current depth map; The depth estimation model is obtained by unsupervised training of a pre-built GAN model; The reinforcement learning module is used for: Based on the current video frame, the previous video frame, and the current depth map, the current state vector and the current total reward are determined; the current state vector includes the error feature vector of the current video frame relative to the previous video frame and the dynamic feature vector corresponding to the dynamic region. The current state vector is input into the policy network and the value network respectively to obtain the current action vector and the current value estimate; When the number of consecutively processed video frames reaches N, the PPO loss is calculated based on the state vector, action vector, total reward, and value estimate corresponding to the N consecutive video frames. After updating the parameters of the value network and policy network through the Adam optimizer, the parameters of the depth estimation model are updated based on the updated policy network to complete one round of online reinforcement learning.

9. An electronic device, characterized in that, include: A memory and a processor, wherein the memory stores a software program, and the processor executes the software program when the electronic device is running to implement the reinforcement learning method for the deep estimation model as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the reinforcement learning method for the depth estimation model according to any one of claims 1-7.

Images