Method and device for optimizing depth estimation model, electronic equipment and storage medium

Abstract

The application provides an optimization method and device of a depth estimation model, electronic equipment and a computer readable storage medium. The method comprises: acquiring a target image containing a dynamic object in a current environment by using a monocular camera, and extracting continuous target frames and reference frames from the target image; reconstructing the target frames according to the reference frames and a preset depth estimation model to obtain reconstructed frames, and acquiring reconstruction errors between the target frames and the reconstructed frames; preprocessing the target image to obtain point cloud data and instance segmentation data in the target image; fusing the point cloud data and the instance segmentation data in the target image to obtain mask data; obtaining a loss function according to the reconstruction errors and the mask data, and training the depth estimation model based on the loss function. The loss function is obtained by using the reconstruction errors and the mask data to optimize the depth estimation model, and more accurate dynamic scene depth estimation results can be obtained.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to an optimization method, apparatus, electronic device, and computer-readable storage medium for a depth estimation model. Background Technology

[0002] Recovering the depth information of a scene from a sequence of 2D images captured from a 3D scene is a crucial research area in computer vision. Monocular depth estimation is an important method for understanding the geometric relationships within a 3D scene. Monocular depth estimation refers to the process of processing images or videos captured by a monocular camera to obtain the corresponding depth data. Videos captured using a monocular camera are called monocular videos. When capturing monocular videos, unstable factors such as lens shake, movement of objects in the scene, and noise can cause differences between adjacent frames. These differences lead to significant jitter during monocular depth estimation, meaning that the depth data between two adjacent video frames may differ considerably.

[0003] To suppress this jitter, current deep learning methods for monocular depth estimation mainly utilize the SFM (Structure From Motion) principle. By providing the model with images from different times and viewpoints, the model infers the target depth and reconstructs the target's viewpoint image using reference images. Depth estimation with lower error will reconstruct images that are closer to the original target image. However, the similarity of the reconstructed image cannot accurately represent the degree of depth error in the following situations: moving objects do not conform to the viewpoint pose transformation of SFM and cannot be reconstructed correctly.

[0004] Existing technologies cannot completely filter out moving objects during the training of monocular depth estimation models, resulting in significant errors in the accuracy of depth estimation and preventing the model parameters from reaching their optimal level. Summary of the Invention

[0005] In view of this, this application proposes an optimization method, apparatus, electronic device and computer-readable storage medium for a depth estimation model. By utilizing reconstruction error and mask data to obtain a loss function, the depth estimation model can be optimized, resulting in more accurate depth estimation results for dynamic scenes.

[0006] In a first aspect, one embodiment of this application provides an optimization method for a depth estimation model, comprising:

[0007] A monocular camera is used to acquire target images containing dynamic objects in the current environment, and continuous target frames and reference frames are extracted from the target images.

[0008] The target frame is reconstructed based on the reference frame and a preset depth estimation model to obtain a reconstructed frame, and the reconstruction error between the target frame and the reconstructed frame is obtained.

[0009] The target image is preprocessed to obtain point cloud data and instance segmentation data in the target image;

[0010] The point cloud data in the target image is fused with the instance segmentation data to obtain mask data;

[0011] A loss function is obtained based on the reconstruction error and the mask data, and the depth estimation model is trained based on the loss function until the loss function converges to optimize the depth estimation model.

[0012] In one embodiment, the method of preprocessing the target image includes: obtaining instance segmentation data of each pixel in the target image based on an instance segmentation network, and scanning the target image using a lidar to obtain point cloud data of the lidar.

[0013] In one embodiment, fusing the point cloud data with the instance segmentation data to obtain radar mask data includes: acquiring the extrinsic parameters of the monocular camera and the lidar, the extrinsic parameters including rotation and translation matrices; projecting the point cloud data of the lidar onto the three-dimensional coordinate system of the monocular camera according to the extrinsic parameters; acquiring the intrinsic parameters of the monocular camera, the intrinsic parameters including an intrinsic parameter matrix and a distortion parameter matrix; projecting the points in the three-dimensional coordinate system of the monocular camera onto an imaging plane according to the intrinsic parameters to obtain the correspondence between the point cloud data of the lidar and the pixels of the target image; and appending the instance segmentation data of each pixel in the target image to the point cloud data according to the correspondence to obtain the radar mask data.

[0014] In one embodiment, the correspondence between the point cloud data of the lidar and the pixels of the target image involves the following calculation formula:

[0015]

[0016] Wherein, P represents the three-dimensional coordinates of the point cloud data, Z represents the Z component of the three-dimensional coordinates of the point cloud data, and P uv X represents the two-dimensional coordinates of the imaging plane, Y represents the X component of the three-dimensional coordinates of the point cloud data, and K represents the intrinsic parameters of the monocular camera.

[0017] In one embodiment, the step of reconstructing the target frame based on the reference frame and a preset depth estimation model to obtain a reconstructed frame includes: estimating the depth of the target frame using the depth estimation model to obtain depth information of the target frame; inputting the target frame and the reference frame into a preset pose estimation model to obtain the camera pose change between the target frame and the reference frame; and reconstructing the target frame based on the depth information and the camera pose change to obtain the reconstructed frame of the target frame.

[0018] In one embodiment, obtaining the reconstruction error between the target frame and the reconstructed frame includes: calculating the brightness difference between the target frame and the reconstructed frame, and obtaining the reconstruction error based on the brightness difference.

[0019] In one embodiment, the optimization method of the depth estimation model further includes: using the optimized depth estimation model for forward propagation to calculate a new predicted depth map of the target frame.

[0020] Secondly, one embodiment of this application provides an optimization method and apparatus for a depth estimation model, comprising: an acquisition module, configured to acquire a target image containing dynamic objects in the current environment using a monocular camera, and extract continuous target frames and reference frames from the target image; a reconstruction module, configured to reconstruct the target frame according to the reference frame and a preset depth estimation model to obtain a reconstructed frame, and acquire the reconstruction error between the target frame and the reconstructed frame; a processing module, configured to preprocess the target image to obtain point cloud data and instance segmentation data in the target image; a fusion module, configured to fuse the point cloud data and the instance segmentation data in the target image to obtain radar mask data; and a calculation module, configured to obtain a loss function according to the reconstruction error and the radar mask data, and train the depth estimation model based on the loss function until the loss function converges, thereby optimizing the depth estimation model.

[0021] Thirdly, one embodiment of this application provides an electronic device, the electronic device including a processor and a memory, the memory being used to store instructions, and the processor being used to call the instructions in the memory, causing the electronic device to execute the optimization method of the depth estimation model as described in the first aspect.

[0022] Fourthly, one embodiment of this application provides a computer-readable storage medium that stores computer instructions that, when executed on an electronic device, cause the electronic device to perform the optimization method for the depth estimation model as described in the first aspect.

[0023] The depth estimation model optimization method, apparatus, electronic device, and computer-readable storage medium provided in this application obtain a reconstructed frame using a reference frame and a preset depth estimation model, and acquire the reconstruction error between the target frame and the reconstructed frame. Simultaneously, mask data is obtained by fusing point cloud data and instance segmentation data from the target image. The loss value is calculated by combining the reconstruction error and the mask data to train the depth estimation model. The depth estimation model trained using the above method significantly improves the accuracy of the depth map obtained when estimating the depth map of a target image in a video sequence, thereby enabling more accurate depth estimation results for dynamic scenes. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating the steps of an optimization method for a depth estimation model according to an embodiment of this application.

[0025] Figure 2 This is a schematic diagram illustrating an application scenario of the optimization method for a depth estimation model according to an embodiment of this application.

[0026] Figure 3 This is a schematic diagram of the functional modules of an optimization device for a depth estimation model according to an embodiment of this application.

[0027] Figure 4 This is a schematic diagram of the architecture of an electronic device according to an embodiment of this application.

[0028] Explanation of main component symbols

[0029] Optimization device 10 for depth estimation model

[0030] Get Module 11

[0031] Reconstruction Module 12

[0032] Processing module 13

[0033] Fusion Module 14

[0034] Calculation Module 15

[0035] Electronic devices 20

[0036] Processor 21

[0037] Memory 22

[0038] The following detailed description, in conjunction with the accompanying drawings, will further illustrate this application. Detailed implementation method:

[0039] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them.

[0040] It should be noted that, in the embodiments of this application, "at least one" refers to one or more, and "more than one" refers to two or more. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the specification of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application.

[0041] It should be noted that in the embodiments of this application, the terms "first," "second," etc., are used only for descriptive purposes and should not be construed as indicating or implying relative importance, nor as indicating or implying order. Features specified as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0042] Please see Figure 1 , Figure 1 This is a flowchart illustrating the steps of an optimization method for a depth estimation model provided in an embodiment of this application.

[0043] The depth estimation model optimization method provided in this embodiment is mainly applied to dynamic environments containing dynamic objects, such as... Figure 2 As shown, a dynamic object refers to an object whose position is P in the view of monocular camera O1 at one moment, but whose position is P' in the view of monocular camera O2 at the next moment. The projection point of P' in the view of monocular camera O2 is P3, and (P1,P3) is the feature point matching of the dynamic object. The feature point matching of the dynamic object (P1,P3) is different from the viewpoint transformation pose relationship obtained by the feature point matching of static objects such as (P1,P2), so SFM technology is mostly used for modeling static objects.

[0044] Specifically, the optimization method for the depth estimation model in this embodiment may include:

[0045] S100 uses a monocular camera to acquire target images containing moving objects in the current environment, and extracts continuous target frames and reference frames from the target images.

[0046] Specifically, monocular cameras, such as RGB or IR cameras, can be used to acquire images of targets containing moving objects in the current environment. Due to their advantages such as ease of installation, small size, and low cost, monocular cameras have a broader application prospect in the field of depth estimation.

[0047] Preferably, before training the depth estimation model using the target frame and reference frame, the target frame and reference frame can be calibrated to ensure the training effect of the subsequent depth model.

[0048] It should be noted that extracting continuous target frames and reference frames from the target image and using them as training data during the training of the depth estimation model helps reduce the cost of collecting training data required for training the depth estimation model. Furthermore, it can effectively avoid relying on too much external image information, thereby effectively ensuring the learning and modeling capabilities of the depth estimation model while effectively reducing the training cost of the depth estimation model.

[0049] S200: Reconstruct the target frame based on the reference frame and the preset depth estimation model to obtain the reconstructed frame, and obtain the reconstruction error between the target frame and the reconstructed frame.

[0050] Specifically, the depth of the target frame is estimated using a depth estimation model to obtain its depth information. The target frame and a reference frame are then input into a pre-defined pose estimation model to obtain the camera pose change between them. Based on the depth information and the camera pose change, the target frame is reconstructed to obtain a reconstructed frame. The brightness difference between the target frame and the reconstructed frame is calculated, and the reconstruction error is obtained based on this difference. The pre-defined pose estimation model is a mature existing technology and will not be described in detail here.

[0051] It should be noted that image brightness can be understood as the degree of lightness or darkness of an image, while brightness difference information can be determined by the target frame captured by the monocular camera at the previous moment and the reference frame captured by the monocular camera at the next moment. Brightness difference information can be used to describe the error between the calculated brightness and the actual brightness during the image brightness recognition process, that is, the reconstruction error between the target frame image and the reconstructed frame image.

[0052] It should be noted that in this embodiment, any possible method can be used to determine the brightness difference between the target frame and the reconstructed frame, such as model matching, engineering methods, image processing methods, etc., without any limitation.

[0053] S300 preprocesses the target image to obtain point cloud data and instance segmentation data from the target image.

[0054] Specifically, instance segmentation data for each pixel in the target image is obtained based on an instance segmentation network. The target image is then scanned using a LiDAR scanner to obtain point cloud data.

[0055] In other embodiments, point cloud data and instance segmentation data can be obtained in other ways, and this application does not limit them.

[0056] S400 fuses point cloud data from the target image with instance segmentation data to obtain mask data.

[0057] Specifically, the extrinsic parameters of the monocular camera and LiDAR are obtained, including rotation and translation matrices. The point cloud data of the LiDAR is projected onto the three-dimensional coordinate system of the monocular camera based on the extrinsic parameters. The intrinsic parameters of the monocular camera are obtained, including an intrinsic parameter matrix and a distortion parameter matrix. The points in the three-dimensional coordinate system of the monocular camera are projected onto the imaging plane based on the intrinsic parameters to obtain the correspondence between the LiDAR point cloud data and the pixels of the target image. Based on the correspondence, instance segmentation data of each pixel in the target image is appended to the point cloud data to obtain the radar mask data.

[0058] Specifically, the calculation formula involved in the correspondence between the point cloud data of the lidar and the pixels of the target image is as follows:

[0059]

[0060] Where P represents the 3D coordinates of the point cloud data, Z represents the Z component of the 3D coordinates of the point cloud data, and P uv X represents the two-dimensional coordinates of the imaging plane, Y represents the X component of the three-dimensional coordinates of the point cloud data, and K represents the intrinsic parameters of the monocular camera.

[0061] Point cloud data contains velocity information, which can be used to determine whether an object is moving.

[0062] S500, obtain the loss function based on the reconstruction error and mask data, and train the depth estimation model based on the loss function until the loss function converges, so as to optimize the depth estimation model.

[0063] Specifically, the loss function is obtained based on the reconstruction error and mask data, which can eliminate all moving objects in the target image. This overcomes the drawback of the original method, which can only eliminate objects moving at a constant speed relative to the camera. It also eliminates the influence of moving targets in dynamic scenes. At the same time, the systematic errors in the depth estimation model are corrected through iterative solution, thereby improving the depth estimation accuracy of the depth estimation model.

[0064] In this embodiment, the method for calculating the loss value can be:

[0065] Assuming the instance segmentation data is L, the reconstruction error is D, the mask data is M, and the loss function is Loss, the relevant calculation formulas are as follows:

[0066]

[0067] These are the pixel coordinates of the point cloud data.

[0068] Loss = DM

[0069] Where (u,v) or (u′,v′) are the pixel coordinates of any image, I is the set of serial numbers for instance segmentation, the reconstruction error D is an image of size W*H, the pixels represent the error values, and the mask data M is also an image of size W*H, with pixels ∈ {0,1}.

[0070] The loss function Loss is the product of the reconstruction error D and the mask data M. The result of the loss function Loss retains the error value of the reconstruction error D when the pixel value in the mask data M is 1. The average of the retained error values ​​is used as the final loss value. Based on this loss value, a preset depth estimation model is trained to obtain an optimized depth estimation model, and there are no restrictions on this.

[0071] In this embodiment, the reconstruction error between the target frame and the reconstructed frame is obtained through a preset pose estimation model and a depth estimation model. At the same time, mask data is obtained by fusing point cloud data and instance segmentation data in the target image. The loss function is obtained by combining the reconstruction error and the mask data to train the depth estimation model.

[0072] The depth estimation model trained using the above training method can exclude all moving objects in monocular images, overcoming the drawback of the original method which can only exclude objects moving at a constant speed relative to the camera. It eliminates the influence of moving targets in dynamic scenes. At the same time, through iterative solution, it corrects the systematic errors in the preset depth estimation model, thereby improving the depth estimation accuracy of the depth estimation model.

[0073] S600 uses the optimized depth estimation model for forward propagation to calculate the predicted depth map of the new target frame.

[0074] Specifically, by using the optimized depth estimation model for forward propagation to calculate the predicted depth map of the new target frame, more accurate depth estimation results for dynamic scenes can be obtained. The depth estimation model trained by the above method significantly improves the accuracy of the depth map when estimating the depth map of new target images in a video sequence, thus enabling more accurate depth estimation results for dynamic scenes.

[0075] Any process or method description in the flowcharts of the above embodiments or otherwise described herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order according to the functions involved, as should be understood by those skilled in the art to which the embodiments of this application pertain.

[0076] Please see Figure 3 , Figure 3 A schematic diagram of the functional modules of the depth estimation model optimization device 10 provided in an embodiment of this application.

[0077] In this embodiment, based on the same idea as the depth estimation model optimization method in the above embodiments, this application also provides a depth estimation model optimization device 10, which can be used to execute the depth estimation model optimization method in the above embodiments. For ease of explanation, the functional block diagram of the depth estimation model optimization device 10 embodiment only shows the parts related to the embodiments of this application. Those skilled in the art will understand that the illustrated structure does not constitute a limitation on the depth estimation model optimization device 10, and may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0078] Specifically, in this embodiment, the depth estimation model optimization device 10 includes: an acquisition module 11, used to acquire a target image containing dynamic objects in the current environment using a monocular camera, and extract continuous target frames and reference frames from the target image; a reconstruction module 12, used to reconstruct the target frame according to the reference frame and a preset depth estimation model to obtain a reconstructed frame, and to acquire the reconstruction error between the target frame and the reconstructed frame; a processing module 13, used to preprocess the target image to obtain point cloud data and instance segmentation data in the target image; a fusion module 14, used to fuse the point cloud data and instance segmentation data in the target image to obtain radar mask data; and a calculation module 15, used to obtain a loss function according to the reconstruction error and the radar mask data, and to train the depth estimation model based on the loss function until the loss function converges, so as to optimize the depth estimation model.

[0079] Please see Figure 4 , Figure 4 This is a schematic diagram of an electronic device 20 provided in an embodiment of this application.

[0080] Specifically, the electronic device 20 includes a processor 21 and a memory 22. The memory 22 is used to store computer instructions, and the processor 21 is used to call the computer instructions in the memory 22 to cause the electronic device 20 to execute the steps of the optimization method of the depth estimation model as described in the above embodiments.

[0081] For example, computer instructions can be divided into one or more modules / units, one or more of which are stored in memory 22 and executed by processor 21. One or more modules / units can be a series of computer instruction segments capable of performing a specific function; these instruction segments describe the execution process of the computer instructions in electronic device 20. For example, they can be divided into... Figure 3 The module shown is the acquisition module 11, the reconstruction module 12, the processing module 13, the fusion module 14, and the calculation module 15.

[0082] Electronic device 20 can be a desktop computer, laptop, handheld computer, industrial computer, tablet computer, server, or other computing device. Those skilled in the art will understand that the schematic diagram is merely an example of electronic device 20 and does not constitute a limitation on electronic device 20. It may include more or fewer components than shown in the diagram, or combine certain components, or use different components. For example, electronic device 20 may also include input / output devices, network access devices, buses, etc.

[0083] Processor 21 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, a single-chip microcomputer, or any conventional processor.

[0084] The memory 22 can be used to store computer instructions and / or modules / units. The processor 21 implements various functions of the electronic device 20 by running or executing the computer instructions and / or modules / units stored in the memory 22 and by calling the data stored in the memory 22. The memory 22 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device 20 (such as audio data), etc. In addition, the memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other non-volatile solid-state storage device.

[0085] This application also discloses a computer-readable storage medium that stores computer instructions. When the computer instructions are executed on the electronic device 20, the electronic device 20 performs the steps of the optimization method for the depth estimation model as described in the above embodiments. The storage medium can be any medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0086] The depth estimation model optimization method, apparatus, electronic device, and computer-readable storage medium provided in this application obtain a reconstructed frame through a reference frame and a preset depth estimation model, and acquire the reconstruction error between the target frame and the reconstructed frame. At the same time, mask data is obtained by fusing point cloud data and instance segmentation data in the target image. The loss value is calculated by combining the reconstruction error and the mask data to train the depth estimation model.

[0087] The optimization method, apparatus, electronic device, and computer-readable storage medium of the depth estimation model in the above embodiments significantly improve the accuracy of the depth map obtained when performing depth map estimation on target images in a video sequence, thereby enabling more accurate depth estimation results for dynamic scenes.

[0088] Those skilled in the art should recognize that the specific embodiments described in this specification may be given different names, and the above description is merely illustrative of the structure of this application. All equivalent or simple variations made based on the construction, features, and principles of this application are included within the scope of protection of this application. Those skilled in the art can make various modifications or additions to the described specific examples or adopt similar methods, as long as they do not deviate from the structure of this application or exceed the scope defined by the claims, and all such modifications and additions should fall within the scope of protection of this application.

Claims

1. An optimization method for a depth estimation model, characterized in that, include: A monocular camera is used to acquire target images containing dynamic objects in the current environment, and continuous target frames and reference frames are extracted from the target images. The target frame is reconstructed based on the reference frame and a preset depth estimation model to obtain a reconstructed frame, and the reconstruction error between the target frame and the reconstructed frame is obtained. The target image is preprocessed to obtain point cloud data and instance segmentation data in the target image. The preprocessing method of the target image includes: obtaining instance segmentation data of each pixel in the target image based on an instance segmentation network, and scanning the target image with a lidar to obtain the point cloud data of the lidar. The process of fusing the point cloud data in the target image with the instance segmentation data to obtain mask data includes: acquiring the extrinsic parameters of the monocular camera and the lidar, the extrinsic parameters including rotation and translation matrices; projecting the point cloud data of the lidar onto the three-dimensional coordinate system of the monocular camera according to the extrinsic parameters; acquiring the intrinsic parameters of the monocular camera, the intrinsic parameters including an intrinsic parameter matrix and a distortion parameter matrix; projecting the points in the three-dimensional coordinate system of the monocular camera onto the imaging plane according to the intrinsic parameters to obtain the correspondence between the point cloud data of the lidar and the pixels of the target image; and appending the instance segmentation data of each pixel in the target image to the point cloud data according to the correspondence to obtain the lidar mask data. A loss function is obtained based on the reconstruction error and the mask data, and the depth estimation model is trained based on the loss function until the loss function converges to optimize the depth estimation model.

2. The optimization method for the depth estimation model as described in claim 1, characterized in that, The calculation formula involved in the correspondence between the point cloud data of the lidar and the pixels of the target image is as follows: in, The three-dimensional coordinates of the point cloud data are represented by Z, where Z represents the Z component of the three-dimensional coordinates of the point cloud data. X represents the two-dimensional coordinates of the imaging plane, Y represents the X component of the three-dimensional coordinates of the point cloud data, and K represents the intrinsic parameters of the monocular camera.

3. The optimization method for the depth estimation model as described in claim 1, characterized in that, The step of reconstructing the target frame based on the reference frame and a preset depth estimation model to obtain a reconstructed frame includes: The depth of the target frame is estimated using the depth estimation model to obtain the depth information of the target frame; The target frame and the reference frame are input into a preset pose estimation model to obtain the camera pose change between the target frame and the reference frame; The target frame is reconstructed based on the depth information and the camera pose change to obtain the reconstructed frame of the target frame.

4. The optimization method for the depth estimation model as described in claim 1, characterized in that, The step of obtaining the reconstruction error between the target frame and the reconstructed frame includes: Calculate the brightness difference between the target frame and the reconstructed frame, and obtain the reconstruction error based on the brightness difference.

5. The optimization method for the depth estimation model as described in claim 1, characterized in that, Also includes: The optimized depth estimation model is used for forward propagation to calculate the predicted depth map of the new target frame.

6. An optimization method apparatus for a depth estimation model, characterized in that, include: The acquisition module is used to acquire target images containing dynamic objects in the current environment using a monocular camera, and to extract continuous target frames and reference frames from the target images. The reconstruction module is used to reconstruct the target frame based on the reference frame and a preset depth estimation model to obtain a reconstructed frame, and to obtain the reconstruction error between the target frame and the reconstructed frame. The processing module is used to preprocess the target image to obtain point cloud data and instance segmentation data in the target image. The preprocessing method of the target image includes: obtaining instance segmentation data of each pixel in the target image based on an instance segmentation network, and scanning the target image with a lidar to obtain the point cloud data of the lidar. A fusion module is used to fuse the point cloud data in the target image with the instance segmentation data to obtain radar mask data. The fusion module includes: acquiring the extrinsic parameters of the monocular camera and the lidar, the extrinsic parameters including rotation and translation matrices; projecting the point cloud data of the lidar onto the three-dimensional coordinate system of the monocular camera according to the extrinsic parameters; acquiring the intrinsic parameters of the monocular camera, the intrinsic parameters including an intrinsic parameter matrix and a distortion parameter matrix; projecting the points in the three-dimensional coordinate system of the monocular camera onto an imaging plane according to the intrinsic parameters to obtain the correspondence between the point cloud data of the lidar and the pixels of the target image; and appending the instance segmentation data of each pixel in the target image to the point cloud data according to the correspondence to obtain the radar mask data. The calculation module is used to obtain a loss function based on the reconstruction error and the radar mask data, and to train the depth estimation model based on the loss function until the loss function converges, so as to optimize the depth estimation model.

7. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory being used to store instructions, and the processor being used to invoke the instructions in the memory, causing the electronic device to execute the optimization method of the depth estimation model as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed on an electronic device, cause the electronic device to perform an optimization method for the depth estimation model as described in any one of claims 1 to 5.

Images