Depth image acquisition method, electronic device, and storage medium
By using deep learning network models to process adjacent frames, the problem of inaccurate depth image acquisition is solved, thus improving the safety of autonomous driving.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HON HAI PRECISION INDUSTRY CO LTD
- Filing Date
- 2022-06-22
- Publication Date
- 2026-07-28
AI Technical Summary
Existing methods for acquiring depth images are not accurate enough for autonomous driving, which affects vehicle driving safety.
The deep learning network model processes adjacent frame images, calculates the error value of the predicted depth map, performs instance segmentation and masking, obtains an accurate transformation matrix, converts it into a point cloud image, and adjusts the network model to improve the accuracy of the depth image.
This improves the accuracy and detail of depth images, ensuring the safety of autonomous driving.
Smart Images
Figure CN117333827B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of machine vision, and more particularly to a method for acquiring depth images, an electronic device, and a storage medium. Background Technology
[0002] During autonomous driving, vehicles need to use depth images to monitor pedestrians, vehicles, or obstacles in their vicinity. Currently, methods for acquiring depth images include binocular vision, structured light, and time-of-flight methods. However, the depth images obtained by these methods are not accurate enough, thus affecting the safety of autonomous driving. Summary of the Invention
[0003] In view of the above, it is necessary to provide a depth image acquisition method, electronic device and storage medium to solve the problem of insufficient accuracy of the acquired depth image, especially to address the technical problem of driving safety caused by insufficient accuracy of depth images in the field of autonomous vehicle driving.
[0004] This application provides a depth image acquisition method, which includes: acquiring a first image and a second image, wherein the first image and the second image are adjacent frame images; processing the first image based on a deep learning network model to obtain a predicted depth map, and calculating a first error value of the predicted depth map using a first preset loss function; determining a first transformation matrix between the first image and the second image, wherein the first transformation matrix includes a transformation matrix for transforming the first image to each pixel in the second image; performing instance segmentation processing on the first image to obtain an instance segmentation image, and performing masking processing on the instance segmentation image to obtain a first mask image. The first transformation matrix is averaged based on the first and second mask images to obtain a target transformation matrix; the predicted depth map is converted into a first point cloud image based on the camera's intrinsic parameters; the first point cloud image is converted into a second point cloud image based on the target transformation matrix, and the second point cloud image is converted into a third image; a second error value is calculated between the second image and the third image using a second preset loss function; the deep learning network model is adjusted based on the first and second error values to obtain a target deep learning network model; the image to be detected is input into the target deep learning network model to obtain a depth image corresponding to the image to be detected.
[0005] In one optional implementation, the step of performing masking processing on the instance segmentation image to obtain a first mask image and a second mask image includes: performing masking processing on all moving objects in the instance segmentation image to obtain the first mask image; and performing masking processing on all stationary objects in the instance segmentation image to obtain the second mask image.
[0006] In one optional implementation, the step of averaging the first transformation matrix based on the first mask image and the second mask image to obtain the target transformation matrix includes: averaging the first transformation matrix based on the first mask image to obtain a second transformation matrix; averaging the first transformation matrix based on the second mask image to obtain a third transformation matrix; and adding the second transformation matrix and the third transformation matrix to obtain the target transformation matrix.
[0007] In one optional implementation, the step of averaging the first transformation matrix based on the first mask image to obtain the second transformation matrix includes: obtaining the number of pixels of each moving object in the first mask image and the first transformation matrix corresponding to the pixels of each moving object; averaging the first transformation matrix according to the number of pixels of each moving object and the first transformation matrix corresponding to the pixels of each moving object to obtain the transformation matrix of each moving object, and using the transformation matrix of each moving object as the second transformation matrix.
[0008] In one optional implementation, the deep learning network model includes a depth estimation network and a pose estimation network, and the process of processing the first image based on the deep learning network model to obtain a predicted depth map includes: performing convolution and deconvolution operations on the first image based on the depth estimation network to obtain the predicted depth map.
[0009] In one optional implementation, determining the first transformation matrix between the first image and the second image includes: performing a convolution operation on the first image and the second image based on the pose estimation network to obtain the first transformation matrix.
[0010] In one optional implementation, the step of performing instance segmentation processing on the first image to obtain an instance segmented image includes: inputting the first image into a pre-trained instance segmentation model to obtain an output image; detecting the segmentation accuracy of the instance segmentation model on the first image based on the output image; and if the segmentation accuracy is greater than a preset threshold, determining the output image as the instance segmented image.
[0011] In one alternative implementation, converting the predicted depth map into a first point cloud image based on the camera's intrinsic parameters includes using a formula... The predicted depth map is converted into the first point cloud image, where D represents the depth value of the predicted depth map, (x1, y1) represents the pixel coordinates corresponding to the depth value in the predicted depth map, K represents the intrinsic parameters of the camera, and U represents the point coordinates (x, y, z) in the first point cloud image.
[0012] This application also provides an electronic device, which includes a processor and a memory. The processor is used to execute a computer program stored in the memory to implement the depth image acquisition method.
[0013] This application also provides a computer-readable storage medium storing at least one instruction that, when executed by a processor, implements the depth image acquisition method.
[0014] As can be seen from the above technical solution, this application obtains the predicted depth map of adjacent frame images and the first transformation matrix between adjacent frame images through a deep learning network model, and calculates the first error value of the predicted depth map. Subsequently, the image is processed by instance segmentation, which can accurately segment moving and stationary objects in the image. The image after instance segmentation is processed by masking, which can obtain a more accurate description of the object contour, and then obtain the pixels representing the object and the number of pixels representing the object. The first transformation matrix in the masked image is averaged to obtain a more accurate target transformation matrix. The predicted depth map is transformed into a first point cloud image according to the camera's intrinsic parameters, and the first point cloud image is transformed into a second point cloud image according to the target transformation matrix. The difference between the first point cloud image and the second point cloud image is calculated as a second error value. The deep learning network model is adjusted based on the first error value and the second error value to obtain a target deep learning network model. After the deep learning network model is adjusted by the first error value and the second error value, the accuracy and refinement of the depth image can be improved. Thus, the obtained depth image can be applied to the field of autonomous driving of vehicles, which can ensure that autonomous driving is safer. Attached Figure Description
[0015] Figure 1 This is an application scenario diagram of a depth image acquisition method provided in an embodiment of this application.
[0016] Figure 2 A flowchart illustrating a depth image acquisition method provided in an embodiment of this application.
[0017] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0018] To better understand the above-mentioned objectives, features, and advantages of this application, the application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this application.
[0019] Numerous specific details are set forth in the following description to provide a thorough understanding of this application. The described embodiments are merely some, not all, of the embodiments described herein. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.
[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.
[0021] Currently, a neural network for depth estimation, known as a depth estimation network, can be trained using a large number of sample images captured by a camera. This network is then used to estimate the depth information of the image to be estimated, thus achieving depth estimation. The training of the depth estimation network can be unsupervised.
[0022] Unsupervised depth estimation training mainly includes two categories of methods. The first is based on image sets captured by a monocular camera. For adjacent frames in the image set, the depth and pose of the preceding and following frames are estimated separately. The images are then remapped using the depth and pose, and the network training is guided by the error in the remapped images. The second is based on image sets captured by a camera. A disparity map is learned by predicting the disparity map between the left and right binocular images in the image set and the left-right consistency of the disparity. The depth value is then calculated based on the camera's baseline and focal length.
[0023] However, methods based on fore-and-aft pose estimation are susceptible to the influence of moving objects, leading to inaccurate pose estimation and consequently affecting the computational remapping error. Methods based on binocular video require an additional camera, increasing computational complexity. Furthermore, the calibration and correction accuracy of the left and right binocular cameras also impacts the depth estimation accuracy.
[0024] like Figure 1 As shown, Figure 1 This diagram illustrates the application environment of a depth image acquisition method provided in this embodiment. The depth image acquisition method can be applied to one or more electronic devices 1, which communicate with a monocular camera 2.
[0025] The depth image acquisition method is applied to one or more electronic devices. The electronic device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored computer-readable instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.
[0026] The electronic device can be any electronic product that can interact with the user, such as a personal computer, tablet computer, smartphone, personal digital assistant (PDA), game console, interactive network television (IPTV), smart wearable device, etc.
[0027] The electronic devices may include network devices and / or user devices. The network devices include, but are not limited to, single network electronic devices, groups of multiple network electronic devices, or cloud computing-based systems consisting of a large number of hosts or network electronic devices.
[0028] The network in which the electronic device is located includes, but is not limited to: the Internet, wide area network, metropolitan area network, local area network, virtual private network (VPN), etc.
[0029] like Figure 2 As shown, Figure 2 This is a flowchart illustrating a depth image acquisition method provided in an embodiment of this application. The order of steps in the flowchart can be adjusted according to different needs, and some steps may be omitted. The method is executed by an electronic device, such as... Figure 1 The electronic device 1 shown. The method specifically includes the following:
[0030] 200, acquire the first image and the second image, wherein the first image and the second image are adjacent frame images.
[0031] In at least one embodiment of this application, the first image and the second image are obtained by a camera, which may be a monocular camera. For example, the first image and the second image may be images taken by a monocular camera while the vehicle is in motion.
[0032] In at least one embodiment of this application, the method for acquiring the first image and the second image further includes:
[0033] The vehicle's forward field of view during driving is recorded using a monocular camera, and images of adjacent frames in the video are obtained using OpenCV technology as the first image and the second image.
[0034] In at least one embodiment of this application, the adjacent frame image includes either the frame image preceding or following the first image. For example, if the first image is at time t, then the second image can be the image at time t+1 or the image at time t-1.
[0035] 201. Obtain the predicted depth map of the first image based on the deep learning network model, and calculate the first error value of the predicted depth map using the first preset loss function.
[0036] In at least one embodiment of this application, the deep learning network model includes a depth estimation network and a pose estimation network.
[0037] In at least one embodiment of this application, obtaining the predicted depth map of the first image based on a deep learning network model includes:
[0038] The predicted depth map is obtained by processing the first image using the depth estimation network.
[0039] In this embodiment, the depth estimation network includes an input layer, a convolutional layer, and a deconvolutional layer.
[0040] In this embodiment, the first image is input into the depth estimation network through the input layer of the depth estimation network, and a feature map of the first image is obtained by performing convolution operation through the convolutional layer of the depth estimation network. The feature map is then input into the deconvolutional layer for deconvolution operation to obtain a first high-dimensional feature map. Finally, each pixel in the first high-dimensional feature map is mapped to the corresponding depth and the predicted depth map is output.
[0041] In at least one embodiment of this application, the first preset loss function is:
[0042]
[0043] Where L1 represents the first error, D t (u, v) represent the pixel coordinates of the predicted depth map, I t (u, v) represents the pixel coordinates of the first image.
[0044] 202, Determine the first transformation matrix between the first image and the second image.
[0045] In at least one embodiment of this application, determining the first transformation matrix between the first image and the second image includes:
[0046] The pose estimation network is used to process the first image and the second image to obtain a first transformation matrix between the first image and the second image.
[0047] In this embodiment, the pose estimation network includes an input layer and a convolutional layer.
[0048] In this embodiment, the first image and the second image are input into the pose estimation network through the input layer of the pose estimation network. After convolution operation through the convolution layer of the pose estimation network, feature maps of the first image and the second image are obtained. The camera pose between the first image and the second image, i.e., the first transformation matrix, is obtained using a camera pose estimation algorithm.
[0049] In this embodiment, the camera pose includes a rotation matrix and a translation matrix, represented by six degrees of freedom, meaning that the pixel coordinates in the first image are used to determine the corresponding pixel coordinates in the second image through the camera pose information.
[0050] In this embodiment, the first transformation matrix includes the transformation matrix for transforming the first image to each pixel in the second image.
[0051] 203. Perform instance segmentation processing on the first image to obtain an instance segmentation image, and perform masking processing on the instance segmentation image to obtain a first mask image and a second mask image.
[0052] In at least one embodiment of this application, the instance segmentation image includes the label information of the first image. In at least one embodiment of this application, if the first image carries image information of a car, then the instance segmentation image carries block diagram information indicating the car.
[0053] In at least one embodiment of this application, the step of performing instance segmentation processing on the first image to obtain an instance segmentation image includes:
[0054] The first image is input into a pre-trained instance segmentation model to obtain the output image;
[0055] The segmentation accuracy of the instance segmentation model on the first image is detected based on the output image;
[0056] If the segmentation accuracy is greater than a preset threshold, then the output image is determined as the instance segmentation image.
[0057] In this embodiment, the instance segmentation model includes a feature extraction network layer, a feature mapping layer, and a classification layer. The feature extraction network layer is used to extract feature information from the first image, and may include multiple convolutional layers. The feature mapping layer is used to map feature information of the same subject into feature maps. For example, if the first image contains both a pedestrian and a car, the feature map obtained by the feature mapping layer includes a feature map corresponding to the pedestrian and a feature map corresponding to the car. The classification layer is used to identify the type corresponding to the feature map. The classification layer includes fully connected layers.
[0058] The preset threshold can be set according to actual needs; for example, the preset threshold can be 90%.
[0059] This application enables automated instance segmentation of vehicle driving images using the instance segmentation model, thereby improving the generation efficiency of the output image. Furthermore, by detecting the output image, the accuracy of the instance segmentation image can be improved.
[0060] In this embodiment, the step of detecting the segmentation accuracy of the instance segmentation model for the first image based on the output image includes:
[0061] Identify the subject location of the subject label in the first image, wherein the subject label is the label of the object in the first image. For example, the objects in the first image include a puppy, a car, and a tree. Let 1 represent the puppy, 2 represent the car, and 3 represent the tree. Then the subject label corresponding to the puppy is 1, the subject label corresponding to the car is 2, and the subject label corresponding to the tree is 3.
[0062] Obtain the predicted position corresponding to the main label from the output image;
[0063] The segmentation accuracy is obtained by calculating the ratio of the difference between the predicted position and the subject position to the ratio at the subject position.
[0064] This application analyzes the subject position and the predicted position using the same subject label, thereby improving the accuracy of the segmentation calculation.
[0065] In at least one embodiment of this application, the step of performing masking processing on the instance segmentation image to obtain a first mask image and a second mask image includes:
[0066] The first mask image is obtained by masking all moving objects in the segmented image of the instance.
[0067] A masking process is applied to all static objects in the instance segmentation image to obtain the second mask image. In this embodiment, the instance segmentation image may include objects such as roads, trees, pedestrians, vehicles, cyclists, and streetlights. Moving objects such as pedestrians, vehicles, and cyclists in the instance segmentation image are masked to obtain the first mask image, while static objects such as roads, trees, and streetlights in the instance segmentation image are masked to obtain the second mask image.
[0068] In this embodiment, the masking process includes representing the pixel values of moving or stationary objects using two pixel values, such as 0 and 1, or 0 and 255. Specifically, the pixel values of moving objects are represented by 0, and the pixel values of other areas besides the moving object are represented by 255.
[0069] 204. The first transformation matrix is averaged based on the first mask image and the second mask image to obtain the target transformation matrix.
[0070] In at least one embodiment of this application, the step of averaging the first transformation matrix based on the first mask image and the second mask image to obtain the target transformation matrix includes:
[0071] The first transformation matrix is averaged based on the first mask image to obtain the second transformation matrix;
[0072] The first transformation matrix is averaged based on the second mask image to obtain the third transformation matrix;
[0073] The target transformation matrix is obtained by adding the second transformation matrix to the third transformation matrix;
[0074] In at least one embodiment of this application, the step of averaging the first transformation matrix based on the first mask image to obtain the second transformation matrix includes:
[0075] Obtain the number of pixels of each moving object in the first mask image and the first transformation matrix corresponding to each pixel of the moving object;
[0076] Based on the number of pixels of each moving object and the first transformation matrix corresponding to each pixel of each moving object, the first transformation matrix is averaged to obtain the transformation matrix of each moving object, and the transformation matrix of each moving object is used as the second transformation matrix. For example, in the first mask image, there are three moving objects: a vehicle, a pedestrian, and a kitten. The number of pixels representing the vehicle is 100, the number of pixels representing the pedestrian is 123, and the number of pixels representing the kitten is 114. The first transformation matrices corresponding to the pixels representing the vehicle, the pedestrian, and the kitten are obtained respectively. All the first transformation matrices representing the vehicle pixels are summed, and the sum is divided by the number of pixels representing the vehicle to obtain the vehicle's transformation matrix, which is used as the second transformation matrix. Similarly, all the first transformation matrices representing the pedestrian pixels are summed, and the sum is divided by the number of pixels representing the pedestrian to obtain the pedestrian's transformation matrix, which is used as the second transformation matrix. Finally, all the first transformation matrices representing the kitten pixels are summed, and the sum is divided by the number of pixels representing the kitten to obtain the kitten's transformation matrix, which is used as the second transformation matrix.
[0077] In at least one embodiment of this application, the step of averaging the first transformation matrix based on the second mask image to obtain the third transformation matrix includes:
[0078] Obtain the number of pixels of each stationary object in the second mask image and the first transformation matrix corresponding to each pixel of the stationary object;
[0079] Based on the number of pixels of each stationary object and the first transformation matrix corresponding to each pixel of each stationary object, the first transformation matrix is averaged to obtain the transformation matrix of each stationary object, and the transformation matrix of each stationary object is used as the third transformation matrix.
[0080] In at least one embodiment of this application, adding the second transformation matrix to the third transformation matrix to obtain the target transformation matrix includes:
[0081] The target transformation matrix is obtained by adding the second transformation matrix of all moving objects and the third transformation matrix of all stationary objects.
[0082] It is understood that the embodiments of this application average the transformation matrices of all moving objects in the image and the transformation matrices of all stationary objects in the image, and finally add the averaged transformation matrices together to obtain a more accurate transformation matrix and obtain a more accurate pose, thereby improving the accuracy of training the depth estimation network.
[0083] 205. Based on the camera's intrinsic parameters, the predicted depth map is converted into a first point cloud image, and based on the target transformation matrix, the first point cloud image is converted into a second point cloud image, and the second point cloud image is converted into a third image.
[0084] In at least one embodiment of this application, both the first point cloud image and the second point cloud image are three-dimensional (3D) point cloud maps.
[0085] In at least one embodiment of this application, converting the predicted depth map into a first point cloud image includes:
[0086] Based on the camera's intrinsic parameters and the predicted depth map, the predicted depth map is converted into a first point cloud image.
[0087] In this embodiment, the predicted depth map is expressed using a formula. Convert to a 3D point cloud image; where D represents the depth value of the depth image, (x1, y1) represents the pixel coordinates corresponding to the depth value in the depth image, K represents the camera's intrinsic parameters, and U represents the coordinates (x, y, z) of a point in the first point cloud image.
[0088] In at least one embodiment of this application, converting the first point cloud image into a second point cloud image based on the target transformation matrix includes:
[0089] Obtain the coordinates of points in the first point cloud image and combine them into a coordinate set;
[0090] The second point cloud image is obtained by multiplying the target transformation matrix with the coordinate set.
[0091] In at least one embodiment of this application, converting the second point cloud image into a third image includes:
[0092] The second point cloud image is processed using the formula Convert to a third image, where (a, b, c) represent the coordinates of a point in the second point cloud image, and K... -1 Let K be the inverse matrix, and (m, n) represent the pixel coordinates in the third image.
[0093] 206. Calculate the second error value between the second image and the third image using the second preset loss function.
[0094] In at least one embodiment of this application, the second preset loss function is:
[0095]
[0096] Where L2 represents the second error value, and α represents a constant. The pixel coordinates of the third image are represented by l(u, v), and the pixel coordinates of the second image are represented by l(u, v). SSIM (Structural Similarity index) is an index used to measure the similarity between two images.
[0097] 207. Adjust the deep learning network model according to the first error value and the second error value to obtain the target deep learning network model.
[0098] In at least one embodiment of this application, adjusting the deep learning network model based on the first error value and the second error value to obtain the target deep learning network model includes:
[0099] The sum of the first error value and the second error value is calculated to obtain the total training error value of the deep learning network model;
[0100] Adjust the configuration parameters of the deep learning network model until the total training error value no longer decreases, and obtain the target deep learning network model.
[0101] The configuration parameters refer to the initial configuration parameters in the depth estimation network and the pose estimation network. For example, the configuration parameters include the number of convolutional and deconvolutional layers in the depth estimation network and the size of the convolutional kernels in the convolutional and deconvolutional layers. The configuration parameters also include the number of convolutional layers in the pose estimation network and the size of the convolutional kernels in the convolutional layers.
[0102] 208. Input the image to be detected into the target deep learning network model to obtain the depth image corresponding to the image to be detected.
[0103] In at least one embodiment of this application, the image to be detected includes an image corresponding to the field of vision of the autonomous vehicle during driving.
[0104] In at least one embodiment of this application, the depth image corresponding to the image to be detected can be obtained by inputting the image to be detected into the target deep learning network model, thereby achieving the technical effect of acquiring a depth image based on a monocular image.
[0105] As can be seen from the above technical solution, this application obtains the predicted depth map of adjacent frame images and the first transformation matrix between adjacent frame images through a deep learning network model, and calculates the first error value of the predicted depth map. Subsequently, the image is processed by instance segmentation, which can accurately segment moving and stationary objects in the image. The image after instance segmentation is processed by masking, which can obtain a more accurate description of the object contour, and then obtain the pixels representing the object and the number of pixels representing the object. The first transformation matrix in the masked image is averaged to obtain a more accurate target transformation matrix. The predicted depth map is transformed into a first point cloud image according to the camera's intrinsic parameters, and the first point cloud image is transformed into a second point cloud image according to the target transformation matrix. The difference between the first point cloud image and the second point cloud image is calculated as a second error value. The deep learning network model is adjusted based on the first error value and the second error value to obtain a target deep learning network model. After the deep learning network model is adjusted by the first error value and the second error value, the accuracy and refinement of the depth image can be improved. Thus, the obtained depth image can be applied to the field of autonomous driving of vehicles, which can ensure that autonomous driving is safer.
[0106] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. For those skilled in the art, improvements can be made without departing from the inventive concept of this application, but these all fall within the scope of protection of this application.
[0107] like Figure 3 As shown, Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 1 includes a memory 301, at least one processor 302, a computer program 303 stored in the memory 301 and executable on the at least one processor 302, and at least one communication bus 304.
[0108] Those skilled in the art will understand that Figure 3 The schematic diagram shown is merely an example of the electronic device 1 and does not constitute a limitation on the electronic device 1. It may include more or fewer components than shown, or combine certain components, or different components. For example, the electronic device 1 may also include input / output devices, network access devices, etc.
[0109] The at least one processor 302 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The at least one processor 302 may be a microprocessor or any conventional processor, etc. The at least one processor 302 is the control center of the electronic device 1, connecting various parts of the electronic device 1 via various interfaces and lines.
[0110] The memory 301 can be used to store the computer program 303. The at least one processor 302 implements various functions of the electronic device 1 by running or executing the computer program 303 stored in the memory 301 and calling data stored in the memory 301. The memory 301 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device 1 (such as audio data), etc. In addition, the memory 301 may include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other non-volatile solid-state storage device.
[0111] If the modules / units integrated in the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).
[0112] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within this application. No appended diagram markings in the claims should be construed as limiting the scope of the claims.
[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application.
Claims
1. A depth image acquisition method, characterized in that, The depth image acquisition method includes: Acquire a first image and a second image, wherein the first image and the second image are adjacent frame images; and The first image is processed based on a deep learning network model to obtain a predicted depth map, and a first error value of the predicted depth map is calculated using a first preset loss function. The first error value is the error between the first image and the predicted depth map. A first transformation matrix is determined between the first image and the second image, wherein the first transformation matrix includes a transformation matrix that transforms the first image to each pixel in the second image; The first image is subjected to instance segmentation processing to obtain an instance segmentation image, and the instance segmentation image is subjected to masking processing to obtain a first mask image and a second mask image; The first transformation matrix is averaged based on the first mask image and the second mask image to obtain the target transformation matrix; The predicted depth map is converted into a first point cloud image based on the camera's intrinsic parameters; Based on the target transformation matrix, the first point cloud image is converted into a second point cloud image, and the second point cloud image is converted into a third image; A second error value between the second image and the third image is calculated using a second preset loss function; The deep learning network model is adjusted based on the first error value and the second error value to obtain the target deep learning network model. The image to be detected is input into the target deep learning network model to obtain the depth image corresponding to the image to be detected.
2. The depth image acquisition method of claim 1, wherein, The step of performing masking processing on the instance segmentation image to obtain a first mask image and a second mask image includes: The first mask image is obtained by masking all moving objects in the segmented image of the instance. The second mask image is obtained by masking all static objects in the segmented image.
3. The depth image acquisition method of claim 2, wherein, The step of averaging the first transformation matrix based on the first mask image and the second mask image to obtain the target transformation matrix includes: The first transformation matrix is averaged based on the first mask image to obtain the second transformation matrix; The first transformation matrix is averaged based on the second mask image to obtain the third transformation matrix; The target transformation matrix is obtained by adding the second transformation matrix to the third transformation matrix.
4. The depth image acquisition method of claim 3, wherein, The step of averaging the first transformation matrix based on the first mask image to obtain the second transformation matrix includes: Obtain the number of pixels of each moving object in the first mask image and the first transformation matrix corresponding to each pixel of the moving object; Based on the number of pixels of each moving object and the first transformation matrix corresponding to each pixel of each moving object, the first transformation matrix is averaged to obtain the transformation matrix of each moving object, and the transformation matrix of each moving object is used as the second transformation matrix.
5. The depth image acquisition method of claim 1, wherein, The deep learning network model includes a depth estimation network and a pose estimation network. The process of processing the first image based on the deep learning network model to obtain the predicted depth map includes: The predicted depth map is obtained by performing convolution and deconvolution operations on the first image based on the depth estimation network.
6. The depth image acquisition method of claim 5, wherein, The determination of the first transformation matrix between the first image and the second image includes: The first transformation matrix is obtained by performing convolution operations on the first image and the second image based on the pose estimation network.
7. The depth image acquisition method of claim 1, wherein, The step of performing instance segmentation processing on the first image to obtain an instance segmentation image includes: The first image is input into a pre-trained instance segmentation model to obtain the output image; The segmentation accuracy of the instance segmentation model on the first image is detected based on the output image; If the segmentation accuracy is greater than a preset threshold, the output image is determined as the instance segmentation image.
8. The depth image acquisition method of claim 1, wherein, The process of converting the predicted depth map into a first point cloud image based on the camera's intrinsic parameters includes: Through formula The predicted depth map is converted into the first point cloud image, wherein, This represents the depth value of the predicted depth map. This represents the pixel coordinates corresponding to the depth value in the predicted depth map. This indicates the intrinsic parameters of the camera. Represents the coordinates of a point in the first point cloud image. .
9. An electronic device, characterized in that, The electronic device includes a processor and a memory, the processor being configured to execute a computer program stored in the memory to implement the depth image acquisition method as described in any one of claims 1 to 8.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction, which, when executed by a processor, implements the depth image acquisition method as described in any one of claims 1 to 8.