Visual distance measurement method, device and equipment and computer readable storage medium
By combining monocular cameras and deep learning technology, the depth information of obstacle points is obtained, and the existing ranging methods are solved, with poor stability, high computational complexity and high equipment costs, achieving higher ranging accuracy and stability.
Patent Information
- Application Number
- CN202411857259.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-16
- Publication Date
- 2025-03-28
AI Technical Summary
The existing ranging methods have problems such as poor stability, high computational complexity, high equipment cost, strict light source restrictions and sensitive to ambient light interference.
The target pixel coordinates of the obstacle point are obtained through a monocular camera, the first absolute depth is calculated, and the pre-trained monocular depth estimation model is used to process the image to be detected, the target depth map is obtained, the relative depth is determined, and it is converted into the second absolute depth, and the distance measurement result is finally obtained by fusing the two depth information.
Improve the stability and accuracy of the ranging results, reduce equipment costs, and reduce dependence on light sources and ambient light.
Smart Images
Figure CN119722798A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer vision technology, and in particular to a visual ranging method, device, robot and computer-readable storage medium. Background Art
[0002] Currently, commonly used distance measurement methods include monocular distance measurement, binocular distance measurement, lidar distance measurement, structured light visual distance measurement and time of flight (ToF) distance measurement. Monocular distance measurement uses a single camera to obtain environmental images, extracts feature points through image processing algorithms, and obtains distance through camera parameters. Binocular distance measurement uses two monocular cameras to simultaneously obtain environmental images with common areas, performs stereo matching to obtain parallax, and then calculates the depth of objects in the picture through known fixed parameters. Lidar distance measurement obtains the distance of the target by emitting a laser beam and measuring the reflection time. Structured light visual distance measurement calculates the depth of the object by projecting a known light pattern onto the surface of the object being measured and analyzing the deformation of the light spot on the surface of the object. ToF distance measurement calculates the distance by emitting a light pulse and measuring the time required for it to reflect back.
[0003] However, all of the above ranging methods have certain disadvantages. First, monocular ranging has poor stability. Second, binocular ranging has high computational complexity. Third, the equipment cost of lidar ranging and ToF ranging is high. Finally, structured light vision ranging has strong restrictions on light sources and is more sensitive to interference from ambient light.
[0004] In summary, how to effectively solve the problems of poor ranging stability, high computational complexity, high equipment cost, strong restrictions on light sources, and high sensitivity to interference from ambient light is an issue that technical personnel in this field urgently need to solve. Summary of the invention
[0005] The purpose of this application is to provide a visual ranging method, which improves the stability and accuracy of the ranging results; another purpose of this application is to provide a visual ranging device, a robot and a computer-readable storage medium.
[0006] In order to solve the above technical problems, this application provides the following technical solutions:
[0007] A visual distance measurement method, comprising:
[0008] Obtain the target pixel coordinates of each obstacle point in the pixel coordinate system through a monocular camera;
[0009] Calculate the first absolute depth of each obstacle point according to each target pixel coordinate;
[0010] Get the image to be detected containing each obstacle point;
[0011] Using the pre-trained monocular depth estimation model to process the image to be detected to obtain a target depth map;
[0012] Determine the target relative depth of each obstacle point according to the target depth map;
[0013] Convert each target relative depth into a corresponding second absolute depth;
[0014] The first absolute depths and the second absolute depths are fused to obtain a target distance measurement result.
[0015] In a specific implementation of the present application, obtaining the target pixel coordinates of each obstacle point in the pixel coordinate system by a monocular camera includes:
[0016] Acquire the initial pixel coordinates of each obstacle point in the ground coordinate system through the monocular camera;
[0017] Obtaining the camera intrinsic parameters of the monocular camera;
[0018] The initial pixel coordinates are converted into target pixel coordinates according to the camera intrinsic parameters.
[0019] In a specific implementation of the present application, determining the target relative depth of each obstacle point according to the target depth map includes:
[0020] Extracting the target pixel coordinates of each obstacle point from the target depth map;
[0021] Taking the target pixel coordinates of each obstacle point as the center, select obstacle points contained in a preset range from the target depth map;
[0022] For each obstacle point, extracting relative depth data corresponding to each selected obstacle point from the target depth map to obtain an initial relative depth set;
[0023] Performing relative depth data screening on the initial relative depth set to obtain a target relative depth set;
[0024] The relative depth data in the target relative depth set are averaged to obtain the target relative depth.
[0025] In a specific implementation of the present application, the initial relative depth set is subjected to relative depth data screening to obtain a target relative depth set, including:
[0026] Sorting the relative depth data in the initial relative depth set by size to obtain an initial relative depth sequence;
[0027] Relative depth data of a preset proportion are deleted from both ends of the initial relative depth sequence to obtain an adjusted relative depth sequence;
[0028] The adjusted relative depth sequence is determined as the target relative depth set.
[0029] In a specific implementation of the present application, the first absolute depths and the second absolute depths are merged, including:
[0030] Get the preset weight hyperparameters;
[0031] The first absolute depth and the second absolute depth corresponding to each obstacle point are weightedly calculated according to the weight hyperparameter.
[0032] In a specific implementation of the present application, a training process of the monocular depth estimation model is also included, and the training process of the monocular depth estimation model includes:
[0033] Collect continuous video frames to obtain a video frame sequence;
[0034] Determine the first frame of two consecutive frames in the video frame sequence as a source frame, and determine the second frame as a target frame;
[0035] Use the depth estimation network to obtain the depth estimation output of each target frame;
[0036] Use the pose estimation network to estimate the camera pose of each target frame to the corresponding source frame;
[0037] Obtaining pixel coordinates of each first ground coordinate system in each target frame, and converting each first ground coordinate system pixel coordinate into first pixel coordinate system pixel coordinate;
[0038] Calculate the pixel coordinates of each second pixel coordinate system in each source frame according to the camera posture and the pixel coordinates of each first pixel coordinate system, and convert the pixel coordinates of each second pixel coordinate system into pixel coordinates of a second ground coordinate system;
[0039] Obtaining pixel values corresponding to pixel coordinates of each second ground coordinate system from each source frame;
[0040] Reconstructing each target frame according to each first ground coordinate system pixel coordinate and each pixel value to obtain each reconstructed image;
[0041] Calculate the photometric loss between each target frame and the corresponding reconstructed image respectively;
[0042] The depth estimation network is iteratively trained based on each photometric loss to obtain the monocular depth estimation model.
[0043] In a specific implementation of the present application, converting each relative depth into a corresponding second absolute depth includes:
[0044] Each relative depth was converted to a corresponding second absolute depth using a pre-built multiple linear regression model.
[0045] A visual distance measuring device, comprising:
[0046] The target pixel coordinate acquisition module is used to obtain the target pixel coordinates of each obstacle point in the pixel coordinate system through a monocular camera;
[0047] A first absolute depth calculation module, used to calculate the first absolute depth of each obstacle point according to each target pixel coordinate;
[0048] The module for acquiring the image to be detected is used to obtain the image to be detected containing each obstacle point;
[0049] A target depth map acquisition module is used to process the image to be detected using a pre-trained monocular depth estimation model to obtain a target depth map;
[0050] A target relative depth determination module, used to determine the target relative depth of each obstacle point according to the target depth map;
[0051] A second absolute depth acquisition module, used for converting the relative depth of each target into a corresponding second absolute depth;
[0052] The target distance measurement result acquisition module is used to fuse each first absolute depth and each second absolute depth to obtain a target distance measurement result.
[0053] A robot, comprising:
[0054] Memory for storing computer programs;
[0055] A processor is used to implement the steps of the visual distance measurement method as described above when executing the computer program.
[0056] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the visual distance measurement method described above are implemented.
[0057] The visual ranging method provided by the present application obtains the target pixel coordinates of each obstacle point in a pixel coordinate system through a monocular camera; calculates the first absolute depth of each obstacle point according to each target pixel coordinate; obtains a to-be-detected image containing each obstacle point; uses a pre-trained monocular depth estimation model to process the to-be-detected image to obtain a target depth map; determines the target relative depth of each obstacle point according to the target depth map; converts each target relative depth into a corresponding second absolute depth; and fuses each first absolute depth and each second absolute depth to obtain a target ranging result.
[0058] It can be seen from the above technical solution that by using a monocular camera to obtain the target pixel coordinates of each obstacle point in the pixel coordinate system, and calculating the first absolute depth of each obstacle point according to each target pixel coordinate, the ranging result calculated by the traditional ranging algorithm is obtained. The target depth map is obtained by pre-training the monocular depth estimation model to process the image to be detected, and the second absolute depth of each obstacle point is determined according to the target depth map, so as to obtain the ranging result calculated by deep learning. The first absolute depth and the second absolute depth are fused to obtain the target ranging result. The final target ranging result is obtained by combining deep learning with the traditional ranging algorithm, which makes full use of the advantages of monocular ranging based on deep learning that is less affected by camera posture changes and the high precision of the traditional ranging algorithm, improves the stability and accuracy of the ranging result, and saves costs.
[0059] Correspondingly, the present application also provides a visual ranging device, a robot and a computer-readable storage medium corresponding to the above-mentioned visual ranging method, which have the above-mentioned technical effects and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0060] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0061] Figure 1 This is an implementation flow chart of the visual ranging method in the embodiment of the present application;
[0062] Figure 2 Another implementation flow chart of the visual ranging method in the embodiment of the present application;
[0063] Figure 3 This is a structural block diagram of a visual distance measurement device in an embodiment of the present application;
[0064] Figure 4This is a structural block diagram of a robot in an embodiment of the present application;
[0065] Figure 5 A schematic diagram of the specific structure of a robot provided in an embodiment of the present application. DETAILED DESCRIPTION
[0066] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.
[0067] See also Figure 1 , Figure 1 This is an implementation flow chart of the visual distance measurement method in the embodiment of the present application. The method may include the following steps:
[0068] S101: Obtain target pixel coordinates of each obstacle point in a pixel coordinate system through a monocular camera.
[0069] When obstacle point ranging is required, such as when obstacle point ranging is required during autonomous driving and advanced assisted driving, or when obstacle point ranging is required during mowing by a lawn mower robot, the target pixel coordinates of each obstacle point in the pixel coordinate system are obtained through a monocular camera.
[0070] S102: Calculate the first absolute depth of each obstacle point according to each target pixel coordinate.
[0071] After obtaining the target pixel coordinates of each obstacle point in the pixel coordinate system, the first absolute depth of each obstacle point is calculated according to each target pixel coordinate, thereby obtaining a distance measurement result calculated using a traditional distance measurement algorithm.
[0072] S103: Obtain a picture to be detected that includes each obstacle point.
[0073] An image acquisition device is used in advance to obtain a picture to be detected containing each obstacle point.
[0074] Image acquisition devices can include cameras, camcorders, scanners, mobile phone cameras, micro-single cameras, drone cameras, industrial cameras, etc.
[0075] S104: Process the image to be detected using the pre-trained monocular depth estimation model to obtain a target depth map.
[0076] A monocular depth estimation model is pre-trained. After obtaining the image to be detected containing each obstacle point, the pre-trained monocular depth estimation model is used to process the image to be detected to obtain a target depth map.
[0077] A depth map is an image that represents the distance between each pixel in the scene and the camera.
[0078] S105: Determine the target relative depth of each obstacle point according to the target depth map.
[0079] The target depth map contains relative depth data corresponding to each obstacle point. After the target depth map is obtained, the target relative depth of each obstacle point is determined according to the target depth map.
[0080] S106: Convert each target relative depth into a corresponding second absolute depth.
[0081] Since relative depth can only indicate the relative position between each obstacle point and the reference object, and cannot directly determine the distance between each obstacle point and the camera, after determining the target relative depth of each obstacle point, each target relative depth is converted into the corresponding second absolute depth, thereby obtaining the distance measurement result calculated by deep learning.
[0082] S107: Fusing each first absolute depth and each second absolute depth to obtain a target distance measurement result.
[0083] For each obstacle point, after obtaining the first absolute depth calculated by the traditional ranging algorithm and the second absolute depth calculated by deep learning, the first absolute depth and the second absolute depth are fused to obtain the target ranging result. The final target ranging result is obtained by combining deep learning with the traditional ranging algorithm, which makes full use of the advantages of monocular ranging based on deep learning that is less affected by camera posture changes and the high precision of the traditional ranging algorithm, thereby improving the stability and accuracy of the ranging result.
[0084] It can be seen from the above technical solution that by using a monocular camera to obtain the target pixel coordinates of each obstacle point in the pixel coordinate system, and calculating the first absolute depth of each obstacle point according to each target pixel coordinate, the ranging result calculated by the traditional ranging algorithm is obtained. The target depth map is obtained by pre-training the monocular depth estimation model to process the image to be detected, and the second absolute depth of each obstacle point is determined according to the target depth map, so as to obtain the ranging result calculated by deep learning. The first absolute depth and the second absolute depth are fused to obtain the target ranging result. The final target ranging result is obtained by combining deep learning with the traditional ranging algorithm, which makes full use of the advantages of monocular ranging based on deep learning that is less affected by camera posture changes and the high precision of the traditional ranging algorithm, improves the stability and accuracy of the ranging result, and saves costs.
[0085] It should be noted that, based on the above embodiments, the embodiments of the present application also provide corresponding improved solutions. In the subsequent embodiments, the same steps or corresponding steps as those in the above embodiments can be referenced to each other, and the corresponding beneficial effects can also be referenced to each other, which will not be repeated one by one in the following improved embodiments.
[0086] See also Figure 2 , Figure 2 This is another implementation flow chart of the visual ranging method in the embodiment of the present application. The method may include the following steps:
[0087] S201: Obtain the initial pixel coordinates of each obstacle point in the ground coordinate system through a monocular camera.
[0088] The initial pixel coordinates of each obstacle point in the ground coordinate system are obtained through a monocular camera.
[0089] After the monocular camera captures the picture, it is fed into the semantic segmentation model to obtain the outlines of all obstacles in the picture and extract the pixel homogeneous coordinates of all ground points of the outline, that is, the initial pixel coordinates of each obstacle point in the ground coordinate system. .
[0090] S202: Obtain the camera intrinsic parameters of the monocular camera.
[0091] After obtaining the initial pixel coordinates of each obstacle point in the ground coordinate system, the camera intrinsic parameters of the monocular camera are obtained, such as obtaining the camera intrinsic parameter k through camera calibration.
[0092] S203: Convert each initial pixel coordinate into each target pixel coordinate according to the camera intrinsic parameter.
[0093] Get the camera installation height After obtaining the camera intrinsic parameters of the monocular camera, the initial pixel coordinates are converted into the target pixel coordinates in the pixel coordinate system according to the camera intrinsic parameters. ,in, .
[0094] The pixel coordinates of each obstacle point are converted from the ground coordinate system to the pixel coordinate system.
[0095] S204: Calculate the first absolute depth of each obstacle point according to each target pixel coordinate.
[0096] After obtaining the pixel coordinates of each target, the first absolute depth of each obstacle point is calculated according to the pixel coordinates of each target. , as shown below:
[0097] ;
[0098] Thus, the first absolute depth of each obstacle point is calculated.
[0099] S205: Obtain the image to be detected that contains each obstacle point.
[0100] S206: Process the image to be detected using the pre-trained monocular depth estimation model to obtain a target depth map.
[0101] After obtaining each image to be detected, the pre-trained monocular depth estimation model is used to process the image to be detected to obtain the target depth map. .
[0102] S207: Extracting the target pixel coordinates of each obstacle point from the target depth map.
[0103] The target depth map contains the target pixel coordinates of each obstacle point in the pixel coordinate system. After processing the image to be detected and obtaining the target depth map, the target pixel coordinates of each obstacle point are extracted from the target depth map. .
[0104] S208: Taking the target pixel coordinates of each obstacle point as the center, select obstacle points contained in a preset range from the target depth map.
[0105] After the target pixel coordinates of each obstacle point are extracted, obstacle points within a preset range are selected from the target depth map with the target pixel coordinates of each obstacle point as the center.
[0106] It should be noted that the preset range area can be set and adjusted according to actual conditions, and the embodiment of the present application does not limit this. For example, it can be set to take the target pixel coordinates of each obstacle point as the center, and The length of the upper side is The square area can also be set to the target pixel coordinates of each obstacle point as the center. Select a circular area with a preset radius.
[0107] S209: For each obstacle point, extract the relative depth data corresponding to each selected obstacle point from the target depth map to obtain an initial relative depth set.
[0108] After selecting the obstacle points contained in the preset range area, for each obstacle point, the relative depth data corresponding to each selected obstacle point is extracted from the target depth map to obtain an initial relative depth set. By preliminarily screening the relative depth data according to the preset range area, the relative depth data in the screened initial relative depth set has a greater correlation with the obstacle point.
[0109] S210: Screening the initial relative depth set for relative depth data to obtain a target relative depth set.
[0110] After the initial relative depth set is obtained, the relative depth data of the initial relative depth set is screened to obtain a target relative depth set, thereby achieving further screening of the relative depth data.
[0111] In a specific implementation of the present application, step S210 may include the following steps:
[0112] Step 1: sort the relative depth data in the initial relative depth set to obtain an initial relative depth sequence;
[0113] Step 2: Deleting a preset proportion of relative depth data from both ends of the initial relative depth sequence to obtain an adjusted relative depth sequence;
[0114] Step 3: Determine the adjusted relative depth sequence as the target relative depth set.
[0115] For the convenience of description, the above three steps can be combined for explanation.
[0116] After obtaining the initial relative depth set, the relative depth data in the initial relative depth set are sorted to obtain the initial relative depth sequence. , subscript Indicates relative depth, superscript Represents each element as a set where:
[0117] , Indicates sorting the collection. Indicates taking coordinate The element at .
[0118] Relative depth data of a preset proportion are deleted from both ends of the initial relative depth sequence to obtain an adjusted relative depth sequence, and the adjusted relative depth sequence is determined as the target relative depth set. By deleting the relative depth data at both ends of the initial relative depth sequence and retaining only the relative depth data in the middle, the similarity between each relative depth data in the target relative depth set and the relative depth of the obstacle point is further improved.
[0119] S211: Calculate the mean of each relative depth data in the target relative depth set to obtain the target relative depth.
[0120] After the target relative depth set is obtained by screening, the mean value of each relative depth data in the target relative depth set is calculated to obtain the target relative depth. By calculating the mean value of each relative depth data in the target relative depth set, the accuracy of the calculated target relative depth is further improved.
[0121] For example, for the initial relative depth set According to the percentage interception, the front and back exclusion ratios Relative depth data, Between 0 and 0.5, keep the middle The data is then averaged to obtain the relative depth of the current obstacle point. , and finally get the relative depth of all obstacle points .
[0122] S212: Convert each relative depth into a corresponding second absolute depth using a pre-built multiple linear regression model.
[0123] A multiple linear regression model is pre-constructed, and after the target relative depth is calculated, each relative depth is converted into a corresponding second absolute depth using the pre-constructed multiple linear regression model.
[0124] The construction and calibration process of the multivariate linear regression model can include:
[0125] (1) Construct a multiple linear regression model as shown below:
[0126] ;
[0127] in, is a hyperparameter, i.e., the highest power, is the parameter that needs to be optimized, is the absolute depth predicted by the multivariate linear regression model.
[0128] (2) Obtaining the loss function value: Here, the residual sum of squares error is used as the loss function of the multivariate linear regression model, as shown in the following formula:
[0129] ;
[0130] (3) Least squares optimization to obtain the best model parameters .
[0131] By using the output results for online calibration, the mapping relationship from the relative depth to the absolute depth output by the deep learning model is obtained.
[0132] S213: Obtain preset weight hyperparameters.
[0133] After each relative depth is converted into a corresponding second absolute depth, a preset weight hyperparameter is obtained.
[0134] S214: Perform weighted calculation on the first absolute depth and the second absolute depth corresponding to each obstacle point according to the weight hyperparameter to obtain a target distance measurement result.
[0135] After the weight hyperparameters are obtained, the first absolute depth and the second absolute depth corresponding to each obstacle point are weightedly calculated according to the weight hyperparameters to obtain the target ranging result.
[0136] The weighted calculation can be performed using the following formula:
[0137] ;
[0138] in, is a weight hyperparameter between 0 and 1.
[0139] In a specific implementation of the present application, the method may further include a training process of a monocular depth estimation model, and the training process of the monocular depth estimation model may include the following steps:
[0140] Step 1: Collect continuous video frame sequences;
[0141] Step 2: Determine the first frame of two consecutive frames in the video frame sequence as the source frame, and determine the second frame as the target frame;
[0142] Step 3: Use the depth estimation network to obtain the depth estimation output of each target frame;
[0143] Step 4: Use the posture estimation network to estimate the camera posture of each target frame to the corresponding source frame;
[0144] Step 5: Obtain the pixel coordinates of each first ground coordinate system in each target frame, and convert the pixel coordinates of each first ground coordinate system into the pixel coordinates of the first pixel coordinate system;
[0145] Step 6: Calculate the pixel coordinates of each second pixel coordinate system in each source frame according to the camera posture and the pixel coordinates of each first pixel coordinate system, and convert the pixel coordinates of each second pixel coordinate system into pixel coordinates of the second ground coordinate system;
[0146] Step 7: Obtain pixel values corresponding to pixel coordinates of each second ground coordinate system from each source frame;
[0147] Step 8: reconstruct each target frame according to each pixel coordinate of the first ground coordinate system and each pixel value to obtain each reconstructed image;
[0148] Step 9: Calculate the photometric loss between each target frame and the corresponding reconstructed image;
[0149] Step 10: Iteratively train the depth estimation network based on various photometric losses to obtain a monocular depth estimation model.
[0150] For the convenience of description, the above ten steps can be combined for explanation.
[0151] Collect a continuous video frame sequence and determine the previous frame of two consecutive frames in the video frame sequence as the source frame , and determine the next frame as the target frame , using the depth estimation network Get each target frame separately The depth estimation output , as shown below:
[0152] ;
[0153] Using pose estimation network Estimate each target frame separately To the corresponding source frame Camera pose , as shown below:
[0154] ;
[0155] Get the pixel coordinates of each first ground coordinate system in each target frame , and convert each pixel coordinate of the first ground coordinate system into the pixel coordinate of the first pixel coordinate system , as shown below:
[0156] ;
[0157] Among them, k is the camera intrinsic parameter.
[0158] According to the camera posture and the pixel coordinates of each first pixel coordinate system Calculate the pixel coordinates of each second pixel coordinate system in each source frame , as shown below:
[0159] ;
[0160] Convert each second pixel coordinate system pixel coordinate to the second ground coordinate system pixel coordinate , as shown below:
[0161] ;
[0162] in, express of Axis coordinates.
[0163] Get the pixel coordinates of each second ground coordinate system from each source frame The corresponding pixel values ,if If it is a decimal, use The difference between the pixel values in the area is obtained.
[0164] Reconstruct each target frame according to each first ground coordinate system pixel coordinate and each pixel value to obtain each reconstructed image , as shown below:
[0165] ;
[0166] The photometric loss between each target frame and the corresponding reconstructed image is calculated separately, as shown in the following formula:
[0167] ;
[0168] in, is the structural similarity loss, It is the preset weight, and its value is between 0 and 1.
[0169] The depth estimation network is iteratively trained based on each photometric loss to obtain a monocular depth estimation model. For example, a photometric loss threshold can be set in advance. When the source frame and the target frame are selected in sequence according to the generation timing of each video frame in the video frame sequence to iteratively train the depth estimation network, when the calculated photometric loss is greater than or equal to the photometric loss threshold, the source frame and the target frame are continued to be iteratively trained according to the generation timing of the video frame; when it is determined that the calculated photometric loss is less than the photometric loss threshold, the iteration is stopped. By setting the photometric loss threshold as the judgment condition for whether to stop iterative training of the depth estimation network, overfitting is avoided while ensuring the accuracy of the trained model.
[0170] By iteratively training the depth estimation network, a monocular depth estimation model is obtained, which improves the accuracy of the ranging results calculated by deep learning.
[0171] Corresponding to the above method embodiment, the present application also provides a visual distance measurement device. The visual distance measurement device described below and the visual distance measurement method described above can be referenced to each other.
[0172] See also Figure 3 , Figure 3 This is a structural block diagram of a visual distance measurement device in an embodiment of the present application, and the device may include:
[0173] The target pixel coordinate acquisition module 31 is used to obtain the target pixel coordinates of each obstacle point in the pixel coordinate system through a monocular camera;
[0174] A first absolute depth calculation module 32, used to calculate the first absolute depth of each obstacle point according to each target pixel coordinate;
[0175] The to-be-detected image acquisition module 33 is used to acquire the to-be-detected image containing each obstacle point;
[0176] A target depth map acquisition module 34 is used to process the image to be detected using the pre-trained monocular depth estimation model to obtain a target depth map;
[0177] A target relative depth determination module 35 is used to determine the target relative depth of each obstacle point according to the target depth map;
[0178] A second absolute depth obtaining module 36, used to convert each target relative depth into a corresponding second absolute depth;
[0179] The target distance measurement result obtaining module 37 is used to fuse each first absolute depth and each second absolute depth to obtain a target distance measurement result.
[0180] It can be seen from the above technical solution that by using a monocular camera to obtain the target pixel coordinates of each obstacle point in the pixel coordinate system, and calculating the first absolute depth of each obstacle point according to each target pixel coordinate, the ranging result calculated by the traditional ranging algorithm is obtained. The target depth map is obtained by pre-training the monocular depth estimation model to process the image to be detected, and the second absolute depth of each obstacle point is determined according to the target depth map, so as to obtain the ranging result calculated by deep learning. The first absolute depth and the second absolute depth are fused to obtain the target ranging result. The final target ranging result is obtained by combining deep learning with the traditional ranging algorithm, which makes full use of the advantages of monocular ranging based on deep learning that is less affected by camera posture changes and the high precision of the traditional ranging algorithm, improves the stability and accuracy of the ranging result, and saves costs.
[0181] In a specific implementation of the present application, the target pixel coordinate obtaining module 31 includes:
[0182] The initial pixel coordinate acquisition submodule is used to obtain the initial pixel coordinates of each obstacle point in the ground coordinate system through a monocular camera;
[0183] The camera intrinsic parameter acquisition submodule is used to obtain the camera intrinsic parameters of the monocular camera;
[0184] The target pixel coordinate acquisition submodule is used to convert each initial pixel coordinate into each target pixel coordinate according to the camera intrinsic parameters.
[0185] In a specific implementation of the present application, the target relative depth determination module 35 includes:
[0186] The target pixel coordinate extraction submodule is used to extract the target pixel coordinates of each obstacle point from the target depth map;
[0187] The obstacle point selection submodule is used to select obstacle points contained in a preset range from the target depth map with the target pixel coordinates of each obstacle point as the center;
[0188] The initial relative depth set acquisition submodule is used to extract the relative depth data corresponding to each selected obstacle point from the target depth map for each obstacle point, and obtain an initial relative depth set;
[0189] The target relative depth set acquisition submodule is used to screen the relative depth data of the initial relative depth set to obtain the target relative depth set;
[0190] The target relative depth acquisition submodule is used to calculate the mean of each relative depth data in the target relative depth set to obtain the target relative depth.
[0191] In a specific implementation of the present application, the target relative depth set acquisition submodule includes:
[0192] An initial relative depth sequence obtaining unit is used to sort the relative depth data in the initial relative depth set to obtain an initial relative depth sequence;
[0193] The adjusted relative depth sequence obtaining unit is used to delete relative depth data of a preset proportion from both ends of the initial relative depth sequence to obtain an adjusted relative depth sequence;
[0194] The target relative depth set determining unit is configured to determine the adjusted relative depth sequence as a target relative depth set.
[0195] In a specific implementation of the present application, the target ranging result obtaining module 37 includes:
[0196] The weight hyperparameter acquisition submodule is used to obtain the preset weight hyperparameters;
[0197] The weighted calculation submodule is used to perform weighted calculation on the first absolute depth and the second absolute depth corresponding to each obstacle point according to the weight hyperparameter.
[0198] In a specific implementation of the present application, the device may further include a monocular depth estimation model training module, and the monocular depth estimation model training module includes:
[0199] The video frame sequence acquisition submodule is used to collect continuous video frames to obtain a video frame sequence;
[0200] A source frame and target frame determination submodule, used to determine the first frame of two consecutive frames in the video frame sequence as the source frame, and determine the second frame as the target frame;
[0201] A depth estimation output acquisition submodule is used to obtain the depth estimation output of each target frame using the depth estimation network;
[0202] The camera pose estimation submodule is used to estimate the camera pose of each target frame to the corresponding source frame using the pose estimation network;
[0203] The first pixel coordinate acquisition submodule is used to obtain the pixel coordinates of each first ground coordinate system in each target frame, and convert the pixel coordinates of each first ground coordinate system into the pixel coordinates of the first pixel coordinate system;
[0204] A second pixel coordinate obtaining submodule is used to calculate the pixel coordinates of each second pixel coordinate system in each source frame according to the camera posture and the pixel coordinates of each first pixel coordinate system, and convert the pixel coordinates of each second pixel coordinate system into pixel coordinates of the second ground coordinate system;
[0205] A pixel value acquisition submodule, used for acquiring pixel values corresponding to pixel coordinates of each second ground coordinate system from each source frame;
[0206] The reconstructed image acquisition submodule is used to reconstruct each target frame according to each first ground coordinate system pixel coordinate and each pixel value to obtain each reconstructed image;
[0207] A photometric loss calculation submodule, used to calculate the photometric loss between each target frame and the corresponding reconstructed image;
[0208] The monocular depth estimation model acquisition submodule is used to iteratively train the depth estimation network based on various photometric losses to obtain a monocular depth estimation model.
[0209] In a specific implementation of the present application, the second absolute depth obtaining module 36 is specifically a module that converts each relative depth into a corresponding second absolute depth using a pre-built multiple linear regression model.
[0210] Corresponding to the above method embodiment, see Figure 4 , Figure 4 A schematic diagram of a robot provided for this application, the device may include:
[0211] A memory 332, for storing computer programs;
[0212] The processor 322 is used to implement the steps of the visual distance measurement method of the above method embodiment when executing the computer program.
[0213] For details, please refer to Figure 5 , Figure 5 A schematic diagram of a specific structure of a robot provided in this embodiment, which may have relatively large differences due to different configurations or performances, may include a processor (central processing units, CPU) 322 (for example, one or more processors) and a memory 332, and the memory 332 stores one or more computer programs 342 or data 344. Among them, the memory 332 can be a temporary storage or a permanent storage. The program stored in the memory 332 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the data processing device. Furthermore, the processor 322 can be configured to communicate with the memory 332 to execute a series of instruction operations in the memory 332 on the robot 301.
[0214] The robot 301 may further include one or more power supplies 326 , one or more wired or wireless network interfaces 350 , one or more input and output interfaces 358 , and / or one or more operating systems 341 .
[0215] The steps in the visual distance measurement method described above can be implemented by the structure of a robot.
[0216] Corresponding to the above method embodiment, the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the following steps can be implemented:
[0217] The target pixel coordinates of each obstacle point in the pixel coordinate system are obtained by a monocular camera; the first absolute depth of each obstacle point is calculated according to each target pixel coordinate; the image to be detected containing each obstacle point is obtained; the pre-trained monocular depth estimation model is used to process the image to be detected to obtain a target depth map; the target relative depth of each obstacle point is determined according to the target depth map; each target relative depth is converted into a corresponding second absolute depth; each first absolute depth and each second absolute depth are fused to obtain a target ranging result.
[0218] The computer-readable storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.
[0219] For an introduction to the computer-readable storage medium provided in this application, please refer to the above method embodiment, and this application will not go into details here.
[0220] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device, robot and computer-readable storage medium disclosed in the embodiment, since they correspond to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0221] Specific examples are used herein to illustrate the principles and implementation methods of the present application, and the description of the above embodiments is only used to help understand the technical solution and core ideas of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the present application.
Claims
1. A visual distance measurement method, characterized in that: include: Obtain the target pixel coordinates of each obstacle point in the pixel coordinate system through a monocular camera; Calculate the first absolute depth of each obstacle point according to each target pixel coordinate; Get the image to be detected containing each obstacle point; Using the pre-trained monocular depth estimation model to process the image to be detected to obtain a target depth map; Determine the target relative depth of each obstacle point according to the target depth map; Convert each target relative depth into a corresponding second absolute depth; The first absolute depths and the second absolute depths are fused to obtain a target distance measurement result.
2. The visual distance measurement method according to claim 1, characterized in that: The target pixel coordinates of each obstacle point in the pixel coordinate system are obtained through a monocular camera, including: Acquire the initial pixel coordinates of each obstacle point in the ground coordinate system through the monocular camera; Obtaining the camera intrinsic parameters of the monocular camera; The initial pixel coordinates are converted into target pixel coordinates according to the camera intrinsic parameters.
3. The visual distance measurement method according to claim 1, characterized in that: Determining the target relative depth of each obstacle point according to the target depth map includes: Extracting the target pixel coordinates of each obstacle point from the target depth map; Taking the target pixel coordinates of each obstacle point as the center, select obstacle points contained in a preset range from the target depth map; For each obstacle point, extracting relative depth data corresponding to each selected obstacle point from the target depth map to obtain an initial relative depth set; Performing relative depth data screening on the initial relative depth set to obtain a target relative depth set; The relative depth data in the target relative depth set are averaged to obtain the target relative depth.
4. The visual distance measurement method according to claim 3, characterized in that: The initial relative depth set is subjected to relative depth data screening to obtain a target relative depth set, including: Sorting the relative depth data in the initial relative depth set by size to obtain an initial relative depth sequence; Relative depth data of a preset proportion are deleted from both ends of the initial relative depth sequence to obtain an adjusted relative depth sequence; The adjusted relative depth sequence is determined as the target relative depth set.
5. The visual distance measurement method according to claim 1, characterized in that: The first absolute depths and the second absolute depths are fused, including: Get the preset weight hyperparameters; The first absolute depth and the second absolute depth respectively corresponding to each obstacle point are weightedly calculated according to the weight hyperparameter.
6. The visual distance measurement method according to claim 1, characterized in that: The method also includes a training process of the monocular depth estimation model, wherein the training process of the monocular depth estimation model includes: Collect continuous video frames to obtain a video frame sequence; Determine the first frame of two consecutive frames in the video frame sequence as a source frame, and determine the second frame as a target frame; Use the depth estimation network to obtain the depth estimation output of each target frame; Use the pose estimation network to estimate the camera pose of each target frame to the corresponding source frame; Obtaining pixel coordinates of each first ground coordinate system in each target frame, and converting each first ground coordinate system pixel coordinate into first pixel coordinate system pixel coordinate; Calculating the pixel coordinates of each second pixel coordinate system in each source frame according to the camera posture and the pixel coordinates of each first pixel coordinate system, and converting the pixel coordinates of each second pixel coordinate system into pixel coordinates of a second ground coordinate system; Obtaining pixel values corresponding to pixel coordinates of each second ground coordinate system from each source frame; Reconstructing each target frame according to each first ground coordinate system pixel coordinate and each pixel value to obtain each reconstructed image; Calculate the photometric loss between each target frame and the corresponding reconstructed image respectively; The depth estimation network is iteratively trained based on each photometric loss to obtain the monocular depth estimation model.
7. The visual distance measurement method according to claim 1, characterized in that: Convert each relative depth to the corresponding second absolute depth, including: Each relative depth was converted to a corresponding second absolute depth using a pre-built multiple linear regression model.
8. A visual distance measuring device, characterized in that: include: The target pixel coordinate acquisition module is used to obtain the target pixel coordinates of each obstacle point in the pixel coordinate system through a monocular camera; A first absolute depth calculation module, used to calculate the first absolute depth of each obstacle point according to each target pixel coordinate; The module for acquiring the image to be detected is used to obtain the image to be detected containing each obstacle point; A target depth map acquisition module is used to process the image to be detected using a pre-trained monocular depth estimation model to obtain a target depth map; A target relative depth determination module, used to determine the target relative depth of each obstacle point according to the target depth map; A second absolute depth acquisition module, used for converting the relative depth of each target into a corresponding second absolute depth; The target distance measurement result acquisition module is used to fuse each first absolute depth and each second absolute depth to obtain a target distance measurement result.
9. A robot, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the visual distance measurement method according to any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the visual distance measurement method according to any one of claims 1 to 7 are implemented.