A depth estimation method, device, electronic device and storage medium

Abstract

The present application provides a depth estimation method, device, electronic device and storage medium, which are used to improve the problem that the accuracy of the obtained depth information is difficult to meet driving requirements. The method includes: obtaining an image to be inferred, which is collected using a monocular camera; using a neural network model to infer the image to be inferred, and obtaining a predicted target frame in the image to be inferred and the estimated depth information corresponding to the predicted target frame. The neural network model is obtained after training with a monocular image. The monocular image is obtained by monocularizing an image collected by a binocular camera. The predicted target frame represents the location area of ​​the target object in the image to be inferred, and the estimated depth information is the distance between the target object and the monocular camera.

Description

Technical Field

[0001] The present application relates to the technical field of image processing and autonomous driving, and more specifically, to a depth estimation method, device, electronic device and storage medium. Background Art

[0002] In the field of assisted driving, automatic driving or unmanned driving, a monocular camera is usually used to collect the image directly in front of the car, and target detection is performed on the image directly in front of the car to identify the area position of the target object in the image directly in front of the car, and then a ranging sensor such as a laser radar and a millimeter-wave radar is used to measure the distance between the target object and the monocular camera. Since the target object is in the depth direction of the image directly in front of the car (which can be understood as the Z coordinate direction perpendicular to the XY coordinate), the distance between the target object and the monocular camera is usually also called the depth information of the target object.

[0003] In the specific practice, it is found that when the height of the monocular camera relative to the ground and the angle between the optical axis and the ground remain unchanged, the depth information determined by the above-mentioned ranging sensor is relatively accurate. However, if the car is always moving in an environment such as a mountain road or a bumpy road, it will brake, start, accelerate, decelerate, and go up and downhill. These situations cause the error of the depth information determined by the above-mentioned ranging sensor to increase rapidly, and the accuracy of the obtained depth information is difficult to meet driving requirements. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a depth estimation method, device, electronic device and storage medium, which are used to improve the problem that the accuracy of the obtained depth information is difficult to meet driving requirements.

[0005] The embodiment of the present application provides a depth estimation method, including: obtaining an image to be inferred, which is collected using a monocular camera; using a neural network model to infer the image to be inferred, obtaining a predicted target frame in the image to be inferred and estimated depth information corresponding to the predicted target frame, the neural network model is obtained after training using a monocular image, the monocular image is obtained by monocular processing of an image collected by a binocular camera, the predicted target frame represents the location area of ​​the target object in the image to be inferred, and the estimated depth information is the distance between the target object and the monocular camera. In the above implementation process, the image to be inferred collected by the monocular camera is inferred by using a neural network model. Since the neural network model is trained using an image processed by monocular processing of a more accurate binocular image, and the pixel points representing the real distance are screened out during the monocular processing, the accuracy of the trained neural network model is higher. The depth estimation method does not rely on external ranging sensors, etc., reduces the error caused by the time synchronization or spatial misalignment of the ranging sensor and the monocular camera, thereby improving the accuracy of the estimated depth information.

[0006] Optionally, in an embodiment of the present application, before using the neural network model to reason about the inference image, it also includes: acquiring a first image and a second image, the first image and the second image are different images captured by a binocular camera for a target object, and the distance between the aperture center points of the binocular camera is equal to a pre-calculated baseline value; performing monocular processing on the first image and the second image to obtain multiple monocular images, and target frames and depth information corresponding to the monocular images; using multiple monocular images as training data, and using target frames and depth information corresponding to multiple monocular images as training labels, the neural network is trained to obtain a neural network model. In the above implementation process, since the training data during collection is collected using a binocular camera, twice the amount of data can be obtained in the same time relative to obtaining monocular data, and the first image and the second image are monocularly processed to obtain more training data, and then more training data is used to train the neural network, so that the obtained neural network model has a higher accuracy rate during reasoning.

[0007] Optionally, in an embodiment of the present application, the first image and the second image are monocularly processed, including: marking a first target frame of the target object on the first image, and marking a second target frame of the target object on the second image, the target frame representing the position area of ​​the target object in the image; calculating the depth information of the target object according to the first target frame and the second target frame. In the above implementation process, the first image and the second image of the target object are acquired by using a binocular camera with a baseline value, and the baseline value is an optimal baseline value calculated in advance. When the distance between the center points of the binocular camera aperture is set to the optimal baseline value, the binocular camera can collect more pixels representing the real distance, thereby obtaining a more accurate image.

[0008] Optionally, in an embodiment of the present application, the depth information of the target object is calculated according to the first target frame and the second target frame, including: filtering out a pixel target frame with a smaller number of pixels in the target frame from the first target frame and the second target frame; calculating the depth information of all pixels in the pixel target frame according to the binocular ranging principle, the depth information of the pixel is the distance between the position point of the pixel in the target object and the binocular camera; removing outlier pixels and background pixels from all pixels in the pixel target frame to obtain multiple representative pixels; calculating the average value of the depth information of the multiple representative pixels, and determining the average value of the depth information as the depth information of the target object. In the above implementation process, the depth information of all pixels in the pixel target frame is calculated according to the binocular ranging principle, and the depth information of the target object is determined according to the depth information of all pixels in the pixel target frame, so that there is no need to use an additional sensor (such as an infrared rangefinder, etc.) to obtain depth information in the data acquisition stage, avoiding the problem of increased error caused by the need to synchronize or align multi-sensor data in the time and space dimensions, and effectively improving the accuracy of the estimated depth information.

[0009] Optionally, in an embodiment of the present application, the binocular camera includes: a first camera and a second camera; the depth information of all pixels in the pixel target frame is calculated according to the binocular ranging principle, including: feature matching of the first target frame and the second target frame to obtain the disparity between the first camera and the second camera; the depth information of all pixels in the pixel target frame is calculated according to the baseline value and the disparity. In the above implementation process, the disparity between the first camera and the second camera is obtained by feature matching of the first target frame and the second target frame, and the depth information of all pixels in the pixel target frame is calculated according to the baseline value and the disparity, so that there is no need to use an additional sensor for obtaining depth information (such as an infrared rangefinder, etc.) in the data acquisition stage, avoiding the problem of increased errors caused by the need to synchronize or align multi-sensor data in the time and space dimensions, and effectively improving the accuracy of the estimated depth information.

[0010] Optionally, in an embodiment of the present application, after obtaining the predicted target frame in the image to be inferred and the estimated depth information corresponding to the predicted target frame, it also includes: determining whether the distance corresponding to the estimated depth information is less than a preset distance; if so, generating and outputting an auxiliary warning signal, or decelerating through a braking system. In the above implementation process, by generating and outputting an auxiliary warning signal when the distance corresponding to the estimated depth information is less than a preset distance, or decelerating through a braking system, the problem of traffic accidents caused by not having enough time to react or not having enough time to brake and accelerate is avoided, and the safety of electronic products such as smart cars, drones, unmanned vehicles or robots in the assisted driving system is effectively improved.

[0011] Optionally, in an embodiment of the present application, after obtaining the predicted target frame in the image to be inferred and the estimated depth information corresponding to the predicted target frame, it also includes: determining whether the ratio of the predicted target frame to the image to be inferred is greater than a preset ratio; if so, replanning the path to avoid the target object in the predicted target frame. In the above implementation process, by replanning the path when the ratio of the predicted target frame to the image to be inferred is greater than a preset ratio, the target object in the predicted target frame is avoided, effectively improving the safety of electronic products such as smart cars, drones, unmanned vehicles or robots in the assisted driving system.

[0012] An embodiment of the present application also provides a depth estimation device, including: an inference image acquisition module, used to acquire an image to be inferred, which is collected using a monocular camera; an image depth estimation module, used to use a neural network model to infer the image to be inferred, and obtain a predicted target box in the image to be inferred and estimated depth information corresponding to the predicted target box, wherein the neural network model is obtained after training using a monocular image, and the monocular image is obtained by monocularizing an image collected by a binocular camera, wherein the predicted target box represents the location area of ​​the target object in the image to be inferred, and the estimated depth information is the distance between the target object and the monocular camera.

[0013] Optionally, in an embodiment of the present application, the depth estimation device also includes: a binocular image acquisition module, used to acquire a first image and a second image, the first image and the second image are different images captured by a binocular camera for a target object, and the distance between the aperture center points of the binocular camera is equal to a pre-calculated baseline value; an image monocular processing module, used to perform monocular processing on the first image and the second image to obtain multiple monocular images, and target frames and depth information corresponding to the monocular images; a network model training module, used to train a neural network using multiple monocular images as training data and target frames and depth information corresponding to the multiple monocular images as training labels to obtain a neural network model.

[0014] Optionally, in an embodiment of the present application, the image monocular processing module includes: a target image annotation module, used to annotate a first target frame of the target object on the first image, and to annotate a second target frame of the target object on the second image, the target frame representing a position area of ​​the target object in the image; and a depth information calculation module, used to calculate the depth information of the target object based on the first target frame and the second target frame.

[0015] Optionally, in an embodiment of the present application, a depth information calculation module includes: a target border screening module, used to screen out pixel target frames with fewer pixel points in the target frames from the first target frame and the second target frame; a depth information calculation module, used to calculate the depth information of all pixel points in the pixel target frame according to the binocular ranging principle, the depth information of the pixel point is the distance between the position point of the pixel point in the target object and the binocular camera; a depth information screening module, used to eliminate outlier pixels and background pixels from all pixel points in the pixel target frame to obtain multiple representative pixels; a depth information determination module, used to calculate the average depth information of multiple representative pixels, and determine the average depth information as the depth information of the target object.

[0016] Optionally, in an embodiment of the present application, the binocular camera includes: a first camera and a second camera; a depth information calculation module, including: a camera parallax acquisition module, used for the camera to perform feature matching on the first target frame and the second target frame to obtain the parallax between the first camera and the second camera; a pixel depth calculation module, used to calculate the depth information of all pixels in the pixel target frame based on the baseline value and the parallax.

[0017] Optionally, in an embodiment of the present application, the depth estimation device also includes: a depth distance judgment module, used to judge whether the distance corresponding to the estimated depth information is less than a preset distance; an auxiliary warning deceleration module, used to generate and output an auxiliary warning signal if the distance corresponding to the estimated depth information is less than the preset distance, or to decelerate through a braking system.

[0018] Optionally, in an embodiment of the present application, the depth estimation device also includes: a border ratio judgment module, used to judge whether the ratio value of the predicted target box to the image to be inferred is greater than a preset ratio; a path re-planning module, used to re-plan the path if the ratio value of the predicted target box to the image to be inferred is greater than a preset ratio, so as to avoid the target object in the predicted target box.

[0019] An embodiment of the present application further provides an electronic device, including: a processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the machine-readable instructions are executed by the processor, the method described above is performed.

[0020] An embodiment of the present application further provides a storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method described above is executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 A schematic diagram of the process of obtaining data provided in an embodiment of the present application is shown;

[0023] Figure 2 A schematic diagram of the binocular ranging principle provided by an embodiment of the present application is shown;

[0024] Figure 3 A schematic diagram showing the distribution of pixel depth information provided by an embodiment of the present application is shown;

[0025] Figure 4 A schematic diagram of a training neural network model provided in an embodiment of the present application is shown;

[0026] Figure 5 A schematic diagram of the monocular processing process provided by an embodiment of the present application is shown;

[0027] Figure 6 The flowchart of the model reasoning stage provided by the embodiment of the present application is shown;

[0028] Figure 7 A schematic diagram of the structure of a depth estimation device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0029] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0030] Before introducing the depth estimation method provided in the embodiment of the present application, some concepts involved in the embodiment of the present application are first introduced:

[0031] The target detection network refers to a neural network that detects target objects in an image, that is, it detects the target objects in the image and gives the position range, classification and probability of the target objects in the image. The position range can be specifically annotated in the form of a detection box, the classification refers to the specific category of the target object, and the probability refers to the probability that the target object in the detection box is of a specific category.

[0032] A server is a device that provides computing services over a network. Examples of servers include x86 servers and non-x86 servers. Non-x86 servers include mainframes, minicomputers, and UNIX servers.

[0033] It should be noted that the depth estimation method provided in the embodiment of the present application can be executed by an electronic device, where the electronic device refers to a device terminal or the above-mentioned server with the function of executing a computer program, and the device terminal is, for example: a smart phone, a personal computer (PC), a tablet computer, a personal digital assistant (PDA) or a mobile Internet device (MID), etc.

[0034] Before introducing the depth estimation method provided in the embodiment of the present application, the application scenarios applicable to the depth estimation method are first introduced. The application scenarios here include but are not limited to the fields of assisted driving, automatic driving or unmanned driving, etc., for example: the depth estimation method is used in systems such as assisted driving systems, drones, unmanned vehicles or robots to estimate the depth information in the image captured by the monocular camera, so as to improve the accuracy of depth estimation of the image captured by the monocular camera. In specific practice, the image target frame and image depth information obtained by the depth estimation method can also be used to train a neural network, and then the trained neural network model is used to infer the target frame and depth information of the target object in the image, for example: the depth information of the obstacle in front is inferred by the trained neural network model in systems such as assisted driving systems, drones, unmanned vehicles or robots, where the depth information can be understood as the distance between the obstacle in front and the acquisition camera, so that the assisted driving system, drone, unmanned vehicle or robot can perform obstacle avoidance processing, such as outputting auxiliary warnings, braking and deceleration, changing lanes or replanning paths, etc.

[0035] The following is an analysis of the reasons why the accuracy of determining depth information using the above-mentioned ranging sensor is poor: the first reason is that in environments such as mountain roads or bumpy roads, due to the different degrees of shaking of the ranging sensor and the monocular camera, when multiple targets to be detected appear on the monocular camera, the depth information measured by the multiple ranging sensors is difficult to match and align with the multiple targets to be detected in the monocular camera. Therefore, an additional correction algorithm is required to avoid the influence of the viewing angle range, alignment angle, and shaking degree of the ranging sensor and the monocular camera, and to perform matching and alignment on the basis of correction; the second reason is that the ranging action of the ranging sensor and the shooting action of the monocular camera are difficult to synchronize in time, that is, the depth information measured by the ranging sensor and the image shot by the monocular camera are difficult to obtain at the same time; if a high-precision control chip is used to control both, it is also difficult to achieve an error of less than milliseconds between the two.

[0036] In order to overcome the above defects, the embodiment of the present application proposes a depth estimation method. The main idea of ​​the depth estimation method is to use a neural network model to infer the image to be inferred collected by a monocular camera. Since the neural network model is trained using images processed by monocularization of binocular images with higher accuracy, and the pixels representing the real distance are screened out during the monocularization process, the accuracy of the trained neural network model is higher. The depth estimation method does not rely on external ranging sensors, etc., reduces the errors caused by the time synchronization or spatial misalignment of the ranging sensor and the monocular camera, thereby improving the accuracy of the estimated depth information.

[0037] It can be understood that the above-mentioned depth estimation method may include four stages: data acquisition, model training, model reasoning and depth information application; data acquisition refers to acquiring a binocular image, and converting the binocular image into a monocular image, a target frame and depth information. The binocular image here can be collected by a binocular camera set with an optimal baseline value; model training refers to using the above-mentioned monocular image, target frame and depth information to train a neural network to obtain a neural network model; model reasoning refers to using the neural network model to reason about the target frame and depth information of the image to be reasoned; depth information application refers to using the target frame and depth information for practical applications such as early warning, deceleration or re-planning of the path.

[0038] See also Figure 1 The flowchart of the data acquisition stage provided in the embodiment of the present application is shown; the data acquisition stage may specifically include:

[0039] Step S110: Acquire a first image and a second image, where the first image and the second image are different images captured by a binocular camera with respect to a target object.

[0040] A binocular image refers to different images captured by a binocular camera of a target object. For example, a binocular image may include a first image and a second image.

[0041] There are many implementation methods of the above-mentioned step S110, including but not limited to the following: a first implementation method, obtaining a pre-stored binocular image, for example: obtaining a binocular image from a file system, or obtaining a binocular image from a database, or obtaining a binocular image from a mobile storage device; a second acquisition method, using a browser or other software to obtain a binocular image on the Internet, or using other applications to access the Internet to obtain a binocular image; a third acquisition method, using a binocular camera to collect binocular images of multiple targets to be detected in front, and calculating an optimal baseline value based on these binocular images; then, setting the distance between the aperture center point of the first camera and the aperture center point of the second camera as the optimal baseline value; finally, using the binocular camera set with the optimal baseline value to collect binocular images of multiple targets to be detected in front.

[0042] The third implementation method described above is specifically as follows: a plurality of targets to be detected are placed at equal intervals on the axis within the farthest detection distance of the binocular camera, and the real distance of each target to be detected is obtained; the baseline value of the binocular camera is set to a smaller initial value, and then the binocular camera is used to collect the binocular image in front. Finally, the first camera is used to obtain the first image of the target object, and the second camera is used to obtain the second image of the target object, the first camera and the second camera are binocular cameras, and the distance between the aperture center point of the first camera and the aperture center point of the second camera is equal to the pre-calculated baseline value.

[0043] It should be noted that the first camera and the second camera can be just binocular cameras installed on a drone, an unmanned vehicle, an intelligent car with an assisted driving system or a robot, or they can be two cameras in a multi-eye camera on a drone, an unmanned vehicle, an intelligent car with an assisted driving system or a robot. For example, the first camera and the second camera are selected according to the baseline value from a three-eye camera, a four-eye camera or more than four-eye camera. The multi-eye camera here refers to multiple cameras that obtain pictures of the target object almost at the same time, for example, the controller simultaneously sends instructions to the multi-eye cameras to collect images.

[0044] See also Figure 2The schematic diagram of the binocular ranging principle provided by the embodiment of the present application is shown; it is assumed that the binocular camera includes a first camera on the left and a second camera on the right, and the first camera and the second camera are both controlled by the same single chip to simultaneously expose and collect images. Depth information can be calculated using the formula z=f*b / d, where z represents the measured distance from each target object O point to the binocular camera, f represents the focal length of the first camera or the second camera (the focal lengths of the two can be the same), b represents the baseline value of the current binocular camera, that is, the distance between the aperture center point C1 of the first camera and the aperture center point C2 of the second camera, and d represents the distance between the imaging center point P1 of the target object on the first camera and the imaging center point P2 on the first camera, that is, the parallax between the first camera and the second camera.

[0045] From the above analysis, it can be seen that, assuming that the focal length of the binocular camera remains unchanged and the distance between the binocular camera and the target object (i.e., the depth information of the target object) remains unchanged, increasing the baseline value of the binocular camera (i.e., the current distance between the first camera and the second camera) increases the parallax value of the first camera and the second camera, and more pixels representing the real distance can be obtained on the imaging plane. However, if the baseline value of the binocular camera is too large, the target objects closer to the edges on both sides cannot be simultaneously presented as planar images in the binocular camera. Therefore, it is necessary to calculate the optimal baseline value to avoid the situation where the target object cannot simultaneously present a planar image in the binocular camera.

[0046] The above method of calculating the optimal baseline value may be calculated by giving a given error range, and the specific process may include:

[0047] Step S111: Place multiple targets to be detected at equal intervals on the axis within the farthest detection distance of the binocular camera, set the baseline value of the binocular camera to a smaller initial value, then adjust the baseline value of the binocular camera, and use the binocular camera to collect binocular images of multiple target objects in front.

[0048] The implementation method of the above step S111 is, for example: adjusting the baseline value of the binocular camera, and using the binocular camera to collect binocular images of multiple target objects in front to obtain a binocular image, the binocular image includes a first image and a second image, the first image and the second image are simultaneously exposed and collected for the same target object. It can be understood that the above multiple target objects are all within the farthest detection distance of the binocular camera, and the farthest detection distance here is Figure 2 Dm can be used to represent it.

[0049] Step S112: label each target on the binocular image to obtain a target frame pair, where the target frame pair includes a first target frame and a second target frame.

[0050] The implementation of the above step S112 is, for example: marking a first target frame of each target object on the first image in the binocular image, and marking a second target frame of each target object on the second image in the binocular image. It can be understood that since the number of targets to be detected and binocular images is relatively small, the marking action here can be performed manually, thereby effectively improving the marking accuracy and image quality.

[0051] Step S113: Calculate the first target frame and the second target frame using the binocular ranging principle to obtain the measured distance from each target object to the binocular camera.

[0052] An implementation example of the above-mentioned step S113 is as follows: a pixel target frame with a smaller number of pixels in the target frame is screened out from the first target frame and the second target frame; the depth information of all pixels in the pixel target frame is calculated according to the binocular ranging principle; outlier pixels and background pixels are eliminated from all pixels in the pixel target frame to obtain multiple representative pixels; the average value of the depth information of the multiple representative pixels is calculated, and the average value of the depth information is determined as the measured distance between the target object and the binocular camera. The measured distance here is the depth information between the target object and the binocular camera. The specific calculation process of calculating the depth information of all pixels in the pixel target frame according to the binocular ranging principle will be described in detail in the following step S130.

[0053] Step S114: Count the errors between the measured distance and the actual distance of each target object, and then filter out the maximum error value from the errors between the measured distance and the actual distance.

[0054] The implementation method of the above step S114 is, for example: obtaining the actual distance from each target object to the binocular camera, and using the formula Count the error between the measured distance and the actual distance of each target object, and then use formula E m =max(E 1 ,E 2 ,E 3 ,…E i ,…E n ) The maximum error value is selected from the error between the measured distance and the true distance; where E i represents the error between the measured distance and the actual distance of the i-th target object, z i represents the measured distance from the i-th target object to the binocular camera, represents the real distance from the i-th target object to the binocular camera. The real distance here refers to the distance between the target to be detected and the binocular camera when the target to be detected is placed above. n represents the total number of target objects. The total number of target objects can be set according to the specific situation, such as 5 to 10, etc. E m Indicates the maximum error value filtered out from the error between the measured distance and the true distance.

[0055] Step S115: If the maximum error value is greater than the preset limit value, gradually increase the baseline value of the binocular camera, and then repeat steps S111 to S115 until the maximum error value is less than or equal to the preset limit value; wherein, the gradually increased baseline value of the binocular camera can be set according to the specific situation, for example, set to 0.1, 0.2, 0.4, 1, 3 or 7, etc., and the preset limit value can also be set according to the specific situation, for example, set to 2% or 5%, etc.

[0056] Step S116: If the maximum error value is less than or equal to the preset limit value, the baseline value of the binocular camera at this time is determined as the optimal baseline value.

[0057] After step S110 , step S120 is performed: a first target frame of the target object is marked on the first image, and a second target frame of the target object is marked on the second image.

[0058] The implementation method of the above step S120 is, for example: using a target detection network to detect a first target frame of the target object on the first image, and using the target detection network to detect a second target frame of the target object on the second image; wherein the target frame represents the position area of ​​the target object in the image, and the target detection network in this embodiment can use a regional convolutional neural network (RCNN), fast RCNN, faster RCNN and feature fusion single shot multibox detector (FSSD) and other networks. Of course, in the specific implementation process, manual annotation can also be used to annotate the target frame of the first image and the target frame of the second image.

[0059] After step S120, step S130 is performed: depth information of the target object is calculated according to the first target frame and the second target frame, where the depth information of the target object is the distance between the target object and the binocular camera.

[0060] The implementation of calculating the depth information of the target object according to the first target frame and the second target frame in the above step S130 may include:

[0061] Step S131: Filter out a pixel target frame having fewer pixel points from the first target frame and the second target frame.

[0062] Step S132: Calculate the depth information of all pixels in the pixel target frame according to the binocular ranging principle. The depth information of the pixel is the distance between the position of the pixel in the target object and the binocular camera.

[0063] An implementation example of the above-mentioned step S132 is: performing feature matching on the first target frame and the second target frame to obtain the disparity between the first camera and the second camera; calculating the focal length, baseline value and disparity of the first camera or the second camera according to the binocular ranging principle to obtain the depth information of all pixel points in the pixel target frame; wherein the focal length of the first camera and the focal length of the second camera may be equal.

[0064] Step S133: Remove outlier pixels and background pixels from all pixels in the pixel target frame to obtain multiple representative pixels.

[0065] See also Figure 3 The distribution diagram of pixel depth information provided by the embodiment of the present application is shown; the implementation method of the above step S133 is, for example: re-sorting the depth information of all pixels in the pixel target frame from small to large to obtain a sorted pixel set, where the sorted pixel set can be represented by Dw, and the number of elements in the set Dw is recorded as w. Then, outlier pixels and background pixels are removed from all pixels in the pixel target frame. Specifically, the Tth pixel can be directly selected from the sorted pixel set Dw. l to T h The multiple pixel points here can be represented as a set Ds, and the number of elements in the set Ds is represented as a; where T l =w·p l (truncation), T h =w·p h (Truncation),P l 、p h It can be set according to the specific application scenario. l 、p h It can be used to remove outlier pixels and background pixels in the target frame. For example, set P l 1%-3%, the p h Set it to 40%-60% etc.

[0066] In the above implementation process, by screening and averaging the depth information of all pixels within the target frame, it is avoided to determine the depth information of the target object only by the depth information of a single pixel, thereby effectively improving the accuracy of the depth information of the target object.

[0067] Step S134: Calculate an average value of depth information of a plurality of representative pixels, and determine the average value of the depth information as the depth information of the target object.

[0068] The implementation method of the above step S134 is, for example, to use the formula Calculate the average value of depth information of multiple pixels, and determine the average value of depth information as the depth information of the target object; where Z represents the average value of depth information, a represents the number of multiple pixels, and D k Represents the depth information of the kth pixel among multiple pixels. In the above implementation process, due to the volume limitation of many products, binocular cameras are usually difficult to play a good role in many products. For example, a binocular camera with a large baseline value (for example, a baseline value greater than 10 times the fuselage) cannot be installed on a drone, and the image parallax collected by a camera with a too small baseline value is difficult to meet the accuracy requirements. Therefore, the embodiment of the present application reflects the role of the binocular camera in the data acquisition stage, so that the baseline value of the binocular camera can be set to infinity in theory, effectively ensuring that the image parallax collected by the binocular camera meets the accuracy requirements.

[0069] See also Figure 4 The schematic diagram of the training neural network model provided by the embodiment of the present application is shown; the depth estimation method mentioned above may include four stages: data acquisition, model training, model reasoning and depth information application. The data acquisition stage is introduced above, and the model training stage is introduced in detail below. Optionally, in the embodiment of the present application, after estimating the depth information of the target object according to the first target frame and the second target frame, the depth information can also be used as a training label to train the model. The specific process of training the neural network model may include:

[0070] Step S210: Acquire a first image and a second image, where the first image and the second image are different images captured by a binocular camera with respect to a target object.

[0071] Among them, the implementation principle and implementation method of step S210 are similar to those of step S110. Therefore, its implementation principle and implementation method are not explained here. If there are any unclear points, please refer to the description of step S110.

[0072] Step S220: performing monocular processing on the first image and the second image to obtain a plurality of monocular images, and target frames and depth information corresponding to the monocular images.

[0073] See also Figure 5The process schematic diagram of the monocular processing provided by the embodiment of the present application is shown; the monocular processing refers to marking the location of the target frame and the depth information corresponding to the target frame in each image in the binocular image. The implementation method of the above step S220 is, for example: the first image is monocularly processed to obtain a first monocular image, and the second image is monocularly processed to obtain a second monocular image. Thereafter, the first monocular image and the second monocular image are independent of each other and have no connection, hereinafter referred to as monocular images. The process of the above monocular processing is, for example: the target frame of the target object on the first image is identified by using a target detection network, where the target frame on the first image can be represented as P1 (xL, y, wL, h); wherein xL represents the horizontal coordinate of the center point of the target frame of the target object in the first image, y represents the vertical coordinate of the target object in the first image, wL represents the width of the target frame of the target object in the first image, and h represents the height of the target frame of the target object in the first image. Similarly, the target frame P2 (xR, y, wR, h) on the second image is marked; the depth information of the target object is calculated in combination with the implementation of step S130, that is, the distance from the target object to the binocular camera can be represented as Z in the figure. The target frame P1 (xL, y, wL, h, Z) with depth information on the first monocular image and the target frame P2 (xR, y, wR, h, Z) with depth information on the second monocular image can be obtained.

[0074] Step S230: Using multiple monocular images as training data, and using depth information and target frames corresponding to the multiple monocular images as training labels, the neural network is trained to obtain a neural network model.

[0075] The implementation method of the above step S230 is, for example: using multiple monocular images as training data, using the target frames and depth information corresponding to the multiple monocular images as training labels, training the neural network, and using the neural network to estimate the predicted distance between the target object and the camera, and the predicted frame of the target object in the image. Then, the distance loss between the depth information of the target object and the predicted distance, and the regression loss between the target frame and the predicted frame are calculated respectively. Finally, the total loss value is calculated based on the distance loss and the regression loss, and then the network weight parameters of the neural network are updated according to the total loss value until the loss value is less than a preset ratio (for example, 5% or 10%, etc.) or the number of training batches (epochs) is greater than a preset threshold (for example, 100 or 1000, etc.), and the neural network model can be obtained. Among them, the neural network here can include: VGG network, ResNet network, YOLO network, MobileNet network, Wide ResNet network and Inception network, etc.

[0076] In the above implementation process, the neural network model trained with the monocular image is used to infer the image to be inferred collected by the monocular camera. The monocular image is obtained by monocularizing the image collected by the binocular camera. In other words, since the neural network model is trained with the monocularized image of the binocular image with higher accuracy, and the pixels representing the real distance are screened out during the monocularization process, the accuracy of the trained neural network model is higher. The depth estimation method does not rely on external ranging sensors, etc., which reduces the errors caused by the time synchronization or spatial misalignment of the ranging sensor and the monocular camera, thereby improving the accuracy of the estimated depth information.

[0077] See also Figure 6 The flowchart of the model reasoning stage provided by the embodiment of the present application is shown; the depth estimation method mentioned above may include four stages: data acquisition, model training, model reasoning and depth information application. The model training stage is introduced above, and the model reasoning stage is introduced in detail below. Optionally, in the embodiment of the present application, after obtaining the neural network model, the trained neural network model can also be used to reason about the image captured by the monocular camera (also called a monocular image). The process of reasoning about the monocular image may include:

[0078] Step S310: Acquire an image to be inferred, where the image to be inferred is captured using a monocular camera.

[0079] The methods for obtaining the image to be inferred in the above step S310 include: a first obtaining method, using a terminal device such as a monocular camera, a video recorder or a color camera to shoot the target object to obtain the image to be inferred; then the terminal device sends the image to be inferred to the electronic device, and then the electronic device receives the image to be inferred sent by the terminal device, and the electronic device can store the image to be inferred in a file system, a database or a mobile storage device; a second obtaining method, obtaining a pre-stored image to be inferred, for example: obtaining the image to be inferred from a file system, or obtaining the image to be inferred from a database, or obtaining the image to be inferred from a mobile storage device; a third obtaining method, using a browser or other software to obtain the image to be inferred on the Internet, or using other applications to access the Internet to obtain the image to be inferred.

[0080] Step S320: Use a neural network model to infer the image to be inferred to obtain a predicted target box and estimated depth information corresponding to the predicted target box. The neural network model is obtained after training using a monocular image. The monocular image is obtained by monocularizing an image captured by a binocular camera.

[0081] The implementation method of the above-mentioned step S320 includes: using a neural network model to reason about the image to be inferred, obtaining a predicted target box and estimated depth information corresponding to the predicted target box; wherein the predicted target box represents the location area of ​​the target object in the image to be inferred, and the estimated depth information is the distance between the target object in the image to be inferred and the acquisition camera of the image to be inferred. The above-mentioned depth estimation method can also be applied to electronic products such as assisted driving systems, drones, unmanned vehicles or robots.

[0082] The depth estimation method mentioned above may include four stages: data acquisition, model training, model reasoning, and depth information application. The model reasoning stage is introduced above, and the depth information application stage is introduced in detail below.

[0083] Optionally, after obtaining the predicted target frame and the estimated depth information corresponding to the predicted target frame, electronic products such as drones, unmanned vehicles or robots can also be warned or decelerated. The warning or deceleration process may include: judging whether the distance corresponding to the estimated depth information is less than the preset distance; if the distance corresponding to the estimated depth information is less than the preset distance, then generating and outputting an auxiliary warning signal, or decelerating through the braking system. This implementation example: electronic products such as smart cars, drones, unmanned vehicles or robots of the assisted driving system judge whether the distance corresponding to the estimated depth information is less than the preset distance; if the distance corresponding to the estimated depth information is less than the preset distance, then the electronic products such as smart cars, drones, unmanned vehicles or robots of the assisted driving system generate and output auxiliary warning signals, or the electronic products such as smart cars, drones, unmanned vehicles or robots of the assisted driving system decelerate through the braking system; wherein the above-mentioned preset distance can be set according to the specific situation, for example, the preset distance is set to 100 meters or 150 meters, etc.

[0084] In the above implementation process, when the distance corresponding to the estimated depth information is less than the preset distance, an auxiliary warning signal is generated and output, or the vehicle is decelerated through the braking system, thereby avoiding the problem of traffic accidents caused by not having enough time to react or brake or accelerate, and effectively improving the safety of electronic products such as smart cars, drones, unmanned vehicles or robots with assisted driving systems.

[0085] Optionally, if the above-mentioned depth estimation method is applied to electronic products such as drones, unmanned vehicles or robots, then after obtaining the predicted target frame and the estimated depth information corresponding to the predicted target frame, the path can be replanned to actively avoid the target object in the predicted target frame (the target object can be a roadblock, a pedestrian or an oncoming vehicle, etc.), and the process of replanning the path can include: judging whether the ratio value of the predicted target frame to the image to be inferred is greater than a preset ratio; if the ratio value of the predicted target frame to the image to be inferred is greater than the preset ratio, then replanning the path to avoid the target object in the predicted target frame. This implementation example: the electronic products such as smart cars, drones, unmanned vehicles or robots of the assisted driving system judge whether the ratio value of the predicted target frame to the image to be inferred is greater than the preset ratio; if the ratio value of the predicted target frame to the image to be inferred is greater than the preset ratio, then the electronic products such as smart cars, drones, unmanned vehicles or robots of the assisted driving system replan the path to avoid the target object in the predicted target frame, where the target object can be an obstacle in the road; wherein the preset ratio here can be set according to the specific situation, for example, the preset ratio is set to 80% or 90%, etc.

[0086] In the above implementation process, when the ratio of the predicted target box to the image to be inferred is greater than the preset ratio, the path is replanned to avoid the target object in the predicted target box, thereby effectively improving the safety of electronic products such as smart cars, drones, unmanned vehicles or robots in the assisted driving system.

[0087] See also Figure 7 The schematic diagram of the structure of the depth estimation device provided by the embodiment of the present application is shown. The embodiment of the present application provides a depth estimation device 400, including:

[0088] The inference image acquisition module 410 is used to acquire the image to be inferred, where the image to be inferred is collected using a monocular camera;

[0089] The image depth estimation module 420 is used to use a neural network model to infer the image to be inferred, and obtain a predicted target box in the image to be inferred and the estimated depth information corresponding to the predicted target box. The neural network model is obtained after training using a monocular image. The monocular image is obtained by monocularizing the image captured by the binocular camera. The predicted target box represents the location area of ​​the target object in the image to be inferred, and the estimated depth information is the distance between the target object and the monocular camera.

[0090] Optionally, in the embodiment of the present application, the depth estimation device further includes:

[0091] A binocular image acquisition module, used to acquire a first image and a second image, where the first image and the second image are different images captured by a binocular camera for a target object, and a distance between the aperture center points of the binocular camera is equal to a pre-calculated baseline value;

[0092] An image monocular processing module, used to perform monocular processing on the first image and the second image to obtain a plurality of monocular images, and target frames and depth information corresponding to the monocular images;

[0093] The network model training module is used to train the neural network using multiple monocular images as training data and target frames and depth information corresponding to the multiple monocular images as training labels to obtain a neural network model.

[0094] Optionally, in an embodiment of the present application, the image monocular processing module includes:

[0095] The target image annotation module is used to annotate a first target frame of the target object on the first image and to annotate a second target frame of the target object on the second image, wherein the target frame represents a position area of ​​the target object in the image.

[0096] The depth information calculation module is used to calculate the depth information of the target object according to the first target frame and the second target frame.

[0097] Optionally, in the embodiment of the present application, the depth information calculation module includes:

[0098] The target frame screening module is used to screen out a pixel target frame with a smaller number of pixel points in the target frame from the first target frame and the second target frame.

[0099] The depth information calculation module is used to calculate the depth information of all pixels in the pixel target frame based on the binocular ranging principle. The depth information of the pixel is the distance between the position of the pixel in the target object and the binocular camera.

[0100] The depth information filtering module is used to remove outlier pixels and background pixels from all pixels in the pixel target frame to obtain multiple representative pixels.

[0101] The depth information determination module is used to calculate an average value of depth information of a plurality of representative pixel points, and determine the average value of the depth information as the depth information of the target object.

[0102] Optionally, in an embodiment of the present application, the binocular camera includes: a first camera and a second camera; and a depth information calculation module includes:

[0103] The camera parallax acquisition module is used for the camera to perform feature matching on the first target frame and the second target frame to obtain the parallax between the first camera and the second camera.

[0104] The pixel depth calculation module is used to calculate the depth information of all pixels in the pixel target box based on the baseline value and disparity.

[0105] Optionally, in the embodiment of the present application, the depth estimation device may further include:

[0106] The depth distance judgment module is used to judge whether the distance corresponding to the estimated depth information is less than a preset distance.

[0107] The auxiliary warning and deceleration module is used to generate and output an auxiliary warning signal if the distance corresponding to the estimated depth information is less than a preset distance, or to decelerate through the braking system.

[0108] Optionally, in the embodiment of the present application, the depth estimation device may further include:

[0109] The frame ratio judgment module is used to judge whether the ratio value of the predicted target frame to the image to be inferred is greater than a preset ratio.

[0110] The path replanning module is used to replan the path to avoid the target object in the predicted target box if the ratio of the predicted target box to the image to be inferred is greater than a preset ratio.

[0111] It should be understood that the device corresponds to the above-mentioned depth estimation method embodiment and can perform the various steps involved in the above-mentioned method embodiment. The specific functions of the device can be referred to in the above description. To avoid repetition, the detailed description is appropriately omitted here. The device includes at least one software function module that can be stored in a memory in the form of software or firmware or solidified in the operating system (OS) of the device.

[0112] An electronic device provided in an embodiment of the present application includes: a processor and a memory, the memory storing machine-readable instructions executable by the processor, and the above method is performed when the machine-readable instructions are executed by the processor.

[0113] An embodiment of the present application further provides a storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the above method is executed.

[0114] Among them, the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, disk or optical disk.

[0115] In several embodiments provided by the embodiments of the present application, it should be understood that the disclosed device and method can also be implemented in other ways. The device embodiments described above are only schematic, for example, the flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a part of a module, a program segment or a code, and a part of a module, a program segment or a code includes one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also be different from the order of occurrence marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which is mainly determined by the functions involved.

[0116] In addition, the functional modules of the various embodiments in the embodiments of the present application can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part.

[0117] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0118] The above description is only an optional implementation manner of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the embodiments of the present application, which should be covered within the protection scope of the embodiments of the present application.

Claims

1. A depth estimation method, characterized in that: include: Acquire an image to be inferred, where the image to be inferred is collected using a monocular camera; Inferring the image to be inferred using a neural network model to obtain a predicted target frame in the image to be inferred and estimated depth information corresponding to the predicted target frame, wherein the neural network model is obtained after training using a monocular image, and the monocular image is obtained by monocularizing an image captured by a binocular camera, the predicted target frame represents the location area of ​​the target object in the image to be inferred, and the estimated depth information is the distance between the target object and the monocular camera; Wherein, before using the neural network model to infer the image to be inferred, it also includes: acquiring a first image and a second image, wherein the first image and the second image are different images collected by the binocular camera for the target object, and the distance between the aperture center points of the binocular camera is equal to a pre-calculated baseline value; marking a first target frame of the target object on the first image, and marking a second target frame of the target object on the second image, the target frame representing the position area of ​​the target object in the image; filtering out a pixel target frame with a small number of pixel points in the target frame from the first target frame and the second target frame; calculating the depth information of all pixels in the pixel target frame; removing outlier pixels and background pixels from all pixels in the pixel target frame to obtain multiple representative pixels; calculating the average value of the depth information of the multiple representative pixels, and determining the average value of the depth information as the depth information of the target object; using multiple monocular images as training data, and using the target frames and depth information corresponding to the multiple monocular images as training labels, to train the neural network.

2. The method according to claim 1, characterized in that: The binocular camera includes: a first camera and a second camera; the depth information of all pixels in the pixel target frame is calculated, including: Performing feature matching on the first target frame and the second target frame to obtain a disparity between the first camera and the second camera; Depth information of all pixels within the pixel target frame is calculated according to the baseline value and the disparity.

3. The method according to any one of claims 1-2, characterized in that: After obtaining the predicted target frame in the image to be inferred and the estimated depth information corresponding to the predicted target frame, the method further includes: Determining whether the distance corresponding to the estimated depth information is less than a preset distance; If so, an auxiliary warning signal is generated and output, or the vehicle is decelerated through the braking system.

4. The method according to any one of claims 1-2, characterized in that: After obtaining the predicted target frame in the image to be inferred and the estimated depth information corresponding to the predicted target frame, the method further includes: Determine whether the ratio of the predicted target frame to the image to be inferred is greater than a preset ratio; If so, the path is replanned to avoid the target object in the predicted target frame.

5. A depth estimation device, characterized in that: include: An inference image acquisition module is used to acquire an image to be inferred, where the image to be inferred is collected using a monocular camera; An image depth estimation module is used to use a neural network model to infer the image to be inferred, and obtain a predicted target frame in the image to be inferred and estimated depth information corresponding to the predicted target frame, wherein the neural network model is obtained after training using a monocular image, and the monocular image is obtained by monocularizing an image captured by a binocular camera, and the predicted target frame represents the location area of ​​the target object in the image to be inferred, and the estimated depth information is the distance between the target object and the monocular camera; Wherein, before using the neural network model to infer the image to be inferred, it also includes: acquiring a first image and a second image, wherein the first image and the second image are different images collected by the binocular camera for the target object, and the distance between the aperture center points of the binocular camera is equal to a pre-calculated baseline value; marking a first target frame of the target object on the first image, and marking a second target frame of the target object on the second image, the target frame representing the position area of ​​the target object in the image; filtering out a pixel target frame with a small number of pixel points in the target frame from the first target frame and the second target frame; calculating the depth information of all pixels in the pixel target frame; removing outlier pixels and background pixels from all pixels in the pixel target frame to obtain multiple representative pixels; calculating the average value of the depth information of the multiple representative pixels, and determining the average value of the depth information as the depth information of the target object; using multiple monocular images as training data, and using the target frames and depth information corresponding to the multiple monocular images as training labels, to train the neural network.

6. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the machine-readable instructions are executed by the processor, the method according to any one of claims 1 to 4 is performed.

7. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 4 is executed.

Images