Monocular vision-based target depth estimation method and device, and vehicle

Abstract

The method comprises the steps of: acquiring original video frame data of a target object collected by a camera; extracting a plurality of target images from the original video frame data, wherein the target video frame data is composed of the plurality of target images; identifying the target object in the target images to obtain actual pixel coordinates of the target object; sequentially determining predicted pixel coordinates of the target object in other target images except for the latest target image in the target video frame data in time sequence; and updating the target depth value according to the predicted pixel coordinates and the actual pixel coordinates. The actual pixel coordinates of the target object in the target image can be used to determine the predicted pixel coordinates of the target object in the next target image adjacent to the target image, and then the target depth value is updated according to the distance between the actual pixel coordinates and the predicted pixel coordinates, so that the real-time requirement can be met.

Description

Monocular vision-based target depth estimation methods, devices, and vehicles Technical Field

[0001] This disclosure relates to autonomous driving technology, and more particularly to a target depth estimation method, device, and vehicle based on monocular vision. Background Technology

[0002] With the increasing demand for vehicle safety and comfort, intelligent driving technology has received widespread attention and research. Environmental perception is a crucial foundation for realizing vehicle intelligence. Target depth estimation, which estimates the longitudinal distance between a target object and the camera installed in the vehicle (or approximates the longitudinal distance between the target object and the vehicle), is a vital step in environmental perception.

[0003] Existing technologies primarily utilize spatial measurement methods and rely on camera pose data for calculations to estimate target depth. However, the reliance on camera pose data results in a large computational load, leading to slow calculation speeds and making it difficult to meet real-time requirements. Summary of the Invention

[0004] This disclosure provides a target depth estimation method, device, and vehicle based on monocular vision to solve the problem that existing technologies using spatial measurement methods for target depth estimation are difficult to meet real-time requirements.

[0005] According to a first aspect of this disclosure, a target depth estimation method based on monocular vision is provided, comprising:

[0006] The system acquires raw video frame data of the target object captured by a camera installed on the vehicle during its operation; and extracts multiple target images from the raw video frame data in chronological order to form target video frame data.

[0007] Identify the target objects in each target image to obtain the actual pixel coordinates of the target objects;

[0008] In chronological order, the predicted pixel coordinates of the target objects in the target video frame data, excluding the latest target image, are determined sequentially. The predicted pixel coordinates of the target objects in each other target image are determined based on the actual pixel coordinates of the target objects in the adjacent previous target image, the initial depth value obtained, the parameter information of the camera, the driving information of the vehicle, and a preset motion model.

[0009] The target depth value of the target object in the latest target image is updated based on the predicted pixel coordinates and actual pixel coordinates of the target object in other target images; wherein the depth value is used to characterize the estimated longitudinal distance between the target object and the camera when the latest target image was acquired.

[0010] According to a second aspect of this disclosure, a target depth estimation device based on monocular vision is provided, comprising:

[0011] The acquisition unit is used to acquire the original video frame data of the target object captured by the camera installed on the vehicle during the vehicle's operation; and to extract multiple target images from the original video frame data in chronological order, so as to form target video frame data from the multiple target images.

[0012] The recognition unit is used to identify the target objects in each target image to obtain the actual pixel coordinates of the target objects;

[0013] The prediction unit is used to determine the predicted pixel coordinates of target objects in other target images (excluding the latest target image) in the target video frame data in chronological order. The predicted pixel coordinates of the target objects in each other target image are determined based on the actual pixel coordinates of the target objects in the adjacent previous target image, the initial depth value obtained, the parameter information of the camera, the driving information of the vehicle, and a preset motion model.

[0014] An update unit is used to update the target depth value of the target object in the latest target image based on the predicted pixel coordinates and actual pixel coordinates of the target object in other target images; wherein the depth value is used to characterize the estimated longitudinal distance between the target object and the camera when the latest target image was acquired.

[0015] According to a third aspect of this disclosure, a vehicle control device is provided, including a memory and a processor; wherein,

[0016] The memory is used to store computer programs;

[0017] The processor is configured to read the computer program stored in the memory and execute the target depth estimation method based on monocular vision as described in the first aspect according to the computer program in the memory.

[0018] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided, wherein computer-executable instructions are stored therein, and when a processor executes the computer-executable instructions, the target depth estimation method based on monocular vision as described in the first aspect is implemented.

[0019] According to a fifth aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the monocular vision-based target depth estimation method as described in the first aspect.

[0020] The target depth estimation method, apparatus, and vehicle based on monocular vision disclosed herein include: acquiring original video frame data of a target object captured by a camera mounted on the vehicle during vehicle operation; extracting multiple target images from the original video frame data in chronological order to form target video frame data; identifying the target object in each target image to obtain the actual pixel coordinates of the target object; sequentially determining the predicted pixel coordinates of the target objects in other target images in the target video frame data, excluding the latest target image, in chronological order; wherein the predicted pixel coordinates of the target object in each other target image are determined based on the actual pixel coordinates of the target object in the adjacent previous target image, the initial depth value obtained, the camera parameter information, the vehicle's driving information, and a preset motion model; updating the target depth value of the target object in the latest target image based on the predicted and actual pixel coordinates of the target objects in the other target images; wherein the depth value is used to characterize the estimated longitudinal distance between the target object and the camera when the latest target image was acquired. The target depth estimation method, device, and vehicle based on monocular vision provided in this solution can determine the predicted pixel coordinates of the target object in the next adjacent target image by using the actual pixel coordinates of the target object in the target image and a preset motion model. Then, the acquired target depth value can be updated based on the distance between the actual pixel coordinates and the predicted pixel coordinates of the target object in the acquired target image. Compared with spatial measurement methods, this method does not rely on pose data, optimizes only the single parameter of depth, has low computational load, is fast, and can meet real-time requirements. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 is a flowchart illustrating a target depth estimation method based on monocular vision, as shown in an exemplary embodiment of this disclosure.

[0023] Figure 2 is a flowchart illustrating a target depth estimation method based on monocular vision, as shown in another exemplary embodiment of this disclosure;

[0024] Figure 3 is a schematic diagram illustrating, in an exemplary embodiment of the present disclosure, the third target image arranged in chronological order from late to early.

[0025] Figure 4 is a schematic diagram illustrating, in an exemplary embodiment of the present disclosure, the second target image is sorted in chronological order from late to early.

[0026] Figure 5 is a schematic diagram illustrating, in an exemplary embodiment of the present disclosure, the first target image is sorted in chronological order from late to early.

[0027] Figure 6 is a schematic diagram of a motion model illustrated in an exemplary embodiment of this disclosure;

[0028] Figure 7 is a schematic diagram illustrating the camera mounting angle in an exemplary embodiment of this disclosure;

[0029] Figure 8 is a schematic diagram illustrating a target depth estimation process based on monocular vision in an exemplary embodiment of this disclosure;

[0030] Figure 9 is a structural diagram of a target depth estimation device based on monocular vision, as shown in an exemplary embodiment of this disclosure;

[0031] Figure 10 is a structural diagram of a vehicle control device illustrated in an exemplary embodiment of this disclosure. Detailed Implementation

[0032] With the increasing demand for vehicle safety and comfort, intelligent driving technology has received widespread attention and research. Environmental perception is a crucial foundation for realizing vehicle intelligence. Target depth estimation, which estimates the longitudinal distance between a target object and the camera installed in the vehicle (or approximates the longitudinal distance between the target object and the vehicle), is a vital step in environmental perception.

[0033] In existing technologies, target depth estimation is generally based on deep learning or spatial measurement. The former requires a large amount of image data for training and needs real depth values ​​for annotation; the latter relies on camera pose data for calculation.

[0034] However, spatial measurement methods rely on camera pose data, resulting in high computational costs and slow processing speeds, making it difficult to meet real-time requirements. Deep learning methods, on the other hand, have high demands on computational resources and data, making them more complex to implement.

[0035] To address the aforementioned technical issues, the solution provided in this disclosure utilizes the actual pixel coordinates of the target object in the target image and a preset motion model to determine the predicted pixel coordinates of the target object in the next adjacent target image. Then, the target depth value can be updated based on the distance between the actual and predicted pixel coordinates of the target object in the acquired target image. Compared to spatial measurement methods, this approach does not rely on pose data, optimizes only the depth value as a single parameter, has low computational complexity, and is fast, meeting real-time requirements. Compared to deep learning methods, it significantly reduces the required data volume, eliminates the need for manual annotation, and has low requirements for computational resources and data, making it easy to implement.

[0036] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0037] The technical solutions of this disclosure and how they solve the aforementioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of this disclosure will now be described with reference to the accompanying drawings.

[0038] Figure 1 is a flowchart illustrating a target depth estimation method based on monocular vision, as shown in an exemplary embodiment of this disclosure.

[0039] As shown in Figure 1, the target depth estimation method based on monocular vision provided in this embodiment includes:

[0040] Step 101: Obtain the original video frame data of the target object captured by the camera installed on the vehicle during the vehicle's operation; and extract multiple target images from the original video frame data in chronological order to form target video frame data.

[0041] The method provided in this disclosure can be implemented by vehicle control equipment.

[0042] This vehicle control device can acquire raw video frame data of target objects collected by cameras installed on the vehicle while it is in motion.

[0043] In this solution, "monocular vision" refers to visual perception based on a single camera installed on the vehicle.

[0044] The target object can be a stationary object on the roadside during vehicle movement. Examples include poles and delineators (map elements). Alternatively, it can be other stationary targets in the road environment, such as traffic signs and traffic lights.

[0045] Specifically, multiple target images can be extracted from the original video frame data in chronological order (e.g., from late to early, or from early to late) to form target video frame data.

[0046] Optionally, the time for acquiring the target image can be close to the current time. Two consecutive frames of the target image can have the same time interval.

[0047] Step 102: Identify the target objects in each target image to obtain the actual pixel coordinates of the target objects.

[0048] Specifically, a pre-set method can be used to identify the target objects in each target image, and then the actual pixel coordinates of the target objects in each target image can be identified.

[0049] Step 103: In chronological order, determine the predicted pixel coordinates of the target objects in the target video frame data, except for the latest target image. The predicted pixel coordinates of the target objects in each other target image are determined based on the actual pixel coordinates of the target objects in the adjacent previous target image, the initial depth value, the camera parameter information, the vehicle driving information, and the preset motion model.

[0050] The initial depth value represents the estimated longitudinal distance between the target and the camera when the target image was acquired latest.

[0051] Specifically, the initial depth value is given by experience, usually in the range of 10 to 60 meters, because at this distance the target object is usually already within the camera's field of view.

[0052] The camera's parameter information can include both intrinsic and extrinsic parameters. The acquired camera parameter information is the calibrated information.

[0053] The vehicle's driving information may include its speed.

[0054] The preset motion model is a motion model pre-set according to the actual situation. This motion model is obtained by modeling the vehicle's trajectory and the target object's trajectory in the video.

[0055] Specifically, the predicted pixel coordinates of the target objects in the target video frame data, except for the latest target image, can be determined sequentially according to the time order (e.g., from late to early or from early to late).

[0056] Specifically, the predicted pixel coordinates of the target object in each other target image are determined based on the actual pixel coordinates of the target object in the adjacent previous target image, the obtained initial depth value, camera parameter information, vehicle driving information, and a preset motion model.

[0057] Specifically, theoretically, depth estimation can be performed using only two target images. However, the more frames involved in the calculation, the more accurate the depth estimation, but the computational efficiency decreases. For example, the target video frame data may include four target images arranged in ascending order of acquisition time: target image 1, target image 2, target image 3, and target image 4. Then, the predicted pixel coordinates of the target object in target image 2 can be predicted based on the actual pixel coordinates of the target object in target image 1; the predicted pixel coordinates of the target object in target image 3 can be predicted based on the actual pixel coordinates of the target object in target image 2; and the predicted pixel coordinates of the target object in target image 4 can be predicted based on the actual pixel coordinates of the target object in target image 3.

[0058] Step 104: Update the target depth value of the target object in the latest target image based on the predicted pixel coordinates and actual pixel coordinates of the target object in other target images; wherein, the depth value is used to characterize the estimated value of the longitudinal distance between the target object and the camera when the latest target image was acquired.

[0059] Among them, the target depth value is used to characterize the estimated longitudinal distance between the target object and the camera when the target image is acquired at the latest time.

[0060] Specifically, the cost function can be determined based on the distance between the actual and predicted pixel coordinates of the target object in the obtained target image. The optimal estimate of the target depth corresponds to the minimum solution of this cost function. This is achieved by minimizing the cost function through an optimization method. During this process, the depth value will continuously change until the upper limit of the optimization iterations is reached or the change in depth value falls below a certain range. At this point, the depth value is determined as the target depth value. The depth estimation process is thus complete, and the method has been verified to have high accuracy.

[0061] The target depth estimation method based on monocular vision disclosed herein includes: acquiring original video frame data of a target object captured by a camera mounted on the vehicle during vehicle operation; extracting multiple target images from the original video frame data in chronological order to form target video frame data; identifying the target object in each target image to obtain the actual pixel coordinates of the target object; sequentially determining the predicted pixel coordinates of the target objects in other target images in the target video frame data, excluding the latest target image, in chronological order; wherein the predicted pixel coordinates of the target object in each other target image are determined based on the actual pixel coordinates of the target object in the adjacent previous target image, the initial depth value obtained, the camera parameter information, the vehicle driving information, and a preset motion model; updating the target depth value of the target object in the latest target image based on the predicted pixel coordinates and actual pixel coordinates of the target objects in other target images; wherein the depth value is used to characterize the estimated longitudinal distance between the target object and the camera when the latest target image was acquired. The method disclosed herein utilizes the actual pixel coordinates of the target object in the target image and a preset motion model to determine the predicted pixel coordinates of the target object in the next adjacent target image. Then, the acquired target depth value can be updated based on the distance between the actual and predicted pixel coordinates of the target object in the acquired target image. Compared to spatial measurement methods, this method does not rely on pose data, optimizes only the single parameter of depth, has low computational complexity, and is fast, thus meeting real-time requirements.

[0062] Figure 2 is a flowchart illustrating a target depth estimation method based on monocular vision, as shown in another exemplary embodiment of this disclosure.

[0063] As shown in Figure 2, the target depth estimation method based on monocular vision provided in this embodiment includes:

[0064] Step 201: Obtain the original video frame data of the target object captured by the camera installed on the vehicle during the vehicle's operation; and extract multiple target images from the original video frame data in chronological order to form target video frame data.

[0065] Specifically, the principle and implementation of step 201 are similar to those of step 101, and will not be repeated here.

[0066] Step 202: Identify the target object in each target image and the bounding box containing the target object; determine the pixel coordinates of the intersection of the two diagonals of the bounding box as the actual pixel coordinates of the target object.

[0067] Specifically, after step 202, you can proceed to step 203 or step 204.

[0068] Specifically, a pre-set method can be used to identify the target objects in each target image, and the position of the target object in the target image can be enclosed in a box. The position of this box is then identified. Then, the diagonals of the box are connected, and the pixel coordinates of the intersection of the two diagonals are determined as the actual pixel coordinates of the target object.

[0069] Step 203: When there are M target images, in order from late to early time, from the target video frame data, based on the actual pixel coordinates of the target object in the Nth sorted target image, the initial depth value, camera parameter information, vehicle driving information, and the preset motion model, determine the predicted pixel coordinates in the (N+1)th sorted target image, increment N by 1, and repeat the steps in order from late to early time until the predicted pixel coordinates of the target object in the Mth sorted target image in the target video frame are determined; where N and M are both positive integers, and N is initially 1, and M is greater than 1.

[0070] Specifically, after step 203, step 205 can be executed.

[0071] Specifically, when there are M target images, the predicted pixel coordinates in the (N+1)th target image can be determined from the target video frame data in ascending order of time, based on the actual pixel coordinates of the target object in the Nth target image, the initial depth value, camera parameter information, vehicle driving information, and the preset motion model. Then, N is incremented by 1, and the steps in ascending order of time are repeated until the predicted pixel coordinates of the target object in the Mth target image in the target video frame are determined. Here, N and M are both positive integers, and N is initially 1, while M is greater than 1.

[0072] In one possible implementation, M=3, then the predicted pixel coordinates of the target object in the second ranked target image are determined according to the actual pixel coordinates of the target object in the first ranked target image, the initial depth value obtained, the camera parameter information and the vehicle driving information, as well as the preset motion model, in the order from late to early time.

[0073] Specifically, if M=3 is chosen, then according to the time order from late to early, the predicted pixel coordinates of the target object in the second ranked target image are determined first based on the actual pixel coordinates of the target object in the first ranked target image, the initial depth value obtained, the camera parameter information, the vehicle driving information, and the preset motion model.

[0074] Optionally, the camera's parameter information includes pixel size, focal length, yaw angle, pitch angle, and principal point coordinates; the vehicle's driving information includes vehicle speed information.

[0075] The predicted pixel coordinates of the target object in the second-ranked target image are then determined using the following formula:

[0076]

[0077] Where z′ represents the initial depth value; (x0, y0) represents the difference between the actual pixel coordinates and the principal point coordinates of the target object in the first ranked target image; (x1′, y1′) represents the difference between the predicted pixel coordinates and the principal point coordinates of the target object in the second ranked target image; Δz1 represents the distance traveled by the vehicle from the time of acquiring the second ranked target image to the time of acquiring the first ranked target image, and Δz1 is determined based on the vehicle speed information and the time difference between the time of acquiring the second ranked target image and the time of acquiring the first ranked target image; (f x f y ) represents the equivalent focal length, which is determined by dividing the focal length by the pixel size; θ represents the yaw angle; β represents the pitch angle.

[0078] Specifically, camera parameters may include pixel size, focal length, yaw angle, pitch angle, and principal point coordinates. The focal length is the calibrated focal length of the camera. The yaw and pitch angles are extrinsic parameters of the camera, and are also the calibrated yaw and pitch angles. The principal point coordinates are intrinsic parameters of the camera, and are the calibrated principal point coordinates, located at the center of the camera coordinate system.

[0079] Specifically, vehicle driving information can include vehicle speed information.

[0080] Specifically, Δz1 represents the distance the vehicle travels from the time when the second target image is acquired to the time when the first target image is acquired. Δz1 is determined based on the vehicle speed information (such as the average vehicle speed) between the time when the second target image is acquired and the time when the first target image is acquired, as well as the time difference between the time when the second target image is acquired and the time difference between the first target image and the time difference between the first target image and the time difference between each two adjacent frames.

[0081] Specifically, the equivalent focal length is determined by dividing the focal length by the pixel size.

[0082] Then, based on the actual pixel coordinates of the target object in the second ranked target image, the initial depth value, camera parameter information, vehicle driving information, and the preset motion model, the predicted pixel coordinates of the target object in the third ranked target image are determined.

[0083] Specifically, the predicted pixel coordinates of the target object in the third ranked target image can be determined based on the actual pixel coordinates of the target object in the second ranked target image, the initial depth value, camera parameter information, vehicle driving information, and a preset motion model. For example, taking a pole-shaped object in a road environment as an example, where the target object is the lower pole of a traffic sign, the third ranked target image is shown in Figure 3, the second ranked target image is shown in Figure 4, and the first ranked target image is shown in Figure 5.

[0084] Optionally, the camera's parameter information includes pixel size, focal length, yaw angle, pitch angle, and principal point coordinates; the vehicle's driving information includes vehicle speed information.

[0085] The predicted pixel coordinates of the target object in the third ranked target image are then determined using the following formula:

[0086]

[0087] Where z′ represents the initial depth value; (x1, y1) represents the difference between the actual pixel coordinates and the principal point coordinates of the target object in the second ranked target image; (x′2, y′2) represents the difference between the predicted pixel coordinates and the principal point coordinates of the target object in the third ranked target image; Δz2 represents the distance traveled by the vehicle from the time of acquiring the third ranked target image to the time of acquiring the second ranked target image, and Δz2 is determined based on the vehicle speed information and the time difference between the time of acquiring the third ranked target image and the time of acquiring the second ranked target image; (f x f y ) represents the equivalent focal length, which is determined by dividing the focal length by the pixel size; θ represents the yaw angle; β represents the pitch angle.

[0088] Specifically, Δz2 represents the distance the vehicle travels from the time of acquiring the third target image to the time of acquiring the second target image. Δz2 is determined based on the vehicle speed information (such as the average vehicle speed) between the times of acquiring the third and second target images, and the time difference between the times of acquiring the third and second target images (this time difference can be determined based on the number of frames between the third and second target images and the time difference between each two adjacent frames).

[0089] Optionally, depth estimation requires predicting the pixel coordinates of the target object in the next frame. This necessitates building a motion model that uses the pixel coordinates of the target object in the current frame as input. Since the pixel coordinates originate from the top-left corner of the image, the principal point coordinates can be subtracted from the pixel coordinates to convert them into camera coordinates for calculation.

[0090] As shown in Figure 6, assuming the vehicle is traveling from point O1 to point O2 from a top-down angle, the straight lines O1A and O2B represent the optical axis of the camera. The straight lines AD and BE lie on the imaging planes of the cameras at points O1 and O2, respectively. Point P is the observation point (i.e., the location of the target object), and D and E are the projection points of observation point P onto the image planes O1 and O2, respectively. Given that AD is x (the value in the X direction of the difference between the pixel coordinates of the target object and the principal point coordinates in the current frame), and O1A and O2B are f... x (i.e., the value in the X direction of the equivalent focal length), O1O2 is Δz (i.e., the distance the vehicle travels from the moment the current frame is captured to the moment the next frame is captured), let BE be x′ (i.e., the value in the X direction of the difference between the predicted pixel coordinates of the target object and the principal point coordinates in the next frame). To obtain x′, draw an auxiliary line perpendicular to the optical axis through point P, intersecting at points N and M. Draw an auxiliary line perpendicular to the optical axis through point O2, intersecting at point C. Let O1N be z. Consider the camera mounting angle, as shown in Figure 7, the angle difference between the camera coordinate system and the vehicle carrier coordinate system in the pitch, azimuth, and roll directions. The dark color represents the camera coordinate system; the light color represents the carrier coordinate system. When solving for x′, only the azimuth angle needs to be considered, i.e., the angle ∠CO1O2 between the optical axis and the XOZ plane of the carrier coordinate system. Let this angle be θ, representing the yaw angle of the camera.

[0091] Then the following relationship exists:

[0092] O2C=MN=Δz sinθ (1)

[0093] O1C=Δz cosθ (2)

[0094]

[0095] O2M=O1N-O1C (4)

[0096] MP = MN + NP (5)

[0097]

[0098] From formula (3), we can obtain:

[0099]

[0100] From formulas (2) and (4), we can obtain

[0101] O2M=z-Δz cosθ (8)

[0102] From formulas (1), (5), and (7), we can obtain...

[0103]

[0104] From formulas (6), (8), and (9), we can obtain...

[0105]

[0106] Right now

[0107]

[0108] Similarly, we can obtain

[0109]

[0110] Among them, f y y represents the value in the Y direction of the equivalent focal length; y represents the value in the Y direction of the difference between the pixel coordinates of the target object and the principal point coordinates in the current frame; y′ represents the value in the Y direction of the difference between the predicted pixel coordinates of the target object and the principal point coordinates in the next frame; β is the pitch angle, that is, the angle between the camera optical axis and the XOY plane of the carrier coordinate system in Figure 7.

[0111] The above derivation formula describes the process of predicting a target in a closer frame from a target in a farther frame during vehicle movement. In practical applications, it is necessary to estimate the depth of the nearest frame. Therefore, to predict the pixel coordinates of a target in a farther frame from a target in a closer frame, formula (11) is transformed into...

[0112]

[0113] At this point, O1N becomes an unknown quantity, and O2M becomes a known quantity. Let O2M be z′ (i.e., the initial depth value), then we can obtain the result from formula (8).

[0114] z=z′+Δz cosθ (14)

[0115] Substituting formula (14) into formula (13) yields

[0116]

[0117] Similarly, we can obtain

[0118]

[0119] At this time, x in formula (15) represents the value in the X direction of the difference between the predicted pixel coordinates of the target object and the principal point coordinates in the current frame; x′ represents the value in the X direction of the difference between the pixel coordinates of the target object and the principal point coordinates in the next frame of the current frame.

[0120] Meanwhile, in formula (16), y represents the value in the Y direction of the difference between the predicted pixel coordinates of the target object and the principal point coordinates in the current frame; y′ represents the value in the Y direction of the difference between the pixel coordinates of the target object and the principal point coordinates in the next frame of the current frame.

[0121] Step 204: When there are M target images, in chronological order from earliest to latest, based on the actual pixel coordinates of the target object in the Nth sorted target image, the initial depth value, camera parameter information, vehicle driving information, and the preset motion model, determine the predicted pixel coordinates in the (N+1)th sorted target image from the target video frame data. Increment N by 1, and repeat the steps in chronological order from earliest to latest until the predicted pixel coordinates of the target object in the (M-1)th sorted target image in the target video frame are determined. Here, N and M are both positive integers, and N is initially 1, and M is greater than 1.

[0122] Specifically, when there are M target images, the predicted pixel coordinates in the (N+1)th target image can be determined from the target video frame data based on the actual pixel coordinates of the target object in the Nth target image, the initial depth value, camera parameter information, vehicle driving information, and the preset motion model. Then, N is incremented by 1, and the steps are repeated in the order from earliest to latest until the predicted pixel coordinates of the target object in the (M-1)th target image in the target video frame are determined. Here, N and M are both positive integers, and N is initially 1, while M is greater than 1.

[0123] Step 205: Update the target depth value of the target object in the latest target image based on the predicted pixel coordinates and actual pixel coordinates of the target object in other target images; wherein, the depth value is used to characterize the estimated value of the longitudinal distance between the target object and the camera when the latest target image was acquired.

[0124] Among them, the target depth value is used to characterize the estimated longitudinal distance between the target object and the camera when the target image is acquired at the latest time.

[0125] Specifically, as shown in Figure 8, if the initial depth value is close to the true value, the predicted pixel coordinates will also approach the actual position of the pole in the figure. However, since the initial depth value is given by experience, there is usually a certain gap between the two. The distance between the predicted pixel coordinates and the actual pixel coordinates is used as the residual for optimization, transforming depth estimation into a nonlinear least squares optimization problem. The cost function is...

[0126]

[0127] Where F(z′) represents the cost function; z′ represents the initial depth value; (x0, y0) represents the difference between the actual pixel coordinates and principal point coordinates of the target object in the first ranked target image (i.e., the actual values ​​x0, y0 in Figure 8); (x1, y1) represents the difference between the actual pixel coordinates and principal point coordinates of the target object in the second ranked target image (i.e., the actual values ​​x1, y1 in Figure 8); (x1′, y1′) represents the difference between the predicted pixel coordinates and principal point coordinates of the target object in the second ranked target image (i.e., the predicted values ​​x1′, y1′ in Figure 8); (x2, y2) represents the difference between the actual pixel coordinates and principal point coordinates of the target object in the third ranked target image (i.e., the actual values ​​x2, y2 in Figure 8); and (x2′, y2′) represents the difference between the predicted pixel coordinates and principal point coordinates of the target object in the third ranked target image (i.e., the predicted values ​​x2′, y2′ in Figure 8).

[0128] The optimal estimate of the target depth corresponds to the minimum solution of the cost function. This is achieved by minimizing the cost function through an optimization method. During this process, the depth value will continuously change until the maximum number of optimization iterations is reached or the change in depth value falls below a certain range. At this point, the depth value can be determined as the target depth value. The depth estimation process is thus complete, and the method has been verified to have high accuracy.

[0129] Figure 9 is a structural diagram of a target depth estimation device based on monocular vision, as shown in an exemplary embodiment of this disclosure.

[0130] As shown in Figure 9, the target depth estimation device 900 based on monocular vision provided in this disclosure includes:

[0131] The acquisition unit 910 is used to acquire the original video frame data of the target object captured by the camera installed on the vehicle during the vehicle's operation; and to extract multiple target images from the original video frame data in chronological order so that the multiple target images constitute the target video frame data.

[0132] The recognition unit 920 is used to identify the target objects in each target image to obtain the actual pixel coordinates of the target objects;

[0133] The prediction unit 930 is used to determine the predicted pixel coordinates of the target objects in the target video frame data, except for the latest target image, in chronological order. The predicted pixel coordinates of the target objects in each other target image are determined based on the actual pixel coordinates of the target objects in the adjacent previous target image, the initial depth value, the camera parameter information, the vehicle driving information, and the preset motion model.

[0134] The update unit 940 is used to update the target depth value of the target object in the latest target image by sequentially updating the predicted pixel coordinates and actual pixel coordinates of the target object in other target images according to the time sequence; wherein the depth value is used to characterize the estimated value of the longitudinal distance between the target object in the latest target image and the camera.

[0135] The prediction unit 930 is specifically used to determine the predicted pixel coordinates of the (N+1)th sorted target image from the target video frame data when there are M target images, in order from late to early time, based on the actual pixel coordinates of the target object in the Nth sorted target image, the initial depth value, camera parameter information, vehicle driving information, and a preset motion model. N is incremented by 1, and the steps in order from late to early time are repeated until the predicted pixel coordinates of the target object in the Mth sorted target image in the target video frame are determined. Here, N and M are both positive integers, and N is initially 1, and M is greater than 1.

[0136] Prediction unit 930, specifically used for M=3, determines the predicted pixel coordinates of the target object in the second ranked target image according to the actual pixel coordinates of the target object in the first ranked target image, the initial depth value obtained, the camera parameter information and the vehicle driving information, as well as the preset motion model, in the order from late to early time.

[0137] Based on the actual pixel coordinates of the target object in the second ranked target image, the initial depth value, camera parameter information, vehicle driving information, and the preset motion model, the predicted pixel coordinates of the target object in the third ranked target image are determined.

[0138] The prediction unit 930 is specifically used for camera parameter information including pixel size, focal length, yaw angle, pitch angle, and principal point coordinates; and vehicle driving information including vehicle speed information.

[0139] The predicted pixel coordinates of the target object in the second-ranked target image are then determined using the following formula:

[0140]

[0141] Where z′ represents the initial depth value; (x0, y0) represents the difference between the actual pixel coordinates and the principal point coordinates of the target object in the first ranked target image; (x1′, y1′) represents the difference between the predicted pixel coordinates and the principal point coordinates of the target object in the second ranked target image; Δz1 represents the distance traveled by the vehicle from the time of acquiring the second ranked target image to the time of acquiring the first ranked target image, and Δz1 is determined based on the vehicle speed information and the time difference between the time of acquiring the second ranked target image and the time of acquiring the first ranked target image; (fx f y ) represents the equivalent focal length, which is determined by dividing the focal length by the pixel size; θ represents the yaw angle; β represents the pitch angle.

[0142] The prediction unit 930 is specifically used for camera parameter information including pixel size, focal length, yaw angle, pitch angle, and principal point coordinates; and vehicle driving information including vehicle speed information.

[0143] The predicted pixel coordinates of the target object in the third ranked target image are then determined using the following formula:

[0144]

[0145] Where z′ represents the initial depth value; (x1, y1) represents the difference between the actual pixel coordinates and the principal point coordinates of the target object in the second ranked target image; (x′2, y′2) represents the difference between the predicted pixel coordinates and the principal point coordinates of the target object in the third ranked target image; Δz2 represents the distance traveled by the vehicle from the time of acquiring the third ranked target image to the time of acquiring the second ranked target image, and Δz2 is determined based on the vehicle speed information and the time difference between the time of acquiring the third ranked target image and the time of acquiring the second ranked target image; (f x f y ) represents the equivalent focal length, which is determined by dividing the focal length by the pixel size; θ represents the yaw angle; β represents the pitch angle.

[0146] The recognition unit 920 is specifically used to identify the target object in each target image and the bounding box where the target object is located.

[0147] The pixel coordinates of the intersection of the two diagonals of the box are determined as the actual pixel coordinates of the target object.

[0148] The prediction unit 930 is further configured to, when there are M target images, determine the predicted pixel coordinates in the (N+1)th sorted target image from the target video frame data according to the actual pixel coordinates of the target object in the Nth sorted target image, the initial depth value, camera parameter information, vehicle driving information, and a preset motion model, in the order from earliest to latest time. Then, it increments N by 1 and repeats the steps in the order from earliest to latest time until the predicted pixel coordinates of the target object in the (M-1)th sorted target image in the target video frame are determined. Here, N and M are both positive integers, and N is initially 1 and M is greater than 1.

[0149] Figure 10 is a structural diagram of a vehicle control device illustrated in an exemplary embodiment of this disclosure.

[0150] As shown in Figure 10, the vehicle control device provided in this embodiment includes:

[0151] Memory 1001;

[0152] Processor 1002; and

[0153] Computer programs;

[0154] The computer program is stored in memory 1001 and configured to be executed by processor 1002 to implement any of the monocular vision-based target depth estimation methods described above.

[0155] This embodiment also provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement any of the monocular vision-based target depth estimation methods described above.

[0156] This embodiment also provides a computer program product, including a computer program that, when executed by a processor, implements any of the above-described monocular vision-based target depth estimation methods.

[0157] This embodiment also provides a vehicle, including a vehicle control device. The vehicle control device enables target depth estimation, thereby controlling the vehicle's movement; wherein, the vehicle control device is the aforementioned vehicle control device.

[0158] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0159] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A target depth estimation method based on monocular vision, characterized in that, This includes: acquiring raw video frame data of the target object collected by a camera installed on the vehicle during the vehicle's operation; Multiple target images are extracted from the original video frame data in chronological order to form target video frame data. Target objects in each target image are identified to obtain their actual pixel coordinates. The predicted pixel coordinates of target objects in all target images in the target video frame data (excluding the latest target image) are determined sequentially in chronological order. The predicted pixel coordinates of the target objects in each other target image are determined based on the actual pixel coordinates of the target object in the adjacent previous target image, the initial depth value obtained, the camera's parameter information, the vehicle's driving information, and a preset motion model. The target depth value of the target object in the latest target image is updated based on the actual pixel coordinates. The depth value represents an estimated longitudinal distance between the target object and the camera when the latest target image was acquired. Updating the target depth value of the target object in the latest target image based on the predicted and actual pixel coordinates of the target objects in other target images includes: determining a cost function based on the distance between the actual and predicted pixel coordinates of the target objects in other target images; minimizing the cost function using an optimization method; and continuously changing the depth value during the process until the number of optimizations reaches an upper limit or the change in the depth value is less than a preset range, at which point the current depth value is determined as the target depth value.

2. The method according to claim 1, characterized in that, When there are M target images, the step of determining the predicted pixel coordinates of the target objects in the target video frame data (excluding the latest target image) in chronological order includes: determining the predicted pixel coordinates of the (N+1)th target image in chronological order from late to early, based on the actual pixel coordinates of the target object in the Nth target image, the initial depth value obtained, the camera parameter information, the vehicle driving information, and a preset motion model; incrementing N by 1; and repeating the steps in chronological order until the predicted pixel coordinates of the target object in the Mth target image in the target video frame data are determined; wherein N and M are both positive integers, and N is initially 1, and M is greater than 1.

3. The method according to claim 2, characterized in that, If M=3, then the step of determining the predicted pixel coordinates of the target object in the (N+1)th sorted target image from the target video frame data according to the actual pixel coordinates of the target object in the Nth sorted target image, the initial depth value obtained, the camera parameter information, the vehicle driving information, and the preset motion model, and then incrementing N by 1, repeating the steps according to the latest time order, until the predicted pixel coordinates of the target object in the Mth sorted target image in the target video frame data are determined, including: determining the predicted pixel coordinates of the target object in the second sorted target image according to the actual pixel coordinates of the target object in the first sorted target image, the initial depth value obtained, the camera parameter information, the vehicle driving information, and the preset motion model, and determining the predicted pixel coordinates of the target object in the third sorted target image according to the actual pixel coordinates of the target object in the second sorted target image, the initial depth value obtained, the camera parameter information, the vehicle driving information, and the preset motion model.

4. The method according to claim 3, characterized in that, The camera's parameter information includes pixel size, focal length, yaw angle, pitch angle, and principal point coordinates; the vehicle's driving information includes vehicle speed information; therefore, the predicted pixel coordinates of the target object in the second-ranked target image are determined using the following formula: in, This represents the initial depth value; This represents the difference between the actual pixel coordinates of the target object in the first-ranked target image and the coordinates of the principal point; ) represents the difference between the predicted pixel coordinates of the target object in the second-ranked target image and the principal point coordinates; This represents the distance traveled by the vehicle from the time the second-ranked target image was acquired to the time the first-ranked target image was acquired. This is determined based on the vehicle speed information and the time difference between the time of acquiring the second-ranked target image and the time of acquiring the first-ranked target image; () represents the equivalent focal length, which is determined by dividing the focal length by the pixel size; Indicates the yaw angle; This indicates the pitch angle.

5. The method according to claim 3, characterized in that, The camera's parameter information includes pixel size, focal length, yaw angle, pitch angle, and principal point coordinates; the vehicle's driving information includes vehicle speed information; therefore, the predicted pixel coordinates of the target object in the third-ranked target image are determined using the following formula: in, This represents the initial depth value; This represents the difference between the actual pixel coordinates of the target object in the second-ranked target image and the coordinates of the principal point; ) represents the difference between the predicted pixel coordinates of the target object in the third ranked target image and the principal point coordinates; This represents the distance traveled by the vehicle from the time the third sorted target image was acquired to the time the second sorted target image was acquired. This is determined based on the vehicle speed information and the time difference between the time of acquiring the third ranked target image and the time of acquiring the second ranked target image; () represents the equivalent focal length, which is determined by dividing the focal length by the pixel size; Indicates the yaw angle; This indicates the pitch angle.

6. The method according to any one of claims 1-5, characterized in that, The step of identifying the target object in each target image to obtain the actual pixel coordinates of the target object includes: identifying the target object in each target image and the bounding box where the target object is located; and determining the pixel coordinates of the intersection of the two diagonals of the bounding box as the actual pixel coordinates of the target object.

7. The method according to claim 1, characterized in that, When there are M target images, the step of determining the predicted pixel coordinates of the target objects in the target video frame data (excluding the latest target image) in chronological order further includes: determining the predicted pixel coordinates of the (N+1)th target image in chronological order from the target video frame data based on the initial depth value obtained from the actual pixel coordinates of the target object in the Nth target image, the parameter information of the camera, the driving information of the vehicle, and a preset motion model; incrementing N by 1; and repeating the steps in chronological order until the predicted pixel coordinates of the target object in the (M-1)th target image in the target video frame data are determined; wherein N and M are both positive integers, and N is initially 1, and M is greater than 1.

8. A target depth estimation device based on monocular vision, characterized in that, include: The acquisition unit is used to acquire the original video frame data of the target object collected by the camera installed on the vehicle during the vehicle's operation. Multiple target images are extracted from the original video frame data in chronological order to form target video frame data. A recognition unit identifies the target objects in each target image to obtain the actual pixel coordinates of the target objects. A prediction unit sequentially determines the predicted pixel coordinates of target objects in all target images in the target video frame data except for the latest target image, in chronological order. The predicted pixel coordinates of the target objects in each other target image are based on the actual pixel coordinates of the target objects in the adjacent previous target image, an initial depth value, camera parameter information, vehicle driving information, and a preset motion parameter. The model is determined by an update unit, which is used to update the target depth value of the target object in the latest target image based on the predicted and actual pixel coordinates of the target object in other target images; wherein the depth value is used to characterize the estimated longitudinal distance between the target object and the camera when the latest target image was acquired; the prediction unit is specifically used to determine a cost function based on the distance between the actual and predicted pixel coordinates of the target object in other target images, and minimize the cost function through an optimization method. During the process, the depth value changes continuously until the number of optimizations reaches the upper limit or the change in the depth value is less than a preset range, then the depth value at this time is determined as the target depth value.

9. A vehicle control device, characterized in that, The method includes a memory and a processor; wherein the memory is used to store a computer program; and the processor is used to read the computer program stored in the memory and execute the method according to any one of claims 1-7.

10. A vehicle, characterized in that, Includes vehicle control equipment; achieves target depth estimation through vehicle control equipment, thereby controlling the vehicle's movement; wherein, the vehicle control equipment is the vehicle control equipment described in claim 9.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the method described in any one of claims 1-7.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-7.

Images