Aerial view target detection method and device based on depth estimation and electronic equipment

Abstract

The application provides a bird's-eye view target detection method and device based on depth estimation and electronic equipment, and relates to the technical field of target detection, and comprises the following steps: determining image features, monocular stereo depth and time sequence surround view depth corresponding to an image sequence based on an image sequence collected by a current vehicle in a traveling direction; determining depth estimation of the image sequence of the current vehicle at a current time according to the monocular stereo depth and the time sequence surround view depth; and determining a target detection result of the current vehicle under a bird's-eye view perspective according to the depth estimation at the current time, so as to alleviate the technical problem of insufficient depth measurement accuracy in the bird's-eye view target detection process.

Description

Technical Field

[0001] This invention relates to the technical field of target detection, and in particular to a method, apparatus, and electronic device for target detection based on depth estimation from a bird's-eye view. Background Technology

[0002] In the past two years, the focus of industry and academia in the field of autonomous driving has been on the research of vision-centered BEV perception algorithms. Compared with LiDAR, vision-centered methods can obtain richer semantic information from images and rely on a deep understanding of images to infer the geometric structure of the scene around the vehicle. The drawback is that such methods lack accurate depth measurement. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide a method, apparatus and electronic device for target detection in bird's-eye view based on depth estimation, so as to alleviate the technical problem of insufficient depth measurement accuracy in the target detection process of bird's-eye view.

[0004] In a first aspect, the embodiments provide a bird's-eye view target detection method based on depth estimation, the method comprising:

[0005] Based on the image sequence acquired by the current vehicle in the direction of travel, determine the image features, monocular stereo depth, and temporal surround view depth corresponding to the image sequence;

[0006] Based on the monocular stereo depth and the temporal surround view depth, the depth estimate of the current vehicle's image sequence at the current moment is determined;

[0007] Based on the depth estimate at the current moment, the target detection result of the current vehicle from the bird's-eye view is determined.

[0008] In an optional implementation, the step of determining the target detection result of the current vehicle from the bird's-eye view perspective based on the depth estimation at the current moment includes:

[0009] The current depth estimate and the image features are combined using a spatial perspective transformation to obtain the current bird's-eye view features.

[0010] The bird's-eye view features are processed based on a preset network model to determine the target detection result of the current vehicle from the bird's-eye view perspective at the current moment.

[0011] In an optional implementation, the step of determining the image features, monocular stereo depth, and temporal surround view depth corresponding to the image sequence acquired based on the current vehicle's direction of travel includes:

[0012] By combining the depth estimation maps of each ring view at the previous k time steps and the depth estimation maps of each view at the current time step, a temporal ring view depth estimation map is obtained.

[0013] In an optional implementation, the step of determining the image features, monocular stereo depth, and temporal surround view depth corresponding to the image sequence acquired based on the current vehicle's direction of travel includes:

[0014] The monocular stereo depth is determined based on the discrete probability distribution of depth at each location point in the current vehicle image sequence.

[0015] In an optional implementation, the step of determining the depth estimate of the current vehicle's image sequence at the current moment based on the monocular stereo depth and the temporal surround view depth includes:

[0016] The temporal surround depth is transformed by a depth transformer and applied to the monocular stereo depth to obtain the depth estimate of the current vehicle's image sequence at the current moment.

[0017] In an optional implementation, the step of determining the image features, monocular stereo depth, and temporal surround view depth corresponding to the image sequence acquired based on the current vehicle's direction of travel includes:

[0018] Acquire the image sequence captured in the current vehicle's direction of travel;

[0019] The image sequence is input into a pre-trained backbone network, which outputs image features, monocular stereo depth, and temporal surround depth.

[0020] In an optional implementation, the method further includes:

[0021] The depth estimate at the current moment is supervised and corrected by carrying depth information from the label.

[0022] Secondly, the embodiments provide a bird's-eye view target detection device based on depth estimation, the device comprising:

[0023] The first determining module determines the image features, monocular stereo depth, and temporal surround view depth corresponding to the image sequence based on the image sequence collected by the current vehicle in the direction of travel.

[0024] The second determining module determines the depth estimate of the current vehicle's image sequence at the current moment based on the monocular stereo depth and the temporal surround view depth.

[0025] The detection module determines the target detection result of the current vehicle from the bird's-eye view perspective based on the depth estimation at the current moment.

[0026] Thirdly, an embodiment provides an electronic device including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the computer program, it implements the steps of the method described in any of the foregoing embodiments.

[0027] Fourthly, an embodiment provides a machine-readable storage medium storing machine-executable instructions, which, when invoked and executed by a processor, cause the processor to perform the steps of the method described in any of the foregoing embodiments.

[0028] This invention provides a method, apparatus, and electronic device for target detection in bird's-eye view based on depth estimation. By collecting image sequences along the current vehicle's travel direction, the corresponding image features, monocular stereo depth, and temporal surround view depth are determined. Based on the monocular stereo depth and the temporal surround view depth, a more accurate depth estimate corresponding to the vehicle's captured image at the current moment can be determined. Furthermore, based on this depth estimate, the vehicle's road conditions from a bird's-eye view perspective can be determined, i.e., the detection distribution of target objects in the captured images, thereby ensuring safer and more reliable vehicle driving.

[0029] Other features and advantages of this disclosure will be set forth in the following description, or some features and advantages may be inferred from the description or determined without doubt, or may be learned by practicing the techniques described above.

[0030] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0031] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0032] Figure 1 A flowchart of a bird's-eye view target detection method based on depth estimation is provided in an embodiment of the present invention;

[0033] Figure 2 A flowchart of another bird's-eye view target detection method based on depth estimation provided in an embodiment of the present invention;

[0034] Figure 3 A schematic diagram illustrating the determination of depth estimation according to an embodiment of the present invention;

[0035] Figure 4 A functional block diagram of a bird's-eye view target detection device based on depth estimation provided in an embodiment of the present invention;

[0036] Figure 5 This is a schematic diagram of the hardware architecture of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] The inventors discovered that the accuracy of a predicted depth distribution is crucial when using it to enhance the 2D features of an image. Pre-training for depth estimation and 3D detection can significantly improve the capabilities of image 2D feature extractors.

[0039] Based on this, the present invention provides a bird's-eye view target detection method based on depth estimation, which makes the depth estimation smoother and obtains more accurate bird's-eye view target detection results by organically combining information from past time points in the time dimension.

[0040] To facilitate understanding of this embodiment, a bird's-eye view target detection method based on depth estimation disclosed in this embodiment of the invention will be described in detail first. This method can be applied to intelligent control devices such as vehicle systems, controllers, and host computers.

[0041] Figure 1 This is a flowchart of a bird's-eye view target detection method based on depth estimation, provided as an embodiment of the present invention.

[0042] like Figure 1 As shown, the method includes the following steps:

[0043] Step S102: Based on the image sequence acquired by the current vehicle in the direction of travel, determine the image features, monocular stereo depth, and temporal surround view depth corresponding to the image sequence.

[0044] The image sequence can be understood as road images along the vehicle's direction of travel at various times, as well as road images from various angles in the driving scene at the same time. These images may include target objects such as pedestrians, vehicles, and road signs. By acquiring image features, monocular stereo depth, and temporal surround view depth, this embodiment of the invention can obtain relatively accurate detection results of target objects, thereby further ensuring driving safety.

[0045] Step S104: Determine the depth estimate of the current vehicle's image sequence at the current moment based on the monocular stereo depth and temporal surround view depth.

[0046] Here, monocular stereo depth can be understood as a depth estimation map of the set of depth probability value distributions of each location point in each image in an image sequence, while temporal surround depth can be understood as a depth estimation map of each viewpoint with a time dimension.

[0047] Step S106: Based on the depth estimation at the current moment, determine the target detection result of the current vehicle from the bird's-eye view perspective.

[0048] It should be noted that the target detection results of the current vehicle in the bird's-eye view can be understood as the detection results of the current vehicle in the bird's-eye view within a certain preset distance range in front of, behind, to the left and right of the vehicle.

[0049] In a preferred embodiment of practical application, the corresponding image features, monocular stereo depth, and temporal surround view depth are determined by collecting image sequences in the current vehicle travel direction. Based on the monocular stereo depth and the temporal surround view depth in the time dimension, a more accurate depth estimate corresponding to the vehicle image at the current moment can be determined. Based on this depth estimate, the vehicle road conditions from the bird's-eye view perspective can be determined, that is, the detection distribution of target objects in the collected images, thereby ensuring safer and more reliable vehicle driving.

[0050] Furthermore, in the 3D object detection task of the bird's-eye view BEV, the branch results of depth estimation can be used to compensate for the shortcomings of the original route in the geometric reasoning of the scene around the vehicle, thus enabling better scene analysis.

[0051] In some embodiments, the image sequence can be processed through three different branches based on a pre-trained backbone network, thereby outputting three different results, and the three output results can be used to assist in target object detection; for example, step S102 may include:

[0052] Step 1.1) Obtain the image sequence captured by the current vehicle in the direction of travel.

[0053] The system can acquire image sequences of the vehicle's front and the direction of travel at the current moment through multiple cameras, while simultaneously collecting sensor data through other sensors (IMU, Odom, etc.), and sending them all into the processing unit. At the same time, the data is stored using the memory bank in the memory.

[0054] Step 1.2) Input the image sequence into the pre-trained backbone network and output image features, monocular stereo depth and temporal surround depth.

[0055] In this scheme, the image sequence can output three branches through a pre-trained backbone network: 1. image features, 2. monocular stereo depth, and 3. temporal look-around depth. The storage contains depth estimation maps of each loop viewpoint for the first k time steps, thus yielding the temporal look-around depth.

[0056] In some embodiments, the temporal lookaround depth branch in step S102 can be determined by the depth information of each viewpoint in the time dimension, specifically including:

[0057] Step 2.1) Combine the depth estimation maps of each ring view at the previous k time steps with the depth estimation maps of each view at the current time step to obtain the temporal ring view depth estimation map.

[0058] The memory bank here stores the view from view 1 to view 2 over the past k time intervals.

[0059] Depth estimation map of view n. The memory bank of different viewpoints is combined with the depth estimation maps of the five viewpoints at the current time to obtain a temporal lookaround depth estimate, such as... Figure 3 As shown.

[0060] This invention proposes using temporal lookaround depth estimation to assist in depth estimation at the current time step, and the temporal information is based on... Figure 3 The memory bank format shown is used for storage. Utilizing past look-through depth estimates stored in the memory bank can make the current depth estimate smoother, organically combining information from past time points in the time dimension.

[0061] 0 In some embodiments, the monocular stereo depth branch in step S102 described above can be implemented through various viewpoints.

[0062] Depth information is determined, specifically including:

[0063] Step 3.1) Determine the monocular stereo depth based on the discrete probability distribution of depth at each location point in the current vehicle image sequence.

[0064] like Figure 3 As shown, the depth information of each location point in each feature map at the same time from each viewpoint (view 1-view 5n) is used to determine the stereo depth of the monocular camera. This branch performs the following for each location point on the feature map:

[0065] A depth estimate is essentially a predefined discrete probability distribution. Assuming a preset depth of 0-40, it is divided into 41 bins. The monocular stereo depth is then given as the probability distribution across these 41 bins.

[0066] In some embodiments, with the assistance of temporal loop depth in the time dimension, it is possible to obtain the current...

[0067] A more accurate depth estimation at the previous time step can be achieved through the following steps, specifically including: (Step 4.1) Transforming the temporal look-around depth using a depth transformer and applying it to...

[0068] In monocular stereo depth, the depth estimate of the current vehicle image sequence at the current moment is obtained.

[0069] In this process, the temporal surround depth estimate is transformed and applied to the monocular stereo depth, together determining the depth estimate at the current moment.

[0070] In some embodiments, to further ensure the accuracy of the depth estimate at the current moment, it can also be corrected in a supervised manner. This method further includes:

[0071] Step 5.1) involves supervising and correcting the depth estimate at the current moment by using the depth information carried by the label.

[0072] Step 5.1) can occur during the training phase, where the depth estimate at the current moment is corrected using labeled depth information to enable supervised learning of the model, or it can occur during the application inference phase, where labeled depth information obtained through additional means is used to correct and verify the depth estimate at the current moment to determine a more accurate depth estimate.

[0073] Specifically, the temporal surround view depth estimation map is derived by relying on the depth information of each ring view from the previous k time steps. The monocular stereo depth branch, after being transformed by the Depth-Trans transformer, is fused with the temporal surround view depth branch to obtain the depth estimate, Depth. During the training phase, depth label information is used to supervise its calculation. Figure 2 (The dashed part in the image). The depth estimation and image features were combined with the spatial transformation oxelPooling operation to obtain the bird's-eye view BEV features.

[0074] like Figure 2As shown, the image sequence acquired by the vehicle in its current direction of travel is processed by the backbone network to output three branches: image features, temporal surround-view depth, and monocular stereo depth. The depth estimation result at the current moment is obtained by fusing the monocular stereo depth estimation and surround-view depth estimation at the current moment. The surround-view depth estimation originates from a temporal memorybank structure, which can utilize depth information from past moments to better assist the current depth estimation result. Furthermore, to enable the network to better learn the temporal surround-view depth branch and the monocular stereo depth branch, labeled depth information can be used for supervision during the training phase of the depth estimation branch. The monocular stereo depth branch, after being transformed by the Depth-Trans transformer, is fused with the temporal surround-view depth branch to obtain the depth estimation. The depth estimation and image features are then subjected to an oxel pooling spatial transformation operation to obtain the bird's-eye view BEV features.

[0075] Based on the foregoing embodiments, and based on this precise bird's-eye view feature, step S106 includes:

[0076] Step 6.1) Perform spatial perspective transformation on the depth estimate and image features at the current moment to obtain the bird's-eye view features at the current moment.

[0077] Among them, the depth information at the current moment is further obtained based on the monocular stereo depth branch and the temporal surround depth branch, and the spatial perspective transformation is performed in conjunction with image features to obtain BEV features.

[0078] Step 6.2) Process the bird's-eye view features based on the preset network model to determine the target detection result of the current vehicle in the bird's-eye view at the current moment.

[0079] Here, the preset network model may include a pre-trained head network. The object detection result can be understood as the head network outputting the detection result of the task-related category from the BEV perspective. It should be noted that each feature map in the image sequence may include multiple target objects, while the user expects to detect and identify only one or more target categories from these objects. This target category is the category of interest to the task. By inputting accurate bird's-eye view features into the head network for processing, a highly reliable object detection result from the bird's-eye view perspective is output. This scheme can be used for single BEV perception tasks (such as object detection) or multi-BEV perception tasks (3D object detection, map segmentation).

[0080] In the training phase, this invention utilizes supervised information on depth to fully learn the branch network for depth estimation. During the inference phase, the branch used for depth estimation effectively assists in 3D object detection tasks. The depth estimation includes two branches: one path processes the image sequence acquired by multiple cameras at the current moment through the backbone network to obtain a monocular stereo depth estimation map; the other path combines the depth estimation maps from the past k moments to obtain the current surround-view depth estimation map.

[0081] like Figure 4 As shown, this embodiment of the invention also provides a bird's-eye view target detection device 200 based on depth estimation, the device comprising:

[0082] The first determining module 201 determines the image features, monocular stereo depth, and temporal surround view depth corresponding to the image sequence based on the image sequence collected by the current vehicle in the direction of travel.

[0083] The second determining module 202 determines the depth estimate of the current vehicle's image sequence at the current moment based on the monocular stereo depth and the temporal surround view depth.

[0084] The detection module 203 determines the target detection result of the current vehicle from the bird's-eye view perspective based on the depth estimation at the current moment.

[0085] In some embodiments, the detection module 203 is further configured to: perform spatial perspective transformation on the depth estimate at the current moment and the image features to obtain the bird's-eye view features at the current moment; process the bird's-eye view features based on a preset network model to determine the target detection result of the current vehicle at the current moment under the bird's-eye view. It should be noted that the target detection result of the current vehicle under the bird's-eye view can be understood as the detection result of the current vehicle within a certain preset distance range in front of, behind, to the left and right of the vehicle.

[0086] In some embodiments, the first determining module 201 is further specifically used to combine the depth estimation maps of each ring view at the previous k time points of the current vehicle with the depth estimation maps of each view at the current time point to obtain a temporal ring view depth estimation map.

[0087] In some embodiments, the first determining module 201 is further specifically used to determine the monocular stereo depth based on the depth discrete probability distribution of each location point in the current vehicle image sequence.

[0088] In some embodiments, the second determining module 202 is further specifically used to transform the temporal surround depth using a depth transformer and apply it to the monocular stereo depth to obtain a depth estimate of the current vehicle's image sequence at the current moment.

[0089] In some embodiments, the first determining module 201 is further specifically used to acquire an image sequence collected by the current vehicle in the direction of travel; input the image sequence into a pre-trained backbone network, and output image features, monocular stereo depth, and temporal surround view depth.

[0090] In some embodiments, a supervision module is further included, which is used to supervise and correct the depth estimate at the current moment by carrying depth information of the label.

[0091] Figure 5 This is a schematic diagram of the hardware architecture of the electronic device 300 provided in an embodiment of the present invention. See also... Figure 5 As shown, the electronic device 300 includes a machine-readable storage medium 301 and a processor 302, and may also include a non-volatile storage medium 303, a communication interface 304, and a bus 305; wherein the machine-readable storage medium 301, the processor 302, the non-volatile storage medium 303, and the communication interface 304 communicate with each other through the bus 305. The processor 302 can execute the depth-estimation-based bird's-eye view target detection method described in the above embodiments by reading and executing machine-executable instructions for depth estimation-based bird's-eye view target detection in the machine-readable storage medium 301.

[0092] The machine-readable storage medium mentioned in this article can be any electronic, magnetic, optical, or other physical storage device that can contain or store information such as executable instructions, data, etc. For example, machine-readable storage media can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0093] Non-volatile media can be non-volatile memory, flash memory, storage drives (such as hard disk drives), any type of storage disk (such as optical discs, DVDs, etc.), or similar non-volatile storage media, or combinations thereof.

[0094] It is understood that the specific operation methods of each functional module in this embodiment can be referred to the detailed description of the corresponding steps in the above method embodiment, and will not be repeated here.

[0095] The computer-readable storage medium provided in the embodiments of the present invention stores a computer program. When the computer program code is executed, it can implement the bird's-eye view target detection method based on depth estimation as described in any of the above embodiments. For specific implementation, please refer to the method embodiments, which will not be repeated here.

[0096] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system and apparatus described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0097] Furthermore, in the description of the embodiments of the present invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention based on the specific circumstances.

[0098] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0099] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit them. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the scope of the technology disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention.

Claims

1. A target detection method based on depth estimation in bird's-eye view, characterized in that, The method includes: Based on the image sequence acquired by the vehicle in the current direction of travel, the image features, monocular stereo depth, and temporal surround view depth corresponding to the image sequence are determined; the monocular stereo depth is determined based on the depth discrete probability distribution of each position point in the image sequence; the temporal surround view depth is determined by combining the depth estimation maps of each viewpoint of the current vehicle at the previous k time points with the depth estimation maps of each viewpoint at the current time point to obtain a temporal surround view depth estimation map. Based on the monocular stereo depth and the temporal surround view depth, the depth estimate of the current vehicle's image sequence at the current moment is determined; Based on the depth estimate at the current moment, the target detection result of the current vehicle from the bird's-eye view is determined.

2. The method according to claim 1, characterized in that, The step of determining the target detection result of the current vehicle from the bird's-eye view perspective based on the depth estimation at the current moment includes: The current depth estimate and the image features are combined using a spatial perspective transformation to obtain the current bird's-eye view features. The bird's-eye view features are processed based on a preset network model to determine the target detection result of the current vehicle from the bird's-eye view perspective at the current moment.

3. The method according to claim 1, characterized in that, The step of determining the depth estimate of the current vehicle's image sequence at the current moment based on the monocular stereo depth and the temporal surround view depth includes: The temporal surround depth is transformed by a depth transformer and applied to the monocular stereo depth to obtain the depth estimate of the current vehicle's image sequence at the current moment.

4. The method according to claim 1, characterized in that, The steps of determining the image features, monocular stereo depth, and temporal surround view depth corresponding to the image sequence acquired based on the current vehicle's direction of travel include: Acquire the image sequence captured in the current vehicle's direction of travel; The image sequence is input into a pre-trained backbone network, which outputs image features, monocular stereo depth, and temporal surround depth.

5. The method according to claim 1, characterized in that, The method further includes: The depth estimate at the current moment is supervised and corrected by carrying depth information from the label.

6. A target detection device based on depth estimation in a bird's-eye view, characterized in that, The device includes: The first determining module determines the image features, monocular stereo depth, and temporal surround view depth corresponding to the image sequence based on the image sequence collected by the current vehicle in the direction of travel. The monocular stereo depth is determined based on the depth discrete probability distribution of each position point in the image sequence. The temporal surround view depth is determined by combining the depth estimation maps of each viewpoint of the current vehicle at the previous k time points with the depth estimation maps of each viewpoint at the current time point to obtain a temporal surround view depth estimation map. The second determining module determines the depth estimate of the current vehicle's image sequence at the current moment based on the monocular stereo depth and the temporal surround view depth. The detection module determines the target detection result of the current vehicle from the bird's-eye view perspective based on the depth estimation at the current moment.

7. An electronic device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method described in any one of claims 1 to 5.

8. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores machine-executable instructions that, when invoked and executed by a processor, cause the processor to perform the steps of the method according to any one of claims 1 to 5.

Images