Target detection method and system for improving lateral perception accuracy, and electronic device
By combining multi-layer neural networks and laser point cloud information, the three-dimensional bounding box and lateral position set of the truck target are obtained, which solves the problem of insufficient accuracy in truck lateral position detection in existing technologies, and realizes high-precision lateral perception, which is suitable for autonomous driving.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-08-31
- Publication Date
- 2026-03-31
AI Technical Summary
Existing technologies are insufficient to meet the high-precision detection requirements for the lateral position of target vehicles on the side when trucks are passing or driving side by side. Direct regression detection box positions are affected by the accuracy of neural networks, and the planar assumption and prior information on target obstacles are difficult to meet high-precision requirements in reality.
By acquiring image information, a multi-layer neural network is used to obtain the target's 3D bounding box and lateral position set. Combined with laser point cloud information and lane line distance, a Kalman filter is used for fusion to correct the target position and improve the accuracy of lateral perception.
It achieves high-precision lateral position detection of truck targets, improves lateral perception accuracy, and is suitable for autonomous driving scenarios.
Smart Images
Figure CN115546748B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of target detection technology, and in particular to a target detection method, system, and electronic device for improving lateral sensing accuracy. Background Technology
[0002] Trucks are wider than passenger cars, with a typical truck width of 2.8 meters. Highway lane widths are 3.75 meters. When two trucks are centered in their lanes and meet, the closest distance between them is only 95 centimeters. In contrast, a passenger car, with a width of 1.6 meters, has 155 centimeters of space when passing a truck, approximately 1.6 times the passing distance of a passenger car. Therefore, when trucks are passing or driving side-by-side, the accuracy required for detecting the lateral position of target vehicles on the side is much higher than for passenger cars.
[0003] Existing technologies mainly employ three solutions: 1) Directly regressing the lateral position of an object in the vehicle coordinate system based on deep learning; 2) Mapping the object to a point cloud bird's-eye view (BEV) based on the planar assumption to obtain its lateral position; 3) Obtaining the lateral position of an object based on projective geometry and prior information about the target obstacle. The shortcomings of existing technologies mainly include: 1) The position of the bounding box (bbox) obtained through direct regression is affected by the accuracy of the neural network, making it difficult to meet the high lateral accuracy requirements of trucks; 2) The planar assumption and the prior information about the target obstacle are both strong assumptions, which are difficult to satisfy in real-world situations, and the prior information about the target obstacle may be biased, resulting in insufficient lateral accuracy to meet the high accuracy requirements of autonomous truck driving scenarios. Summary of the Invention
[0004] To address the problems existing in the prior art, this invention provides a target detection method, system, and electronic device that improves the accuracy of lateral sensing.
[0005] This invention provides a target detection method for improving lateral sensing accuracy, the method comprising:
[0006] Acquire image information of targets in the side direction, wherein the image information includes lane lines;
[0007] Based on the image information, a first three-dimensional bounding box of the target and a first lateral position set corresponding to the first three-dimensional bounding box are obtained through a first neural network.
[0008] Based on the image information, a distance set between the second three-dimensional bounding box of the target and the lane line is obtained through a second neural network;
[0009] Based on the distance set, obtain the second lateral position set corresponding to the second three-dimensional bounding box;
[0010] The second horizontal position set and the first horizontal position set are merged to form a third horizontal position set.
[0011] According to the target detection method for improving lateral perception accuracy provided by the present invention, the first neural network includes an end-to-end neural network.
[0012] According to the present invention, a target detection method for improving lateral perception accuracy, based on the image information, obtains the distance information between the second three-dimensional bounding box of the target and the lane line through a second neural network, including:
[0013] The pixel distance set between the first two-dimensional bounding box of the target and the lane line in the image space is obtained by the second neural network.
[0014] By transforming the pixel distance set to three-dimensional space through visual projection relationship transformation, the distance set is used as the distance set.
[0015] According to the present invention, a target detection method for improving lateral sensing accuracy is provided, the method further includes:
[0016] Based on the image information, the target is classified at the pixel level by a third neural network to obtain the second two-dimensional bounding box of the target.
[0017] Based on the projection relationship, the second two-dimensional bounding box is transformed into a third three-dimensional bounding box;
[0018] The third three-dimensional bounding box and the first three-dimensional bounding box are merged to obtain the fourth three-dimensional bounding box;
[0019] Based on the correction amount of the fourth three-dimensional bounding box compared to the first three-dimensional bounding box, the third lateral position set is adjusted to obtain the fourth lateral position set.
[0020] According to the present invention, a target detection method for improving lateral sensing accuracy is provided, the method further includes:
[0021] Obtain the laser point cloud information of the target;
[0022] Based on the laser point cloud information, a local contour of the target on the side closest to the vehicle is obtained;
[0023] The local contour and the fourth three-dimensional bounding box are merged to obtain the fifth three-dimensional bounding box;
[0024] Based on the correction amount of the fifth three-dimensional bounding box compared to the fourth three-dimensional bounding box, the fourth lateral position set is adjusted to obtain the fifth lateral position set.
[0025] According to the target detection method for improving lateral perception accuracy provided by the present invention, the second lateral position set and the first lateral position set are fused, and / or the third three-dimensional bounding box and the first three-dimensional bounding box are fused, and / or the local contour and the fourth three-dimensional bounding box are fused, all by means of a Kalman filter.
[0026] This invention also provides a target detection system for improving lateral sensing accuracy, the system comprising:
[0027] The acquisition module is used to acquire image information of targets in the side direction, wherein the image information includes lane lines;
[0028] The first perception module is used to obtain, based on the image information, the first three-dimensional bounding box of the target and the first lateral position set corresponding to the first three-dimensional bounding box through the first neural network;
[0029] The lane line module is used to obtain the distance set between the second three-dimensional bounding box of the target and the lane line through a second neural network based on the image information.
[0030] The second perception module is used to obtain the second lateral position set corresponding to the second three-dimensional bounding box based on the distance set;
[0031] The fusion module is used to merge the second horizontal position set and the first horizontal position set to form a third horizontal position set.
[0032] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the target detection method for improving lateral perception accuracy as described in any of the preceding claims.
[0033] The present invention also provides an autonomous driving truck, the truck including the aforementioned electronic equipment.
[0034] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the target detection method for improving lateral perception accuracy as described in any of the preceding claims.
[0035] The target detection method, system, and electronic device provided by this invention improves the accuracy of lateral perception. By acquiring the three-dimensional bounding box of the target through a neural network, the lateral position set of the target is obtained, and the target instance is segmented to obtain more accurate lateral position information of the target. Furthermore, based on the distance relationship between the lane line and the target, the vehicle's lateral positioning is assisted to correct the lateral position information, thereby further improving the accuracy of the acquired lateral position of the target. Attached Figure Description
[0036] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0037] Figure 1 A schematic diagram of a target detection method for improving lateral sensing accuracy provided by the present invention;
[0038] Figure 2 A schematic diagram of a target detection system for improving lateral sensing accuracy is provided by the present invention.
[0039] Figure 3 This is a schematic diagram of the physical structure of an electronic device provided by the present invention. Detailed Implementation
[0040] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0041] The target detection method for improving lateral perception accuracy provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.
[0042] Figure 1 A schematic diagram of a target detection method for improving lateral sensing accuracy provided by the present invention is shown below. Figure 1 As shown, the present invention provides a target detection method for improving lateral sensing accuracy, which may include the following steps.
[0043] It should be noted that this method can be applied to scenarios where vehicles meet each other or drive side by side, and is particularly suitable for larger vehicles such as trucks. Furthermore, it is even more suitable for meeting the needs of autonomous driving, where these vehicles are equipped with cameras and lidar.
[0044] S100. Acquire image information of targets in the side direction, wherein the image information includes lane lines.
[0045] Preferably, the target is a vehicle or other obstacle.
[0046] Preferably, the image information is acquired through the vehicle's camera.
[0047] S200. Based on image information, obtain the first three-dimensional bounding box of the target and the first horizontal position set corresponding to the first three-dimensional bounding box through the first neural network.
[0048] Optionally, the first neural network includes an end-to-end neural network.
[0049] It should be noted that, in order to obtain more detailed lateral position information of the target, such as the lateral position of the front of the vehicle being different from that of the middle and rear of the vehicle, a set of position information about the target, namely the first lateral position set, is obtained based on the first three-dimensional bounding box.
[0050] S300: Based on image information, obtain the distance set between the second three-dimensional bounding box of the target and the lane lines through the second neural network.
[0051] Optionally, based on image information, the distance information between the second three-dimensional bounding box of the target and the lane lines is obtained through a second neural network, including:
[0052] The pixel distance set between the first two-dimensional bounding box of the target and the lane line in the image space is obtained by the second neural network.
[0053] By transforming the visual projection relationship, the pixel distance set is transformed into a three-dimensional space and then used as the distance set.
[0054] Preferably, the pixel distance set is transformed into a three-dimensional space through visual projection relationship transformation, and then used as a distance set. You can refer to the article "Photogrammetry and Computer Vision (7) Geometric Transformation Level" in Ji Shunping's ScienceNet blog.
[0055] S400. Based on the distance set, obtain the second horizontal position set corresponding to the second three-dimensional bounding box.
[0056] Preferably, based on the lane line positioning information detected by the vehicle, and then based on the distance set, a second lateral position set corresponding to the second three-dimensional bounding box is obtained. Compared with the target vehicle, the lane line positioning information obtained by the vehicle has more accurate positioning information due to the continuity of the lane lines. The position of the target vehicle is indirectly obtained based on the lane line positioning information, which contributes to improving the positioning accuracy of the target vehicle.
[0057] It should be noted that the second three-dimensional bounding box can be obtained without the second neural network, but only by relying on the distance between the edge of the first two-dimensional bounding box and the lane line in the image information to obtain the second lateral position set corresponding to the second three-dimensional bounding box. That is, the second three-dimensional bounding box is proposed only to distinguish it from the first three-dimensional bounding box.
[0058] S500, merge the second horizontal position set and the first horizontal position set to form the third horizontal position set.
[0059] Optionally, the method further includes:
[0060] Based on image information, the target is classified at the pixel level by a third neural network to obtain the second two-dimensional bounding box of the target.
[0061] Based on the projection relationship, the second two-dimensional bounding box is transformed into a third three-dimensional bounding box;
[0062] The third 3D bounding box and the first 3D bounding box are merged to obtain the fourth 3D bounding box;
[0063] Based on the correction amount of the fourth 3D bounding box compared to the first 3D bounding box, the third lateral position set is adjusted to obtain the fourth lateral position set.
[0064] Preferably, the third neural network includes a deep learning neural network. Deep learning performs pixel-level classification of targets in the image information to obtain a more accurate second two-dimensional bounding box, thereby obtaining a more accurate third three-dimensional bounding box. Because the size and shape of the three-dimensional bounding box are more precise, the accuracy of the lateral position set of various points on the target is directly improved.
[0065] Optionally, the method further includes:
[0066] Obtain the laser point cloud information of the target;
[0067] Based on laser point cloud information, obtain the local contour of the target on the side closest to the vehicle;
[0068] The local contour and the fourth 3D bounding box are merged to obtain the fifth 3D bounding box;
[0069] Based on the correction amount of the fifth 3D bounding box compared to the fourth 3D bounding box, the fourth lateral position set is adjusted to obtain the fifth lateral position set.
[0070] Preferably, laser point cloud information is acquired through the vehicle's lidar.
[0071] Preferably, acquiring the laser point cloud information of the target includes: performing noise reduction processing on the laser point cloud information.
[0072] It should be noted that by obtaining the local contour of the target on the side closest to the vehicle based on laser point cloud information, a more accurate external contour of the target can be obtained.
[0073] Optionally, the second lateral position set and the first lateral position set are fused, and / or the third 3D bounding box and the first 3D bounding box are fused, and / or the local contour and the fourth 3D bounding box are fused, all by using a Kalman filter.
[0074] This embodiment obtains the three-dimensional bounding box of the target through a neural network, and then obtains the target's lateral position set to achieve target instance segmentation, thereby obtaining more accurate target lateral position information. Furthermore, based on the distance relationship between the lane line and the target, it assists in vehicle lateral positioning and corrects the lateral position information, thereby further improving the accuracy of the obtained target lateral position.
[0075] The target detection system for improving lateral perception accuracy provided by the present invention is described below. The target detection system for improving lateral perception accuracy described below and the target detection method for improving lateral perception accuracy described above can be referred to in correspondence.
[0076] Figure 2 A schematic diagram of a target detection system for improving lateral sensing accuracy provided by the present invention is shown below. Figure 2 As shown, the present invention also provides a target detection system for improving lateral sensing accuracy, the system comprising:
[0077] The acquisition module is used to acquire image information of targets in the side direction, including lane lines;
[0078] The first perception module is used to obtain, based on image information, the first three-dimensional bounding box of the target and the first lateral position set corresponding to the first three-dimensional bounding box through the first neural network.
[0079] The lane line module is used to obtain the distance set between the second three-dimensional bounding box of the target and the lane line based on image information and through a second neural network.
[0080] The second perception module is used to obtain the second lateral position set corresponding to the second three-dimensional bounding box based on the distance set;
[0081] The fusion module is used to merge the second horizontal position set and the first horizontal position set to form the third horizontal position set.
[0082] This embodiment obtains the three-dimensional bounding box of the target through a neural network, and then obtains the target's lateral position set to achieve target instance segmentation, thereby obtaining more accurate target lateral position information. Furthermore, based on the distance relationship between the lane line and the target, it assists in vehicle lateral positioning and corrects the lateral position information, thereby further improving the accuracy of the obtained target lateral position.
[0083] Figure 3A schematic diagram of the physical structure of an electronic device provided by the present invention, such as... Figure 3 As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communication interface 320, and the memory 330 communicate with each other via the communication bus 340. The processor 310 can call logical instructions in the memory 330 to execute a target detection method to improve lateral perception accuracy, the method including:
[0084] Acquire image information of targets in the side direction, wherein the image information includes lane lines;
[0085] Based on the image information, a first three-dimensional bounding box of the target and a first lateral position set corresponding to the first three-dimensional bounding box are obtained through a first neural network.
[0086] Based on the image information, a distance set between the second three-dimensional bounding box of the target and the lane line is obtained through a second neural network;
[0087] Based on the distance set, obtain the second lateral position set corresponding to the second three-dimensional bounding box;
[0088] The second horizontal position set and the first horizontal position set are merged to form a third horizontal position set.
[0089] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0090] On the other hand, the present invention also provides an autonomous driving truck, which includes electronic equipment.
[0091] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer is able to execute the target detection method for improving lateral perception accuracy provided by the above methods, the method comprising:
[0092] Acquire image information of targets in the side direction, wherein the image information includes lane lines;
[0093] Based on the image information, a first three-dimensional bounding box of the target and a first lateral position set corresponding to the first three-dimensional bounding box are obtained through a first neural network.
[0094] Based on the image information, a distance set between the second three-dimensional bounding box of the target and the lane line is obtained through a second neural network;
[0095] Based on the distance set, obtain the second lateral position set corresponding to the second three-dimensional bounding box;
[0096] The second horizontal position set and the first horizontal position set are merged to form a third horizontal position set.
[0097] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the target detection methods for improving lateral sensing accuracy provided above, the methods comprising:
[0098] Acquire image information of targets in the side direction, wherein the image information includes lane lines;
[0099] Based on the image information, a first three-dimensional bounding box of the target and a first lateral position set corresponding to the first three-dimensional bounding box are obtained through a first neural network.
[0100] Based on the image information, a distance set between the second three-dimensional bounding box of the target and the lane line is obtained through a second neural network;
[0101] Based on the distance set, obtain the second lateral position set corresponding to the second three-dimensional bounding box;
[0102] The second horizontal position set and the first horizontal position set are merged to form a third horizontal position set.
[0103] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0104] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0105] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A target detection method for improving lateral perception accuracy, characterized in that, The method comprises: acquiring image information of a target in a side direction, wherein the image information comprises lane lines; based on the image information, acquiring, by a first neural network, a first three-dimensional contour frame of the target and a first lateral position set corresponding to the first three-dimensional contour frame; based on the image information, acquiring, by a second neural network, a distance set between a second three-dimensional contour frame of the target and the lane lines; based on the distance set, acquiring a second lateral position set corresponding to the second three-dimensional contour frame; fusing the second lateral position set and the first lateral position set to form a third lateral position set; The method further comprises: based on the image information, performing pixel-level classification on the target by a third neural network to acquire a second two-dimensional contour frame of the target; based on a projection relationship, converting the second two-dimensional contour frame into a third three-dimensional contour frame; fusing the third three-dimensional contour frame and the first three-dimensional contour frame to obtain a fourth three-dimensional contour frame; based on a correction amount of the fourth three-dimensional contour frame compared to the first three-dimensional contour frame, adjusting the third lateral position set to obtain a fourth lateral position set.
2. The method of claim 1, wherein, The first neural network comprises an end-to-end neural network.
3. The method of claim 1, wherein, Based on the image information, acquiring, by a second neural network, a distance set between a second three-dimensional contour frame of the target and the lane lines comprises: acquiring, by the second neural network, a pixel distance set between a first two-dimensional contour frame of the target in an image space and the lane lines; after transformation of the pixel distance set to a three-dimensional space through a visual projection relationship transformation, taking the pixel distance set as the distance set.
4. The method of claim 1, wherein, The method further comprises: acquiring laser point cloud information of the target; based on the laser point cloud information, acquiring a local contour of the target close to a side of the vehicle; fusing the local contour and the fourth three-dimensional contour frame to obtain a fifth three-dimensional contour frame; based on a correction amount of the fifth three-dimensional contour frame compared to the fourth three-dimensional contour frame, adjusting the fourth lateral position set to obtain a fifth lateral position set.
5. The method of claim 4, wherein, The fusions of the second lateral position set and the first lateral position set, and / or the fusions of the third three-dimensional contour frame and the first three-dimensional contour frame, and / or the fusions of the local contour and the fourth three-dimensional contour frame are all performed through a Kalman filter.
6. A target detection system that improves lateral perception accuracy, characterized by, The system comprises: an acquisition module configured to acquire image information of a target in a side direction, wherein the image information comprises lane lines; a first perception module configured to acquire, based on the image information, a first three-dimensional contour frame of the target and a first lateral position set corresponding to the first three-dimensional contour frame through a first neural network; a lane line module configured to acquire, based on the image information, a distance set between a second three-dimensional contour frame of the target and the lane lines through a second neural network; a second perception module configured to acquire, based on the distance set, a second lateral position set corresponding to the second three-dimensional contour frame; a fusion module configured to fuse the second lateral position set and the first lateral position set to form a third lateral position set; Also comprising: based on the image information, performing pixel-level classification on the target through a third neural network to obtain a second two-dimensional contour box of the target; based on the projection relationship, converting the second two-dimensional contour box into a third three-dimensional contour box; fusing the third three-dimensional contour box and the first three-dimensional contour box to obtain a fourth three-dimensional contour box; based on a correction amount of the fourth three-dimensional contour box compared to the first three-dimensional contour box, adjusting the third lateral position set to obtain a fourth lateral position set.
7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the steps of the target detection method for improving lateral perception accuracy according to any one of claims 1-5 when executing the program.
8. An autonomously driving truck, characterized in that The truck comprises the electronic device of claim 7. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program implements the steps of the target detection method for improving lateral perception accuracy according to any one of claims 1-5 when executed by the processor.
Citation Information
Patent Citations
Method and device for identifying lane changing trend of vehicle
CN112753038A