3D object detection method and 3D object detection system

Abstract

This invention provides a 3D object detection system and a 3D object detection method that offer improved sensitivity to the surrounding environment for use in autonomous vehicles, advanced driver assistance systems, and roadside units. [Solution] The 3D object detection method includes the steps of: detecting multiple feature points of an object using a camera; obtaining initial values ​​of multiple map feature points using map information; calculating a low-dimensional positioning position by the camera by coarse matching of the point cloud detected by the camera and the initial map feature point cloud; forming a pyramid mesh set using the detected feature points and multiple camera parameters; correcting the initial values ​​of the map feature points using the pyramid mesh set to obtain the detected, matched multiple 3D feature points and the accurate positioning position by the camera; and obtaining the 3D positions of other unmatched objects using the above calculation results and map features.

Description

Technical Field

[0001] The present disclosure relates to a 3D object detection method and a 3D object detection system.

Background Art

[0002] In the fields of autonomous vehicles, Advanced Driver Assistance Systems (ADAS), and Roadside Units (RSU), cameras are typically used to assist in making autonomous driving decisions. However, since conventional image detection technology is limited to the image plane and even with the introduction of artificial intelligence technology, it is limited to predicting 3D information, it cannot meet the system requirements of autonomous driving in terms of both accuracy and verifiability, causes distance errors in image-based positioning and object detection, is likely to cause errors in autonomous driving decision-making, and poses a risk to the safety of passengers.

Summary of the Invention

[0003] One technical aspect of the present disclosure is a 3D object detection method.

[0004] In one embodiment of the present disclosure, the 3D object detection method includes steps of detecting a plurality of detected feature points of an object from an image captured by a camera, forming a plurality of map feature points using map information, calculating a low-dimensional positioning position by the camera through rough matching between a map feature point group and a detected feature point group, forming a pyramid mesh set using the detected feature points and a plurality of parameters of the camera, correcting the map feature points using the pyramid mesh set to obtain a plurality of 3D feature points of the detected object and an accurate positioning position by the camera, and obtaining 3D positions of other objects by expanding using the 3D feature points of the detected object and map information.

[0005] In one embodiment of the present disclosure, the 3D object detection method further includes a step of expanding map information to obtain 3D positions of a plurality of feature points of a plurality of meaningful objects not matched in the map information.

[0006] In one embodiment of the present disclosure, the 3D object detection method further includes the step of updating map information using the 3D locations of feature points of meaningful objects.

[0007] In one embodiment of the present disclosure, the correction of map feature points includes iterative calculations.

[0008] In one embodiment of the present disclosure, the iterative calculation includes mapping and matching.

[0009] In one embodiment of the present disclosure, if the outer contour of an object is circular, the feature points to be detected include a plurality of points distributed along the circular boundary.

[0010] In one embodiment of this disclosure, the feature point to be detected lies on the image plane.

[0011] Another technical aspect of this disclosure is a 3D object detection system.

[0012] In one embodiment of this disclosure, a 3D object detection system includes a camera, a map information module, a coarse matching module, a computation module, and a matching module. The camera is configured to detect a plurality of detectable feature points of an object. The map information module is configured to store map information and to use the map information to form a plurality of map feature points. In some embodiments, the map information module may be electrically connected to the coarse matching module, but this is not required, and the map information module and camera may be electrically connected to the computation module. The coarse matching module is electrically connected to the camera and the map information module and is configured to match the map feature point cloud with the detectable feature point cloud to calculate the low-dimensional positioning position by the camera. The computation module is electrically connected to the coarse matching module and is configured to use the detectable feature points and a plurality of camera parameters to form a pyramidal mesh set. The matching module is electrically connected to the computation module and is configured to correct the map feature points using the pyramidal mesh set to obtain a plurality of 3D feature points of the detected object and the accurate positioning position by the camera.

[0013] In one embodiment of the present disclosure, correction of map feature points includes mapping and matching.

[0014] In one embodiment of the present disclosure, the matching module is further configured to extend the map information to obtain multiple 3D locations of multiple feature points of unmatched meaningful objects in the map information.

[0015] In the embodiments described above, the 3D position of an object is obtained by correcting the initial values ​​of map feature points using a pyramidal mesh set. By continuously updating the map information through iterative calculations, the error value of the detected object's 3D position is reduced, improving the accuracy of the object detection system's judgment and increasing the sensitivity of the 3D object detection system to the surrounding environment, thereby enabling the 3D object detection system to be applied to autonomous vehicles, advanced driver-assistance systems, and roadside units. [Brief explanation of the drawing]

[0016] The aspects of this disclosure will be best understood by reading the following embodiments in conjunction with the accompanying drawings. In accordance with standard practice in this industry, various features are not depicted to scale. In practice, the sizes of various features may be arbitrarily increased or decreased for clarity in the description.

[0017] [Figure 1] This is a flowchart showing a 3D object detection method according to one embodiment of the present disclosure. [Figure 2] This is a block diagram showing a 3D object detection system according to one embodiment of the present disclosure. [Figure 3] Figure 2 is a schematic diagram illustrating a practical application of the 3D object detection system. [Figure 4] Figure 1 is a 3D schematic diagram showing an intermediate process of the 3D object detection method. [Figure 5] Figure 1 is a 3D schematic diagram showing an intermediate process of the 3D object detection method. [Figure 6] Figure 1 is a 3D schematic diagram showing an intermediate process of the 3D object detection method. [Figure 7] Figure 1 is a 3D schematic diagram showing an intermediate process of the 3D object detection method. [Figure 8] Figure 1 is a 3D schematic diagram showing an intermediate process of the 3D object detection method. [Modes for carrying out the invention]

[0018] The embodiments disclosed below provide many different embodiments or examples for carrying out different features of the subject matter provided. Specific examples of elements and structures are described below for brevity of this disclosure. Of course, these examples are illustrative and not limiting. Reference numbers and / or letters may be repeated in various examples within this disclosure. This repetition is for brevity and clarity and does not, in itself, define relationships between the various embodiments and / or structures described.

[0019] In this specification, spatially relative terms such as “… below”, “… beneath”, “lower part”, “… above”, “upper part” may be used for the purpose of describing the relationship between one element or feature and another element or feature as shown in the accompanying drawings. Spatially relative terms are intended to encompass different directions of the device during use or operation in addition to the directions depicted in the accompanying drawings. The device may be oriented in other ways (rotated 90 degrees or in other directions), and the spatially relative descriptors used herein may be interpreted similarly.

[0020] FIG. 1 is a flowchart showing a 3D object detection method according to an embodiment of the present disclosure. As shown in FIG. 1, the 3D object detection method includes the following steps S1 to S6. First, in step S1, a plurality of detected feature points of an object are detected from an image captured by a camera. Next, in step S2, a plurality of map feature points are formed using map information. Subsequently, in step S3, a low-dimensional positioning position by the camera is calculated by rough matching between the map feature point group and the detected feature point group. Then, in step S4, a pyramid mesh set is formed using the detected feature points and a plurality of parameters of the camera. Thereafter, in step S5, the map feature points are corrected using the pyramid mesh set to obtain a plurality of 3D feature points of the detected object and an accurate positioning position by the camera. Finally, in step S6, the 3D feature points of the detected object are used to expand and obtain the 3D positions of other objects.

[0021] In some embodiments, the 3D object detection method is not limited to the above steps S1 to S6. For example, in some embodiments, among steps S1 to S6, other steps may be further included between two consecutive steps, other steps may be further included before step S1, and other steps may be further included after step S6.

[0022] Figure 2 is a block diagram showing a 3D object detection system 100 according to one embodiment of the present disclosure. As shown in Figure 2, the object detection system 100 includes a camera 110, a map information module 120, a coarse matching module 130, a calculation module 140, and a matching module 150. The camera 110 is configured to detect multiple feature points of an object. The map information module 120 is configured to store map information and to use the map information to form multiple map feature points. In some embodiments, the map information module 120 may be electrically connected to a coarse matching module 130, but this is not required, and the map information module 120 and the camera 110 may be electrically connected to a calculation module 140. The coarse matching module 130 is electrically connected to the camera 110 and the map information module 120 and is configured to coarsely match the detected feature point cloud with the map feature point cloud to calculate the low-dimensional positioning position by the camera 110. The calculation module 140 is electrically connected to the coarse matching module 130 and is configured to form a pyramidal mesh set using feature points detected by the camera and multiple parameters of the camera 110. The matching module 150 is electrically connected to the calculation module 140 and is configured to correct the map feature points using the pyramidal mesh set and to obtain multiple 3D feature points of the detected object and the precise positioning location by the camera 110. For example, the pyramidal mesh set and the map feature points are used to calculate a 6-axis transformation between them. In some embodiments, the matching module 150 is further configured to backproject the map information to obtain multiple 3D locations of multiple feature points of unmatched meaningful objects in the map information.

[0023] Figure 3 is a schematic diagram showing a practical application of the 3D object detection system 100 of Figure 2. As shown in Figure 3, in some embodiments, the object detection system 100 may be integrated into the electronic system of a vehicle 210 or the electronic system of roadside equipment 220. The position and number of cameras 110 can be adjusted according to the needs of the system and are not limited to this disclosure. In some embodiments, the vehicle 210 may be an autonomous vehicle or a human-driven vehicle equipped with an Advanced Driver Assistance System (ADAS). In some embodiments, the roadside equipment 220 may be a Roadside Unit (RSU) which can interact with the vehicle 210 and provide information to the vehicle 210. The method can be applied to any electronic system equipped with cameras and can also be applied to scenes in which relevant map information has already been built.

[0024] Figures 4 to 8 are 3D schematic diagrams showing the intermediate process of the 3D object detection method in Figure 1. As shown in Figure 4, first, an image I is captured of an object using the camera 110 located at the optical center. Image I is projected onto a single image plane 320. Next, the object detection system 100 prepares to reconstruct the position of the object corresponding to image I using the parameters of the camera 110.

[0025] Next, as shown in Figure 5, after capturing image I, the object detection system 100 detects feature points C1, C2, C3, and C4 in image I. The feature points are located in the image plane 320. In some embodiments, to facilitate later steps, there may be multiple feature points C1, C2, C3, and C4 for a single object, and the number is not limited to this disclosure. The locations of the feature points C1, C2, C3, and C4 may be, but are not limited to, locations on the outer contour of the object where the direction changes, such as an angle of rotation. For example, if the outer contour of the object is circular, the feature points C1, C2, C3, and C4 include multiple points distributed along the boundary of the circle. In some embodiments, if the object is circular, the number of feature points C1, C2, C3, and C4 may be, for example, 8 or 10, but is not limited to this disclosure.

[0026] Then, as shown in Figure 6, a pyramid mesh set is formed using feature points C1, C2, C3, and C4 and multiple parameters of the camera 110. The pyramid mesh set is arranged around the optical center where the camera 110 is located, linked to the low-dimensional positioning location, and the feature points C1, C2, C3, and C4 are transformed into the side extensions L1, L2, L3, and L4 of the pyramid's sides by multiple parameters of the camera. Before this step, the method for calculating the low-dimensional positioning location by the camera 110 first includes a step of coarse matching the feature points C1, C2, C3, and C4 with the initial values ​​of multiple feature points in the map information.

[0027] Subsequently, as shown in Figure 7, the pyramid mesh assembly starts correcting the initial values ​​of map feature points using the high-precision map information stored in the map information module 120 as initial positions I1, I2, I3, and I4, thereby obtaining multiple 3D feature points of the detected object and the accurate positioning position by the camera 110. The correction of the initial values ​​of map feature points includes iterative calculations. In some embodiments, the iterative calculations include mapping and matching. The iterative calculations are performed in pairs of matching results, with coarse matching being the iteration and matching of points between two point clouds, and second matching including iteration, matching, and mapping. Taking Figure 8 as an example, the second matching coarsely matches the back-projection points of C with the map feature points to obtain point I, which is then matched, mapped, and iterated to the surface composed of L, and further approximated to obtain point F. The algorithms used for coarse matching and matching may be, for example, based on the Iterative Closest Point (ICP) algorithm, but are not limited to this disclosure. Taking Figure 8 as an example, two point clouds are given.

[0028]

number

[0029]

number

[0030]

number

[0031]

number

[0032]

number

[0033]

number

[0034] Finally, as shown in both Figures 7 and 8, after iterative calculations, the 3D positions F1, F2, F3, and F4 of the object can be obtained using the map feature points. During the rough matching process, if there are unmatched feature points in meaningful objects in the map information (meaningful objects include static features in the environment such as road lines, road markings, traffic lights, road signs, and building features), such as moving pedestrians, vehicles, or corrected static features, the map information can be expanded to obtain multiple 3D positions of multiple feature points of the unmatched meaningful object in the map information. Then, the map information is updated using the 3D positions of these feature points of the meaningful object. This allows the object detection system 100 to perform the next detection using the updated map information, ensuring that it is not affected by changes between itself and the environment (e.g., vehicle movement, appearance of obstacles, etc.).

[0035] In summary, the 3D positions F1, F2, F3, and F4 of objects are obtained by correcting the initial values ​​of map feature points using a pyramid mesh set. By continuously updating the map information through iterative calculations, the error values ​​of the detected object's 3D positions F1, F2, F3, and F4 are reduced, improving the accuracy of the object detection system 100's judgment and its sensitivity to the surrounding environment. This allows the 3D object detection system 100 to be applied to autonomous vehicles, advanced driver-assistance systems, and roadside units.

[0036] The above is a summary of some embodiments to help those skilled in the art better understand aspects of this disclosure. Those skilled in the art should understand that this disclosure can be readily used as a basis for designing or modifying other processes and structures to achieve the same objectives and / or advantages as the embodiments described herein. Those skilled in the art should also understand that such equivalent configurations do not deviate from the spirit and scope of this disclosure, and that various changes, substitutions, and modifications can be made herein without departing from the spirit and scope of this disclosure. [Explanation of Symbols]

[0037] 100 3D Object Detection Systems 110 Camera 120 Map Information Module 130 Coarse Matching Module 140 Computing Modules 150 Matching Modules 210 vehicles 220 Roadside equipment 320 Image Plane C1, C2, C3, C4 characteristic points F1, F2, F3, F4 3D position Image I I1, I2, I3, I4 initial position L1, L2, L3, L4 lines S1, S2, S3, S4, S5, S6 steps

Claims

1. The steps include detecting multiple feature points of an object from an image captured by a camera, The steps include forming multiple map feature points using map information, The steps include calculating the low-dimensional positioning position by the camera by coarse matching of the map feature point cloud and the detected feature point cloud, The steps include forming a pyramidal mesh set using a plurality of the detected feature points and a plurality of parameters of the camera, The steps include correcting a plurality of map feature points using the pyramidal mesh set to obtain a plurality of 3D feature points of the detected object and the accurate position determined by the camera, A 3D object detection method comprising the step of obtaining multiple 3D positions of other objects by augmenting them using multiple 3D feature points of the detected object.

2. The 3D object detection method according to claim 1, further comprising the step of extending the map information to obtain multiple 3D positions of multiple feature points of unmatched meaningful objects in the map information.

3. The 3D object detection method according to claim 2, further comprising the step of updating the map information using a plurality of 3D positions of a plurality of feature points of the meaningful object.

4. The 3D object detection method according to claim 1, wherein the correction of the plurality of map feature points includes iterative calculations.

5. The 3D object detection method according to claim 4, wherein the iterative calculation includes mapping and matching.

6. The 3D object detection method according to claim 1, wherein, when the outer contour of the object is circular, the plurality of detected feature points include a plurality of points distributed on the circular boundary.

7. The 3D object detection method according to claim 1, wherein the plurality of detected feature points are located on the image plane.

8. A camera configured to detect multiple feature points of an object, A map information module configured to store map information and to form multiple map feature points using the map information, A coarse matching module is electrically connected to the camera and the map information module and is configured to match the map feature point cloud with the detected feature point cloud to calculate the low-dimensional positioning position by the camera. A computing module electrically connected to the coarse matching module and configured to form a pyramidal mesh set using a plurality of the detected feature points and a plurality of parameters of the camera, A 3D object detection system comprising: a matching module electrically connected to the calculation module and configured to correct a plurality of map feature points using the pyramidal mesh set, and to acquire a plurality of 3D feature points of a detected object and an accurate position determined by the camera.

9. The 3D object detection system according to claim 8, wherein the correction of the plurality of map feature points includes mapping and matching.

10. The 3D object detection system according to claim 8, wherein the matching module is further configured to back-project the map information to obtain a plurality of 3D positions of a plurality of feature points of a plurality of unmatched meaningful objects in the map information.

Images