System and method for sensor fusion using plurality of radars and cameras

Abstract

A system for sensor fusion using a plurality of radars and cameras according to an embodiment of the present invention includes: a plurality of radars; a plurality of cameras; and a processor for matching a radar and a camera installed in the same direction into one group, fusing recognition data and image data collected through the radar and the camera matched into each group to recognize surrounding objects, and integrating recognition results of the surrounding objects for each group to generate integrated information about the surrounding objects.

Description

Sensor fusion system and method using multiple radars and cameras

[0001] The present invention relates to a sensor fusion system and method using a plurality of radars and cameras, and more particularly, to a system and method capable of fusion of data collected through a plurality of vehicle radars and cameras installed in a vehicle.

[0002] Radar (Radio Detection And Ranging) detects the location and direction of objects, and measures their distance and speed. Radar is widely used for environmental detection using radio waves. However, automotive radar has a low angular recognition performance, so objects are detected as a single point regardless of type. This limits the ability to distinguish between different types of objects using radar alone.

[0003] Meanwhile, with the recent development of deep learning technology, autonomous driving technology that recognizes the surrounding environment through multiple cameras in the vehicle and creates a driving path based on this is rapidly emerging.

[0004] Additionally, sensor fusion technology, which combines multiple sensors to understand the surrounding environment, is also receiving a lot of attention.

[0005] Therefore, it is necessary to find a way to recognize surrounding objects using vehicle radar and multiple cameras within the vehicle and to determine the location of the recognized surrounding objects.

[0006] The present invention has been devised to solve the above problems, and the purpose of the present invention is to provide a system and method capable of integrating data collected through a plurality of radars and cameras installed in a vehicle.

[0007] In addition, another object of the present invention is to provide a system and method capable of improving detection (recognition) accuracy by removing objects that are detected repeatedly when recognizing surrounding objects through a plurality of radars and cameras.

[0008] According to one embodiment of the present invention for achieving the above object, a sensor fusion system using a plurality of radars and cameras includes: a plurality of radars for collecting recognition data on surrounding objects; a plurality of cameras for collecting image data on surrounding objects; and a processor for matching radars and cameras installed in the same direction into one group, fusing recognition data and image data collected through the radars and cameras matched to each group, recognizing surrounding objects, and integrating recognition results of surrounding objects by group to generate integrated information on surrounding objects.

[0009] And the radar may include a pair of side radars installed on the right and left sides of the vehicle, respectively, and a forward radar installed at the front of the vehicle, and the camera may include a pair of side cameras installed on the right and left sides of the vehicle, respectively, and a forward camera installed at the front of the vehicle.

[0010] Additionally, the processor can generate sensor fusion information including the position coordinates of an object by fusing recognition data and image data collected through radar and cameras installed in the same direction.

[0011] And, when generating sensor fusion information, the processor can generate sensor fusion information by converting recognition data into image points, identifying objects within the image data, and reflecting the image points in the object tracking results (object recognition parameters) produced.

[0012] Additionally, the processor can generate integrated information about surrounding objects by integrating first sensor fusion information generated based on data collected through the side radar and side camera installed on the right, second sensor fusion information generated based on data collected through the side radar and side camera installed on the left, and third sensor fusion information generated based on data collected through the front radar and front camera.

[0013] And, in the process of generating integrated information about surrounding objects by integrating the generated first to third sensor fusion information, the processor can determine whether there is a duplicate object (same object) in each sensor fusion information, and, if there is a duplicate object, perform a task of removing the duplicate object.

[0014] In addition, when determining whether there are overlapping objects, the processor integrates the generated first to third sensor fusion information to detect objects having similar object recognition parameters, and when similar objects are detected, calculates an average value of the object recognition parameters of the detected objects and designates it as a parameter of a representative object representing the detected objects to perform a task of removing overlapping objects, and when the parameter of the representative object is designated, reflects the parameter of the designated representative object in the result of object recognition and removes the object recognition parameters of the detected objects.

[0015] And, when integrating the first to third sensor fusion information, the processor can integrate the position coordinate values ​​of surrounding objects included in each fusion information to generate integrated information reflecting the position coordinate values ​​of surrounding objects.

[0016] Additionally, when integrated information is generated, the processor can separately prepare the generated integrated information and transmit it to a control system that provides the vehicle's autonomous driving function or driving assistance function.

[0017] Meanwhile, a sensor fusion method using a plurality of radars and cameras according to another embodiment of the present invention includes: a step in which a processor divides a plurality of radars and a plurality of cameras based on the direction in which they are installed, so that radars and cameras installed in the same direction are matched as one group; a step in which the processor collects recognition data and image data on surrounding objects through the radars and cameras matched to each group; a step in which the processor recognizes surrounding objects by integrating the collected recognition data and image data; and a step in which the processor integrates the recognition results of surrounding objects by group to generate integrated information on surrounding objects.

[0018] As described above, according to embodiments of the present invention, data collected through multiple radars and cameras installed in a vehicle can be fused, and further, when recognizing surrounding objects through multiple radars and cameras, objects that are detected repeatedly can be removed, thereby improving detection (recognition) accuracy.

[0019] FIG. 1 is a diagram provided to explain the configuration of a sensor fusion system using multiple radars and cameras according to one embodiment of the present invention;

[0020] FIG. 2 is a flowchart provided for explaining a sensor fusion method using multiple radars and cameras according to one embodiment of the present invention;

[0021] FIG. 3 is a flowchart providing a more detailed description of the process of generating integrated information using a system according to one embodiment of the present invention;

[0022] FIG. 4 is a flowchart provided to explain a process of removing duplicate detected objects using a system according to one embodiment of the present invention; and

[0023] FIG. 5 is a flowchart providing a more detailed description of the process of removing duplicate detected objects illustrated in FIG. 4.

[0024] Hereinafter, the present invention will be described in more detail with reference to the drawings.

[0025] FIG. 1 is a diagram provided to explain the configuration of a sensor fusion system using multiple radars and cameras according to one embodiment of the present invention.

[0026] A sensor fusion system using multiple radars and cameras according to the present embodiment (hereinafter collectively referred to as the “system”) can recognize surrounding objects, determine the location of recognized surrounding objects, and remove objects that are detected repeatedly by fusion of data collected through multiple radars and cameras installed in a vehicle.

[0027] For this purpose, the system may include a radar (110), a camera (120), a processor (130), and a storage unit (140).

[0028] The radar (110) can be implemented as a vehicle radar (110) installed in a vehicle to collect recognition data on surrounding objects.

[0029] A camera (120) is installed in a vehicle and can collect image data on surrounding objects.

[0030] Here, the radar (110) and camera (120) may be provided in multiple numbers, and the radar (110) and camera (120) installed in the same direction may be matched as one group by the processor (130).

[0031] For example, the radar (110) may include a pair of side radars (right side radar (110a) and left side radar (110b)) installed on the right and left sides of the vehicle, respectively, and a front radar (110c) installed on the front of the vehicle.

[0032] Additionally, the camera (120) may include a pair of side cameras (right-side camera (120a) and left-side camera (120b)) installed on the right and left sides of the vehicle, respectively, and a front camera (120c) installed on the front of the vehicle.

[0033] The storage unit (140) is a storage medium that can store data collected from the radar (110) and the camera (120) and programs and data required for the processor (130) to operate.

[0034] The processor (130) can match radars and cameras installed in the same direction into one group, and recognize surrounding objects by fusing recognition data and image data collected through radars and cameras matched to each group, and generate integrated information about surrounding objects by integrating the recognition results of surrounding objects by group.

[0035] Additionally, the processor (130) can remove objects that are detected repeatedly when recognizing surrounding objects through multiple radars (110) and cameras (120).

[0036] FIG. 2 is a flowchart provided to explain a sensor fusion method using multiple radars and cameras according to one embodiment of the present invention.

[0037] The sensor fusion method using multiple radars and cameras according to the present embodiment can be executed by the system described above with reference to FIG. 1.

[0038] Referring to FIG. 2, the processor (130) divides a plurality of radars and a plurality of cameras based on the direction in which they are installed, so that radars and cameras installed in the same direction are matched as one group (S210), and then, through the radars and cameras matched to each group, recognition data and image data for surrounding objects can be collected (S220).

[0039] The processor (130) can recognize surrounding objects by group by fusing the recognition data and image data collected by group (S230), and can generate integrated information about surrounding objects by integrating the recognition results of surrounding objects by group (S240).

[0040] That is, the processor (130) can recognize surrounding objects by group, generate sensor fusion information by group, and integrate each sensor fusion information generated by group to generate integrated information that includes the position coordinate values ​​(object recognition parameters) of each recognized object.

[0041] At this time, the integrated information about the generated surrounding objects is information in which the position coordinate values ​​of the surrounding objects included in each piece of fusion information are integrated and the position coordinate values ​​of the surrounding objects are reflected, and when the integrated information is generated, it can be transmitted to a control system (separately provided) that provides the autonomous driving function or driving assistance function of the vehicle by the processor (130).

[0042] FIG. 3 is a flowchart providing a more detailed description of the process of generating integrated information using a system according to one embodiment of the present invention.

[0043] Referring to FIG. 3, the processor (130) matches radars and cameras installed in the same direction as a group, and then when recognition data individually collected through multiple radars and image data individually collected through multiple cameras are transmitted, the recognition data and image data collected through the radars and cameras installed in the same direction can be fused to generate sensor fusion information.

[0044] Specifically, a plurality of radars (e.g., right-side radar (110a), left-side radar (110b), and front radar (110c)) and a plurality of cameras (e.g., right-side camera (120a), left-side camera (120b), and front camera (120c)) can collect recognition data and image data for surrounding objects, respectively (S310), and transmit them to the processor (130).

[0045] The processor (130) can generate sensor fusion information including the position coordinate values ​​of an object by fusion of recognition data individually collected through multiple radars and image data individually collected through multiple cameras by group (S320).

[0046] At this time, the processor (130) can convert recognition data collected through each radar (110) for each group into image points and identify objects within the image data collected through each camera (120).

[0047] For example, when identifying objects by group, the processor (130) can recognize objects within image data, track recognized objects, and produce object tracking results based on image data by group.

[0048] And the processor (130) determines whether the converted image point is included in the object tracking result based on image data for each group, and if the image point is included in the object tracking result produced for each group, the image point is reflected in the object tracking result produced for each group, thereby generating sensor fusion information for each group that includes the object's position coordinate value (object recognition parameter).

[0049] And the processor (130) can generate integrated information by integrating the position coordinate values ​​of surrounding objects included in each sensor fusion information (S330) (S340).

[0050] That is, the processor (130) can generate integrated information on surrounding objects by integrating first sensor fusion information generated based on data collected through the side radar and side camera installed on the right, second sensor fusion information generated based on data collected through the side radar and side camera installed on the left, and third sensor fusion information generated based on data collected through the front radar (110c) and the front camera (120c).

[0051] FIG. 4 is a flowchart provided to explain a process of removing duplicate detected objects using a system according to one embodiment of the present invention.

[0052] Referring to FIG. 4, a plurality of radars (e.g., right-side radar (110a), left-side radar (110b), and front radar (110c)) and a plurality of cameras (e.g., right-side camera (120a), left-side camera (120b), and front-side camera (120c)) can collect recognition data and image data for surrounding objects, respectively (S410), and transmit them to the processor (130).

[0053] The processor (130) can generate multiple sensor fusion information for each group by recognizing surrounding objects by group (S420) by integrating recognition data individually collected through multiple radars and image data individually collected through multiple cameras.

[0054] Additionally, the processor (130) can perform a task of removing duplicate objects in the process of integrating multiple generated sensor fusion information.

[0055] For example, when a plurality of sensor fusion information (e.g., first to third sensor fusion information) is generated, the processor (130) may determine whether there is an overlapping object (same object) in each sensor fusion information in the process of generating integrated information about surrounding objects by integrating the generated plurality of sensor fusion information (S430), and if there is an overlapping object (S430-Yes), perform a task of removing the overlapping object (S440).

[0056] And the processor (130) can generate integrated information about surrounding objects by integrating information from which duplicate objects have been removed (S450).

[0057] FIG. 5 is a flowchart providing a more detailed description of the process of removing duplicate detected objects illustrated in FIG. 4.

[0058] Referring to FIG. 5, when a plurality of sensor fusion information (e.g., first sensor fusion information to third sensor fusion information) is generated (S510), the processor (130) integrates the plurality of sensor fusion information to detect objects having similar object recognition parameters (S520), and when similar objects are detected, the processor calculates an average value of the object recognition parameters of the detected objects and designates the average value as a parameter of a representative object representing the detected objects (S530).

[0059] In addition, when parameters of a representative object are specified, the processor (130) can reflect the parameters of the specified representative object in the result of object recognition and remove object recognition parameters of detected objects (S540).

[0060] And, whenever integrated information on surrounding objects is generated, the processor (130) stores information on objects whose current locations have been determined in a surrounding object information list, and separately prepares the stored surrounding object information list and transmits it to a control system that provides an autonomous driving function or a driving assistance function of the vehicle.

[0061] Here, when multiple sensor fusion information is generated for each group, the processor (130) performs a task of removing overlapping objects in the process of generating integrated information about surrounding objects by integrating the multiple sensor fusion information, and then stores the generated integrated information in a surrounding object information list.

[0062] Meanwhile, it goes without saying that the technical idea of ​​the present invention can also be applied to a computer-readable recording medium containing a computer program that performs the functions of the device and method according to the present embodiment. In addition, the technical idea according to various embodiments of the present invention can be implemented in the form of computer-readable code recorded on a computer-readable recording medium. The computer-readable recording medium can be any data storage device that can be read by a computer and store data. For example, the computer-readable recording medium can be a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical disk, a hard disk drive, etc. In addition, the computer-readable code or program stored on the computer-readable recording medium can be transmitted through a network connected between computers.

[0063] In addition, although the preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above, and various modifications can be made by a person having ordinary skill in the art to which the present invention pertains without departing from the gist of the present invention as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present invention.

Claims

1. Multiple radars that collect perception data on surrounding objects; Multiple cameras that collect image data about surrounding objects; and A sensor fusion system using multiple radars and cameras, including a processor that matches radars and cameras installed in the same direction into one group, fuses recognition data and image data collected through the radars and cameras matched to each group, recognizes surrounding objects, and integrates the recognition results of surrounding objects by group to generate integrated information about surrounding objects.

2. In claim 1, The radar, It includes a pair of side radars installed on the right and left sides of the vehicle, respectively, and a forward radar installed at the front of the vehicle. The camera, A sensor fusion system using multiple radars and cameras, characterized by including a pair of side cameras installed on the right and left sides of the vehicle, respectively, and a front camera installed at the front of the vehicle.

3. In claim 2, The processor is, A sensor fusion system using multiple radars and cameras, characterized in that it generates sensor fusion information including the position coordinate values ​​of an object by fusion of recognition data and image data collected through radars and cameras installed in the same direction.

4. In claim 3, The processor is, A sensor fusion system using multiple radars and cameras, characterized in that when generating sensor fusion information, recognition data is converted into image points, an object is identified within the image data, and the image points are reflected in the object tracking results (object recognition parameters) produced to generate sensor fusion information.

5. In claim 3, The processor is, A sensor fusion system using multiple radars and cameras, characterized in that it generates integrated information on surrounding objects by integrating first sensor fusion information generated based on data collected through a side radar and a side camera installed on the right, second sensor fusion information generated based on data collected through a side radar and a side camera installed on the left, and third sensor fusion information generated based on data collected through a front radar and a front camera.

6. In claim 5, The processor is, A sensor fusion system using multiple radars and cameras, characterized in that, in the process of generating integrated information about surrounding objects by integrating the generated first to third sensor fusion information, it is determined whether there is a duplicate object (same object) in each sensor fusion information, and if there is a duplicate object, the task of removing the duplicate object is performed.

7. In claim 6, The processor is, When determining whether there are duplicate objects, the generated first to third sensor fusion information is integrated to detect objects with similar object recognition parameters. When similar objects are detected, the average value of the object recognition parameters of the detected objects is calculated to perform the task of removing duplicate objects, and is designated as the parameter of a representative object representing the detected objects. A sensor fusion system using multiple radars and cameras, characterized in that when parameters of a representative object are specified, the parameters of the specified representative object are reflected in the result of object recognition and the object recognition parameters of the detected objects are removed.

8. In claim 5, The processor is, A sensor fusion system using multiple radars and cameras, characterized in that when integrating first to third sensor fusion information, the position coordinate values ​​of surrounding objects included in each fusion information are integrated to generate integrated information reflecting the position coordinate values ​​of surrounding objects.

9. In claim 1, The processor is, A sensor fusion system using multiple radars and cameras, characterized in that when integrated information is generated, the generated integrated information is separately prepared and transmitted to a control system that provides autonomous driving functions or driving assistance functions of a vehicle.

10. A step in which the processor divides multiple radars and multiple cameras based on the direction in which they are installed, so that radars and cameras installed in the same direction are matched as one group; A step of collecting recognition data and image data on surrounding objects through radar and cameras matched to each group; A step in which the processor recognizes surrounding objects by fusing the collected recognition data and image data; and A sensor fusion method using a plurality of radars and cameras, comprising: a step of generating integrated information about surrounding objects by integrating the recognition results of surrounding objects by group by a processor;

Images