Multi-sensor autonomous driving system

Abstract

The present invention discloses a multi-sensor automatic driving system, comprising an image information receiving module for receiving continuous image information, a sensor information receiving module for receiving continuous sensing information, and a processing unit for analyzing the continuous image information or / and the continuous sensing information to determine whether it contains an identified target, and generating trigger information when it is confirmed that the identified target is contained, wherein the processing unit marks the corresponding image information or / and sensing information according to the trigger information.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving, and in particular to a multi-sensor autonomous driving system. Background Art

[0002] Environmental perception capability is a very important key technology for autonomous driving technology. A robust autonomous driving system needs to be equipped with sensors with different characteristics and judged through artificial intelligence to grasp the dynamics of the environment.

[0003] Artificial intelligence needs to collect a large amount of information through different sensor information, mark and learn, and adjust the parameters to get the optimized performance of the system to improve the ability to perceive the environment. Therefore, when developing an ideal autonomous driving system, information collection efficiency is more important. Today's in-vehicle intelligence has sensors with different functions such as cameras, millimeter-wave radars, ultrasonic radars and LiDAR. The collection of different amounts of information requires the construction of an integrated platform to achieve efficient information collection, which can be used as a function of information synchronization and integration. Sensor information, among which image information, occupies a large storage space when collected. Therefore, how to effectively record a large amount of sensor information at the same time and then classify, mark and learn, improve the utilization rate of storage space, and even for classification and screening before back-end information processing, will be a problem that must be solved when developing autonomous driving technology. In addition, in order to verify whether the developed artificial intelligence can effectively realize the functions required by autonomous driving, it is necessary to integrate the developed artificial intelligence into a hardware platform. The relevant specifications and settings of the hardware platform are different from the information collection system, and the hardware platform needs to be installed on a real vehicle for verification. Summary of the invention

[0004] In order to solve the above technical problems, a multi-sensor automatic driving system is provided.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] A multi-sensor autonomous driving system includes an image information receiving module for receiving continuous image information, a sensor information receiving module for receiving continuous sensing information, and a processing unit for analyzing the continuous image information and / or the continuous sensing information to determine whether it contains an identified target, and generating trigger information when it is confirmed that the identified target is contained, wherein the processing unit marks the corresponding image information and / or sensing information according to the trigger information.

[0007] A storage module is also included, and the processing unit stores the continuous image information and / or the continuous sensing information in the storage module.

[0008] A storage module is also included, and the processing unit stores the marked image information and / or the marked sensing information in the storage module.

[0009] The identification target may be set based on multiple conditions, and may be set based on the intersection or union of the multiple conditions.

[0010] The image information is in LVDS format.

[0011] The sensing information is at least one of ultrasonic information, lidar information, millimeter wave radar information, GPS information, and driving data information.

[0012] The sensor information receiving module further includes a sensor information receiving block having an Ethernet interface for receiving the lidar information.

[0013] The processing unit performs time synchronization on the continuous image information and the continuous sensing information. Compared with the prior art, the present invention has the following advantages:

[0014] 1. After setting the recognition target and the storage method, the multi-sensor autonomous driving system of the present invention marks the received continuous image information and continuous sensor information. Finally, the stored information can generate classification labels through this system. The autonomous driving system developer can then perform effective back-end information processing on the collected image information and / or sensor information, which is conducive to the development of vehicle-mounted artificial intelligence;

[0015] 2. The system configuration of the present invention includes an image information receiving module and a sensor information receiving module, and further includes a sensor information receiving block with an Ethernet interface, which is used to receive the lidar information. The image information is in the LVDS information format, and the transmitted image quality is better, which is more beneficial for the development of image artificial intelligence. In addition, the system provides a sensor information receiving module with a CAN bus interface, which can simultaneously receive sensing information from different types of sensors, and a fast Ethernet interface is also provided for the lidar signal as a transmission. The developer can directly use the system of the present invention to load the developed artificial intelligence on the vehicle to achieve real vehicle verification. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a structural schematic diagram of a multi-sensor automatic driving system of the present invention.

[0017] Figure 2 It is a schematic diagram of a process of using a multi-sensor autonomous driving system of the present invention for artificial intelligence information collection.

[0018] Explanation of reference numerals: 100 - multi-sensor autonomous driving system; 101 - image information receiving module; 102 - sensor information receiving module; 103 - processing unit; 104 - storage module; 1020 - sensor information receiving block of Ethernet interface. DETAILED DESCRIPTION

[0019] In the description of the present invention, it is necessary to understand that the terms "upper" and "lower" etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or component referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0020] In the description of the present invention, “plurality” means two or more than two, unless otherwise clearly and specifically defined.

[0021] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two components or the interaction relationship between two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0022] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments:

[0023] Please refer to Figure 1 The present invention discloses a multi-sensor autonomous driving system 100 including an image information receiving module 101 , a sensor information receiving module 102 , a processing unit 103 and a storage module 104 .

[0024] The image information receiving module 101 is used to receive continuous image information and has an image processing unit (ISP, Image Signal Processor), which can process functions such as lens correction, pixel correction, color interpolation, Bayer noise removal, white balance correction, color correction, gamma correction, color space conversion, etc. The image information receiving module 101 can generally have an LVDS (Low Voltage Differential Signaling) or MIPI CSI transmission interface. The image information comes from one or more cameras connected to the vehicle. The cameras are usually installed around the vehicle to capture driving images. In order to obtain better image quality, the image information can be LVDS format information.

[0025] The sensor information receiving module 102 is used to receive continuous sensing information, and the sensing information is at least one of ultrasonic information, lidar information, millimeter wave radar information (mmWave Radar), GPS information, and driving data information. Among them, ultrasonic information, millimeter wave radar information, GPS information, driving data information (including, for example, vehicle speed, engine speed, steering angle, acceleration, file position and other sensor information, etc.) can be transmitted through the CAN bus interface of the sensor information receiving module 102. The sensor information receiving module 102 also includes a sensor information receiving block 1020 with an Ethernet interface, which is used to receive the lidar information, and high-speed lidar signal transmission can be achieved through the Ethernet interface.

[0026] The processing unit 103 is the main computing unit of the system of the present invention, which has an optimized artificial intelligence built in. The processing core is usually a DSP (digital signal processor). The DSP is suitable for performing various multiplication and addition operations (SOP: Sum of Products), such as: finite impulse response filter operation (FIR: Finite Impulse Response), infinite impulse response filter operation (IIR: Infinite Impulse Response), discrete Fourier transform (DFT: Discrete Fourier Transform), discrete cosine transform (DCT: Discrete Cosine Transform), dot product operation (Dot product), convolution operation (Convolution), and matrix polynomial evaluation operation. The processing unit 103 has a memory transmission interface, which can execute the instructions built in the memory. The processing unit 103 also has an image transmission interface, which is connected to the image information receiving module to calculate the image information converted from the image information receiving module. The processing unit 103 also includes a CAN bus interface, which is connected to the CAN bus of the sensor information receiving module to calculate the sensing signal such as ultrasonic information, millimeter wave radar information, GPS information, driving data information, etc. In addition, the processing unit 103 also has a high-speed transmission interface, supports high-speed Ethernet interface transmission, and is mainly used to calculate the light-receiving information from the sensor information receiving module.

[0027] The main function of the storage module 104 is to store image information and / or various sensing information. The storage module 104 can be a built-in integrated circuit memory or an external storage device, such as an SSD or SD card. The storage file is stored in the form of frames, and each piece of data in the file must conform to the corresponding unit format. For example, the format of the field identification data beginning, the field data format, such as Camera image, Radar, LiDAR..., the field data length, the field synchronization current time, etc.

[0028] Please refer to Figure 1 and Figure 2 According to the embodiment of the process of artificial intelligence information collection by the multi-sensor autonomous driving system 100 of the present disclosure. The multi-sensor autonomous driving system 100 of the present invention is installed on a real vehicle and has the following steps:

[0029] Step S1, setting the recognition target and storage mode. The setting method can be operated through a user interface (user interface) (not shown) when the system is started to operate the multi-sensor automatic driving system 100, or when the system is started, the recognition target is pre-loaded into the system through a program. The loading method can be through a physical transmission device, such as a USB port, a memory card, etc., or through wireless transmission, such as Bluetooth, wireless network, etc. The recognition target can be set as a single setting or a setting of multiple conditions, and can be set in an intersection or a union manner according to multiple conditions, that is, more than two different recognition targets can be set. The intersection represents that different recognition targets exist at the same time, and the union means that any recognition target exists. The storage mode is based on the storage of different data types according to the recognition target, which can be determined according to different sensor information, the format of the storage information, the size of the storage information data, etc. In the storage mode of the present invention, it can be set whether the system should store the relevant images or other sensor information when the recognition target is detected, and even the number of frames to be stored can be adjusted according to the setting, such as the number of frames before and after the system detects the recognition target. In contrast, the system of the present invention can be configured to store relevant images or other sensor information when the identified target is not present.

[0030] Generally speaking, the recognition target may include specific objects, events, manipulations and other behaviors, especially specific targets that are mainly judged by artificial intelligence, and may also be targets that require artificial intelligence to judge other sensing information. Examples of specific objects may include pedestrians, various vehicles, road signs, signal signs, telephone poles, lane lines, sidewalks, lane types, potholes, curbstones, parking spaces, ground locks, limit levers, etc. Examples of specific events may include driving weather conditions such as rain, fog or snow in the environment, as well as ambient illumination during driving, and even road congestion, vehicle failure, vehicle collision, sensor failure or shielding affecting function, etc. Examples of specific manipulations may be failures of automatic driving functions, such as failures of automatic parking and automatic lane change, or driver behavior intervention, such as emergency braking, rapid steering, etc. The above recognition targets are mainly targets that are mainly judged by artificial intelligence, but the present invention is not limited to this, and of course may also include sensor information, such as GPS setting range, vehicle speed, ambient temperature and humidity, driver intervention control time, etc.

[0031] Step S2, receiving image information and / or sensing information. After the above steps are set, as the vehicle moves, the system of the present invention configures an image information receiving module 101 to receive continuous image information and a sensor information receiving module 102 to receive continuous sensing information. The continuous image information and continuous sensing information are temporarily stored in a shared memory area of ​​a dynamic memory (not shown) built into the system before entering the processing unit 103 of the present invention to analyze and determine whether there is a recognized target. At this time, the processing unit 103 performs time synchronization on the continuous image information and the continuous sensing information stored in the shared memory area.

[0032] Step S3, processing unit analysis. The multi-sensor autonomous driving system 100 of the present invention has a processing unit 103, and the processing unit 103 has built-in artificial intelligence, which can identify the recognition target. In other words, the processing unit 103 of the multi-sensor autonomous driving system 100 of the present invention has built-in optimized artificial intelligence, which can identify the existing recognition targets of the system, improve the efficiency of information collection or the convenience of subsequent information screening and processing, so as to develop other autonomous driving functions and perform information collection operations. The processing unit 103 analyzes the continuous image information or / and the continuous sensing information for the recognition target set in step S1 to determine whether the recognition target is included, and generates trigger information when it is confirmed that the recognition target is included. That is, the processing unit 103 of the present invention can select the information to be analyzed according to the recognition target features selected in step S1. If the calculation is only the artificial intelligence calculation of the image information, only the image information will be analyzed, and other sensing information will not be analyzed to save the system's calculation amount.

[0033] The trigger information is generated after calculation by the processing unit 103 and stored in the dynamic memory (not shown) built into the system of the present invention, and the trigger information may include the relevant actions performed by the processing unit 103 on the system of the present invention, such as controlling the system (or vehicle), stopping the system (or vehicle), etc., or marking, storing and other information processing of the continuous image information and / or the continuous sensing information.

[0034] Step S4, mark for storage. The processing unit 103 marks the continuous image information or / and the continuous sensing information corresponding to the trigger information generated in the above steps. The mark is mainly generated according to the setting of step S1, and its specific practice can be that the processing unit 103 generates at least one of text data, time start and end data, type mark of the identified target, circle mark of the identified target, stored frames or confidence index of each frame number, etc. for the continuous image information or / and the continuous sensing information. For the marked continuous image information or / and the continuous sensing information, the processing unit 103 calls the storage instruction of the dynamic memory built into the system of the present invention to store these image information or / and sensing information in the storage module 104 of the system of the present invention. The information storage can be performed according to the setting of step S1, and the optional methods are: (1) the processing unit 103 stores the continuous image information or / and the continuous sensing information in the storage module 104, that is, after the storage system is activated according to the setting of step S1, it starts to store the image information or / and the sensing information, but part of the continuous image information and part of the continuous sensing information already have corresponding marking information. (2) the processing unit 103 stores the marked image information or / and the marked sensing information in the storage module 104, that is, only the marked information is stored, and other information without marking is not stored in the storage module 104.

[0035] The system of the present invention classifies and marks the stored information obtained through step S4. The autonomous driving system developer can then conduct artificial intelligence learning and training on the collected image information and / or sensor information through information re-marking, screening, parameter adjustment, algorithm improvement, etc. Finally, the trained artificial intelligence can be updated for offline verification, or the artificial intelligence can be verified on a real vehicle through the hardware platform built by the system of the present invention. Information collection can repeat the aforementioned steps S1 to S4 until the developed autonomous driving function is optimized.

[0036] Now, the following implementation is made for the multi-sensor automatic driving system 100 of the present invention, and please refer to Figure 1 and Figure 2 Description.

[0037] The multi-sensor automatic driving system 100 is used as an implementation description of the storage of the mark of the recognition target as a grid. When the common automatic driving system realizes the automatic parking function, it is necessary to collect the corresponding grid information. Therefore, when the system of the present invention is in step S1, it is necessary to set the recognition target and storage mode. In the recognition target, since the grid information includes the grid line, grid number, other vehicles, ground obstacles (such as ground locks, limit levers, etc.) and other obstacles (such as pedestrians, animals, etc.), or environmental illumination, etc. Developers can select single or multiple conditions according to different recognition targets. For example, it can be a single recognition target when only the grid line is double-line, or a recognition target when the grid line is selected and the environmental illumination is below 100LUX, or even a recognition target when the grid line is single-line or double-line. The storage mode can be set to store the information type with the recognition target, and further, it can be selected to store only the image information without storing other sensor information, or to store the image information and other sensor information at the same time. Furthermore, the storage size for storing the image information may be set, for example, the number of all frames from when the system (processing unit 103) detects the start of the recognition target to when the recognition target is finished.

[0038] When the target recognition and storage mode setting are completed, the image information receiving module 101 is turned on to receive continuous image information, and the sensor information receiving module 102 is turned on to receive continuous sensing information in step S2.

[0039] Then, the processing unit 103 performs step S3 analysis on the continuous image information received by the image information receiving module 101 or / and the continuous sensing information received by the sensor information receiving module 102. Since the processing unit 103 of the multi-sensor automatic driving system 100 of the present invention already has artificial intelligence, it can distinguish information according to the setting of step S1. For example, when a single recognition target is selected when the vehicle grid is double-line and only the image information is stored without storing other sensor information, the processing unit 103 will calculate the artificial intelligence according to the characteristics of the vehicle grid, and only analyze the image information without analyzing other sensing information. If the processing unit 103 analyzes that the image information has the recognition target set in step S1, a trigger information is generated. When the trigger information is a storage instruction in this example, the information is stored in the storage module 104 according to this trigger information, that is, the step S4 is entered, and only the image information is stored without storing other sensing information.

[0040] The information stored in step S4 can be stored in different ways according to the setting of step S1. The following optional ways are available: (1) the processing unit 103 stores the continuous image information or / and the continuous sensing information in the storage module 104, that is, after the system is started, the image information or / and the sensing information are stored, but part of the continuous image information and part of the continuous sensing information have the marking information completed in step S3. (2) the processing unit 103 stores the marked image information or / and the marked sensing information in the storage module 104, that is, only the marked information is stored, and other information without marking is not stored in the storage module 104. It is particularly noted that the storage method of the present invention can store the relevant image or other sensing information when there is no recognition target. For example, the system of the present invention can choose not to store the image information when the vehicle grid is a single line that has been identified by the processing unit 103, and only store the vehicle grid line image information preset by the system other than the vehicle grid line is a single line.

[0041] The following is an implementation description of the multi-sensor autonomous driving system 100, for example, as a marker storage using the identified target as an obstacle. In the development of the autonomous driving function, the sensing of obstacles usually requires more than two sensor information for analysis and judgment. The sensors are usually cameras and ultrasonic sensors. The purpose is to have better recognition results for obstacles through artificial intelligence calculations after receiving information from different sensors, such as mastering the obstacle type and obstacle distance, and even fusing different sensor information. Taking the spatial detection during the development of automatic parking as an example, in step S1, multiple obstacle types can be set as identification targets for information collection, such as vehicles parked in the parking space, ground locks, limit levers, etc., and these multiple identification targets are selected in a joint manner. The storage mode can be set to only store sensor information when the processing unit 103 generates trigger information after analyzing the identified target. Here, the sensor information includes all image information and ultrasonic information, and the storage method is all frames from the start of detecting the identified target to the end of detecting the identified target.

[0042] When the setting is completed, similarly, in step S2 , the image information receiving module 101 is turned on to receive continuous image information, and the sensor information receiving module 102 is turned on to receive continuous sensing information. In this example, the sensor information at least includes ultrasonic information.

[0043] In step S3, the processing unit 103 analyzes the continuous image information received by the image information receiving module 101 or / and the continuous sensing information received by the sensor information receiving module 102. Since the processing unit 103 of the multi-sensor automatic driving system 100 of the present invention has artificial intelligence, it can analyze the image information and ultrasonic information at the same time according to the obstacle setting status of step S1. If the processing unit 103 analyzes that the image information has the identification target set in step S1, it will generate a trigger information (storage instruction), and then according to this trigger information, the information is stored in the storage module 104, that is, entering step S4. According to step S1, what is stored at this time is the image information and ultrasonic information. Specifically, when the processing unit 103 generates a trigger information for the identification target (the vehicle parked in the vehicle grid, the ground lock, and the limit lever), the collected signal at this time includes the distance information corresponding to the ultrasonic signal in addition to the identification target type of the image signal and the distance information after the image calculation. Therefore, for the mark storage of obstacles or the information collection before the fusion of different sensor information, the system of the present invention can effectively filter, classify and store these huge amounts of information.

[0044] For developers, in addition to making the collection of sensor information more efficient through the system of the present invention, the system setting of the present invention includes an image information receiving module 101 and a sensor information receiving module 102, and further includes a sensor information receiving block 1020 with an Ethernet interface for receiving the lidar information. After the artificial intelligence is developed, the developer can directly use the system of the present invention to load it on the vehicle to realize actual vehicle verification.

[0045] The above description is only a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the present invention has been disclosed as a preferred embodiment as above, it is not intended to limit the present invention. Any technician familiar with the profession can make some changes or modifications to equivalent embodiments of equivalent changes using the methods and technical contents disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A multi-sensor autonomous driving system, characterized in that: include: The image information receiving module is used to receive continuous image information. The sensor information receiving module is used to receive continuous sensing information. User interface, used to pre-set recognition targets and storage modes, a processing unit that analyzes the continuous image information or / and the continuous sensing information to determine whether the identification target is included, generates trigger information when it is confirmed that the identification target is included, and marks the image information or / and sensing information containing the identification target according to the trigger information, a storage module, storing the image information and / or the sensing information according to the storage mode, The storage mode determines the storage format and data size of the image information and / or the sensing information according to the recognition target, and includes a setting of whether to store the image information or the sensing information with and / or without the recognition target when the processing unit detects the recognition target.

2. The multi-sensor autonomous driving system according to claim 1, characterized in that: The identification target is a setting of multiple conditions, and is set in an intersection or union manner according to the multiple conditions.

3. The multi-sensor automatic driving system according to claim 1, characterized in that: The image information is in LVDS format.

4. The multi-sensor autonomous driving system according to claim 1, wherein: The sensing information is at least one of ultrasonic information, lidar information, millimeter wave radar information, GPS information, and driving data information.

5. The multi-sensor automatic driving system according to claim 4, characterized in that: The sensor information receiving module further includes a sensor information receiving block having an Ethernet interface for receiving the lidar information.

6. The multi-sensor automatic driving system according to claim 1, characterized in that: The processing unit performs time synchronization on the continuous image information and the continuous sensing information.

Images