Automatic driving control system and automatic driving control device

Abstract

The present invention relates to an autonomous driving control system, comprising: a data exchange unit configured to acquire a video data stream and distribute the video data stream to at least one computing unit; at least one computing unit configured to calculate perception result data from the video data stream; a sensor fusion unit configured to fuse the perception result data with sensor data to obtain fused result data; and a planning control unit configured to generate driving control instructions based on the fused result data. The planning control unit or the sensor fusion unit is configured to provide a bypass for the data exchange unit. If an abnormality occurs in either unit, the autonomous driving control system can provide a corresponding bypass to maintain the autonomous driving function.

Description

Technical Field

[0001] The present invention relates to an automatic driving control system. Background Art

[0002] Autonomous driving control systems require capabilities such as perception, fusion, positioning, path planning, decision-making and control, V2X, and high-speed data transmission. In practical applications, these systems often require the connection of multiple cameras, millimeter-wave radars, lidars, and inertial measurement units to implement advanced driver assistance functions and various levels of autonomous driving. Consequently, autonomous driving applications and algorithmic software place demands on control systems for high computing power, high bandwidth, real-time performance, and security.

[0003] Existing autonomous driving control systems often find it difficult to meet both real-time and safety requirements, and are also difficult to recover quickly when an abnormality occurs in the system. Summary of the Invention

[0004] According to one aspect of the present invention, an automatic driving control system is provided, which includes: a data exchange unit configured to acquire a video data stream and distribute the video data stream to at least one computing unit; at least one computing unit configured to calculate perception result data from the video data stream; a sensor fusion unit configured to fuse the perception result data with sensor data from an external sensor to obtain fusion result data; and a planning control unit configured to generate driving control instructions based on the fusion result data; wherein the planning control unit or the sensor fusion unit is configured to provide a bypass of the data exchange unit.

[0005] Optionally, the sensor fusion unit is further configured to: calculate the second perception data when the data exchange unit is in a non-working state, and generate fusion result data based on the second perception data and the sensor data.

[0006] Optionally, the sensor fusion unit is further configured to generate a second control instruction for driving control when the planning control unit is in a non-working state.

[0007] Optionally, the planning control unit is further configured to: calculate the third perception data when the data exchange unit is in a non-working state, and provide the third perception data to the sensor fusion unit.

[0008] Optionally, the planning control unit is further configured to: provide a bypass of the sensor fusion unit to calculate fusion result data when the sensor fusion unit is in a non-working state.

[0009] Optionally, the data exchange unit is further configured to provide fusion result data when the sensor fusion unit is in a non-working state.

[0010] Optionally, the data exchange unit is further configured to obtain at least one of the following from an external storage device: configuration parameters of the data exchange unit; configuration parameters of the sensor fusion unit; and configuration parameters of the planning control unit.

[0011] According to another aspect of the present invention, an automatic driving control device is provided, which includes the automatic driving control system as described above.

[0012] The autonomous driving control system provided by the present invention can balance the real-time and safety requirements of driving control commands. If an anomaly occurs in the data exchange unit, a bypass provided by the sensor fusion unit or planning control unit can replace the data exchange unit. Consequently, the vehicle's autonomous driving function remains unaffected, and the user does not notice the switch from the main circuit to the bypass. If an anomaly occurs in other units, the control system can also provide a corresponding bypass to maintain autonomous driving functionality. This autonomous driving control system can be implemented on a single integrated circuit, resulting in low manufacturing costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 A module diagram of an automatic driving control system according to some embodiments of the present invention is shown.

[0014] Figure 2 Shows the effective block diagram of the automatic driving control system when the data exchange unit is not working. DETAILED DESCRIPTION

[0015] Specific details are provided in the following description to provide a thorough understanding of the present invention. However, it will be apparent to those skilled in the art that embodiments of the present invention can be practiced without these specific details. Specific numerical references may be made throughout the present invention, such as "first element," "second device," and the like. However, these numerical references should not be construed as necessarily being in the order in which they appear, but rather as meaning that "first element" is distinct from "second element."

[0016] The specific details presented herein are merely exemplary and may vary while remaining within the spirit and scope of the present invention. The term "coupled" is defined as meaning directly connected to a component or indirectly connected to a component via another component. A "bypass" refers to a signal path (including electrical signals, digital signals, etc.) distinct from the main circuit (primary signal path). When switching is required, the signal originally passing through the main circuit can be transferred and processed between the two units via the bypass without affecting the functionality of the original system. Therefore, a bypass generally includes a processing unit and a signal path. The processing unit can reuse the processing unit of the main circuit or be a separate processing unit.

[0017] Preferred embodiments of methods, systems, and devices suitable for implementing the present invention are described below with reference to the accompanying drawings. Although each embodiment is described with respect to a single combination of elements, it should be understood that the present invention encompasses all possible combinations of the disclosed elements. Thus, if one embodiment includes elements A, B, and C, and a second embodiment includes elements B and D, the present invention should also be considered to include any remaining combinations of A, B, C, or D, even if not explicitly disclosed.

[0018] Figure 1 The present invention illustrates an autonomous driving control system comprising a data exchange unit 200, multiple computing units 211, a sensor fusion unit 22, and a planning control unit 24. This autonomous driving control system is implemented on an integrated circuit, with each unit coupled via a CAN bus. For example, the data exchange unit 200 can be implemented using a field-programmable gate array (FPGA) circuit and written in real-time language (RTL)-level code. FPGAs offer flexible interfaces and large-scale parallel computing capabilities. Using this architecture, the data exchange unit 200 can meet the real-time requirements of high-bandwidth data transmission while also offering flexible data flow configuration.

[0019] Specifically, the data exchange unit 200 can obtain a video data stream from at least one camera device 101, 102, 103 provided on the vehicle, and distribute the video data stream to the corresponding computing unit 211. Each computing unit 211 can calculate the perception result data from the video data stream, and the perception result data is forwarded to the sensor fusion unit 22 via the data exchange unit 200. The perception result data includes the perception results of pedestrians, obstacles, and road conditions around the vehicle, which are used to generate driving control instructions in real time. Therefore, the computing power and bandwidth of the data exchange unit 200 and the computing unit 211 should meet the real-time requirements. Among them, the computing unit 211 can be implemented using a dedicated neural network acceleration chip (NPU), which specifically optimizes the neural network-based algorithm to improve the execution efficiency of the algorithm.

[0020] In some improved embodiments, the video data streams captured by each camera 101, 102, and 103 are provided to the corresponding image signal processing unit 206 (ISP) for preprocessing. This preprocessing can filter out noise signals and background image information and further extract information about regions of interest. This reduces the computational burden on the data exchange unit 200 and the computing unit 211, improving the operational efficiency of the control system and the real-time nature of the generated driving control commands.

[0021] The sensor fusion unit 22 can fuse the perception result data with its own sensor data obtained from the forward-looking camera 101, millimeter-wave radar 105, and ultrasonic sensor 106 in the camera device to produce fused result data. As external sensors, the millimeter-wave radar 105 and ultrasonic sensor 106 can obtain highly accurate sensor data, which helps ensure the safety of autonomous driving. In some embodiments, the millimeter-wave radar 105 and ultrasonic sensor 106 are only activated to perform further sensing to determine the location, size, and distance of one or more pedestrians or obstacles when the perception result data indicates that there are pedestrians or obstacles near the vehicle. As a result, the fused result data is more accurate than the perception result data.

[0022] The fusion result data is transmitted to the planning control unit 24. The planning control unit 24 can generate driving control instructions based on the fusion result data. The driving control instructions are transmitted to the actuators 33 on the vehicle, thereby realizing autonomous driving or assisted driving. Actuators include, for example, the accelerator pedal, brake pedal, and steering wheel. Driving control instructions include acceleration control instructions, steering wheel angle control instructions, etc. The planning control unit 24 may include a proportional-integral-derivative (PID) controller, a model predictive control (MPC) controller, and a machine learning module that can learn the driver's driving habits.

[0023] In various embodiments of the present invention, the planning control unit 24 or the sensor fusion unit 22 is configured to provide or implement a bypass of the data exchange unit 200. Thus, when the data exchange unit 200 is not functioning or experiencing an abnormality, at least a portion of the video data stream and sensor data can be bypassed to the sensor fusion unit 22 or the planning control unit 24, thereby preventing the automated (assisted) driving function from being seriously affected and improving the safety and reliability of the system.

[0024] In some embodiments, as Figure 2 As shown, when the data exchange unit 200 is in a non-working state, the sensor fusion unit 22 can obtain video data from the front-view camera 101 in the camera device through the bypass provided by the sensor fusion unit 22 (as a bypass of the data exchange unit), and execute the backup perception program to calculate the second perception data, thereby replacing the functionality of the data exchange unit 200 and the calculation unit 211. The sensor fusion unit 22 further generates fusion result data based on the second perception data and the sensor data; the operation of the planning control unit 24 will not be affected. Even if the second perception data may be degraded in accuracy compared to the perception result data generated by the data exchange unit 200 and the calculation unit 211, the system can recover quickly when an abnormality occurs in the data exchange unit 200, and the user may not even notice the switch from the main loop of the data exchange unit 200 to its bypass, and the vehicle's automatic driving function can always work normally. Figure 2Inactive modules are shown in dashed boxes. In other embodiments, if an abnormality occurs in one or more computing units 211, a substantially similar processing strategy as described above for the case where an abnormality occurs in the data exchange unit 200 can be adopted, that is, the functionality of the data exchange unit 200 and the computing unit 211 can be replaced by a bypass provided by the sensor unit, or the functionality of the data exchange unit 200 and the computing unit 211 can be replaced by a bypass provided by the planning control unit.

[0025] According to some embodiments of the present invention, the sensor fusion unit 22 can also replace the planning control unit 24. Specifically, if the planning control unit 24 experiences an abnormality or is in an inoperative state, the sensor fusion unit 22 can execute a backup planning program to generate a second control instruction via a bypass provided by the sensor fusion unit 22 (which acts as a bypass of the planning control unit). In this way, the sensor fusion unit 22 performs two different calculations: the calculation performed via the main signal path fuses the perception result data with the sensor data, while the calculation performed via the bypass executes the backup planning program. These two different calculations can be performed by two independent processors or by the same processor in a time-division multiplexing manner. The generated second control instruction is transmitted to the actuator 33 to implement autonomous (or assisted) driving. Therefore, if the planning control unit 24 experiences an abnormality, the sensor fusion unit 22 can, via this bypass, at least partially complete the work of the planning control unit without affecting the autonomous driving function.

[0026] As a further improvement, the sensor fusion unit 22 also acquires V2X-based sensor signals via the road test unit (RSU) and onboard unit (OBU) 107. These sensor signals may include, for example, traffic light information, road condition information, speed limit information, and other road sign information. The sensor fusion unit 22 fuses these various sensor signals with the perception data provided by the data exchange unit 200 to produce fused data. This data not only indicates the location, size, and distance of pedestrians or obstacles, but also the current road conditions. Accordingly, the planning control unit 24 (if functioning properly) can also generate driving control instructions that are more appropriate to the current road conditions, thereby improving the safety of autonomous driving.

[0027] Both the planning control unit 24 and the sensor fusion unit 22 can be implemented based on a common chip architecture. For example, both the planning control unit 24 and the sensor fusion unit 22 can be implemented as a system-on-chip (SoC). In this way, the two SoCs and the FPGA chip used by the data exchange unit 200 are integrated on a printed circuit board to form an autonomous driving control system capable of generating driving control commands.

[0028] In some embodiments of the present invention, the planning control unit 24 provides a bypass to replace the data exchange unit 200. Specifically, when the data exchange unit 200 is in an inoperative state, it is not replaced by the sensor fusion unit 22, but by the planning control unit 24. The planning control unit 24 uses the bypass it provides (as a bypass of the data exchange unit) to obtain a portion of the video data stream, execute the backup perception program, generate third-party perception data, and provide it to the sensor fusion unit 22. After receiving the third-party perception data, the sensor fusion unit 22 fuses it with the sensor data to generate fusion result data. Subsequently, the main signal path of the planning control unit 24 generates driving control instructions based on the fusion result data. In other words, the bypass provided by the planning control unit 24 is logically prior to the sensor fusion unit 22, while the main signal path of the planning control unit 24 is logically after the sensor fusion unit 22.

[0029] As a further improvement, the planning control unit 24 can also obtain body signals from the body module 120 and generate driving control instructions based on the body signals and the fusion data provided by the sensor fusion unit 22. Body signals include, for example, wheel speed signals, steering wheel angle signals, and motor torque signals. This allows for more detailed driving control instructions, enabling the autonomous driving control system to upgrade from providing assisted driving to providing autonomous driving.

[0030] According to some embodiments of the present invention, the planning control unit 24 can also operate in place of the sensor fusion unit 22. Specifically, the planning control unit 24 provides a bypass for the sensor fusion unit 22. When the sensor fusion unit 22 is in an inactive state, the bypass allows the millimeter-wave signals (from the millimeter-wave radar 105) and ultrasonic signals (from the ultrasonic sensor 106) that originally entered the sensor fusion unit 22 to enter the bypass. Within this bypass, they are fused with the perception result data (from the data exchange unit 200) to generate fused result data. The main signal path of the planning control unit 24 then generates driving control commands based on the fused result data.

[0031] In some embodiments of the present invention, the data exchange unit 200 can be connected to an external storage device 112, such as a mobile hard drive, flash memory, various non-transitory computer-readable storage devices, or even a smartphone. The data exchange unit 200 can be equipped with a USB interface, a PCIE interface, a Lightning interface, or the like for connection to the external storage device 112. The external storage device 112 can store configuration parameters for each unit, including those for the data exchange unit 200, the sensor fusion unit 22, and the planning control unit 24. These configuration parameters are not only used in the calculation processes of the corresponding units, but also for user customization. After the vehicle is started, the data exchange unit 200 automatically reads the configuration parameters of each unit from the external storage device. When performing assisted or automated driving, the data exchange unit 200 applies these configuration parameters to itself, the sensor fusion unit 22, and the planning control unit 24. After the vehicle is parked, the various parameters used by each unit during the previous driving process can be saved to the external storage device 112. A user can carry the external storage device 112 with them to another vehicle, allowing the user's personalized parameters to be shared between different vehicles.

[0032] In some embodiments of the present invention, the data exchange unit 200 can execute a backup fusion program to output fusion result data when the sensor fusion unit 22 is in an inactive state. Thus, the data exchange unit 200 can provide a bypass for the sensor fusion unit 22. When necessary, at least a portion of the sensor data originally entering the sensor fusion unit 22 can be forwarded to the bypass provided by the data exchange unit 200, whereupon a computing unit 211 executes the backup fusion program to generate fusion result data. The backup fusion program can be a simplified version of the fusion program executed by the sensor fusion unit 22, configured to be executed by the data exchange unit 200 and the computing unit 211.

[0033] In addition to providing a main signal path and a bypass between the two units, multiple output paths can be provided to ensure the safety of autonomous driving. These paths are directed to various actuators in the vehicle. This allows the other output path to take over if one output communication fails. For example, the planning control unit 24 is coupled to the vehicle's actuators via a first output path, while the sensor fusion unit 22 is coupled to the actuators via a second output path. The two output paths are independent of each other.

[0034] The autonomous driving control system, capable of generating driving control commands, provides corresponding bypasses for the data exchange unit 200, computing unit 211, sensor fusion unit 22, and planning control unit 24. This provides redundant backup functionality. If any of these units experiences a malfunction, a remedial strategy is in place to maintain the vehicle's autonomous driving functionality and ensure driving safety. Furthermore, the autonomous driving control system can be implemented on a single integrated circuit, utilizing commercially available chips as much as possible. This reduces implementation costs and facilitates widespread adoption within the industry.

[0035] Other embodiments of the present invention provide autonomous driving control devices that employ the aforementioned autonomous driving control system to generate driving control instructions. The autonomous driving control device can be installed on a single vehicle to provide assistance or autonomous driving functionality for the vehicle. The autonomous driving control device includes an integrated circuit comprising a data exchange unit 200, a plurality of computing units 211, a sensor fusion unit 22, and a planning control unit 24, all coupled to each other via a CAN bus. The data exchange unit 200 and the computing unit are implemented using FPGA chips, while the sensor fusion unit 22 and the planning control unit 24 are implemented using SOC chips.

[0036] In some embodiments, at least a portion of the autonomous driving control system can also be implemented using a group of distributed computing devices connected by a communication network, or based on the "cloud." In such a system, multiple distributed computing devices operate together to provide services by using shared resources. As an example, multiple vehicles traveling on a highway can be connected to a common server that provides current road information, speed limit information, etc., so that each vehicle can generate corresponding driving control instructions based on this common information and the video data streams and sensor data obtained by each vehicle. This partially cloud-based driving control system is open, flexible, and scalable.

[0037] The present invention further provides a vehicle with an automatic driving function, which uses the above-mentioned automatic driving control device to help the driver drive the vehicle or automatically drive so that the driver can have a rest.

[0038] Those skilled in the art will appreciate that the various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the aspects disclosed herein can be implemented as electronic hardware, computer software, or a combination thereof. In order to demonstrate the interchangeability between hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software will depend on the specific application and the design limitations imposed on the overall system. Technicians can implement the described functionality in a varied manner for specific specific applications, but such implementation decisions should not be understood as causing a departure from the scope of the invention.

[0039] The above description is only directed to the preferred embodiments of the present invention and is not intended to limit the scope of protection of the present invention. Those skilled in the art may make various modified designs without departing from the spirit of the present invention and the accompanying claims.

Claims

1. An automatic driving control system, comprising: a data exchange unit configured to acquire a video data stream and distribute the video data stream to at least one computing unit; The at least one computing unit is configured to compute perception result data from the video data stream; a sensor fusion unit configured to fuse the perception result data with sensor data from an external sensor to obtain fusion result data; as well as a planning control unit configured to generate a driving control instruction based on the fusion result data; wherein the planning control unit or the sensor fusion unit is configured to provide a bypass of the data exchange unit, the planning control unit is coupled to an actuator of the vehicle via a first output path, and the sensor fusion unit is coupled to the actuator via a second output path; When the data exchange unit does not work or an abnormality occurs, at least a part of the video data stream and sensor data can be connected to the sensor fusion unit or the planning control unit through a bypass, and the two output paths are independent of each other; when an abnormality occurs in one output path, the other output path can replace it.

2. The control system according to claim 1, characterized in that: The sensor fusion unit is further configured to: When the data exchange unit is in a non-working state, second perception data is calculated, and the fusion result data is generated based on the second perception data and the sensor data.

3. The control system according to claim 1, characterized in that: The sensor fusion unit is further configured to: A second control instruction for driving control is generated when the planning control unit is in an inoperative state.

4. The control system according to claim 1, characterized in that: The planning control unit is further configured to: When the data exchange unit is in a non-working state, third sensing data is calculated to provide the third sensing data to the sensor fusion unit.

5. The control system according to claim 1, characterized in that: The planning control unit is further configured to: A bypass of the sensor fusion unit is provided to calculate the fusion result data when the sensor fusion unit is in a non-operating state.

6. The control system according to claim 1, characterized in that: The data exchange unit is further configured to: The fusion result data is provided when the sensor fusion unit is in a non-working state.

7. The control system according to any one of claims 1 to 6, characterized in that: The data exchange unit is implemented by a field programmable gate array circuit.

8. The control system according to any one of claims 1 to 6, characterized in that: The planning control unit and the sensor fusion unit are implemented by a first chip and a second chip respectively.

9. The control system according to claim 1, characterized in that: The data exchange unit is configured to: The video data stream is acquired from at least one camera device disposed on the vehicle.

10. The control system according to claim 9, characterized in that: The data exchange unit is further configured to obtain at least one of the following from the external storage device: Configuration parameters of the data exchange unit; Configuration parameters of the sensor fusion unit; and Configuration parameters of the planning control unit.

11. The control system according to claim 9, characterized in that: The external sensor includes a forward-looking camera in the at least one camera device.

12. The control system according to claim 11, characterized in that: The external sensor includes: millimeter-wave radar; Ultrasonic sensors; a drive test unit; and On-board unit.

13. The control system according to claim 9, characterized in that: The planning control unit is further configured to acquire a vehicle body signal and generate the driving control instruction based on the vehicle body signal and the fusion result data, wherein the vehicle body signal includes at least one of the following: wheel speed; Steering wheel angle; and Motor torque.

14. An automatic driving control device, comprising the automatic driving control system according to any one of claims 1 to 13.

15. A vehicle configured to automatically drive or assist in driving using the automatic driving control device according to claim 14.

Images