Image processing device

Abstract

This invention provides an image processing device that is easy to build and inexpensive, while also offering multiplexing of image information processing systems. [Solution] In one embodiment, the image processing apparatus according to the present invention comprises a first image processing chip that processes image information obtained from an external information acquisition sensor, and a second image processing chip that processes the image information, wherein the first image processing chip and the second image processing chip are each capable of processing the image information independently, and the processing information processed by the first image processing chip or the second image processing chip is sent to an operation processing unit that performs vehicle control based on the processing information.

Description

Technical Field

[0005] , , , , , ,

[0001] The present invention relates to an image processing apparatus.

Background Art

[0002] Patent Document 1 discloses a vehicle control apparatus including a signal processing IC (Integrated Circuit) unit that receives an output of a camera installed in a vehicle and performs image processing, and a recognition processing IC unit that is configured as a separate unit from the signal processing IC unit, receives image data output from the signal processing IC unit and an output of a radar, and performs recognition processing of an external environment of the vehicle, and multiplexes each of them.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By multiplexing (redundantizing) a processing system that processes image information obtained from an external information acquisition sensor such as a camera, it is possible to continue the recognition processing of the external environment even when a failure occurs and ensure vehicle safety. However, in a conventional apparatus in which a signal processing IC unit that processes image data used for recognition processing of an external environment in a recognition processing IC unit is added as a separate unit from the recognition processing IC unit, and both the signal processing IC unit and the recognition processing IC unit are multiplexed, there is a problem that the configuration becomes complicated.

[0005] Therefore, an object of the present invention is to provide an image processing apparatus having a configuration that is easy to construct and inexpensive while multiplexing a processing system for image information.

Means for Solving the Problems

[0006] Therefore, in one embodiment, the image processing apparatus according to the present invention comprises a first image processing chip that processes image information obtained from an external information acquisition sensor, and a second image processing chip that processes the image information, wherein the first image processing chip and the second image processing chip are each capable of processing the image information independently, and the processing information processed by the first image processing chip or the second image processing chip is sent to an operation processing unit that performs vehicle control based on the processing information. [Effects of the Invention]

[0007] According to the present invention, it is possible to implement an image information processing system that is multiplexed while maintaining an easy-to-build and inexpensive configuration. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing one aspect of an automated driving system. [Figure 2] This is a diagram showing one aspect of the target that the main image processing chip and the sub-image processing chip each recognize. [Figure 3] This is a flowchart illustrating one aspect of the image processing process in a sub-image processing chip. [Figure 4] This diagram shows the signal flow during tracking processing in the normal operating state of the main image processing chip. [Figure 5] This diagram shows the signal flow during tracking processing in a faulty state where a malfunction of the main image processing chip is detected. [Modes for carrying out the invention]

[0009] Hereinafter, embodiments of the image processing apparatus according to the present invention will be described with reference to the drawings. Figure 1 is a block diagram showing one embodiment of the driving automation system 10 installed in vehicle 1. The automated driving system 10 functions as an Advanced Driving Assistant System (ADAS) or an Autonomous Driving System (ADS).

[0010] The automated driving system 10 includes a camera 20 as an external information acquisition sensor, an image processing device 30, and a vehicle control device 40 as a driving processing unit. Camera 20 takes pictures of the area in front of vehicle 1 (in other words, the vehicle itself), generates an image, and sends the image information to image processing device 30.

[0011] The image processing device 30 is a device that processes image information acquired from the camera 20, and recognizes the external environment of the vehicle 1 based on the acquired image information. In detail, the image processing device 30 performs recognition processing of the external environment based on image information acquired from the camera 20, recognizing landmarks such as the road on which vehicle 1 travels (in other words, the passable space), surrounding vehicles of vehicle 1 (in other words, other vehicles around vehicle 1), road markings on the road on which vehicle 1 travels, free space around vehicle 1, pedestrians, traffic lights, and other obstacles.

[0012] The free space around the vehicle recognized by the image processing device 30 is, for example, an area on the road on which the vehicle 1 is traveling where there are no dynamic obstacles such as other vehicles or pedestrians, and no static obstacles such as median strips or center poles, and may include space on the shoulder of the road where emergency stopping is possible. Furthermore, the image information processing in the image processing device 30 may include not only processing to recognize the external environment, but also distortion correction processing, white balance adjustment processing, and so on.

[0013] The image processing device 30 sends information about the external environment as processing information, that is, the recognition results of targets such as passable space, surrounding vehicles, lane markings, and free space, to the vehicle control device 40, which performs vehicle control such as automatic driving control and driver assistance control. The vehicle control device 40 then acquires information on various targets recognized by the image processing device 30, as well as map information, vehicle 1 position information, vehicle 1's acceleration in the three axes, angular velocity around the three axes, and driving condition information such as vehicle speed. Based on the acquired information, the vehicle control device 40 controls the vehicle 1's movement by controlling its steering angle (tire angle) and braking / driving force.

[0014] For example, in the case of automated driving control, the vehicle control device 40 plans a target route and target speed for vehicle 1 based on information about the external environment, the position information of vehicle 1, and information about the driving state of vehicle 1, and outputs control commands to the vehicle 1's driving actuators to drive vehicle 1 along the target route and target speed. Here, the driving actuator of vehicle 1 is an electronically controllable actuator and includes steering devices such as a steer-by-wire system (SBW system), drive devices such as a motor or internal combustion engine, hydraulic brakes capable of increasing or decreasing the supplied hydraulic pressure, electric calipers, and braking devices such as a regenerative motor.

[0015] The image processing device 30 has, as an IC (Integrated Circuit) chip (in other words, an image processing semiconductor chip or image processing unit) that performs image processing, a main image processing chip 31 (first image processing chip) and a sub-image processing chip 32 (second image processing chip) arranged in parallel, each capable of processing image information independently. The main image processing chip 31 and the sub-image processing chip 32 each acquire image information from the camera 20, perform image processing including recognition of the external environment (various targets), and send the recognition information of the external environment as processing information to the vehicle control device 40.

[0016] In other words, the image processing device 30 is configured to have multiplexed (redundant) IC chips for processing image information obtained from the camera 20, enabling parallel execution of image processing. Furthermore, the main image processing chip 31 is powered by the main power supply circuit 33, and the sub-image processing chip 32 is powered by the sub-power supply circuit 34.

[0017] Here, as the sub-image processing chip 32, an image processing chip with a smaller arithmetic processing capacity (processing ability) than the main image processing chip 31 and a lower cost than the main image processing chip 31 is adopted. When a malfunction of the main image processing chip 31 is detected in a failure state, the sub-image processing chip 32 executes limited image processing among the image processing performed by the main image processing chip 31 in a normal operating state, so that when the main image processing chip 31 malfunctions, a minimum image processing function is ensured, and a minimum function in automatic driving or driving support is secured.

[0018] Thereby, the driving safety of the vehicle 1 when the main image processing chip 31 malfunctions is ensured. In other words, by making the image processing of the sub-image processing chip 32 limited, an image processing chip with a smaller arithmetic processing capacity and a relatively low cost compared to the main image processing chip 31 can be adopted as the sub-image processing chip 32.

[0019] According to the image processing apparatus 30 having such a configuration, since it is composed of the main image processing chip 31 and the sub-image processing chip 32, it separately includes an IC unit that performs distortion correction of image information and an IC unit that recognizes the external environment from the image information, and compared with an image processing apparatus in which each IC unit is multiplexed, the configuration is easy to build and inexpensive. Furthermore, since an IC chip with a smaller arithmetic processing capacity than the main image processing chip 31 is used as the sub-image processing chip 32, compared with the case where an IC chip having the same arithmetic processing capacity as the main image processing chip 31 (for example, an IC chip of the same model as the main image processing chip 31) is provided as the sub-image processing chip 32, the image processing apparatus 30 in which the image processing chips are multiplexed can be made inexpensive.

[0020] Furthermore, the detection of malfunction in the main image processing chip 31 is performed by self-diagnosis by the main image processing chip 31, monitoring of the acquisition status of processing information from the main image processing chip 31 by the vehicle control device 40, and comparison of the image processing results of the main image processing chip 31 and the image processing results of the sub-image processing chip 32 by the sub-image processing chip 32 or the vehicle control device 40. In more detail, the vehicle control device 40 can detect a malfunction of the main image processing chip 31 when it becomes unable to obtain processing information from the main image processing chip 31 (when no processing information is obtained for a predetermined period of time).

[0021] Furthermore, the sub-image processing chip 32 or the vehicle control device 40 can detect a malfunction of the main image processing chip 31 if, for example, the image processing results of the main image processing chip 31 and the image processing results of the sub-image processing chip 32 do not match, and the main image processing chip 31 does not recognize a target that the sub-image processing chip 32 has recognized. Furthermore, malfunctions of the main image processing chip 31 include not only those caused by failures of the main image processing chip 31 itself, but also those caused by poor power supply due to failures of the main power supply circuit 33.

[0022] The following explains the differences in image processing between the main image processing chip 31 and the sub-image processing chip 32. Figure 2 is a diagram showing one aspect of the target to be recognized by the main image processing chip 31 and the sub-image processing chip 32, respectively.

[0023] Basically, the sub-image processing chip 32 recognizes only a portion of the multiple targets recognized by the main image processing chip 31, specifically those targets that are minimally necessary to recognize in evasive driving control. This allows the sub-image processing chip 32 to perform minimal image processing to move vehicle 1 to a safe location when the main image processing chip 31 malfunctions.

[0024] Next, we will describe in detail one aspect of the target that the main image processing chip 31 and the sub-image processing chip 32 each recognize. In its normal operating state, the main image processing chip 31 performs image processing based on image information from the camera 20 to recognize the road on which vehicle 1 is traveling (passable space), surrounding vehicles, lane markings on the road on which vehicle 1 is traveling, free space around vehicle 1, pedestrians, traffic lights, and other obstacles, and sends the recognized external environment (targets) information to the vehicle control device 40. The vehicle control device 40 then performs normal vehicle control based on information about the external environment sent from the main image processing chip 31.

[0025] Meanwhile, in a malfunction state in which a failure of the main image processing chip 31 is detected, the sub-image processing chip 32 performs image processing based on image information from the camera 20, prioritizing the recognition of the road on which vehicle 1 is traveling, surrounding vehicles of vehicle 1, lane markings on the road on which vehicle 1 is traveling, and free space around vehicle 1, and sends the information of the recognized external environment (targets) to the vehicle control device 40. Then, the vehicle control device 40 performs vehicle control to move the vehicle 1 to a safe location based on information about the external environment sent from the sub-image processing chip 32.

[0026] In other words, in a failure state in which a malfunction of the main image processing chip 31 is detected, the image processing performed by the sub-image processing chip 32 is evacuation information processing that recognizes the external environment used to move the vehicle 1 to a safe location. In the normal operating state when the main image processing chip 31 is functioning properly, among the targets recognized, the road on which vehicle 1 is traveling, surrounding vehicles, road markings on the road on which vehicle 1 is traveling, and free space around vehicle 1 are relatively easy to detect, while pedestrians, traffic lights, and other obstacles are relatively easier to detect compared to the road on which vehicle 1 is traveling.

[0027] Therefore, the sub-image processing chip 32 is configured to prioritize the recognition of targets with a relatively low detection difficulty, in other words, the recognition of targets that are minimally necessary for evasive driving control, thereby enabling evasive control of vehicle 1 with minimal image processing. In other words, in order to achieve retraction control by image processing performed by the sub-image processing chip 32, which has a smaller processing capacity than the main image processing chip 31, the targets to be recognized by the image processing performed by the sub-image processing chip 32 are limited to targets with a relatively low detection difficulty.

[0028] Furthermore, in the normal operating state when the main image processing chip 31 is functioning correctly, the sub-image processing chip 32 acquires information on targets that the main image processing chip 31 has determined to be tracking targets, and performs tracking processing on those tracking targets. Here, the sub-image processing chip 32 can detect a malfunction in the main image processing chip 31 by acquiring the results of the tracking process performed by the main image processing chip 31 and comparing them with the results of its own tracking process. Furthermore, the sub-image processing chip 32 performs tracking processing from the normal operating state when the main image processing chip 31 is functioning properly, and continues tracking processing even after the main image processing chip 31 malfunctions, thereby enabling smooth evacuation information processing (processing to acquire external environment information for evacuation control).

[0029] Figure 3 is a flowchart showing one aspect of the image processing process in the sub-image processing chip 32. In step S101, the sub-image processing chip 32 compares the image information obtained from the camera 20 with the data of the target determined to be a tracking target by the main image processing chip 31 in normal operation, and performs tracking processing for the target to be tracked. The sub-image processing chip 32 then sends position data of the tracking target obtained in the tracking process to the vehicle control device 40.

[0030] Next, in step S102, the sub-image processing chip 32 determines whether or not the main image processing chip 31 is in a malfunction state that has been detected. If the main image processing chip 31 is functioning normally, the sub-image processing chip 32 returns to step S101 and continues tracking the tracking target identified by the main image processing chip 31.

[0031] On the other hand, if the sub-image processing chip 32 detects a malfunction in the main image processing chip 31, it proceeds from step S102 to step S103 and continues the tracking process of the tracking target that the main image processing chip 31 was performing until just before it failed, as a backup information processing step. Furthermore, even after the main image processing chip 31 fails, the sub-image processing chip 32 continues to send position data of the tracking target obtained during the tracking process to the vehicle control device 40.

[0032] Figure 4 shows the signal flow during tracking processing in step S101 (in other words, the normal operating state of the main image processing chip 31). Here, the main image processing chip 31 identifies the tracking target from the image information obtained from the camera 20 and sends the data of the identified tracking target to the sub-image processing chip 32.

[0033] The sub-image processing chip 32 then compares the image information obtained from the camera 20 with the data of the target object to be tracked, performs tracking processing on the target object, and sends the position data of the target object obtained from the tracking processing to the vehicle control device 40. A tracking target is, for example, a free space where vehicle 1 can move to.

[0034] Figure 5 shows the signal flow during the tracking process in step S103 (in other words, the failure state in which a malfunction of the main image processing chip 31 is detected). If the main image processing chip 31 fails, it enters fail mode and is unable to send data of the tracking target to the sub-image processing chip 32.

[0035] At this time, the sub-image processing chip 32 continues the tracking process of the tracking target object that the main image processing chip 31 was performing until just before it failed, and continues to send position data of the tracking target object and other information to the vehicle control device 40. The vehicle control device 40 performs vehicle control to move the vehicle 1 to a safe location based on position data of tracking targets such as free space sent from the sub-image processing chip 32.

[0036] In other words, if the main image processing chip 31 fails, the sub-image processing chip 32 can perform tracking processing to realize evacuation information processing, making it possible to control the vehicle to move vehicle 1 to a safe location. Furthermore, the sub-image processing chip 32 performs tracking processing from the normal operating state of the main image processing chip 31 and continues tracking processing even after the main image processing chip 31 fails, thus enabling smooth implementation of retraction control through tracking processing.

[0037] The technical concepts described in the above embodiments can be used in appropriate combinations, as long as no contradictions arise. Furthermore, although the contents of the present invention have been specifically described with reference to preferred embodiments, it will be obvious to those skilled in the art that various modifications can be taken based on the basic technical concept and teachings of the present invention.

[0038] For example, the automated driving system 10 is equipped with LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) as an external information acquisition sensor, and the image processing device 30 can process the image information obtained by the LiDAR. Furthermore, since the sub-image processing chip 32 consumes less current than the main image processing chip 31, it can be configured to remain in an active state when the main image processing chip 31 is shut down. [Explanation of Symbols]

[0039] 1...Vehicle, 10...Automated driving system, 20...Camera (external information acquisition sensor), 30...Image processing device, 31...Main image processing chip (first image processing chip), 32...Sub image processing chip (second image processing chip), 40...Vehicle control device (driving processing unit)

Claims

1. An image processing device mounted on a vehicle, A first image processing chip that processes image information obtained from an external information acquisition sensor, A second image processing chip that processes the aforementioned image information, Equipped with, The first image processing chip and the second image processing chip are each capable of processing the image information independently. The processing information processed by the first image processing chip or the second image processing chip is sent to the driving processing unit that performs vehicle control based on the processing information. Image processing device.

2. An image processing apparatus according to claim 1, The second image processing chip has a smaller processing capacity than the first image processing chip. Image processing device.

3. An image processing apparatus according to claim 1 or claim 2, In a failure state in which a malfunction of the first image processing chip is detected, the second image processing chip prioritizes performing evacuation information processing, which is image processing relating to at least one of the following: the space in which the vehicle can pass, surrounding vehicles, road markings on which the vehicle is traveling, or free space around the vehicle. Image processing device.

4. An image processing apparatus according to claim 3, When the first image processing chip is functioning normally, the second image processing chip performs tracking processing on the target that the first image processing chip has determined to be a target to be tracked. Image processing device.

5. An image processing apparatus according to claim 4, The aforementioned evacuation information processing is the tracking processing. Image processing device.

Images