Image processing device and image processing system

Abstract

To provide an image processing device and an image processing system that ensure redundancy and enable control even when the imaging device itself breaks down or malfunctions.SOLUTION: An image processing device 3 recognizes a recognition target on the basis of image data of the outside world captured by a first imaging device 2a and a second imaging device 2b installed vertically away from the first imaging device 2a from the inside of the vehicle cabin through a window glass 6. The image processing device includes a first image processing unit 3a that recognizes a first recognition target on the basis of image data from the first imaging device 2a, and a second image processing unit 3b that recognizes a second recognition target different from the first recognition target on the basis of image data from the first imaging device 2a and the second imaging device 2b. When a predetermined condition is satisfied, the first recognition target is recognized by the second image processing unit 3b.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an image processing device and an image processing system, and more particularly to an image processing device and an image processing system suitable for installation in a vehicle. [Background technology]

[0002] In recent years, interest in automotive safety technology has grown significantly, and various preventive safety systems have been put into practical use, primarily by automotive-related companies. For example, Patent Document 1 discloses a technology that includes an outdoor camera and an indoor camera installed vertically spaced apart, and an image processing device, receives a first image from the outdoor camera and a second image from the indoor camera whose imaging range includes at least a portion of the imaging range of the first image, and determines whether there is an abnormality in the first image or the second image based on the common part of the imaging range of the first image and the second image.The technology disclosed in Patent Document 1 then removes the adhesion of foreign matter or fogging that is the abnormality when an abnormality occurs in either the outdoor camera or the indoor camera. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-125942 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, the technology disclosed in Patent Document 1 does not take into consideration maintaining redundancy and performing control when the outdoor camera or indoor camera (imaging device) itself breaks down or malfunctions. Therefore, the present invention provides an image processing device and an image processing system that can ensure redundancy and perform control even if the imaging device itself breaks down or malfunctions. [Means for solving the problem]

[0005] In order to solve the above problems, the image processing device according to the present invention is to be established a first imaging device and provided on the vehicle, An image processing device that recognizes a recognition target based on image data captured by a second imaging device installed vertically spaced from the first imaging device, the image processing device having: a first image processing unit that recognizes a first recognition target based on image data of the first imaging device; and a second image processing unit that recognizes a second recognition target different from the first recognition target based on image data of the first imaging device and the second imaging device, the first image processing unit or the second image processing unit determines a failure of the second imaging device, and the first image processing unit determines a failure of the second image processing unit; If certain conditions are met, The first image processing unit The first recognition target Continued recognition of It is characterized by:

[0006] The image processing system according to the present invention also includes , approved An image processing system for recognizing a recognition object, comprising: a first imaging device; a second imaging device installed vertically spaced from the first imaging device; a first image processing unit electrically connected to at least the first imaging device; and a second image processing unit electrically connected to the first imaging device and the second imaging device, wherein the first image processing unit recognizes a first recognition object based on image data of the first imaging device, and the second image processing unit recognizes a second recognition object different from the first recognition object based on image data of the first imaging device and the second imaging device; the first image processing unit or the second image processing unit determines a failure of the second imaging device, and the first image processing unit determines a failure of the second image processing unit; If certain conditions are met, The first image processing unit The first recognition target Continued recognition of It is characterized by: [Effects of the Invention]

[0007] According to the present invention, it is possible to provide an image processing device and an image processing system that can ensure redundancy and perform control even if the imaging device itself breaks down or malfunctions. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0008] [Figure 1] 1A and 1B are a front view and a side view of a vehicle equipped with an image processing system according to an embodiment of the present invention. [Figure 2] 1 is a diagram showing an overall schematic configuration of an image processing system according to a first embodiment of the present invention; [Figure 3] 1 is a functional block diagram of an image processing device constituting an image processing system according to a first embodiment. [Figure 4] 1 is a bird's-eye view illustrating an example in which a vehicle equipped with an image processing system according to a first embodiment is located at an intersection with traffic lights. [Figure 5] 4 is a flowchart showing a processing flow of the image processing device constituting the image processing system according to the first embodiment. [Figure 6] 10 is a flowchart showing another processing flow of the image processing device according to the first embodiment. [Figure 7] FIG. 10 is a schematic diagram illustrating an overall configuration of a modified example of the image processing system according to the first embodiment. [Figure 8] FIG. 8 is a functional block diagram of the image processing device shown in FIG. [Figure 9] FIG. 10 is a diagram showing an example of a wiper position in a degenerate mode in a second embodiment according to another embodiment of the present invention. [Figure 10] FIG. 10 is a diagram illustrating another example of the wiper position in the degenerate mode according to the second embodiment. [Figure 11] FIG. 10 is a diagram illustrating another example of the wiper position in the degenerate mode according to the second embodiment. [Figure 12] FIG. 10 is a diagram illustrating the wiper position in a normal mode. DETAILED DESCRIPTION OF THE INVENTION

[0009] 1 is a front view and a side view of a vehicle equipped with an image processing system according to one embodiment of the present invention. As shown in the side view of Fig. 1, the image processing system according to the present invention is composed of a first image capturing device 2a installed above a window glass (windshield) 6, a second image capturing device 2b installed below the window glass (windshield) 6, and an image processing device 3. As shown in the front view of FIG. 1, the second imaging device 2b is installed vertically spaced apart from the first imaging device 2a. It is desirable to make the FOV (Field of View) of the second imaging device 2b narrower than that of the first imaging device 2a. This is to enable the second imaging device 2b to more easily recognize (image) objects at greater distances. However, this is not limiting, and the FOVs of the first imaging device 2a and the second imaging device 2b may be the same. The first imaging device 2a and the second imaging device 2b may be realized, for example, by a camera, a charge-coupled device (CCD), an image sensor, or the like. Furthermore, as shown in the side view of FIG. 1, the first imaging device 2a and the second imaging device 2b are each electrically connected to the image processing device 3 via a signal line. The signal line may be wired or wireless. Hereinafter, an embodiment of the present invention will be described with reference to the drawings. [Example]

[0010] [Configuration of Image Processing System 1] FIG. 2 is a schematic diagram illustrating the overall configuration of an image processing system according to a first embodiment of the present invention. As illustrated in FIG. 2, the image processing system 1 includes a first imaging device 2a, a second imaging device 2b, and an image processing device 3. The image processing device 3 includes a first image processing unit 3a having a recognition function for basic ADAS (Advanced Driver-Assistance Systems) and a second image processing unit 3b having a recognition function for advanced ADAS / ADS (Automated Driving System, hereinafter referred to as AD). The first imaging device 2a is electrically connected to the first image processing unit 3a and the second image processing unit 3b constituting the image processing device 3 via signal lines. Similarly, the second imaging device 2b is electrically connected to the first image processing unit 3a and the second image processing unit 3b constituting the image processing device 3 via signal lines. An output of the first image processing unit 3a constituting the image processing device 3 and an output of the second image processing unit 3b constituting the image processing device 3 are input to a vehicle control unit 4. As will be described in detail later, the first image processing section 3a and the second image processing section 3b are configured to be able to communicate with each other.

[0011] [Configuration of image processing device 3] FIG. 3 is a functional block diagram of an image processing device 3 constituting an image processing system 1 according to this embodiment. As shown in FIG. 3, a first image processing unit 3a constituting the image processing device 3 is composed of an input I / F 11a, a preprocessing unit 12a, a first object recognition unit 13a, an output I / F 14a, a database 15a, and a communication I / F 16a, which are capable of communicating (data transfer) with each other via an internal bus 17a. Here, the preprocessing unit 12a and the first object recognition unit 13a are realized, for example, by a processor such as a CPU (not shown), a ROM (read only memory) for storing various programs, a RAM (random access memory) for temporarily storing data during the calculation process, and a storage device such as an external storage device. The processor such as a CPU reads and executes the various programs stored in the ROM and stores the calculation results, which are the execution results, in the RAM or the external storage device. It is desirable that the first image processing unit 3a be implemented, for example, in a single ECU (Electronic Control Unit).

[0012] The input I / F 11a acquires image data at 30 or 60 frames per second from the first imaging device 2a and the second imaging device 2b. Note that the number of frames per second is not limited to this. If the input I / F 11a does not acquire image data from the first imaging device 2a, it can determine that the first imaging device 2a has failed. Similarly, if the input I / F 11a does not acquire image data from the second imaging device 2b, it can determine that the second imaging device 2b has failed. The input I / F 11a transfers the image data acquired from the first imaging device 2a to the preprocessing unit 12a via the internal bus 17a.

[0013] The pre-processing unit 12a performs, for example, contour enhancement, smoothing, or normalization on image data from the first imaging device 2a transferred from the input I / F 11a. Normalization is effective for image data from the first imaging device 2a because the brightness of the image data varies depending on the time of day or weather. If the brightness of the image data from the first imaging device 2a is clearly abnormal and normalization is difficult, the pre-processing unit 12a can determine that the first imaging device 2a is faulty. The pre-processing unit 12a transfers the pre-processed image data to the first object recognition unit 13a via the internal bus 17a.

[0014] The first object recognition unit 13a recognizes a first recognition target from the preprocessed image data transferred from the preprocessing unit 12a using a CNN (Convolution Neural Network) such as U-Net. Here, the first recognition target is, for example, a lane, a vehicle, a motorcycle, a pedestrian, etc. Note that the recognition of the first recognition target from the preprocessed image data is not limited to using a CNN, and may instead be configured to perform template matching processing using data stored in a database 15a (described later). The first object recognition unit 13a outputs the lane recognition result, vehicle recognition result, motorcycle recognition result, pedestrian recognition result, etc. as the recognition results of the first recognition target to the vehicle control unit 4 via the internal bus 17a and the output I / F 14a, and the results are used to execute the ADAS function. The database 15a stores data on lanes, vehicles, motorcycles, pedestrians, and the like in advance.

[0015] 3, the second image processing unit 3b of the image processing device 3 is composed of an input I / F 11b, a preprocessing unit 12b, a first object recognition unit 13b, an output I / F 14b, a database 15b, a communication I / F 16b, a second object recognition unit 18b, and a database 19b, which are capable of communicating (transferring data) with each other via an internal bus 17b. The preprocessing unit 12b, the first object recognition unit 13b, and the second object recognition unit 18b are implemented, for example, by a processor such as a CPU (not shown), a ROM (read only memory) for storing various programs, a RAM (random access memory) for temporarily storing data during the calculation process, and a storage device such as an external storage device. The processor such as a CPU reads and executes the various programs stored in the ROM and stores the calculation results, which are the execution results, in the RAM or the external storage device. It is desirable that the second image processing unit 3b be implemented, for example, in a single ECU.

[0016] The input I / F 11b acquires image data at 30 or 60 frames per second from the first imaging device 2a and the second imaging device 2b. Note that the number of frames per second is not limited to this. If the input I / F 11b does not acquire image data from the first imaging device 2a, it can determine that the first imaging device 2a has failed. Similarly, if the input I / F 11b does not acquire image data from the second imaging device 2b, it can determine that the second imaging device 2b has failed. The input I / F 11b transfers the image data acquired from the first imaging device 2a and the second imaging device 2b to the preprocessing unit 12b via the internal bus 17b.

[0017] The pre-processing unit 12b processes, for example, the image data from the first imaging device 2a and the image data from the second imaging device 2b transferred from the input I / F 11b. toThe preprocessing unit 12b performs contour enhancement, smoothing, normalization, or the like on the image data from the first imaging device 2a and the second imaging device 2b. Normalization is effective because the brightness of the image data from the first imaging device 2a or the second imaging device 2b fluctuates depending on the time of day or weather. If the brightness of the image data from the first imaging device 2a or the second imaging device 2b is clearly abnormal and normalization is difficult, the preprocessing unit 12b can determine that the first imaging device 2a or the second imaging device 2b is faulty. The preprocessing unit 12b transfers the image data from the first imaging device 2a, which is the preprocessed image data, to the first object recognition unit 13b via the internal bus 17b. The preprocessing unit 12b also transfers the image data from the second imaging device 2b, which is the preprocessed image data, to the second object recognition unit 18b via the internal bus 17b. The first object recognition unit 13b is similar to the first object recognition unit 13a, and therefore a description thereof will be omitted here. The first object recognition unit 13b operates when the first imaging device 2a or the first image processing unit 3a, which will be described later, fails.

[0018] The second object recognition unit 18b recognizes a second recognition target from image data captured by the second image capture device 2b, which is preprocessed image data transferred from the preprocessing unit 12b, using a convolution neural network (CNN) such as U-Net. Here, the second recognition target includes, for example, a traffic light display status, a road sign, free space, 3D sensing distance, etc. Here, free space refers to an area in which the host vehicle can move. In other words, it refers to an area in which there are no objects that may obstruct the movement of the host vehicle. Furthermore, the 3D sensing distance is the distance between a target object (e.g., a traffic light) and the host vehicle, which can be measured with higher accuracy by obtaining three-dimensional information through stereoscopic vision using the first image capture device 2a and the second image capture device 2b. Note that the recognition of the second recognition target from the preprocessed image data is not limited to the use of a CNN; for example, a template matching process may be performed using data stored in a database 19b, which will be described later. The second object recognition unit 18b outputs the recognition results of the second recognition target, such as traffic light recognition results, road sign recognition results, free space recognition results, and 3D sensing distance recognition results, to the vehicle control unit 4 via the internal bus 17b and the output I / F 14b, and is used to execute advanced ADAS / AD functions. The database 19b stores in advance data necessary for recognizing the display status of traffic lights, road signs, free space, 3D sensing distance, and the like.

[0019] The communication I / F 16b transmits and receives monitoring signals to and from the communication I / F 16a constituting the first image processing unit 3a, for example, at a predetermined interval, to detect abnormalities on the other side. That is, if the communication I / F 16b constituting the second image processing unit 3b transmits a monitoring signal to the communication I / F 16a constituting the first image processing unit 3a but no response signal is transmitted from the communication I / F 16a constituting the first image processing unit 3a, it is determined that the first image processing unit 3a has failed. On the other hand, if the communication I / F 16a constituting the first image processing unit 3a transmits a monitoring signal to the communication I / F 16b constituting the second image processing unit 3b but no response signal is transmitted from the communication I / F 16b constituting the second image processing unit 3b, it is determined that the second image processing unit 3b has failed. In this embodiment, the case where the first image processing unit 3a and the second image processing unit 3b mutually transmit and receive monitoring signals has been described as an example, but the present invention is not limited to this. For example, the vehicle control unit 4 may transmit monitoring signals to the first image processing unit 3a and the second image processing unit 3b, and determine that the first image processing unit 3a or the second image processing unit 3b has failed based on whether or not a response signal has been received from the first image processing unit 3a or the second image processing unit 3b.

[0020] FIG. 4 is a bird's-eye view illustrating an example in which a vehicle equipped with the image processing system 1 according to this embodiment is located at an intersection with a traffic light. FIG. 4 illustrates a situation in which the host vehicle SV equipped with the image processing system 1 according to this embodiment is stopped in front of the intersection because the traffic light 5 is displaying a red light. In the example illustrated in FIG. 4, the FOV of the first image capture device 2a and the FOV of the second image capture device 2b are the same. A motorcycle as a first recognition target is stopped in front of the host vehicle SV in the same lane (first recognition target) within the FOV, and another vehicle OV as a first recognition target is entering the intersection from the right. Also present within the FOV is a traffic light 5 as a second recognition target. As typified by the traffic light 5, the second recognition target is an object having a predetermined height from the road surface, and the predetermined height is set to, for example, 3 m or 5 m. Here, 3 m corresponds to the height of a road sign (not shown), which is the second recognition target, and 5 m corresponds to the height of the traffic light 5. However, the predetermined height is not limited to 3 m or 5 m.

[0021] [Processing flow of image processing device 3] Next, a specific processing operation of the image processing device 3 according to this embodiment will be described below. Fig. 5 is a flowchart showing the processing flow of the image processing device 3 constituting the image processing system 1 according to this embodiment.

[0022] In step S110, the input I / F 11a of the first image processing unit 3a and the input I / F 11b of the second image processing unit 3b, which constitute the image processing device 3, acquire image data from the first imaging device 2a and the second imaging device 2b.

[0023] In step S111, a predetermined condition is met, i.e., whether or not the first imaging device 2a or the first image processing unit 3a has failed is determined. As described above, the determination of whether or not the first imaging device 2a has failed is made by the input I / F 11a and pre-processing unit 12a constituting the first image processing unit 3a, or the input I / F 11b and pre-processing unit 12b constituting the second image processing unit 3b. Furthermore, whether or not the first image processing unit 3a has failed is determined by the communication I / F 16b constituting the second image processing unit 3b as described above, or by the vehicle control unit 4. If the determination result satisfies a predetermined condition, the process proceeds to step S112, and if the determination result does not satisfy the predetermined condition, the process proceeds to step S116.

[0024] In step S112, the system transitions to the degenerate mode. Then, in step S113, the preprocessing unit 12b constituting the second image processing unit 3b performs the above-described preprocessing on the image data acquired from the second imaging device 2b, and transfers the preprocessed image data to the first object recognition unit 13b via the internal bus 17b. Next, in step S114, the second object recognition unit 18b constituting the second image processing unit 3b stops, and the first object recognition unit 13b recognizes the first recognition target, such as a lane, a vehicle, a motorcycle, or a pedestrian.

[0025] In step S115, the first object recognition unit 13b constituting the second image processing unit 3b outputs the lane recognition result, vehicle recognition result, motorcycle recognition result, pedestrian recognition result, etc. as the recognition results of the first recognition target to the vehicle control unit 4 via the internal bus 17b and the output I / F 14b. As a result, although the execution of the advanced ADAS / AD function is stopped, the results are used to execute the ADAS function. In other words, redundancy is ensured.

[0026] On the other hand, in step S116, the pre-processing unit 12a constituting the first image processing unit 3a and the pre-processing unit 12b constituting the second image processing unit 3b perform the above-mentioned pre-processing on the image data acquired from the first imaging device 2a and the second imaging device 2b. In step S117, the first object recognition unit 13a constituting the first image processing unit 3a recognizes a first recognition target, that is, a lane, a vehicle, a motorcycle, a pedestrian, or the like, which is the first recognition target.

[0027] In step S118, the second object recognition unit 18b constituting the second image processing unit 3b recognizes a second recognition target, that is, the second recognition target, such as the display state of a traffic light, a road sign, free space, or a 3D sensing distance. In step S119, the first object recognition unit 13a constituting the first image processing unit 3a outputs the lane recognition result, vehicle recognition result, motorcycle recognition result, pedestrian recognition result, etc. as the recognition results of the first recognition target to the vehicle control unit 4 via the internal bus 17a and the output I / F 14a. Also, the second object recognition unit 18b constituting the second image processing unit 3b outputs the traffic light recognition result, road sign recognition result, free space recognition result, 3D sensing distance recognition result, etc. as the recognition results of the second recognition target to the vehicle control unit 4 via the internal bus 17b and the output I / F 14b. This is used to execute the advanced ADAS / AD function and the ADAS function. In this embodiment, step S118 is executed after step S117, but the present invention is not limited to this, and steps S117 and S118 may be executed in parallel.

[0028] 6 is a flowchart showing another processing flow of the image processing device 3 according to this embodiment. As shown in Fig. 6, in step S110, the input I / F 11a of the first image processing unit 3a and the input I / F 11b of the second image processing unit 3b constituting the image processing device 3 acquire image data from the first imaging device 2a and the second imaging device 2b.

[0029] In step S211, a predetermined condition is met, i.e., whether or not the second imaging device 2b or the second image processing unit 3b has failed is determined. As described above, the determination of whether or not the second imaging device 2b has failed is made by the input I / F 11a and pre-processing unit 12a constituting the first image processing unit 3a, or the input I / F 11b and pre-processing unit 12b constituting the second image processing unit 3b. Furthermore, whether or not the second image processing unit 3b has failed is determined, as described above, by the communication I / F 16a constituting the first image processing unit 3a or the vehicle control unit 4. If the result of the determination satisfies a predetermined condition, the process proceeds to step S112, and if the result of the determination does not satisfy the predetermined condition, the process proceeds to step S116.

[0030] In step S112, the system transitions to the degenerate mode. Then, in step S213, the preprocessing unit 12a constituting the first image processing unit 3a performs the above-described preprocessing on the image data acquired from the first imaging device 2a, and transfers the preprocessed image data to the first object recognition unit 13a via the internal bus 17a. Next, in step S214, the first object recognition unit 13a constituting the first image processing unit 3a recognizes the first recognition target, such as a lane, a vehicle, a two-wheeled vehicle, or a pedestrian.

[0031] In step S215, the first object recognition unit 13a constituting the first image processing unit 3a outputs the lane recognition result, vehicle recognition result, motorcycle recognition result, pedestrian recognition result, etc. as the recognition results of the first recognition target to the vehicle control unit 4 via the internal bus 17a and the output I / F 14a. As a result, although the execution of the advanced ADAS / AD function is stopped, the results are used to execute the ADAS function. In other words, redundancy is ensured.

[0032] On the other hand, in step S116, the pre-processing unit 12a constituting the first image processing unit 3a and the pre-processing unit 12b constituting the second image processing unit 3b perform the above-mentioned pre-processing on the image data acquired from the first imaging device 2a and the second imaging device 2b. In step S117, the first object recognition unit 13a constituting the first image processing unit 3a recognizes a first recognition target, that is, a lane, a vehicle, a motorcycle, a pedestrian, or the like, which is the first recognition target.

[0033] In step S118, the second object recognition unit 18b constituting the second image processing unit 3b recognizes a second recognition target, that is, the second recognition target, such as the display state of a traffic light, a road sign, free space, or a 3D sensing distance. In step S119, the first object recognition unit 13a constituting the first image processing unit 3a outputs the lane recognition result, vehicle recognition result, motorcycle recognition result, pedestrian recognition result, etc. as the recognition results of the first recognition target to the vehicle control unit 4 via the internal bus 17a and the output I / F 14a. Also, the second object recognition unit 18b constituting the second image processing unit 3b outputs the traffic light recognition result, road sign recognition result, free space recognition result, 3D sensing distance recognition result, etc. as the recognition results of the second recognition target to the vehicle control unit 4 via the internal bus 17b and the output I / F 14b. This is used to execute the advanced ADAS / AD function and the ADAS function. In this embodiment, step S118 is executed after step S117, but the present invention is not limited to this, and steps S117 and S118 may be executed in parallel.

[0034] <Modification of Image Processing System> FIG. 7 is a schematic diagram illustrating an overall configuration of a modified image processing system according to the present embodiment. As shown in FIG. 7, the image processing system 10 includes a first imaging device 2a, a second imaging device 2b, and an image processing device 3. The image processing device 3 includes a first image processing unit 3a having a recognition function for basic ADAS and a second image processing unit 3b having a recognition function for advanced ADAS / ADS. The first imaging device 2a is electrically connected to the first image processing unit 3a and the second image processing unit 3b constituting the image processing device 3 via signal lines. In contrast, the second imaging device 2b is electrically connected only to the second image processing unit 3b constituting the image processing device 3 via signal lines. This differs from the image processing system 1 illustrated in FIG. 2 above. The outputs of the first image processing unit 3a constituting the image processing device 3 and the second image processing unit 3b constituting the image processing device 3 are input to the vehicle control unit 4.

[0035] FIG. 8 is a functional block diagram of the image processing device 3 shown in FIG. 7. As shown in FIG. 8, the first image processing unit 3a constituting the image processing device 3 is composed of an input I / F 11a, a preprocessing unit 12a, a first object recognition unit 13a, an output I / F 14a, a database 15a, and a communication I / F 16a, which are capable of communicating (transferring data) with each other via an internal bus 17a. Here, the preprocessing unit 12a and the first object recognition unit 13a are realized by, for example, a processor such as a CPU (not shown), a ROM (read only memory) for storing various programs, a RAM (random access memory) for temporarily storing data during the calculation process, and a storage device such as an external storage device. The processor such as the CPU reads and executes the various programs stored in the ROM and stores the calculation results, which are the execution results, in the RAM or the external storage device. Note that the first image processing unit 3a is preferably implemented in, for example, a single ECU.

[0036] The input I / F 11a acquires image data at 30 or 60 frames per second from the first imaging device 2a, although the number of frames per second is not limited to this. If the input I / F 11a does not acquire image data from the first imaging device 2a, it can determine that the first imaging device 2a has failed. The input I / F 11a transfers the image data acquired from the first imaging device 2a to the pre-processing unit 12a via the internal bus 17a.

[0037] The pre-processing unit 12a performs, for example, contour enhancement, smoothing, or normalization on image data from the first imaging device 2a transferred from the input I / F 11a. Normalization is effective for image data from the first imaging device 2a because the brightness of the image data varies depending on the time of day or weather. If the brightness of the image data from the first imaging device 2a is clearly abnormal and normalization is difficult, the pre-processing unit 12a can determine that the first imaging device 2a is faulty. The pre-processing unit 12a transfers the pre-processed image data to the first object recognition unit 13a via the internal bus 17a.

[0038] The first object recognition unit 13a recognizes a first recognition target from the preprocessed image data transferred from the preprocessing unit 12a using a CNN such as U-Net. Here, the first recognition target is, for example, a lane, a vehicle, a motorcycle, a pedestrian, etc. Note that the recognition of the first recognition target from the preprocessed image data is not limited to using a CNN, and may instead be configured to perform template matching processing using data stored in a database 15a (described later). The first object recognition unit 13a outputs the lane recognition result, vehicle recognition result, motorcycle recognition result, pedestrian recognition result, etc. as the recognition results of the first recognition target to the vehicle control unit 4 via the internal bus 17a and the output I / F 14a, and these results are used to execute the ADAS function. The database 15a stores data on lanes, vehicles, motorcycles, pedestrians, and the like in advance.

[0039] The second image processing unit 3b constituting the image processing device 3 is the same as that shown in FIG. 3 above, and therefore a description thereof will be omitted.

[0040] Next, the specific processing operations of the image processing device 3 according to this embodiment are almost the same as those shown in FIGS. 5 and 6, and therefore only the differences will be described below.

[0041] 5, an input I / F 11a of a first image processing unit 3a constituting the image processing device 3 acquires image data from the first imaging device 2a. An input I / F 11b of a second image processing unit 3b constituting the image processing device 3 acquires image data from the first imaging device 2a and the second imaging device 2b.

[0042] 5, the pre-processing unit 12a constituting the first image processing unit 3a performs the above-described pre-processing on the image data acquired from the first imaging device 2a, and the pre-processing unit 12b constituting the second image processing unit 3b performs the above-described pre-processing on the image data acquired from the first imaging device 2a and the second imaging device 2b.

[0043] 6, an input I / F 11a of a first image processing unit 3a constituting the image processing device 3 acquires image data from the first imaging device 2a. An input I / F 11b of a second image processing unit 3b constituting the image processing device 3 acquires image data from the first imaging device 2a and the second imaging device 2b.

[0044] 6, a predetermined condition is met, i.e., whether or not the second imaging device 2b or the second image processing unit 3b has a malfunction is determined. The determination of whether or not the second imaging device 2b has a malfunction is made by the input I / F 11b and pre-processing unit 12b that constitute the second image processing unit 3b.

[0045] 6, the pre-processing unit 12a constituting the first image processing unit 3a performs the above-described pre-processing on the image data acquired from the first imaging device 2a, and the pre-processing unit 12b constituting the second image processing unit 3b performs the above-described pre-processing on the image data acquired from the first imaging device 2a and the second imaging device 2b.

[0046] As described above, the modified example differs from the image processing system 1 described above in that only the second image processing unit 3b determines whether the second imaging device 2b has a malfunction.

[0047] As described above, according to this embodiment, it is possible to provide an image processing device and an image processing system that can ensure redundancy and perform control even if the imaging device itself breaks down or malfunctions. [Example]

[0048] 9 to 11 show an example of wiper positions in the degenerate mode in Example 2 according to another embodiment of the present invention. In addition to Example 1 described above, this Example differs from Example 1 in that a configuration is added in which the wiper is temporarily operated in the degenerate mode and stopped at a position that does not obstruct the FOV (field of view) of the second imaging device 2b. Only the added points will be described below.

[0049] First, Fig. 12 is a diagram showing the wiper position in normal mode. As shown in Fig. 12, there is a concern that the lower part of the FOV (field of view) of the second image capture device 2b is blocked by the wiper L52. As described in the above-mentioned first embodiment, in the degenerate mode due to a failure of the first image capture device 2a, the image processing device 3 must perform processing based on image data from only the second image capture device 2b. To achieve this, it is necessary to make the FOV (field of view) of the second image capture device 2b as wide as possible.

[0050] 9 shows a case where the wiper R21 and the wiper L22 are driven by one actuator and motor. When the mode is shifted to the degenerate mode in step S112 in FIGS. 5 and 6 in the first embodiment described above, the wiper R21 and the wiper L22 temporarily operate in synchronization with each other and stop at a position where the wiper R21 and the wiper L22 sandwich the second image capture device 2b and do not obstruct the FOV (field of view) of the second image capture device 2b, as shown in FIG.

[0051] In the example shown in Fig. 10, the wiper R31 and the wiper L32 are driven by one actuator and motor. When the mode is shifted to the degenerate mode in step S112 in Fig. 5 and Fig. 6 in the first embodiment, the wiper R31 and the wiper L32 temporarily operate in synchronization with each other, and stop at the turning point of their reciprocating motion as shown in Fig. 9.

[0052] In the example shown in FIG. 11, the wiper R41 and the wiper L42 are each provided with a motor, so when the mode is shifted to the degenerate mode, only one of the wipers, the wiper L42, moves to a predetermined position and stops.

[0053] As described above, according to this embodiment, in addition to the effects of the first embodiment, it is possible to reliably ensure the imaging field of view of the second imaging device in the degenerate mode.

[0054] The present invention is not limited to the above-described embodiment, but includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. [Explanation of symbols]

[0055] 1,10...Image processing system 2a...first imaging device 2b...Second imaging device 3...Image processing device 3a...first image processing unit 3b...Second image processing section 4...Vehicle control unit 5...Traffic lights 6...Window glass (windshield) 11a, 11b...Input I / F 12a, 12b...Pre-processing section 13a, 13b...first object recognition unit 14a, 14b...Output I / F 15a, 15b...Database 16a,16b…Communication I / F 17a, 17b...Internal bus 18b...Second object recognition unit 19b...Database 21, 31, 41, 51...Wiper R 22, 32, 42, 52...Wiper L

Claims

1. An image processing device that recognizes a recognition target based on image data captured by a first imaging device provided on a vehicle and a second imaging device provided on the vehicle and spaced apart from the first imaging device in a vertical direction, a first image processing unit that recognizes a first recognition target based on image data of the first imaging device; a second image processing unit that recognizes a second recognition target different from the first recognition target based on image data of the first imaging device and the second imaging device, the first image processing unit or the second image processing unit determines a failure of the second imaging device, the first image processing unit determines a failure of the second image processing unit, 10. An image processing apparatus, comprising: a first image processing unit that continues to recognize the first recognition target when a predetermined condition is satisfied;

2. In the image processing device according to claim 1, The image processing device according to claim 1, wherein the predetermined condition is a failure of the second image capturing device or a failure of the second image processing unit.

3. In the image processing device according to claim 2, When the predetermined condition is satisfied, the image processing device transitions to a degenerate mode, and the first image processing unit recognizes the first recognition target based on image data of the first imaging device.

4. In the image processing device according to claim 1, The image processing device is characterized in that the second recognition target includes at least an object that exists at a predetermined height or more from the road surface.

5. In the image processing device according to claim 4, the first recognition object includes a lane, a vehicle, a motorcycle, and a pedestrian; An image processing device characterized in that the second recognition target includes any of the display status of a traffic light, a road sign, a free space which is an area without any objects that may obstruct the movement of the vehicle, and a 3D sensing distance.

6. An image processing system for recognizing a recognition target, comprising: a first imaging device; a second imaging device installed vertically spaced apart from the first imaging device; a first image processing unit electrically connected to at least the first imaging device; a second image processing unit electrically connected to the first imaging device and the second imaging device, the first image processing unit recognizes a first recognition target based on image data of the first imaging device; the second image processing unit recognizes a second recognition target different from the first recognition target based on image data of the first imaging device and the second imaging device; the first image processing unit or the second image processing unit determines a failure of the second imaging device, the first image processing unit determines a failure of the second image processing unit, An image processing system, characterized in that, when a predetermined condition is satisfied, the first image processing unit continues recognizing the first recognition target.

7. In the image processing system according to claim 6, The image processing system is characterized in that the predetermined condition is a failure of the second image capturing device or a failure of the second image processing unit.

8. In the image processing system according to claim 7, When the predetermined condition is satisfied, the image processing system transitions to a degenerate mode, and the first image processing unit recognizes the first recognition target based on image data from the first imaging device.

9. In the image processing system according to claim 6, An image processing system, wherein the second recognition target includes at least an object that exists at a predetermined height or more from the road surface.

10. In the image processing system according to claim 9, the first recognition object includes a lane, a vehicle, a motorcycle, and a pedestrian; An image processing system characterized in that the second recognition object includes any of the display status of a traffic light, a road sign, a free space which is an area without any objects that may obstruct the movement of the vehicle, and a 3D sensing distance.

Images