Image processing apparatus and image processing method

Abstract

To provide a technique for cutting out an area in which the attention of a driver may be declined from a front image and using it for recognition processing.SOLUTION: An image processing device includes: an acquisition unit that acquires a front image obtained by capturing the front of a vehicle with an in-vehicle front camera and acquires an in-vehicle image obtained by capturing the driver with an in-vehicle camera; a line-of-sight detection unit that detects a line-of-sight direction of the driver on the basis of the in-vehicle image; a setting unit that sets a cutout area in which a part of the front image is cut out based on the detected line-of-sight direction of the driver; and a recognition processing unit that executes image recognition processing on the set cutout area. The setting unit sets a cutout area in an area deviated from the line-of-sight direction of the driver in the front image.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a technique for reducing the data amount of an image used for recognition processing.

Background Art

[0002] Patent Document 1 discloses an image processing apparatus including an image cutting-out unit that cuts out a part from an entire image including an entire subject in an image captured by a camera mounted on a vehicle, an image reduction unit that reduces the entire image in the image captured by the camera to generate a reduced image, a first image recognition unit that recognizes a part of the image cut out by the image cutting-out unit, a second image recognition unit that recognizes the reduced image reduced by the image reduction unit, and an image recognition result integration unit that integrates a first recognition result recognized by the first image recognition unit and a second recognition result recognized by the second image recognition unit and generates information on the integrated recognition result. The image cutting-out unit cuts out a part from a cuttable range located at the center of the entire image.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the technique described in Patent Document 1, the center part of the entire image is cut out and used for recognition processing. However, since the driver basically looks ahead during driving, the driver often already recognizes the cut-out part in the center.

[0005] An object of the present invention is to provide a technique for cutting out an area where the driver's attention may drop from a front image and using it for recognition processing.

Means for Solving the Problems

[0006] In order to solve the above problems, an image processing apparatus according to an aspect of the present invention includes an acquisition unit that acquires a front image obtained by imaging the front of a vehicle with an in-vehicle front camera and an in-vehicle image obtained by imaging a driver with an in-vehicle camera, a gaze detection unit that detects the gaze direction of the driver based on the in-vehicle image, a setting unit that sets a cutout area obtained by cutting out a part of the front image based on the detected gaze direction of the driver, and a recognition processing unit that executes image recognition processing on the set cutout area. The acquisition unit acquires driving state information indicating the driving state of the vehicle from in-vehicle sensors. The setting unit sets the cutout area in an area of the front image that is deviated from the gaze direction of the driver. Then, the height of the cutout area in the front image is adjusted based on the driving state information. The setting unit sets the size of the cutout area according to the position and movement of the obstacle recognized by the previous image recognition process.

[0007] Another aspect of the present invention is an image processing method. This method is an image processing method in which each step is executed by a computer, and includes a step of acquiring a front image obtained by imaging the front of a vehicle with an in-vehicle front camera and an in-vehicle image obtained by imaging a driver with an in-vehicle camera, a step of detecting the gaze direction of the driver based on the in-vehicle image, a step of setting a cutout area obtained by cutting out a part of the front image based on the detected gaze direction of the driver, and a step of executing image recognition processing on the set cutout area. In the step of acquisition, driving state information indicating the driving state of the vehicle is acquired from in-vehicle sensors. In the setting step, the cutout area is set in an area of the front image that is deviated from the gaze direction of the driver. Then, the height of the cutout area in the front image is adjusted based on the driving state information. In the step of setting, the size of the cutout area is set according to the position and movement of the obstacle recognized by the previous image recognition process.

Advantages of the Invention

[0008] According to the present invention, it is possible to provide a technique of cutting out an area where the driver's attention may drop from the front image and using it for recognition processing.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Embodiments for Carrying Out the Invention

[0010] FIG. 1 shows a state where the driver 30 is looking outside the vehicle through the windshield 34 from inside the vehicle. The surrounding situation through the windshield 34 shown in FIG. 1 is imaged by a front camera and acquired as a front image. The acquired front image is used for image recognition processing in order to detect obstacles that may become obstacles to driving.

[0011] In the image recognition process, if the front image is used as it is, it will impose a load. Therefore, it is preferable to process the front image to reduce the amount of data used for the image recognition process. Thus, the image processing apparatus of the embodiment executes the recognition process using an image with a reduced amount of data by cutting out a part of the front image. In particular, it cuts out the area that the driver 30 is not looking at and executes the recognition process. This recognition process result may be used, for example, in automatic driving control.

[0012] In FIG. 1, the line of sight direction 32 of the driver 30 is facing the left side, and the visual field area 36 of the driver 30 is biased to the left side. At this time, since the driver 30 is gazing at the left side, there is a possibility that the attention on the right side is reduced. Therefore, the image processing apparatus of the embodiment generates a cut-out area 38 by cutting out the area that the driver 30 is not looking at, and executes the recognition process on the cut-out area 38. Thereby, the load of the image recognition process can be reduced.

[0013] FIG. 2 is a diagram showing the functional configuration of the image processing system 1. Each function of the image processing system 1 can be configured, in terms of hardware, by circuit blocks, memories, and other LSIs, and in terms of software, can be realized by system software or application programs loaded into the memory. Therefore, it is understood by those skilled in the art that each function of the image processing system 1 can be realized in various forms by hardware only, software only, or a combination thereof, and is not limited to any one of them.

[0014] The image processing system 1 includes an image processing device 10, a front camera 12, an in-vehicle camera 14, an in-vehicle sensor 16, and a vehicle control device 18. The image processing device 10 includes an acquisition unit 20, a gaze detection unit 22, a setting unit 24, and a recognition processing unit 26.

[0015] The front camera 12 captures an image of the front of the vehicle to generate a front image, and sends the front image to the image processing device 10. The in-vehicle camera 14 captures an image of the driver in the vehicle to generate an in-vehicle image, and sends the in-vehicle image to the image processing device 10.

[0016] The in-vehicle sensor 16 may be an acceleration sensor that detects the acceleration of the vehicle, an inclination sensor that detects the inclination of the road surface, a steering angle sensor that detects the steering angle, etc., detects the driving state of the vehicle, and sends the detected driving state information to the image processing device 10.

[0017] The vehicle control device 18 controls the driving of the vehicle based on the information of obstacles and traffic signs recognized by the image processing device 10. For example, the vehicle control device 18 performs control to avoid collisions with obstacles or executes control to drive the vehicle according to traffic signs.

[0018] The acquisition unit 20 of the image processing device 10 acquires a front image from the front camera 12, an in-vehicle image from the in-vehicle camera 14, and driving state information from the in-vehicle sensor 16.

[0019] The gaze detection unit 22 analyzes the in-vehicle image to detect the driver's gaze direction. The gaze detection unit 22 calculates the orientation of the driver's face and the position of the pupils in the eyes based on the in-vehicle image, and detects the driver's gaze direction based on the orientation of the face and the position of the pupils in the eyes. The gaze detection unit 22 detects the driver's gaze direction with a vector extending from the eyes starting from the center of both eyes. The orientation of the driver's face is calculated based on the positions of the facial parts from the center of the face.

[0020] The setting unit 24 sets a cutout area obtained by cutting out a part of the front image based on the line-of-sight direction of the driver detected by the line-of-sight detection unit 22. The setting unit 24 sets the cutout area in an area deviated from the line-of-sight direction 32 of the driver 30 in the front image.

[0021] The setting unit 24 derives the visual field area that the driver 30 is looking at based on the line-of-sight direction 32 of the driver 30, and sets the cutout area in an area deviated from the visual field area. The method for setting this cutout area will be described with reference to new drawings.

[0022] FIG. 3 is a diagram showing a front image 40 including a visual field area 42 and cutout areas 44a and 44b. In the example shown in FIG. 3, the driver 30 is driving facing forward, and the line-of-sight direction 46 is facing the center of the front image 40. The setting unit 24 derives the visual field area 42 based on the line-of-sight direction 46. The position passing through the vector shown in the line-of-sight direction 46 is set as the center C of the visual field area 42. The setting unit 24 may hold a table for calculating the coordinates at which the vector that is the line-of-sight direction 46 penetrates the front image 40. The size of the visual field area 42 is set according to the human visual angle and may be set based on experiments or the like.

[0023] The setting unit 24 sets the cutout areas 44a and 44b to be shifted from the visual field area 42. The cutout areas 44a and 44b shown in FIG. 3 do not overlap with the visual field area 42 and are arranged outside the visual field area 42. Thereby, the setting unit 24 can set the outside of the driver's 30 visual field as the cutout areas 44a and 44b. Note that the cutout areas 44a and 44b may partially overlap with the visual field area 42, but are set to be shifted from the visual field area 42 so that at least more than half of the area of the cutout areas 44a and 44b does not overlap with the visual field area 42.

[0024] The sizes of the cutout areas 44a and 44b, that is, the areas on the front image, are preset. The upper limit of the sizes of the cutout areas 44a and 44b may be determined by the transfer speed to the recognition processing unit 26 and may be determined by the performance of the hardware. When the visual field area 42 of the driver 30 is located at the center of the front image 40, the setting unit 24 sets the right and left areas deviated from the center of the front image 40 as the cutout areas 44a and 44b. When the driver 30 is driving facing forward, the image processing device 10 pays attention to the left and right.

[0025] By the way, when the vehicle is driving on a slope or a curved road, the center position of the front image may deviate from the planned path. The setting unit 24 adjusts the positions of the cutout areas 44a and 44b in the front image 40 based on the driving state information. The height of the center of the cutout areas 44a and 44b may be preset as a reference height. The setting unit 24 adjusts the reference height of the cutout areas 44a and 44b based on the driving state information. Thereby, the cutout areas can be appropriately set according to the driving state of the vehicle.

[0026] The size of the cutout area may be changed according to the position and movement of the obstacle. For example, when it is recognized by the previous image processing that the obstacle is located or approaching in the traveling direction of the vehicle, the setting unit 24 may set the cutout area to be larger than when there is no obstacle. That is, the setting unit 24 sets the size of the cutout area according to the position and movement of the obstacle. The setting unit 24 may indicate the cutout area by coordinates on the front image. When the driver is not facing forward, the setting unit 24 sets the area located at the center of the front image w as the cutout area.

[0027] The setting unit 24 executes a processing operation to cut out the cutout areas 44a and 44b from the front image 40, and sends the cutout areas 44a and 44b to the recognition processing unit 26. The cutout areas 44a and 44b may include coordinate information in the front image 40. Further, the setting unit 24 executes a process of reducing the entire front image 40 to reduce the data amount. For example, the setting unit 24 reduces the front image 40 to one-tenth to generate a reduced image.

[0028] Return to FIG. 2. The recognition processing unit 26 receives the captured image and executes a process of recognizing the object included in the captured image. The recognition processing unit 26 detects the orientation of the driver's face and the position of the pupil in the eyes based on the in-vehicle image. The recognition processing unit 26 analyzes the front image and detects objects such as obstacles and traffic signs included in the front image.

[0029] The recognition processing unit 26 executes an image recognition process on the cutout area set by the setting unit 24 and generates information regarding the object included in the cutout area. The information regarding the object includes the type of the object, the positional relationship between the vehicle and the object, and the like. Thereby, the area outside the driver's visual field can be recognized without degrading the recognition accuracy. Further, by narrowing down the area outside the driver's visual field, the amount of data to be recognized can be reduced.

[0030] The recognition processing unit 26 executes an image recognition process on the reduced image of the front image and generates information regarding the object included in the reduced image. For the recognition results that overlap between the cutout area and the reduced area, the cutout area is prioritized over the reduced image. The recognition processing unit 26 outputs the recognition result to the vehicle control device 18.

[0031] In a modified example, the setting unit 24 may set a cutout area based on the detection result of the side camera. The front camera 12 captures the front of the vehicle, while the side camera captures the areas on both sides of the front image. The recognition processing unit 26 analyzes the side image captured by the side camera and tracks the objects included in the side image. When it is predicted based on the tracking result of the object that the object will enter the viewing angle of the front camera 12 from the side camera, the setting unit 24 sets the area where the object will enter as the cutout area. That is, when the object enters the viewing angle of the front camera 12 from the right side, the setting unit 24 sets the right end area of the front image as the cutout area, and when the object enters the viewing angle of the front camera 12 from the left side, the setting unit 24 sets the left end area of the front image as the cutout area. Thereby, when the object enters the viewing angle, the recognition processing unit 26 can accurately detect the object.

[0032] FIG. 4 is a flowchart of the image recognition processing executed by the image processing apparatus 10. The acquisition unit 20 acquires a front image from the front camera 12 (S10) and acquires an in-vehicle image from the in-vehicle camera 14 (S12). The gaze detection unit 22 detects the driver's gaze direction based on the in-vehicle image (S14). Note that the recognition processing unit 26 may analyze the in-vehicle image to detect the driver's face, eyes, and the position of the pupils in the eyes, respectively, and the gaze detection unit 22 may detect the driver's gaze direction based on the detection results.

[0033] The setting unit 24 derives a visual field area in the front image based on the gaze direction detected by the gaze detection unit 22 (S16). The setting unit 24 sets a cutout area outside the visual field area in the front image (S18). The setting unit 24 cuts out the set cutout area from the front image and sends it to the recognition processing unit 26, and the recognition processing unit 26 executes image recognition processing on the cutout area (S20).

[0034] The setting unit 24 reduces the entire front image to generate a reduced image (S22), and the recognition processing unit 26 executes recognition processing on the reduced image (S24). The recognition processing unit 26 integrates the recognition result of the cut-out area and the recognition result of the reduced image so that the recognition result of the cut-out area takes precedence, generates a recognition result, and outputs the recognition result to the vehicle control device 18 (S26).

[0035] As described above, the present disclosure has been described based on the embodiments. The present disclosure is not limited to the above-described embodiments, and various design changes and other modifications can be made based on the knowledge of those skilled in the art.

Explanation of Reference Numerals

[0036] 1 Image processing system, 10 Image processing device, 12 Front camera, 14 In-vehicle camera, 16 Vehicle-mounted sensor, 18 Vehicle control device, 20 Acquisition unit, 22 Gaze detection unit, 24 Setting unit, 26 Recognition processing unit, 30 Driver, 32 Gaze direction, 34 Windshield, 36 Field of view area, 38 Cut-out area, 40 Front image, 42 Field of view area, 44a, 44b Cut-out area.

Claims

1. An acquisition unit that acquires a front image obtained by imaging the front of the vehicle with an in-vehicle front camera and an in-vehicle image obtained by imaging the driver with an in-vehicle camera; A gaze detection unit that detects the gaze direction of the driver based on the in-vehicle image; A setting unit that sets a cutout area obtained by cutting out a part of the front image based on the detected gaze direction of the driver; An image recognition processing unit that executes image recognition processing on the set cutout area, and The acquisition unit acquires driving state information indicating the driving state of the vehicle from an in-vehicle sensor, The setting unit sets a cutout area in an area deviated from the gaze direction of the driver in the front image, and adjusts the height of the cutout area in the front image based on the driving state information, The setting unit sets the size of the cutout area according to the position and movement of the obstacle recognized by the previous image recognition processing. An image processing apparatus characterized by this.

2. The image processing apparatus according to claim 1, wherein the setting unit derives a visual field area that the driver is looking at based on the gaze direction of the driver, and sets the cutout area by shifting it from the visual field area.

3. The image processing apparatus according to claim 2, wherein when the visual field area of the driver is located at the center of the front image, the setting unit sets the cutout area in the right and left areas deviated from the center of the front image.

4. An image processing method in which each step is executed by a computer, A step of acquiring a front image obtained by imaging the front of the vehicle with an in-vehicle front camera and an in-vehicle image obtained by imaging the driver with an in-vehicle camera; A step of detecting the gaze direction of the driver based on the in-vehicle image; A step of setting a cutout area obtained by cutting out a part of the front image based on the detected gaze direction of the driver; The step of performing image recognition processing on a set cut-out area, and In the step of obtaining, driving state information indicating the driving state of the vehicle is obtained from an in-vehicle sensor, In the step of setting, a cut-out area is set in an area of the front image that is deviated from the line-of-sight direction of the driver, and the height of the cut-out area in the front image is adjusted based on the driving state information, An image processing method characterized by setting the size of a cut-out area according to the position and movement of an obstacle recognized by a previous image recognition process in the step of setting.

Images