Image processing device, image processing method, and computer-readable storage medium

Abstract

The present invention relates to an image processing device, an image processing method, and a computer-readable storage medium capable of appropriately identifying a moving object outside a vehicle based on an image captured from the interior of the vehicle. The image processing device includes: an imaging unit that captures an image including an area corresponding to the interior of the vehicle and an area corresponding to the exterior; an acquisition unit that acquires a plurality of images captured by the imaging unit at predetermined time intervals; a generation unit that generates a mask filter for masking an area corresponding to the interior of the vehicle in the image captured by the imaging unit based on a variation in the plurality of images acquired by the acquisition unit; and a storage unit that stores the mask filter generated by the generation unit.

Description

Technical Field

[0001] The present invention relates to an image processing device and an image processing method for processing an image captured by a capturing means, and a computer-readable storage medium storing the program. Background Art

[0002] Various processing is performed on images captured by a camera mounted on a vehicle. Patent document 1 describes the following: in an image captured by a shooting device installed so as to face rearward from the side rearview mirror's field of view, images are excluded from the processing range based on the fact that the magnitude of the deviation in brightness or hue of each pixel is below a predetermined threshold. Patent document 2 describes the following: a mask image is generated in a manner consistent with the vehicle body portion in an image captured by a shooting unit arranged near the side rearview mirror, thereby more appropriately improving the driver's visual recognition of other vehicles. Patent document 3 describes the following: a graphic for route guidance is displayed superimposed on an image captured by a vehicle-mounted camera arranged around the front window and capturing the front of the vehicle.

[0003] On the other hand, regarding image processing, it is known to remove a moving object from an image to generate a road background image in which only the road background is extracted (Patent Document 4), or to relatively increase the importance of brightness in areas with high importance in object recognition processing, adjusting the brightness so that the areas with higher importance have the optimal brightness (Patent Document 5).

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Laid-Open No. 2016-224649

[0007] Patent Document 2: Japanese Patent Application Laid-Open No. 2016-111509

[0008] Patent Document 3: Japanese Patent Application Laid-Open No. 2007-315861

[0009] Patent Document 4: Japanese Patent Application Laid-Open No. 2003-296709

[0010] Patent Document 5: Japanese Patent Application Publication No. 2019-139471 Summary of the Invention

[0011] Problems to be solved by the invention

[0012] However, none of the patent documents mentions that a moving object outside a vehicle can be appropriately recognized based on an image captured from inside the vehicle and including both the interior and exterior of the vehicle.

[0013] An object of the present invention is to provide an image processing device, an image processing method, and a computer-readable storage medium storing a program that can appropriately recognize a moving object outside a vehicle based on an image captured from the interior of the vehicle.

[0014] Means used to solve problems

[0015] The image processing device involved in the present invention includes: a shooting unit that shoots an image including an area corresponding to the interior and an area corresponding to the exterior of a vehicle; an acquisition unit that acquires a plurality of images shot by the shooting unit at predetermined time intervals; a generation unit that generates a mask filter for masking the area corresponding to the interior of the vehicle in the image shot by the shooting unit based on the amount of change in the plurality of images acquired by the acquisition unit; and a storage unit that stores the mask filter generated by the generation unit.

[0016] The image processing method involved in the present invention is an image processing method performed in an image processing device, wherein the image processing device has a shooting unit for shooting an image including an area corresponding to the interior of a vehicle and an area corresponding to the exterior, wherein the image processing method comprises: an acquisition step, in which a plurality of images shot by the shooting unit are acquired at predetermined time intervals; a generation step, in which a mask filter for masking the area corresponding to the interior of the vehicle in the image shot by the shooting unit is generated based on the amount of change in the plurality of images acquired in the acquisition step; and a storage step, in which the mask filter generated in the generation step is stored in a storage unit.

[0017] The computer-readable storage medium storing a program involved in the present invention stores a program for causing a computer to function in the following manner: acquiring multiple images captured by a capturing unit at predetermined time intervals, the capturing unit capturing images including an area corresponding to the interior of a vehicle and an area corresponding to the exterior, generating a mask filter for masking the area corresponding to the interior of the vehicle in the images captured by the capturing unit based on a change in the multiple acquired images, and storing the generated mask filter in a storage unit.

[0018] Effects of the Invention

[0019] According to the present invention, a moving object outside the vehicle can be appropriately recognized based on an image captured from the interior of the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a diagram showing the configuration of a vehicle control device.

[0021] Figure 2 This is a diagram showing the functional blocks of the control unit.

[0022] Figure 3 A diagram illustrating a driving recorder and an indicator.

[0023] Figure 4 This figure shows an image captured by a drive recorder.

[0024] Figure 5 This is a flowchart showing the display control process.

[0025] Figure 6 3 is a flowchart showing the mask filter generation process.

[0026] Figure 7 This is a flowchart showing the display control process.

[0027] Figure 8 A diagram for explaining the averaged frame image and the mask filter.

[0028] Figure 9 A diagram for explaining object detection using a masked image.

[0029] Figure 10 This is a diagram showing a situation where pedestrians are recognized around a vehicle.

[0030] Figure 11 This is a diagram for explaining the determination of an object that becomes a risk target.

[0031] Figure 12 This is a diagram for explaining the determination of an object that becomes a risk target.

[0032] Description of Reference Numerals

[0033] 1: Vehicle; 2: Control unit; 20, 21, 22, 23, 24, 25, 26, 27, 28, 29: ECU; 200, 220: Control unit; 218: Driving recorder; 219: Indicator. DETAILED DESCRIPTION

[0034] The following embodiments are described in detail with reference to the accompanying drawings. The following embodiments do not limit the inventions described herein. Furthermore, the combinations of features described in the embodiments are not necessarily essential to the invention. Any combination of two or more of the multiple features described in the embodiments may be used. Identical or similar components are denoted by the same reference numerals, and duplicate descriptions are omitted.

[0035] [First embodiment]

[0036] Figure 1 This is a block diagram of a vehicle control device (travel control device) according to one embodiment of the present invention, which controls the vehicle 1. Figure 1 , a top view and a side view schematically illustrate a vehicle 1. As an example, the vehicle 1 is a sedan-type four-wheeled passenger vehicle.

[0037] Figure 1 The control device includes a control unit 2. The control unit 2 includes a plurality of ECUs 20 to 29 that are connected to communicate via an in-vehicle network. Each ECU includes a processor represented by a CPU, a storage device such as a semiconductor memory, an interface with an external device, etc. The storage device stores programs for execution by the processor, data used by the processor in processing, etc. Each ECU may also have multiple processors, storage devices, interfaces, etc. In addition, Figure 1 The control device can be configured as a computer that implements the invention involved in the program.

[0038] The following describes the functions and the like performed by each of the ECUs 20 to 29. The number of ECUs and the functions performed by them can be appropriately designed, and they can be further refined or integrated than in the present embodiment.

[0039] The ECU 20 performs control related to driving assistance and autonomous driving of the vehicle 1. In driving assistance, at least one of steering and acceleration / deceleration of the vehicle 1 is automatically controlled. In autonomous driving, both steering and acceleration / deceleration of the vehicle 1 are automatically controlled.

[0040] The ECU 21 controls the electric power steering system 3. The electric power steering system 3 includes a mechanism for steering the front wheels based on the driver's driving operation (steering operation) of the steering wheel 31. The electric power steering system 3 also includes a motor that generates driving force to assist the steering operation or automatically steer the front wheels, and a sensor that detects the steering angle. When the vehicle 1 is in autonomous driving mode, the ECU 21 automatically controls the electric power steering system 3 in accordance with instructions from the ECU 20 to control the direction of travel of the vehicle 1.

[0041] ECU 22 and ECU 23 control detection units 41 through 43, which detect the vehicle's surrounding conditions, and process the detection results. Detection unit 41 is a camera (hereinafter sometimes referred to as camera 41) that captures images of the vehicle 1 in front of it. In this embodiment, it is mounted on the front roof of vehicle 1, inside the cabin of the front window. By analyzing the images captured by camera 41, it is possible to extract the outlines of objects and lane demarcation lines (such as white lines) on the road.

[0042] The detection unit 42 is a light detection and ranging (LIDAR) that detects targets around the vehicle 1 or measures the distance to the target. In the case of this embodiment, five detection units 42 are provided, one at each corner of the front of the vehicle 1, one at the center of the rear, and one at each side of the rear. The detection unit 43 is a millimeter wave radar (hereinafter sometimes referred to as radar 43) that detects targets around the vehicle 1 or measures the distance to the target. In the case of this embodiment, five radars 43 are provided, one at the center of the front of the vehicle 1, one at each corner of the front, and one at each corner of the rear.

[0043] ECU 22 controls the camera 41 and the detection units 42 on one side, and processes the detection results. ECU 23 controls the camera 41 and the radars 43 on the other side, and processes the detection results. Having two sets of devices for detecting the vehicle's surroundings improves the reliability of the detection results. Furthermore, having different types of detection units, such as cameras and radars, allows for comprehensive analysis of the vehicle's surrounding environment.

[0044] ECU 24 controls the gyro sensor 5, GPS sensor 24b, and communication device 24c, and processes information on detection and communication results. Gyro sensor 5 detects the rotational motion of vehicle 1. The vehicle 1's route can be determined based on the detection results of gyro sensor 5, wheel speed, and other factors. GPS sensor 24b detects the vehicle 1's current location. Communication device 24c wirelessly communicates with a server that provides map information, traffic information, and weather information to obtain this information. ECU 24 can access a database 24a of map information stored in a storage device, and perform tasks such as searching for a route from the current location to the destination. Furthermore, a database of the aforementioned traffic information, weather information, and other information can also be constructed within database 24a.

[0045] The ECU 25 includes a communication device 25a for inter-vehicle communication. The communication device 25a wirelessly communicates with other surrounding vehicles, exchanging information between vehicles. The communication device 25a has various communication functions, such as dedicated short-range communication (DSRC) and cellular communication. The communication device 25a can also be configured as a TCU (Telematics Communication Unit) including transmitting and receiving antennas. DSRC is a one-way or two-way short- to medium-range communication function that enables high-speed data communication between vehicles and between roads and vehicles.

[0046] ECU 26 controls the power unit 6. The power unit 6 is a mechanism that outputs driving force to rotate the drive wheels of vehicle 1 and includes, for example, an engine and a transmission. For example, ECU 26 controls the engine output in response to the driver's driving operation (accelerator operation or accelerator operation) detected by the operation detection sensor 7a provided on the accelerator pedal 7A, or switches the transmission gears based on information such as the vehicle speed detected by the vehicle speed sensor 7c. When vehicle 1 is in automatic driving mode, ECU 26 automatically controls the power unit 6 in response to instructions from ECU 20, thereby controlling the acceleration and deceleration of vehicle 1.

[0047] The ECU 27 controls the lighting devices (headlights, taillights, etc.) including the direction indicator 8 (turn signal). Figure 1 In the example of FIG, the direction indicators 8 are provided at the front, door mirrors, and rear of the vehicle 1.

[0048] ECU28 controls the input-output device 9. The input-output device 9 outputs information to the driver and receives input of information from the driver. The sound output device 91 reports information to the driver by sound. The display device 92 reports information to the driver by displaying an image. The display device 92 is, for example, arranged in front of the driver's seat to constitute an instrument panel, etc. In addition, sound and display are illustrated here, but information can also be reported by vibration or light. In addition, it is also possible to combine multiple of sound, display, vibration or light to report information. Furthermore, it is also possible to make different combinations or different notification methods according to the level of the information to be notified (for example, the urgency). In addition, the display device 92 includes a navigation device.

[0049] The input device 93 is a switch group that is arranged at a position where the driver can operate it and issues instructions to the vehicle 1 , but may also include a voice input device.

[0050] ECU29 controls the braking device 10 and the parking brake (not shown). The braking device 10 is, for example, a disc brake device, which is provided on each wheel of the vehicle 1 and decelerates or stops the vehicle 1 by applying resistance to the rotation of the wheel. ECU29 controls the operation of the braking device 10 in accordance with the driver's driving operation (brake operation) detected by the operation detection sensor 7b provided on the brake pedal 7B. When the driving state of the vehicle 1 is automatic driving, ECU29 automatically controls the braking device 10 in accordance with the instructions from ECU20 to control the deceleration and stopping of the vehicle 1. The braking device 10 and the parking brake can operate in order to maintain the stopped state of the vehicle 1. In addition, when the transmission of the power unit 6 is equipped with a parking lock mechanism, the parking lock mechanism can also be activated in order to maintain the stopped state of the vehicle 1.

[0051] The control performed by ECU 20 related to driving assistance for vehicle 1 will be described. In driving assistance, ECU 20 automatically controls at least one of vehicle 1's steering and acceleration / deceleration. During automatic control, ECU 20 obtains information (external information) related to the surrounding conditions of vehicle 1 from ECU 22 and ECU 23 and, based on this information, instructs ECU 21, ECU 26, and ECU 29 to control vehicle 1's steering, acceleration, and deceleration. Furthermore, even when both steering and acceleration / deceleration of vehicle 1 are controlled by ECU 20, if the driver is requested to monitor the surroundings or system status, this control is also performed as driving assistance-related control. While the above description describes a case where ECU 20 performs control related to driving assistance for vehicle 1, there are also cases where ECU 20 performs control related to autonomous driving for vehicle 1. In this case, if the driver indicates a destination and autonomous driving, ECU 20 automatically controls vehicle 1 toward the destination according to the guidance route searched by ECU 24. In this case, as in the case of executing control related to driving assistance, ECU 20 obtains information related to the surrounding conditions of vehicle 1 (external information) from ECU 22 and ECU 23, and based on the obtained information, instructs ECU 21, ECU 26, and ECU 29 to control the steering, acceleration, and deceleration of vehicle 1. This embodiment is applicable both to the case where ECU 20 executes control related to driving assistance of vehicle 1 and to the case where ECU 20 executes control related to autonomous driving of vehicle 1.

[0052] Figure 2 2 is a diagram showing the functional blocks of the control unit 2. The control unit 200 and Figure 1 The control unit 2 corresponds to the external recognition unit 201, the own position recognition unit 202, the vehicle interior recognition unit 203, the action planning unit 204, the drive control unit 205, and the device control unit 206. Each functional block is connected by Figure 1 The illustrated ECU or multiple ECUs may be used to implement the system.

[0053] The outside world recognition unit 201 recognizes outside information of the vehicle 1 based on signals from the outside world recognition camera 207 and the outside world recognition sensor 208. Here, the outside world recognition camera 207 is, for example, Figure 1 The camera 41, the sensor 208 for external recognition is, for example, Figure 1The detection unit 42 and the detection unit 43 are used. The external recognition unit 201 recognizes scenes such as intersections, railway crossings, tunnels, free spaces such as road shoulders, and the behavior (speed, direction of travel) of other vehicles based on signals from the external recognition camera 207 and the external recognition sensor 208. The own position recognition unit 202 recognizes the current position of the vehicle 1 based on signals from the GPS sensor 211. Here, the GPS sensor 211 is used for example to identify the vehicle 1. Figure 1 The GPS sensor 24b corresponds to the GPS sensor 24b.

[0054] The interior recognition unit 203 identifies passengers in the vehicle 1 and their status based on signals from the interior recognition camera 209 and the interior recognition sensor 210. The interior recognition camera 209 is, for example, a near-infrared camera mounted on the display device 92 inside the vehicle 1, and detects the direction of the passenger's line of sight. The interior recognition sensor 210 is, for example, a sensor that detects the passenger's biological signals. Based on these signals, the interior recognition unit 203 can identify whether the passenger is drowsy or engaged in activities other than driving.

[0055] The action planning unit 204 executes a driving plan for the vehicle 1 such as an optimal path, a risk avoidance path, etc. based on the recognition results made by the external recognition unit 201 and the self-position recognition unit 202. The action planning unit 204 makes an action plan based on, for example, the entry judgment of the starting point and the end point of an intersection, a railway crossing, etc., and the prediction of the behavior of other vehicles. The drive control unit 205 controls the driving force output device 212, the steering device 213, and the braking device 214 based on the action plan made by the action planning unit 204. Here, the driving force output device 212 is, for example, Figure 1 The power device 6 corresponds to the steering device 213 and Figure 1 The electric power steering device 3 corresponds to the electric power steering device 3, and the braking device 214 corresponds to the braking device 10.

[0056] The device control unit 206 controls the devices connected to the control unit 200. For example, the device control unit 206 controls the speaker 215 to output predetermined audio messages such as warnings and navigation messages. Furthermore, for example, the device control unit 206 controls the display device 216 to display a predetermined interface screen. The display device 216 corresponds to the display device 92, for example. Furthermore, for example, the device control unit 206 controls the navigation device 217 to obtain setting information from the navigation device 217.

[0057] The control unit 200 may also include Figure 2Function blocks other than those shown in the figure may include, for example, an optimal route calculation unit that calculates an optimal route to a destination based on map information acquired via the communication device 24c. Figure 2 In addition to acquiring information from the cameras and sensors shown, the control unit 200 can also acquire information from other vehicles via the communication device 25a. Furthermore, the control unit 200 receives detection signals not only from the GPS sensor 211 but also from various sensors installed in the vehicle 1. For example, the control unit 200 receives detection signals from the door opening and closing sensors and door lock mechanism sensors installed in the door sections of the vehicle 1 via the ECU configured in the door sections. This enables the control unit 200 to detect door unlocking and door opening and closing.

[0058] In this embodiment, a driving recorder 218 is installed in the vehicle 1. The driving recorder 218 can be a structure built into the vehicle 1 or a structure installed later. Figure 3 As shown, the driving recorder 218 is installed on the upper part of the front windshield and the back side of the rearview mirror. The driving recorder 218 uses the impact of the sensor 222 on the vehicle 1 above the threshold as a trigger, and stores the dynamic image data captured by the camera 221 in the storage unit 223. Figure 2 In FIG, the drive recorder 218 is shown as a structure in which the camera 221 is built in, but it can also be configured as a separate type in which the main body and the camera 221 are separated.

[0059] The control unit 220 includes a processor and memory, and centrally controls the drive recorder 218. For example, based on detection signals from the sensor 222, the control unit 220 initiates recording with the camera 221 or stores captured image data in the storage unit 223. The operations of this embodiment are achieved, for example, by the processor of the control unit 220 reading and executing a program stored in the memory. In other words, the control unit 220 and the drive recorder 218 can serve as the computer for implementing the invention.

[0060] like Figure 3 As shown, the camera 221 includes a camera 301 formed on the front surface of the driving recorder 218 and a camera 302 formed on the rear surface of the driving recorder 218. The camera 301 is a wide-angle camera capable of photographing the front of the vehicle 1, and the camera 302 is a fisheye camera capable of photographing the rear of the vehicle. As a fisheye camera, for example, a fisheye camera set horizontally can also be used. In addition, as a fisheye camera, for example, a fisheye camera with a field of view angle of 220 to 240 degrees can be used. In Figure 3 In the figure, the camera 301 and the camera 302 are shown in one each, but they can also be configured in multiples. Figure 4The image 401 is an example of an image captured by the camera 301 . Figure 4 Image 402, image 403, and image 404 are examples of images captured by camera 302. Figure 4 As shown, images 402, 403, and 404 include images of the interior of vehicle 1 and images of the exterior scenery visible through the vehicle windows. Image 402 represents an image captured from the right rear of drive recorder 218, while image 403 represents an image captured from the left rear of drive recorder 218. Furthermore, image 404 represents an image captured from the rear of drive recorder 218. Camera 221 transmits the captured image data to control unit 220 at a predetermined frame rate. Based on the transmitted image data, control unit 220 generates a moving image file in a predetermined format, such as MP4, and stores it in storage unit 223.

[0061] Sensors 222 include, for example, acceleration sensors, motion sensors, and GPS sensors. The control unit 220 acquires information such as the vehicle's 1 position, speed, acceleration, and time of day based on detection signals from the sensors 222. Based on this information, the control unit 220 controls the camera 221's image capture and the display of the captured image.

[0062] The storage unit 223 is, for example, an SD card, and is configured to store dynamic image data of a predetermined capacity. In addition, in the present embodiment, the storage unit 223 stores the mask filter generated as described later. The display unit 224 is, for example, a liquid crystal monitor, which displays various user interface screens such as a setting screen. In addition, the driving recorder 218 can also be configured to cooperate with the navigation device 217. For example, the driving recorder 218 can also be set through the setting operation on the screen displayed by the navigation device 217. The communication interface 225 can communicate with the control unit 200 of the vehicle 1 and each electrical unit. For example, the driving recorder 218 can communicate with the control unit 200 and the braking device 214 via Bluetooth (registered trademark) / WiFi (registered trademark). In addition, the driving recorder 218 can also be configured to communicate with devices other than the control unit 200 of the vehicle 1 and each electrical unit, such as a portable terminal such as a smartphone held by the driver.

[0063] like Figure 3 As shown, the indicator 219 is provided on the upper portion of the instrument panel and is configured to light up display areas corresponding to eight directions around the vehicle 1 using LEDs or the like. Figure 11 as well as Figure 12As shown, the eight directions around the vehicle 1 refer to the front (F), right front (FR), left front (FL), right (R), left (L), rear (B), right rear (BR), and left rear (BL) of the vehicle 1. The glass-like circular area on the surface of the indicator 219 is divided into the above eight directions, and a variable LED 304 of, for example, yellow / red is embedded in each area. The LED 304 emits light, and from the driver's point of view, the fan-shaped part corresponding to the LED 304 appears to be glowing. The driving recorder 218 can communicate with the indicator 219 via the communication interface 225. For example, the driving recorder 218 judges the risk object outside the vehicle 1 based on the image data captured by the camera 221, and makes the LED 304 corresponding to the direction where the risk object exists emit light in a predetermined color. The judgment of the risk object will be described later. The driving recorder 218 may also appropriately include Figure 2 Functional blocks other than those shown may include, for example, a microphone for inputting sound.

[0064] In this embodiment, risky objects outside of vehicle 1 are identified based on images captured by dashcam 218, and the identification results are reported to the driver and other passengers. Dashcam 218 captures not only the front of vehicle 1 but also the interior of vehicle 1. Therefore, the images captured by dashcam 218 include not only the interior of vehicle 1 but also the exterior scenery visible through the vehicle windows. In this embodiment, based on these characteristics of the images captured by dashcam 218, risky objects are identified not only in front of vehicle 1 but also to the sides and rear. Furthermore, in this case, the exterior scenery visible through the vehicle windows in the images captured by dashcam 218 can be appropriately identified.

[0065] Figure 5 1 is a flowchart showing the display control process of the drive recorder 218 in this embodiment. Figure 5 The processing is implemented, for example, by the processor of the control unit 220 of the drive recorder 218 reading out and executing a program stored in the memory. Figure 5 The processing is started, for example, when the driver gets on the vehicle 1 and starts driving. Alternatively, the driver may set the setting screen of the driving recorder 218 to report the risk object outside the vehicle 1 through the driving recorder 218, and the process may be started with this setting as a trigger. Figure 5 processing.

[0066] In S101, the control unit 220 determines whether the vehicle 1 is traveling. For example, the control unit 220 determines whether the vehicle 1 is traveling based on the detection signal from the sensor 222 and the image data captured by the camera 221. If it is determined that the vehicle 1 is traveling, the process proceeds to S102. If it is determined that the vehicle 1 is not traveling, the process proceeds to S109. The case where the vehicle 1 is determined not to be traveling includes, for example, a temporary stop at an intersection. In this embodiment, whether the vehicle 1 is traveling is determined in S101, but the determination can also be made based on various vehicle information conditions. For example, the determination in S101 can also be made based on whether the speed of the vehicle 1 is below a predetermined value. In S102, a mask filter is generated.

[0067] Figure 6 This is a flowchart showing the mask filter generation process in S102. In S201, the control unit 220 acquires frame image data from the camera 221. In S202, the control unit 220 determines whether a predetermined number of frame image data has been acquired. If it determines that the predetermined number of frame image data has not been acquired, the control unit 220 waits for a predetermined time to elapse in S203 and then acquires frame image data again in S201. The predetermined time in S203 corresponds to the frame rate. That is, through the processes of S201 to S203, a predetermined number of frame image data is acquired in a time-series manner at predetermined time intervals. After S202, in S204, the control unit 220 generates averaged image data based on the predetermined number of frame image data acquired in S201 to S203.

[0068] Then, in S205, the control unit 220 determines whether to terminate acquisition of the frame image data. For example, if the control unit 220 determines that sufficient averaged image data has been generated to generate the mask filter, the control unit 220 determines to terminate acquisition of the frame image data. The criteria for this determination will be described later. If it is determined in S205 that acquisition of the frame image data is not terminated, the control unit 220 waits for a predetermined time to elapse in S203, and then acquires frame image data again in S201.

[0069] Figure 8 It is a diagram for explaining the operations of S201 to S205. Figure 8, images captured by camera 302 of the left rear of drive recorder 218 are shown. Frame images 801 through 804 represent frame images captured by camera 302 in time sequence at predetermined time intervals. Frame image 801 is acquired in S201 at frame image acquisition time k-3. Frame image 802 is acquired in S201 at frame image acquisition time k-2, a predetermined time later in S203. Frame image 803 is acquired in S201 at frame image acquisition time k-1, a predetermined time later in S203. Frame image 804 is acquired in S201 at frame image acquisition time k, a predetermined time later in S203. That is, frame images 801 through 804 are sequentially acquired in S201 as the predetermined time elapses. Note that the predetermined number determined in S202 is assumed to be "2." Furthermore, each frame image is described as an RGB image, for example, and pixel values are represented as RGB values. The control unit 220 generates an averaged image by sequentially using the frame images 801 to 804. That is, the control unit 220 calculates the average value of each RGB value for each pixel of the acquired predetermined number of frame images.

[0070] For example, when frame images 801 and 802 are acquired, it is determined in S202 that a predetermined number of frame images have been acquired, and an averaged image is generated in S204 using frame images 801 and 802. Subsequently, when frame image 803 is acquired in S201 through S203, it is determined in S202 that a predetermined number of frame images have been acquired. In other words, by acquiring both the averaged image already generated in S204 and frame image 803 acquired in S201, it is determined that a predetermined number of frame images have been acquired in S202. Then, an averaged image is generated in S204 using the averaged image already generated in S204 and frame image 803.

[0071] Then, when frame image 804 is acquired in S201 after S203, it is determined in S202 that the predetermined number of frame images has been acquired. Specifically, by acquiring both the averaged image generated in S204 and frame image 804 acquired in this S201, it is determined that the predetermined number of frame images has been acquired in S202. Then, an averaged image is generated in S204 using the averaged image generated in S204 and frame image 804.

[0072] That is, in this embodiment, for frame images acquired at predetermined time intervals, a moving average of a predetermined number of pixel values is calculated. In the above case, the averaged image 805 represents the averaged image generated when the frame image 804 is acquired. Figure 6 The pixel values of the averaged image generated by the processing show the following tendency.

[0073] As shown in frame images 801 through 804, the images include an area capturing the interior of the vehicle and an area capturing the exterior scenery visible from the vehicle windows. The area capturing the interior includes, for example, images of seats and doors, while the area capturing the exterior scenery visible from the vehicle windows includes, for example, images of pedestrians and vehicles. The image of the area capturing the interior of the vehicle is considered to have little change over time, so the pixel value of each pixel in the frame images remains nearly constant over time. On the other hand, the image of the area capturing the exterior scenery visible from the vehicle windows exhibits significant variation in pixel value across frames 801 through 804, as the subject imaged changes randomly over time. Due to this tendency, the pixel values in the area capturing the interior of the vehicle within the averaged image are based on the subject imaged. Meanwhile, the pixel values in the area capturing the exterior scenery visible from the vehicle windows within the averaged image gradually approach their maximum or minimum values. For example, the addition during averaging approaches white (whitening), which is the maximum RGB pixel value of (255, 255, 255). Alternatively, the addition during averaging approaches black, which is the minimum RGB pixel value of (0, 0, 0). This embodiment describes a case where the pixel values in an area captured from a car window within the averaged image approach the maximum value.

[0074] Figure 8 The area 806 in the averaged image 805 is whitened by repeatedly generating the averaged image in step S204. Figure 8 The averaged image 805 shows a completely whitened state, but may not be completely whitened depending on the number of times of processing in S204.

[0075] As a criterion for determining the completion of acquisition of the frame image data in S205, for example, a threshold value may be set for the RGB value considered to be whitened. When the RGB value of a region where the RGB value fluctuates (e.g., region 806) exceeds the threshold value, whitening is determined, and acquisition of the frame image data is terminated. The threshold value may be determined, for example, by predetermining the number of overlaps of pixels of randomly appearing colors and its relationship to whitening.

[0076] In S206, the control unit 220 performs binarization on the averaged image generated in S204. The pixel value threshold used for binarization can be the same as or different from the threshold used in S205. For example, the threshold used for whitening in S205 can be set higher than the threshold used for binarization in S206. This configuration allows for more appropriate identification of the exterior landscape area and the vehicle interior area captured as the subject of the masking process.

[0077] Figure 8 The binarized image 807 represents the frame image binarized in S206. For example, as shown in the binarized image 807, the region corresponding to the region 806 is binarized to a value of "0", and the other regions are binarized to a value of "1". In S207, the control unit 220 generates a mask filter based on the region corresponding to the value "1" in the binarized image 807 binarized in S206, and stores it in the storage unit 223. Then, the process ends. Figure 6 processing.

[0078] Refer again Figure 5 After generating the mask filter in S102 , in S103 , the control unit 220 performs mask processing on the image data captured by the camera 221 using the mask filter generated in S102 . Figure 9 The masked image 901 represents an image that has been masked using the mask filter generated in S102. As shown in the masked image 901, the image other than the external scenery visible from the car window is in a state of being masked. Then, in S104, the control unit 220 performs image processing on the masked image 901. The image processing performed here is image processing for appropriately detecting a moving object in the later stage, such as brightness adjustment. That is, since the external scenery visible from the car window is captured in the masked image 901, depending on the environment at the time, for example, whitening may occur in a part of the image and grayscale may be lost. When grayscale is lost, the object to be detected may not be detected in the detection of moving objects. Therefore, in this embodiment, level correction and tone curve adjustment are performed on the masked image 901 to prevent grayscale loss.

[0079] In S104, for example, the following image processing may be performed. The control unit 220 detects the brightness distribution in the unmasked area (i.e., the exterior scenery visible from the vehicle window) within the masked image 901. This detection result is extracted as a histogram distribution of the number of pixels at each brightness level. If this brightness distribution is biased toward the minimum brightness value or toward the maximum brightness value, the control unit 220 eliminates the bias by distributing the brightness in a transitional manner from the minimum to the maximum value. As a result, the brightness of the unmasked area within the masked image 901 is improved, enabling appropriate detection of moving objects.

[0080] In S105, control unit 220 performs object detection based on masked image 901. Object detection uses a neural network that has been learned to detect moving objects (traffic participants), such as pedestrians and bicycles. In detected image 902, pedestrian 903 is detected as a moving object. The neural network used here is one that has been learned using an image that meets the image processing conditions of S104, such as an image with a predetermined brightness distribution.

[0081] In this embodiment, as shown in masked image 901, moving object detection is performed using image data that has been masked for the vehicle interior. Moving objects may exist within the vehicle interior, such as decorative objects hanging near windows that may sway due to vibration. If unmasked image data were used, such objects could be mistakenly detected as moving objects outside the vehicle, such as pedestrians or bicycles. However, in this embodiment, masking is performed on areas other than the exterior scenery visible through the vehicle windows based on the frame image data from camera 221. This prevents such objects within the vehicle interior from being mistakenly detected as moving objects outside the vehicle.

[0082] In S106, the control unit 220 detects the direction and distance of the object detected in S105 from the vehicle 1. For example, the control unit 220 may detect the direction and distance of the moving object based on the optical flow using a plurality of frame image data over time, the horizontal position of the object in the image, and the size of the object's detection frame.

[0083] Figure 10 1001 and 1002 are obtained while the vehicle 1 is moving. The control unit 220 detects a pedestrian 1003 from the frame image 1001 and a pedestrian 1004 from the frame image 1002. Figure 10As shown in the lower layer of FIG, pedestrians 1011 and 1012 are recognized as being located in vehicle 1. Pedestrians 1011 and 1012 correspond to pedestrians 1003 and 1004, respectively.

[0084] In S107, the control unit 220 determines which moving objects among the moving objects detected in S105 constitute risk (risk objects) based on the vehicle information of vehicle 1. For example, the control unit 220 determines risk objects based on the various behaviors of vehicle 1 and the moving objects. When vehicle 1 is traveling straight, the control unit 220 determines the risk of collision based on the moving direction and estimated speed of the moving objects identified from images captured to the right and left of vehicle 1, as well as the vehicle speed of vehicle 1. Moving objects identified as high risk are identified as risk objects. Furthermore, when vehicle 1 is turning, the area for determining risk objects is limited to the direction of the turn.

[0085] Figure 11 Is to say Figure 10 As shown in the lower layer of FIG, the control unit 220 recognizes a moving object and the vehicle 1 is turning left. In this case, the control unit 220 sets the risk extraction range 1101 as the target area for judgment in S107. Figure 11 As shown, the direction in which vehicle 1 is turning left overlaps with the direction of movement of pedestrian 1012. Furthermore, if the control unit 220 determines, based on the speed of vehicle 1 and the estimated speed of pedestrian 1012, that the risk of collision between vehicle 1 and pedestrian 1012 over time is high, it determines pedestrian 1012 as a risky object. On the other hand, the direction in which vehicle 1 is turning left overlaps with the direction of movement of pedestrian 1011. However, if the control unit 220 determines, based on the speed of vehicle 1 and the estimated speed of pedestrian 1011, that the risk of collision between vehicle 1 and pedestrian 1011 is extremely low, it does not determine pedestrian 1011 as a risky object.

[0086] Figure 12 Is to say Figure 10 As shown in the lower layer of FIG, the control unit 220 identifies a moving object and the vehicle 1 is turning right. In this case, the control unit 220 sets the risk extraction range 1201 as the target area for determination in S107. In other words, pedestrians 1011 and 1012 are not determined as risk objects. By limiting the target area for determination in S107 according to the turning direction of the vehicle 1, the processing load on the control unit 220 can be reduced.

[0087] In S108 , the control unit 220 performs display control based on the determination result in S107 .

[0088] Figure 7It is a flowchart showing the processing of the display control of S108. In S301, the control unit 220 determines the display area of the indicator 219. For example, the control unit 220 determines the area of the display object from the display area of the indicator 219 divided into eight blocks based on the direction and distance of the moving object determined as a risk object in S106 and S107. Here, the display area after being divided into eight blocks refers to the front (F), right front (FR), left front (FL), right (R), left (L), back (B), right back (BR), and left back (BL). For example, Figure 11 As shown, when it is determined that the pedestrian 1012 is a risk object, the control unit 220 specifies the display area corresponding to the left front in the display area divided into eight sections of the indicator 219 .

[0089] In S302, the control unit 220 determines the display method for the display area determined in S301. In this case, the control unit 220 determines the display method for the display area based on the behavior of the risk object and vehicle 1. For example, if the TTC (Time to Collision) between the risk object and vehicle 1 is less than a threshold, the control unit 220 determines to illuminate the red LED indicating urgency. On the other hand, if the TTC is greater than the threshold, the control unit 220 determines to illuminate the yellow LED indicating caution.

[0090] In S303, the control unit 220 controls the indicator 219 to display the display mode determined in S302 in the display area determined in S301. After S303, the process ends. Figure 7 processing.

[0091] The determination of the display area in S301 and the determination of the display method in S302 are not limited to those described above. For example, if the risk object is located forward, in S301 and S302, the control unit 220 determines the entire display area of the indicator 219, divided into eight sections, as the display area and determines to illuminate the red LED. This configuration allows for a greater degree of warning display, particularly when a risk object is located in an area identified as high risk.

[0092] Furthermore, the control unit 220 may further determine the display area in S301 and the display method in S302 based on information from the control unit 200. For example, the control unit 220 may use information on the driver's line of sight, transmitted from the in-vehicle recognition unit 203 of the control unit 200, to perform S301 and S302. For example, if the driver's line of sight coincides with the predetermined time of a risk object, the display area corresponding to the risk object may not be displayed. This configuration prevents the driver's attention from being diverted when the indicator 219 is displayed, even if the driver is already facing the risk object.

[0093] In addition, S108 and Figure 7 The display control processing of the display control can be performed not only on the indicator 219, but also on the display unit 224, the display device 216, and the navigation device 217. In this case, it is also possible to display Figure 11 、 Figure 12 The warning screen shown above can identify the direction and distance of the dangerous object.

[0094] As described above, if it is determined in S101 that the vehicle 1 is moving, a mask filter is generated in S102. On the other hand, if it is determined in S101 that the vehicle 1 is not moving, such as when temporarily stopping at an intersection, the control unit 220 determines in S109 whether there is a mask filter already stored in the storage unit 223. Then, if it is determined that there is a mask filter already stored, the control unit 220 obtains the mask filter in S110 and performs the subsequent processing. On the other hand, if it is determined that there is no mask filter already stored, the process ends. Figure 5 In this case, the processing from S101 can be repeated again.

[0095] Thus, according to the present embodiment, for example, during the period from the time the vehicle 1 reaches the destination, the images captured by the driving recorder can be used to appropriately detect moving objects (risk objects) outside the vehicle 1. In addition, based on the detection result, the display control of information related to the position of the moving object can be performed. As a result, the processing load in the control unit 200 of the vehicle 1 can be reduced. In addition, it is also possible to set whether to use the images captured by the driving recorder 218 to perform detection and display control of moving objects outside the vehicle 1 in the setting screen of the driving recorder 218. In addition, such settings can also be performed before the vehicle 1 starts moving to the destination or in the middle. In addition, at least a part of the processing of the driving recorder 218 described in the present embodiment can also be implemented by the control unit 200. For example, the control unit 220 can also perform the processing until Figure 5The image data up to and including the image processing in S104 is provided to the control unit 200, which then executes the processing in S105 to S107, thereby detecting risky objects with higher accuracy. For example, when the vehicle 1 turns, risky objects on the side opposite to the turning direction can also be detected.

[0096] In this embodiment, a configuration is described in which the image data captured by camera 221 is masked using the mask filter generated in S102. Furthermore, if the configuration is based on the amount of change in pixel values between multiple images, masking can also be performed using other configurations. For example, a configuration can also be used in which, in multiple images captured over a period of several seconds, masking is performed by masking areas where the variance of pixel value changes is lower than a predetermined value (corresponding to the area where the vehicle interior was captured). In this case, the same effects as those of this embodiment can be achieved.

[0097] <Summary of Implementation Methods>

[0098] The image processing device of the above embodiment comprises: a shooting unit (221) which shoots an image including an area corresponding to the interior of a vehicle and an area corresponding to the exterior; an acquisition unit (220, S102) which acquires a plurality of images shot by the shooting unit at predetermined time intervals; a generation unit (220, S102) which generates a mask filter for masking the area corresponding to the interior of the vehicle in the image shot by the shooting unit based on a change amount in the plurality of images acquired by the acquisition unit; and a storage unit (223) which stores the mask filter generated by the generation unit.

[0099] According to such a configuration, for example, an external risk object can be appropriately detected based on an image captured by the drive recorder 218 .

[0100] The generation unit generates the mask filter based on an averaged image obtained from the plurality of images acquired by the acquisition unit. The generation unit acquires the averaged image by performing a moving average of the pixel values of each pixel in the plurality of images acquired by the acquisition unit along a time series. The generation unit generates the mask filter by binarizing the averaged image. The image processing device further includes a processing unit (220, S103) that performs mask processing on the image captured by the capture unit using the mask filter stored in the storage unit.

[0101] With such a configuration, for example, it is possible to generate a mask filter for appropriately performing masking processing on a region corresponding to the interior of the vehicle in an image captured by the drive recorder 218 .

[0102] Furthermore, the image processed by the processing unit is subjected to image processing (S104). In addition, the image processing includes brightness adjustment.

[0103] According to such a configuration, the masked image can be used as an appropriate image for detecting an external moving object.

[0104] Furthermore, the image processing device further includes a detection unit (220, S105) that detects a moving object outside the vehicle based on the image subjected to the mask processing by the processing unit.

[0105] According to such a configuration, for example, pedestrians outside the vehicle can be appropriately detected using images captured by the drive recorder 218 .

[0106] The image processing device further includes a display control unit (220, S108) that controls a display unit (224, 219, 217) based on the detection result of the detection unit. The display control unit controls the display unit to display information related to the position of the moving object relative to the vehicle detected by the detection unit.

[0107] According to such a configuration, for example, a warning display of a moving object can be performed using an image captured by the drive recorder 218 .

[0108] In addition, the display unit is formed outside the image processing device. In addition, the display unit is an indicator (219). According to such a structure, for example, the image captured by the driving recorder 218 can be used to control the display of the indicator.

[0109] In addition, the image processing device further includes the display unit (224). According to such a configuration, for example, a warning of a moving object can be displayed on the drive recorder 218 using the image captured by the drive recorder 218.

[0110] Furthermore, the acquisition unit acquires a plurality of images captured by the imaging unit at predetermined time intervals while the vehicle is traveling.

[0111] According to such a configuration, it is possible to use frame images captured by the imaging unit at a predetermined frame rate.

[0112] In addition, the image processing device is a driving recorder (218). According to such a configuration, the operation of this embodiment can be realized on the driving recorder 218.

[0113] The present invention is not limited to the above-described embodiment, and various modifications and changes can be made within the scope of the gist of the invention.

Claims

1. An image processing device, wherein: The image processing device comprises: a photographing unit that photographs an image including an area corresponding to the interior and an area corresponding to the exterior of the vehicle; an acquisition unit configured to acquire a plurality of images captured by the capturing unit at predetermined time intervals; a generating unit that generates a mask filter for masking a region corresponding to the interior of the vehicle in the image captured by the capturing unit based on the amount of change in the plurality of images captured by the capturing unit; a storage unit configured to store the mask filter generated by the generation unit; a processing unit configured to apply the mask filter stored in the storage unit to a region corresponding to the interior of the vehicle, thereby performing mask processing on the image captured by the capturing unit; as well as A detection unit detects a moving object outside the vehicle based on a region corresponding to the exterior of the vehicle, to which the mask filter is not applied in the mask processing, included in the image subjected to the mask processing by the processing unit.

2. The image processing apparatus according to claim 1, wherein: The generation unit generates the mask filter based on an averaged image obtained from the plurality of images acquired by the acquisition unit.

3. The image processing apparatus according to claim 2, wherein: The generation unit acquires the averaged image by performing a moving average along a time series on the pixel value of each pixel of each of the plurality of images acquired by the acquisition unit.

4. The image processing device according to claim 2 or 3, wherein: The generation unit generates the mask filter by performing a binarization process on the averaged image.

5. The image processing apparatus according to claim 1, wherein: The image processing device performs image processing on the image subjected to the mask processing by the processing unit. The image processing apparatus according to claim 5 , wherein: The image processing includes brightness adjustment.

7. The image processing apparatus according to claim 1, wherein: The image processing device further includes a display control unit configured to control a display unit based on a detection result of the detection unit.

8. The image processing apparatus according to claim 7, wherein: The display control unit controls the display unit to display information related to a position of the moving object detected by the detection unit relative to the vehicle.

9. The image processing apparatus according to claim 7 or 8, wherein: The display unit is configured outside the image processing device.

10. The image processing apparatus according to claim 9, wherein: The display unit is an indicator.

11. The image processing apparatus according to claim 7 or 8, wherein: The image processing device further includes the display unit.

12. The image processing apparatus according to claim 1 or 2, wherein: The acquisition unit acquires a plurality of images captured by the imaging unit at predetermined time intervals while the vehicle is traveling.

13. The image processing apparatus according to claim 1 or 2, wherein: The image processing device is a driving recorder.

14. An image processing method, wherein: The image processing method has the following features: an acquisition step of acquiring a plurality of images captured by a capturing unit at predetermined time intervals, the capturing unit capturing images including an area corresponding to the interior and an area corresponding to the exterior of the vehicle; a generating step of generating a mask filter for masking a region corresponding to the interior of the vehicle in the image captured by the capturing unit based on the amount of change in the plurality of images captured in the acquiring step; a storing step of storing the mask filter generated in the generating step in a storage unit; a processing step of applying the mask filter stored in the storing step to a region corresponding to the interior of the vehicle, thereby performing mask processing on the image captured by the capturing unit; as well as A detection step is performed in which a moving object outside the vehicle is detected based on a region corresponding to the outside of the vehicle and to which the mask filter is not applied in the mask processing, included in the image subjected to the mask processing in the processing step.

15. A computer-readable storage medium, wherein: The computer-readable storage medium stores a program for causing a computer to function as follows: acquiring a plurality of images captured by a capturing unit at predetermined time intervals, the capturing unit capturing images including an area corresponding to the interior and an area corresponding to the exterior of the vehicle, generating a mask filter for masking a region corresponding to the interior of the vehicle in the image captured by the capturing unit based on the amount of change in the plurality of acquired images; storing the generated mask filter in a storage unit, applying the stored mask filter to a region corresponding to the interior of the vehicle, thereby performing mask processing on the image captured by the capturing unit, Detection of a moving object outside the vehicle is performed based on a region corresponding to the exterior of the vehicle, to which the mask filter is not applied in the mask processing, included in the masked image.

Images