External recognition device

Abstract

Provided is an outside-world recognition device capable of accurately determining, on the basis of an image from a camera installed in a host vehicle, whether a lamp of another vehicle is blinking.　An outside-world recognition device 10 comprises: an image acquiring unit 1 for acquiring a captured image from a camera 21 installed in a host vehicle; an other-vehicle detecting unit 2 for detecting an other vehicle 31 included in at least a portion of the captured image; an estimating unit 3 for generating a three-dimensional detection frame 33 of the other vehicle and estimating an orientation 34 and a size of the other vehicle 31; a processing region setting unit 4 for setting a processing region 35 inside or at the periphery of the three-dimensional detection frame 33 on the basis of the orientation 34 and the size of the other vehicle 31; and a blinking detecting unit 5 for detecting whether a lamp of the other vehicle 31 is blinking on the basis of a time-series variation in brightness information in the processing region 35.

Description

[Technical Field]

[0001] The present invention relates to an external environment recognition device, and more particularly to an external environment recognition device that determines whether or not lighting devices of another vehicle are flashing based on an image from a camera mounted on the vehicle. [Background technology]

[0002] Autonomous driving systems of level 3 or higher must drive in a way that does not obstruct emergency vehicles. They must also recognize the intentions of other vehicles, such as their turn signals, and drive safely and smoothly.

[0003] An example of a technology for recognizing other vehicles is Patent Document 1. In Figures 1 and 5 of Patent Document 1, paragraph 0028, it is stated that "the area of ​​the other vehicle shown in the two-dimensional image is surrounded by an object that resembles a three-dimensional shape, thereby setting a target area that represents the other vehicle as if it were recognized in three-dimensional space."

[0004] Furthermore, for example, Patent Document 2 is an example of a technology relating to the determination of emergency vehicles. The abstract of Patent Document 2 states that "Images captured by CCD cameras 10a, 10b are processed by image processor 20 to calculate distance distribution information, and this distance distribution information is read into controller 30 to detect the road shape and the three-dimensional positions of multiple three-dimensional objects (vehicles, obstacles, etc.) to identify the following vehicle. The sizes of the detected following vehicles are then compared, and the vehicle type of the following vehicle is determined based on the comparison result and displayed on display 9. Furthermore, it is determined whether or not the rotating light specific to each vehicle type is on, and if the rotating light is on, the vehicle is determined to be an emergency vehicle and displayed on display 9."

[0005] Regarding the method for setting the detection frame of a rotating light, paragraph 0045 of Patent Document 2 states that "The detection frame of a rotating light for each vehicle type is set, for example, at the top of the vehicle body in the case of a large vehicle as shown in Figure 8(a), at the top of the vehicle body in the case of a standard / compact vehicle as shown in Figure 8(b), and at the middle of the vehicle body in the case of a two-wheeled vehicle as shown in Figure 8(c)." [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent Publication No. 2021-60661 [Patent Document 2] Japanese Patent Application Laid-Open No. 2002-319091 Summary of the Invention [Problem to be solved by the invention]

[0007] However, Patent Document 1 does not describe how to identify emergency vehicles. Furthermore, Patent Document 2 describes setting the position of the detection frame of the rotating light for emergency vehicle identification for each vehicle type, but only considers the case of a following vehicle photographed by a rear camera. Therefore, for example, when the other vehicle is not facing the camera, such as when photographed by a camera other than the rear camera, the position of the rotating light cannot be correctly recognized, and the position of the rotating light may fall outside the detection frame of the rotating light, or the detection frame of the rotating light may be too wide for the position of the rotating light. As a result, there is a problem that the lighting of the rotating light cannot be detected accurately.

[0008] The problem that the present invention aims to solve is to provide an external environment recognition device that can accurately determine whether or not the lights of another vehicle are flashing based on an image from a camera mounted on the vehicle. [Means for solving the problem]

[0009] In order to solve the above-mentioned problems, the external environment recognition device of the present invention includes, for example, an image acquisition unit that acquires a captured image from a camera mounted on the host vehicle; an other vehicle detection unit that detects another vehicle at least a part of which is included in the captured image; an estimation unit that generates a three-dimensional detection frame of the other vehicle and estimates the orientation and size of the other vehicle; a processing area setting unit that sets a processing area inside or around the three-dimensional detection frame based on the orientation and size of the other vehicle; and a blinking determination unit that determines whether or not a lighting device of the other vehicle is blinking based on a time-series change in luminance information in the processing area. The image acquisition unit acquires the captured images from at least one of a front camera that captures an image in front of the host vehicle and a rear camera that captures an image behind the host vehicle, and a side camera that captures an image on the side of the host vehicle, and the estimation unit generates the three-dimensional detection frame of the other vehicle in the captured image of a second camera different from the first camera at the current time, based on the size or type of the other vehicle estimated in the past using the captured image of a first camera. It is characterized by the following. [Effects of the Invention]

[0010] According to the external environment recognition device of the present invention, a processing area is set for determining whether the lighting devices of another vehicle are flashing or not based on the direction and size of the other vehicle, so that it is possible to accurately determine whether the lighting devices of another vehicle are flashing or not based on an image from a camera mounted on the vehicle. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram illustrating an external environment recognition device according to a first embodiment. [Figure 2A] 5A to 5C are diagrams illustrating the operation of the other vehicle detection unit according to the first embodiment. [Figure 2B] FIG. 4 is a diagram illustrating the operation of an estimation unit according to the first embodiment. [Figure 2C] 5A to 5C are diagrams for explaining the operation of a processing area setting unit according to the first embodiment. [Figure 3] 5A to 5C are diagrams for explaining the operation of a processing area setting unit according to the first embodiment. [Figure 4A] 5A to 5C are diagrams for explaining the operation of a blinking determination unit according to the first embodiment. [Figure 4B] 5A to 5C are diagrams for explaining the operation of a blinking determination unit according to the first embodiment. [Figure 5] 10A to 10C are diagrams illustrating the operation of the external environment recognition device according to the second embodiment. [Figure 6] FIG. 10 is a block diagram illustrating an external environment recognition device according to a third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In each drawing and each embodiment, the same or similar components are designated by the same reference numerals, and redundant explanations will be omitted. [Example]

[0013] Fig. 1 is a block diagram illustrating an external environment recognition device according to a first embodiment. Fig. 2A is a diagram illustrating the operation of a vehicle detection unit according to the first embodiment, Fig. 2B is a diagram illustrating the operation of an estimation unit according to the first embodiment, and Fig. 2C is a diagram illustrating the operation of a processing area setting unit according to the first embodiment.

[0014] The vehicle is equipped with, for example, a camera 21, an external environment recognition device 10, an alarm device 22, and a vehicle control device 23. The number of cameras 21 may be one or more. The functions of the external environment recognition device 10, the alarm device 22, and the vehicle control device 23 can be realized as programs executed by, for example, an ECU (Electronic Control Unit), which is a computer that electronically controls the vehicle. The ECU has, for example, a calculation device, a memory, a bus, an input unit, an output unit, a communication unit, and an external storage device.

[0015] The external environment recognition device 10 of the first embodiment includes an image acquisition unit 1, a vehicle detection unit 2, an estimation unit 3, a processing area setting unit 4, and a blinking determination unit 5.

[0016] The image acquisition unit 1 acquires a captured image from a camera 21 mounted on the vehicle.

[0017] The other vehicle detection unit 2 detects other vehicles 31 that are at least partially included in the captured image. For example, as shown in Fig. 2A, the other vehicle detection unit 2 generates other vehicle detection frames 32 so as to surround the other vehicles 31 in the captured image. It is desirable that the other vehicle detection unit 2 has a function of tracking the detected other vehicles 31.

[0018] 2B , the estimation unit 3 generates a three-dimensional detection frame 33 of the other vehicle 31 and estimates the orientation 34 and size of the other vehicle 31. It is desirable that the estimation unit 3 also has a function of tracking the other vehicle 31.

[0019] The three-dimensional detection frame 33 is also called a 3D bounding box, and is a three-dimensional frame that surrounds the other vehicle 31. There are various methods for generating the three-dimensional detection frame 33, such as a monocular measurement method that uses AI (artificial intelligence) to learn vehicle patterns at various angles and generate the three-dimensional detection frame 33, a compound eye measurement method that uses stereo vision to recognize a three-dimensional shape and generate the three-dimensional detection frame 33, and a method that uses other sensors, such as recognizing a three-dimensional shape using LiDAR (Light Detection And Ranging) and fusing it with camera image information to generate the three-dimensional detection frame 33.

[0020] The direction 34 of the other vehicle 31 indicates the direction ahead of the other vehicle 31, thereby enabling the driver to recognize the direction relative to the vehicle. The size of the other vehicle 31 may be the lengths in three directions of the three-dimensional detection frame (total width, total height, total length), or may be a standardized size such as vehicle class.

[0021] The estimation unit 3 may estimate the vehicle type of the other vehicle 31 in addition to the orientation 34 and size of the other vehicle 31. Examples of vehicle types include motorcycles, light vehicles, sedans, SUVs, minivans, buses, trucks, and trailer trucks.

[0022] 2C , the processing area setting unit 4 sets a processing area 35 inside or around the three-dimensional detection frame 33 based on the orientation 34 and size of the other vehicle 31. The processing area setting unit 4 may set the processing area 35 based on the type of vehicle 31 in addition to the orientation 34 and size of the other vehicle 31.

[0023] The blinking determination unit 5 determines whether the lights of the other vehicle 31 are blinking or not based on the time-series changes in the luminance information in the processing area 35. A term similar to blinking is also called flickering, and although the two are sometimes strictly distinguished, no distinction will be made in this specification. Therefore, the term blinking in this specification is used to mean a state in which a light is repeatedly turned on and off.

[0024] According to the external environment recognition device 10 of the first embodiment, the processing area 35, which is a detection frame for detecting the blinking of the lamps of the other vehicle 31, can be set at a position corresponding to the lamps of the other vehicle 31, taking into consideration the orientation 34 of the other vehicle 31 using the three-dimensional detection frame 33. This makes it possible to correctly recognize the position of the lamps and set the processing area 35 even when the other vehicle 31 is not facing the camera 21. This prevents the position of the lamps from falling outside the processing area 35 or the processing area 35 from being too large for the position of the lamps. In this way, by accurately setting the processing area 35 at a position corresponding to the lamps, the S / N ratio when measuring time-series changes in luminance information is increased, making it easier to recognize the blinking of the lamps. Furthermore, erroneous recognition due to changes in background luminance is less likely. This makes it possible to accurately detect the blinking of the lamps.

[0025] The flashing determination unit 5 can identify the type of lamp based on at least one of the flashing cycle, lighting color, lamp installation position, lamp size, and difference in flashing between the left and right. Examples of lamp types include turn signals, hazard lights, and emergency vehicle warning lights that indicate an emergency situation. Emergency vehicle warning lights are, for example, warning lights using red rotating lights, but since colors vary depending on the country or region, it is desirable to use different determination methods depending on the country or region. Furthermore, the type of lamp is not limited to rotating lights, and may be an LED or the like.

[0026] For example, turn signals have a slow blinking cycle and a set standard, the light color is orange (however, in the US, they may be used in combination with taillights), the lights are installed on the left and right ends of the vehicle, the lights are small, and the difference between the left and right lights is that one of them blinks.Hazard lights are also used as turn signals, so they are basically the same as turn signals, but unlike turn signals, the difference between the left and right lights is that the left and right lights blink simultaneously in sync.

[0027] Emergency vehicle warning lights also have a fast flashing cycle, are red (however, in other countries they may be blue or a combination of blue and red), are mounted on top of the vehicle or near the front grille (however, in the case of motorcycles, there are variations depending on the emergency vehicle, such as one on each side in the middle of the vehicle and one at the rear), are large, and flashing is synchronized even if the left and right lights are separate. Note that emergency vehicle warning lights may not always be separate.

[0028] This allows the flashing determination unit 5 to determine whether the turn signal of the other vehicle 31 is flashing, whether the hazard lights of the other vehicle 31 are flashing, and whether the other vehicle 31 is an emergency vehicle in an emergency.

[0029] This determination result is transmitted to the warning device 22 and the vehicle control device 23 shown in Fig. 1 and is used for automatic driving. For example, if the turn signal of the other vehicle 31 is flashing, the warning device 22 notifies the driver as necessary, and the vehicle control device 23 recognizes the intention indicated by the turn signal of the other vehicle 31 and automatically drives the vehicle so as to travel or stop safely and smoothly. Also, if the hazard lights of the other vehicle 31 are flashing, the warning device 22 warns the driver, and the vehicle control device 23 automatically drives the vehicle so as to travel or stop safely and smoothly. Furthermore, if the other vehicle 31 is an emergency vehicle in an emergency, the warning device 22 warns the driver, and the vehicle control device 23 automatically drives the vehicle so as to travel or stop so as not to interfere with the travel of the emergency vehicle.

[0030] 2C, it is desirable that the processing area setting unit 4 sets, as the processing area 35, a first processing area 35A at a position corresponding to the turn signal and hazard lights, and a second processing area 35B at a position corresponding to the emergency vehicle warning light indicating that an emergency vehicle is in an emergency. It is also desirable that the blinking determination unit 5 use different blinking determination methods for the first processing area 35A and the second processing area 35B. This reduces erroneous recognition and makes it possible to more accurately identify the type of lamp and determine whether the lamp is blinking.

[0031] Fig. 3 is a diagram illustrating the operation of the processing area setting unit of Example 1. Fig. 4A and Fig. 4B are diagrams illustrating the operation of the blinking determination unit of Example 1. The horizontal axis of Fig. 4A and Fig. 4B represents time t, and the vertical axis represents the average brightness value Br.

[0032] 3 shows an example in which the other vehicle 31 is a fire engine, which is an emergency vehicle. The estimation unit 3 estimates that the other vehicle 31 is large and facing forward, and the processing area setting unit 4 sets a first processing area 35A and a second processing area 35B.

[0033] When the emergency vehicle warning lights in the second processing area 35B on the upper part of the vehicle body are flashing red, the R component representing red among the RGB components of the average brightness Br is large and changes significantly and periodically, as shown in Fig. 4A. On the other hand, when the emergency vehicle warning lights are off, the R component is large but changes little and not periodically, as shown in Fig. 4B. Therefore, the flash determination unit 5 can determine whether the other vehicle 31 is an emergency vehicle in an emergency.

[0034] As described above, according to the external environment recognition device 10 of the first embodiment, the processing area 35 for determining whether or not the lighting devices of the other vehicle 31 are flashing is set based on the orientation 34 and size of the other vehicle 31, so it is possible to accurately determine whether or not the lighting devices of the other vehicle 31 are flashing based on the image from the camera 21 mounted on the host vehicle. [Example]

[0035] Example 2 is a modified example of Example 1. Example 2 differs from Example 1 in that in a multi-camera configuration having a plurality of cameras 21, different cameras 21 are coordinated with each other. In Example 2, differences from Example 1 will be mainly described, and other configurations and effects are similar to those of Example 1, so redundant description will be omitted.

[0036] Fig. 5 is a diagram illustrating the operation of the external environment recognition device of Example 2. Fig. 5 shows a rear camera image 41, a rear left side camera image 42, and a front left side camera image 43 at times t1, t2, and t3. These images were acquired by the image acquisition unit 1 from the corresponding cameras 21.

[0037] At time t1, a truck is captured in rear camera image 41 as another vehicle 31. At this time, another vehicle detection unit 2 detects the other vehicle 31, and estimation unit 3 generates a three-dimensional detection frame 33 and estimates the orientation 34 and size of the other vehicle 31. Estimation unit 3 may further estimate the type of vehicle of the other vehicle 31. Processing area setting unit 4 sets a first processing area 35A as the processing area 35 at a position corresponding to the turn signal. A second processing area 35B is also set separately as the processing area 35, but is not shown in FIG. 5.

[0038] After that, at time t2, the other vehicle 31 catches up with the host vehicle, and the other vehicle 31 is now only visible in the rear left side camera image 42. Because the side cameras are close to the other vehicle 31, the entire other vehicle 31 may not fit in the rear left side camera image 42. In this case, it may be difficult for the estimation unit 3 to generate the three-dimensional detection frame 33 or to estimate the orientation 34, size, and model of the other vehicle 31 using only the rear left side camera image 42.

[0039] Therefore, the estimation unit 3 of the second embodiment tracks the other vehicle 31 and coordinates with the different cameras 21, and generates a 3D detection frame 33 of the other vehicle 31 in the rear left camera image 42 at the current time t2 based on the size or vehicle type of the other vehicle 31 estimated using the rear camera image 41 at the past time t1. In other words, the 3D detection frame 33 is taken over between the different cameras 21. This makes it possible to generate the 3D detection frame 33 even if the other vehicle 31 does not entirely fit in the rear left camera image 42.

[0040] At this time, it is desirable that the processing area setting unit 4 also tracks the other vehicle 31 and cooperates with the different cameras 21 in the same way as the estimation unit 3, and takes over the processing area 35.

[0041] Further time passes, and at time t3, the other vehicle 31 appears in both the rear left side camera image 42 and the front left side camera image 43. At this time, as in the case of time t2, it is desirable to link the rear left side camera image 42 and the front left side camera image 43 and take over the three-dimensional detection frame 33 and the processing area 35.

[0042] The above explanations have been given on examples of linking from a rear camera to a side camera and examples of linking between side cameras, but this is not limited to these, and linking from a side camera to a front camera, linking from a front camera to a side camera, or linking from a side camera to a rear camera may also be possible.

[0043] Therefore, in the external environment recognition device 10 of the second embodiment, for example, the image acquisition unit 1 acquires captured images from at least one of a front camera that captures an image in front of the host vehicle or a rear camera that captures an image behind the host vehicle, and a side camera that captures an image to the side of the host vehicle, and the estimation unit 3 can be configured to generate a three-dimensional detection frame 33 of the other vehicle 31 in an image captured by a second camera different from the first camera at the current time, based on the size or model of the other vehicle 31 estimated in the past using an image captured by the first camera.

[0044] Furthermore, in the external environment recognition device 10 of the second embodiment, for example, the processing area setting unit 4 can be configured to set the processing area 35 in the image captured by the second camera at the current time based on the processing area 35 previously set using the image captured by the first camera.

[0045] According to the external environment recognition device 10 of the second embodiment, even if the other vehicle 31 cannot be contained in one captured image, the three-dimensional detection frame 33 and the processing area 35 can be taken over by coordinating between different cameras 21. [Example]

[0046] Example 3 is a modification of Example 1 or Example 2. Example 3 differs from Example 1 or Example 2 in that Example 3 is provided with a database that can accommodate a wide variety of emergency vehicles. Example 3 will be described focusing on the differences from Example 1 or Example 2, and other configurations and effects are similar to Example 1 or Example 2, so redundant description will be omitted.

[0047] FIG. 6 is a block diagram illustrating an external environment recognition device according to a third embodiment.

[0048] The external environment recognition device 10 of the third embodiment has a database 6. The processing area setting unit 4 sets the processing area 35 by referring to the database 6. This makes it possible to deal with a wide variety of emergency vehicles.

[0049] Here, if the database 6 can be updated by, for example, wireless communication, it will be possible to accommodate new emergency vehicles.

[0050] Furthermore, if the database 6 has multiple databases associated with regions and the processing area setting unit 4 sets the processing area 35 by referring to different databases depending on the region, it will be possible to deal with cases where the location of emergency vehicle warning lights differs from region to region.

[0051] Furthermore, the blinking determination unit 5 may also be configured to refer to the database 6. This allows the blinking determination method to be switched, for example, by changing the color of the emergency vehicle warning light depending on the region.

[0052] Although the embodiments of the present invention have been described above, the present invention is not limited to the configurations described in the embodiments, and various modifications are possible within the scope of the technical concept of the present invention. In addition, some or all of the configurations described in each embodiment may be combined and applied. [Explanation of symbols]

[0053] 1...image acquisition unit, 2...other vehicle detection unit, 3...estimation unit, 4...processing area setting unit, 5...flash determination unit, 6...database, 10...external environment recognition device, 21...camera, 22...warning device, 23...vehicle control device, 31...other vehicles, 32...other vehicle detection frame, 33...three-dimensional detection frame, 34...orientation, 35...processing area, 35A...first processing area, 35B...second processing area, 41...rear camera image, 42...rear left side camera image, 43...front left side camera image, Br...average brightness value, t, t1, t2, t3...time.

Claims

1. an image acquisition unit that acquires captured images from a camera mounted on the vehicle; an other vehicle detection unit that detects another vehicle at least a part of which is included in the captured image; an estimation unit that generates a three-dimensional detection frame of the other vehicle and estimates the orientation and size of the other vehicle; a processing area setting unit that sets a processing area inside or around the three-dimensional detection frame based on the orientation and size of the other vehicle; a blinking determination unit that determines whether the lighting device of the other vehicle is blinking based on a time-series change in luminance information in the processing area, the image acquisition unit acquires the captured images from at least one of a front camera that captures an image in front of the host vehicle and a rear camera that captures an image behind the host vehicle, and a side camera that captures an image on a side of the host vehicle; The estimation unit generates the three-dimensional detection frame of the other vehicle in the image captured by a second camera different from the first camera at the current time based on the size or vehicle type of the other vehicle estimated in the past using the image captured by the first camera. An external environment recognition device characterized by the above.

2. In claim 1, the estimation unit estimates the type of the other vehicle; The external environment recognition device, wherein the processing area setting unit sets the processing area based on the direction, size, and type of the other vehicle.

3. In claim 1, The external environment recognition device is characterized in that the flashing determination unit identifies the type of the lighting device based on at least one of the flashing period, lighting color, installation position of the lighting device, size of the lighting device, and difference in flashing between the left and right.

4. In claim 1, The processing area setting unit sets a first processing area at a position corresponding to a turn signal and a hazard light, and sets a second processing area at a position corresponding to an emergency vehicle warning light indicating that an emergency vehicle is in an emergency. An external environment recognition device.

5. In claim 4, The external environment recognition device, wherein the blinking determination unit uses different blinking determination methods for the first processing area and the second processing area.

6. In claim 1, The external environment recognition device is characterized in that the blinking determination unit determines whether or not a turn signal of the other vehicle is blinking.

7. In claim 1, The external environment recognition device is characterized in that the flashing determination unit determines whether the hazard lights of the other vehicle are flashing.

8. In claim 1, The external environment recognition device is characterized in that the blinking determination unit determines whether the other vehicle is an emergency vehicle in an emergency.

9. In claim 1, The processing area setting unit sets the processing area in the captured image of the second camera at the current time based on the processing area set in the past using the captured image of the first camera. An external environment recognition device.

10. In claim 1, It has an updatable database, The external environment recognition device, wherein the processing area setting unit sets the processing area by referring to the database.

11. In claim 1, It has a plurality of databases associated with regions, The external environment recognition device, wherein the processing area setting unit sets the processing area by referring to a database that differs depending on the region.

Images