Peripheral state recognition device and peripheral state recognition method and program

Abstract

To appropriately detect a fallen object that is difficult to detect when focusing solely on the fallen object, such as those with low height.SOLUTION: A peripheral state recognition device 15 comprises an acquisition part 3A that acquires an image in front of a vehicle 1 that is captured by a camera 12 positioned outside the portion below the line segment within the vertical plane containing the line segment connecting vehicle 1's head lamp 11 and a fallen object on the road surface in front of vehicle 1, and a detector 3B that detects the fallen object contained in the image acquired by acquisition part 3A. The detector 3B detects the fallen object contained in the image by utilizing the fact that the fallen object blocks light emitted from headlamp 11, thereby causing a shadow to form on the road surface opposite headlamp 11, which is then captured in the image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a surrounding situation recognition device, a surrounding situation recognition method, and a program. [Background technology]

[0002] Patent Document 1 describes a technology for detecting obstacles from images captured by an on-board camera. The technology described in Patent Document 1 does not utilize the fact that an obstacle blocks light emitted from a headlight, resulting in a shadow formed on the road surface on the opposite side of the obstacle being included in the captured image. In other words, the technology described in Patent Document 1 detects an obstacle by using a portion of the captured image that corresponds to the obstacle, rather than a portion that corresponds to the shadow of the obstacle. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2011-086097 Summary of the Invention [Problem to be solved by the invention]

[0004] However, for example, when the height of a fallen object (obstacle) on the road surface ahead of the vehicle is low, it may be difficult to detect the fallen object (obstacle) by using only the portion of the image captured by the on-board camera that corresponds to the fallen object (obstacle).There is a demand for a technology that can appropriately detect a fallen object (obstacle) even when the height of the fallen object (obstacle) on the road surface ahead of the vehicle is low.

[0005] In view of the above, the present disclosure aims to provide a surrounding situation recognition device, a surrounding situation recognition method, and a program that can properly detect falling objects that are difficult to detect when focusing only on the falling object, such as low-height falling objects. is. [Means for solving the problem]

[0006] (1) One aspect of the present disclosure is a surrounding situation recognition device that includes an acquisition unit that acquires an image of the front of the vehicle captured by a camera positioned at a position that is off the lower portion of a vertical plane that includes a line segment connecting the vehicle's headlights and a fallen object on the road surface in front of the vehicle, and a detection unit that detects the fallen object included in the image acquired by the acquisition unit, wherein the detection unit detects the fallen object included in the image by utilizing the fact that the fallen object blocks light emitted from the headlights, and a shadow formed on the road surface on the opposite side of the headlights, across from the fallen object, is included in the image.

[0007] (2) The surrounding situation recognition device of (1) includes an extraction unit that extracts an area in the image acquired by the acquisition unit where the falling object and the shadow of the falling object that may affect the vehicle's driving may be present, and the detection unit may detect the falling object based on the determination result of whether the shadow is included in the area extracted by the extraction unit.

[0008] (3) In the surrounding situation recognition device of (1) or (2), the detection unit detects the falling object contained in the image based on the image acquired by the acquisition unit by using a model obtained by learning using training data, which is a data set of training images of the front of the training vehicle taken by a camera of the training vehicle and labels indicating whether the training image contains a shadow of a training falling object, and at least an area of ​​the training image where the training falling object and the shadow of the training falling object that may affect the driving of the training vehicle may be present is cut out and used for learning the model.

[0009] (4) One aspect of the present disclosure is a surrounding situation recognition method comprising: an acquisition step in which a surrounding situation recognition device acquires an image of the front of the vehicle taken by a camera positioned at a position that is off the lower part of a vertical plane that includes a line segment connecting the vehicle's headlights and a fallen object on the road surface in front of the vehicle; and a detection step in which the surrounding situation recognition device detects the fallen object included in the image acquired in the acquisition step, wherein the falling object included in the image is detected by utilizing the fact that the fallen object blocks the light irradiated from the headlights, and a shadow formed on the road surface on the opposite side of the headlights, across from the fallen object, is included in the image.

[0010] (5) One aspect of the present disclosure is a program for causing a processor to execute an acquisition step of acquiring an image of the front of the vehicle taken by a camera positioned at a position that is off the lower part of a vertical plane that includes a line segment connecting the vehicle's headlights and a fallen object on the road surface in front of the vehicle, and a detection step of detecting the fallen object included in the image acquired in the acquisition step, wherein the detection step detects the fallen object included in the image by utilizing the fact that the fallen object blocks the light emitted from the headlight, and a shadow formed on the road surface on the opposite side of the headlight, separated by the fallen object, is included in the image. [Effects of the Invention]

[0011] According to the present disclosure, it is possible to appropriately detect falling objects that are difficult to detect when only focusing on the falling objects, such as low-height falling objects. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a diagram illustrating an example of a vehicle 1 to which a surrounding situation recognition device 15 according to a first embodiment is applied. [Figure 2] 2 is a diagram showing the positional relationship between headlights 11 (left headlight 11L, right headlight 11R) and a camera 12 provided on the vehicle 1 shown in FIG. [Figure 3] 5 is a flowchart illustrating an example of processing executed by a processor 153 of the surrounding situation recognition device 15 according to the first embodiment. [Figure 4] 1 is a diagram showing an example of a vehicle 1 to which a surrounding situation recognition device 15 according to a second embodiment is applied. [Figure 5] 10 is a flowchart illustrating an example of processing executed by a processor 153 of the surrounding situation recognition device 15 according to the second embodiment. [Figure 6] 10 is a diagram showing the positional relationship between headlights 11 (left headlight 11L, right headlight 11R) and a camera 12 provided on a vehicle 1 to which a surrounding situation recognition device 15 of a third embodiment is applied. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments of a surrounding situation recognition device, a surrounding situation recognition method, and a program according to the present disclosure will be described with reference to the drawings.

[0014] First Embodiment FIG. 1 is a diagram illustrating an example of a vehicle 1 to which a surrounding situation recognition device 15 according to a first embodiment is applied. FIG. 2 is a diagram illustrating the positional relationship between headlights 11 (left headlight 11L, right headlight 11R) and a camera 12 provided on the vehicle 1 illustrated in FIG. 1. Specifically, FIG. 2(A) is a diagram illustrating an example of the positional relationship between the left headlight 11L and the right headlight 11R and the camera 12. Specifically, FIG. 2(A) is a front view of the vehicle 1. FIG. 2(B) is a diagram illustrating an example of the positional relationship between the vehicle 1, a fallen object FT on a road surface RS in front of the vehicle 1, and a shadow SD formed on the road surface RS on the opposite side of the headlights 11 (left headlight 11L, right headlight 11R) across the fallen object FT. Specifically, FIG. 2(B) is a diagram illustrating the vehicle 1, the fallen object FT, and the shadow SD as seen from above. FIG. 2(C) is a diagram illustrating an example of an image IM of the front of the vehicle 1 captured by the camera 12. 1 and 2, a vehicle 1 includes headlights 11, a camera 12, an HMI (Human Machine Interface) 13, a vehicle control device 14, a steering actuator 14A, a braking actuator 14B, a driving actuator 14C, and a surrounding situation recognition device 15. The headlights 11 include a left headlight 11L and a right headlight 11R. In another example, the headlight 11 may include lamps other than the left headlight 11L and the right headlight 11R, such as fog lamps.

[0015] In the example shown in FIGS. 1 and 2, the camera 12 captures an image IM (see FIG. 2(C)) of the area ahead of the vehicle 1, and transmits the image IM data to the vehicle control device 14 and the surrounding situation recognition device 15. The camera 12 is disposed above (on the upper side of FIG. 2(A)) the headlights 11 (left headlight 11L, right headlight 11R). Therefore, the image IM captured by the camera 12 includes a shadow SD (see FIGS. 2(B) and 2(C)) formed on the road surface RS on the opposite side of the headlight 11 (the lower side of FIG. 2(B) and the upper side of FIG. 2(C)) from the headlight 11 (the left headlight 11L, the right headlight 11R) by a fallen object FT (see FIGS. 2(B) and 2(C)) on the road surface RS ahead of the vehicle 1, which blocks the light emitted from the headlight 11 (the left headlight 11L, the right headlight 11R). 1 and 2, the camera 12 is disposed at a position that is off the lower side of the line segment LLF in a vertical plane that includes the line segment LLF connecting the left headlight 11L and the fallen object FT, as shown in Fig. 2(B). Also, the camera 12 is disposed at a position that is off the lower side of the line segment LRF in a vertical plane that includes the line segment LRF connecting the right headlight 11R and the fallen object FT. The HMI 13 has a function of accepting various operations by the driver of the vehicle 1, and transmits a signal indicating the operation by the driver of the vehicle 1 to the vehicle control device 14. The vehicle control device 14 controls the headlights 11, the steering actuator 14A, the braking actuator 14B, and the drive actuator 14C based on information (data, signals) transmitted from the camera 12 and the HMI 13.

[0016] The surrounding situation recognition device 15 is configured by a microcomputer equipped with a communication interface (I / F) 151, a memory 152, and a processor 153. The communication interface 151 has an interface circuit for connecting the surrounding situation recognition device 15 to the camera 12, the HMI 13, the vehicle control device 14, etc. The memory 152 stores programs and various data used in the processing executed by the processor 153. The processor 153 has a function as an acquisition unit 3A, a function as a detection unit 3B, and a function as a processing unit 3C.

[0017] The acquisition unit 3A acquires an image IM of the area ahead of the vehicle 1 captured by the camera 12. The detection unit 3B detects a falling object FT included in the image IM acquired by the acquisition unit 3 A. For example, the detection unit 3B detects a falling object FT included in the image IM based on the image IM acquired by the acquisition unit 3 A by using a model obtained by performing learning using training data, which is a data set of training images of the front of the training vehicle captured by a camera on the training vehicle and labels indicating whether the training images contain the shadow of a training falling object. In other words, in the example shown in Figures 1 and 2, the detection unit 3B detects the falling object FT included in the image IM by taking advantage of the fact that, as shown in Figure 2(B), the falling object FT blocks the light emitted from the headlight 11 (left headlight 11L, right headlight 11R), and the shadow SD formed on the road surface RS on the opposite side of the headlight 11 (the lower side of Figure 2(B)) from the falling object FT is included in the image IM (i.e., is visible from the camera 12) as shown in Figure 2(C). Therefore, in the example shown in Figures 1 and 2, even if the height of the falling object FT is low, the falling object FT can be detected more appropriately than the technology described in Patent Document 1, which focuses only on the falling object FT contained in the image IM.

[0018] When the detection unit 3B detects a fallen object FT on the road surface RS in front of the vehicle 1, the processing unit 3C executes a process to output a warning to the HMI 13 indicating that a fallen object FT is present on the road surface RS in front of the vehicle 1.

[0019] FIG. 3 is a flowchart illustrating an example of processing executed by the processor 153 of the surrounding situation recognition device 15 according to the first embodiment. In the example shown in FIG. 3, the acquisition unit 3A acquires an image IM of the area ahead of the vehicle 1 captured by the camera 12 in step S10. In step S11, the detection unit 3B determines whether the image IM acquired in step S10 includes a shadow SD formed on the road surface RS on the opposite side of the headlight 11, across the falling object FT, by blocking the light emitted from the headlight 11 (left headlight 11L, right headlight 11R). If the result is YES, the process proceeds to step S12; if the result is NO, the process shown in FIG. 3 is terminated. In step S12, the detection unit 3B detects a falling object FT included in the image IM by utilizing the fact that the shadow SD is included in the image IM. In step S13, the processing unit 3C executes a process of causing the HMI 13 to output a warning indicating that a fallen object FT is present on the road surface RS ahead of the vehicle 1.

[0020] Second Embodiment The vehicle 1 to which the surrounding situation recognition device 15 of the second embodiment is applied is configured in the same manner as the vehicle 1 to which the surrounding situation recognition device 15 of the first embodiment described above is applied, except for the points described below.

[0021] Fig. 4 is a diagram showing an example of a vehicle 1 to which a surrounding situation recognition device 15 according to the second embodiment is applied. In the example shown in Fig. 4, a processor 153 has a function as an acquisition unit 3A, a function as a detection unit 3B, a function as a processing unit 3C, and a function as an extraction unit 3D. The extraction unit 3D extracts an area IM1 (see FIG. 2(C)) from the image IM acquired by the acquisition unit 3A, where a falling object FT and a shadow SD of the falling object FT that may affect the traveling of the vehicle 1 may exist. In other words, the extraction unit 3D does not extract an area IM2 (the area above the horizon in the example shown in FIG. 2(C)) from the image IM acquired by the acquisition unit 3A, where a falling object FT and a shadow SD of the falling object FT may not exist.

[0022] 4, the detection unit 3B detects a falling object FT included in the area IM1 extracted by the extraction unit 3D from the image IM acquired by the acquisition unit 3A. For example, the detection unit 3B detects a falling object FT included in the image IM (area IM1) based on the area IM1 extracted by the extraction unit 3D from the image IM acquired by the acquisition unit 3A, by using a model obtained by performing learning using training data, which is a data set of an area where a training object and a shadow of the training object may exist (an area corresponding to the area IM1 shown in FIG. 2(C)) in a training image of the area in front of the training vehicle captured by a camera on the training vehicle, and a label indicating whether the shadow of the training object is included in that area. That is, an area of ​​the training image captured by the camera of the training vehicle where training objects and their shadows that may affect the running of the training vehicle may exist (an area corresponding to area IM1 shown in Figure 2(C)) is cut out and used for training the model. In the example shown in Figure 4, the area IM1 to be detected as a falling object FT is smaller than the entire image IM, and therefore the computational load on the processor 153 can be reduced compared to the examples shown in Figures 1 and 2, in which the area IM1 to be detected as a falling object FT is the entire image IM. In another example, a portion of the training image captured by the camera of the training vehicle may be cut out and used for training the model, in an area where training objects and their shadows that affect the running of the training vehicle may exist (an area corresponding to area IM1 shown in Figure 2(C)) and an area where training objects and their shadows that affect the running of the training vehicle may not exist (an area corresponding to area IM2 shown in Figure 2(C)).

[0023] In the example shown in FIG. 4, the detection unit 3B detects a falling object FT based on the determination result of whether or not the shadow SD of the falling object FT is included in the area IM1 extracted by the extraction unit 3D. That is, in the example shown in Figure 4, the detection unit 3B detects the falling object FT included in area IM1 of image IM by utilizing the fact that, as shown in Figure 2(B), the falling object FT blocks the light emitted from the headlight 11 (left headlight 11L, right headlight 11R), and the shadow SD formed on the road surface RS on the opposite side of the falling object FT from the headlight 11 (the lower side of Figure 2(B)) is included in area IM1 of image IM.

[0024] FIG. 5 is a flowchart illustrating an example of processing executed by the processor 153 of the surrounding situation recognition device 15 according to the second embodiment. In the example shown in FIG. 5, the acquisition unit 3A acquires an image IM of the area ahead of the vehicle 1 captured by the camera 12 in step S20. In step S21, the extraction unit 3D extracts an area IM1 in the image IM acquired in step S20 where a fallen object FT and a shadow SD of the fallen object FT that may affect the running of the vehicle 1 may exist. In step S22, the detection unit 3B determines whether the area IM1 extracted in step S21 includes a shadow SD formed on the road surface RS on the opposite side of the headlight 11, across the falling object FT, by blocking the light emitted from the headlight 11 (left headlight 11L, right headlight 11R). If the result is YES, the process proceeds to step S23; if the result is NO, the process shown in FIG. 5 is terminated. In step S23, the detection unit 3B detects a falling object FT included in the area IM1 by utilizing the fact that the shadow SD is included in the area IM1. In step S24, the processing unit 3C executes a process of causing the HMI 13 to output a warning indicating that a fallen object FT is present on the road surface RS ahead of the vehicle 1.

[0025] Third Embodiment The vehicle 1 to which the surrounding situation recognition device 15 of the third embodiment is applied is configured in the same manner as the vehicle 1 to which the surrounding situation recognition device 15 of the first embodiment described above is applied, except for the points described below.

[0026] Fig. 6 is a diagram showing the positional relationship between the headlights 11 (left headlight 11L, right headlight 11R) provided on the vehicle 1 to which the surrounding situation recognition device 15 of the third embodiment is applied, and the camera 12. In detail, Fig. 6 is a diagram showing an example of the positional relationship between the left headlight 11L and right headlight 11R of the vehicle 1 to which the surrounding situation recognition device 15 of the third embodiment is applied, and the camera 12. In the example shown in FIG. 6, the camera 12 is disposed to the side of the headlight 11 (to the right of the left headlight 11L (left side in FIG. 6) and to the left of the right headlight 11R (right side in FIG. 6)). In detail, in the example shown in FIG. 6, as in the examples shown in FIGS. 1 and 2, the camera 12 is disposed at a position that is off the lower side of the line segment LLF in a vertical plane that includes the line segment LLF connecting the left headlight 11L and the fallen object FT. In addition, the camera 12 is disposed at a position that is off the lower side of the line segment LRF in a vertical plane that includes the line segment LRF connecting the right headlight 11R and the fallen object FT. Therefore, in the example shown in Figure 6, as in the examples shown in Figures 1 and 2, the image IM captured by the camera 12 includes a shadow SD formed on the road surface RS on the opposite side of the headlight 11, separated by the fallen object FT, due to the fallen object FT on the road surface RS in front of the vehicle 1 blocking the light emitted from the headlight 11 (left headlight 11L, right headlight 11R).

[0027] <Fourth embodiment> The vehicle 1 to which the surrounding situation recognition device 15 of the fourth embodiment is applied is configured in the same manner as the vehicle 1 to which the surrounding situation recognition device 15 of the first embodiment described above is applied, except for the points described below.

[0028] In the example shown in Figures 1 and 2 (an example of a vehicle 1 to which the surrounding situation recognition device 15 of the first embodiment is applied), the vehicle control device 14 does not have an automatic driving function that controls the steering actuator 14A, the braking actuator 14B, and the drive actuator 14C to cause the vehicle 1 to drive autonomously without the need for operation by the driver of the vehicle 1. On the other hand, in an example of a vehicle 1 to which the surrounding situation recognition device 15 of the fourth embodiment is applied, the vehicle control device 14 has an automatic driving function that controls the steering actuator 14A, the braking actuator 14B, and the drive actuator 14C to cause the vehicle 1 to travel autonomously without the need for operation by the driver of the vehicle 1. Specifically, the vehicle control device 14 generates a travel plan for the vehicle 1 to reach a destination based on, for example, map information, position information of the vehicle 1, information indicating the destination of the vehicle 1, etc. Furthermore, the vehicle control device 14 causes the vehicle 1 to travel autonomously in accordance with the travel plan. In more detail, the vehicle control device 14 causes the vehicle 1 to travel autonomously while modifying the travel plan based on, for example, an image IM of the front of the vehicle 1 captured by the camera 12, so as to avoid collisions between the vehicle 1 and surrounding vehicles, etc.

[0029] In the example shown in Figures 1 and 2 (an example of a vehicle 1 to which the surrounding situation recognition device 15 of the first embodiment is applied), as described above, when a fallen object FT on the road surface RS in front of the vehicle 1 is detected by the detection unit 3B, the processing unit 3C executes processing to output a warning to the HMI 13 indicating that a fallen object FT is present on the road surface RS in front of the vehicle 1. On the other hand, in an example of a vehicle 1 to which the surrounding situation recognition device 15 of the fourth embodiment is applied, when a fallen object FT on the road surface RS ahead of the vehicle 1 is detected by the detection unit 3B, the processing unit 3C causes the vehicle control device 14 to modify the driving plan so that the vehicle 1 can travel safely without a collision between the fallen object FT and the vehicle 1. The vehicle control device 14 modifies the driving plan in response to an instruction from the processing unit 3C, and causes the vehicle 1 to travel autonomously in accordance with the modified driving plan.

[0030] As described above, embodiments of the surrounding situation recognition device, the surrounding situation recognition method, and the program of the present disclosure have been described with reference to the drawings. However, the surrounding situation recognition device, the surrounding situation recognition method, and the program of the present disclosure are not limited to the above-described embodiments, and appropriate modifications may be made without departing from the spirit of the present disclosure. The configurations of the above-described embodiments may be combined as appropriate. In the above-described embodiments, the processing performed by the surrounding situation recognition device 15 has been described as software processing performed by executing a program. However, the processing performed by the surrounding situation recognition device 15 may be processing performed by hardware. Alternatively, the processing performed by the surrounding situation recognition device 15 may be processing that combines both software and hardware. Furthermore, the program stored in the memory 152 of the surrounding situation recognition device 15 (a program that realizes the functions of the processor 153 of the surrounding situation recognition device 15) may be recorded on a computer-readable storage medium such as a semiconductor memory, a magnetic recording medium, an optical recording medium, etc., and provided, distributed, etc. [Explanation of symbols]

[0031] 1...vehicle, 11...headlight, 11L...left headlight, 11R...right headlight, 12...camera, 13...HMI, 14...vehicle control device, 14A...steering actuator, 14B...braking actuator, 14C...driving actuator, 15...surrounding situation recognition device, 151...communication interface, 152...memory, 153...processor, 3A...acquisition unit, 3B...detection unit, 3C...processing unit, 3D...extraction unit

Claims

1. an acquisition unit that acquires an image of the area in front of the vehicle captured by a camera disposed at a position that is not below a line segment in a vertical plane that includes the line segment connecting the headlights of the vehicle and a fallen object on the road surface in front of the vehicle; a detection unit that detects the falling object included in the image acquired by the acquisition unit, The detection unit detects the falling object included in the image by utilizing the fact that the falling object blocks the light emitted from the headlight, and a shadow formed on the road surface on the opposite side of the headlight, across from the falling object, is included in the image.

2. an extraction unit that extracts an area in the image acquired by the acquisition unit where the fallen object and the shadow of the fallen object that may affect the running of the vehicle may exist, The surrounding situation recognition device according to claim 1 , wherein the detection unit detects the falling object based on a determination result of whether or not the shadow is included in the area extracted by the extraction unit.

3. the detection unit detects the fallen object included in the image based on the image acquired by the acquisition unit by using a model obtained by performing learning using training data, which is a data set of training images of the front of the training vehicle taken by a camera of the training vehicle and labels indicating whether the training images include a shadow of a training fallen object; 2. The surrounding situation recognition device according to claim 1, wherein an area of ​​the training image where the training object and a shadow of the training object that may affect the driving of the training vehicle may exist is cut out and used for training the model.

4. an acquisition step in which the surrounding situation recognition device acquires an image of the area in front of the vehicle, the image being captured by a camera disposed at a position that is not below a line segment in a vertical plane that includes the line segment connecting the headlights of the vehicle and an object lying on the road surface in front of the vehicle; a detection step in which the surrounding situation recognition device detects the fallen object included in the image acquired in the acquisition step, In the detection step, the falling object included in the image is detected by utilizing the fact that the falling object blocks the light emitted from the headlight, and a shadow formed on the road surface on the opposite side of the headlight, across from the falling object, is included in the image.

5. The processor an acquisition step of acquiring an image of the area in front of the vehicle, the image being captured by a camera disposed at a position that is not below a line segment in a vertical plane including the line segment connecting the headlights of the vehicle and a fallen object on the road surface in front of the vehicle; a detection step of detecting the falling object included in the image acquired in the acquisition step, In the detection step, the program detects the fallen object included in the image by utilizing the fact that the fallen object blocks the light emitted from the headlight, and a shadow formed on the road surface on the opposite side of the headlight, across from the fallen object, is included in the image.

Images