Driving assistance device, driving assistance method, storage medium, and program

Abstract

A driving device assistance device according to the present invention is attached to a moving body, and is provided with: an imaging means that captures an image of a subject including the interior of the moving body and the exterior of the moving body; a recognition means for recognizing a person in the image using the captured image; an estimation means that estimates one or more characteristics of a person in the recognized image; a detection means for detecting, as a pedestrian, a person in the image in which each of the one or more estimated characteristics satisfies a preset reference; and an output control means for outputting, from an output device, a warning relating to the detected presence of the pedestrian, the one or more characteristics including at least one of the distance of the person from the driving assistance device, the relative movement speed of the person with respect to the moving body, and the height of the person.

Description

TECHNICAL FIELD

[0001] The present application relates to a driving assistance device, a driving assistance method, a storage medium, and a program. BACKGROUND

[0002] In recent years, a driving assistance technology has been proposed in which an image of the surroundings of the host vehicle captured by a camera provided inside the vehicle (also referred to as a moving body) is analyzed, and a caution image is displayed at a predicted position of an object requiring attention in the display device in a case where it is determined that the object in the image is an object requiring attention (Patent Literature 1). If a warning relating to an object requiring attention is notified to the driver at all times by the driving assistance technology, the driver can pay attention to an object having high danger, and can appropriately avoid danger.

[0003] PRIOR ART DOCUMENTS

[0004] PATENT LITERATURE

[0005] Patent Literature 1: International Publication No. 2019 / 159344 SUMMARY

[0006] PROBLEMS TO BE SOLVED BY THE INVENTION

[0007] However, it is sometimes desired to provide the driver with a caution for a pedestrian passing by on the side or rear of the host vehicle by means of a drive recorder provided inside the vehicle. In most cases, it is necessary to detect a pedestrian in the surroundings of the vehicle using an image of a camera configured so as to include the interior of the vehicle in the captured range. In such a case, it is necessary to reduce false detection of a person in the interior of the vehicle as a pedestrian or the like, and to detect a pedestrian outside the vehicle with high precision from the captured image.

[0008] The present application has been made in view of the above problems, and aims to achieve a technology capable of detecting a pedestrian outside a vehicle with high precision in a case where a captured image including the interior of the vehicle and the exterior of the vehicle is used.

[0009] MEANS FOR SOLVING THE PROBLEMS

[0010] According to the present application, there is provided a driving assistance device mounted on a moving body, characterized by comprising:

[0011] the driving assistance device comprises:

[0012] a capturing mechanism that captures an image including a subject inside the moving body and outside the moving body;

[0013] a recognition mechanism that recognizes a person in the image using the captured image;

[0014] an inference mechanism that infers one or more characteristics of a person in the recognized image;

[0015] a detection mechanism that detects the person in the image as a pedestrian when the inferred one or more characteristics each satisfy a predetermined criterion;

[0016] an output control mechanism that causes a warning related to the presence of the detected pedestrian to be output from an output device,

[0017] The one or more characteristics include at least any one of a distance of the person from the driving assistance device, a relative movement speed of the person with respect to the moving body, and a height of the person.

[0018] Effects of Invention

[0019] According to the present application, when using a captured image including the interior of a vehicle and the exterior of the vehicle, a pedestrian outside the vehicle can be detected with high accuracy.

[0020] Other features and advantages of the present application will become apparent from the following description, taken in conjunction with the accompanying drawings, wherein like reference numerals refer to like elements in the figures. Also, in the accompanying drawings, like reference numerals will be used to indicate like elements in the figures. BRIEF DESCRIPTION OF DRAWINGS

[0021] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description, serve to explain the principles of the application.

[0022] Figure 1 is a diagram schematically showing a configuration example of a moving body (vehicle) to which the embodiment is applied.

[0023] Figure 2A is a diagram schematically showing an appearance configuration example of a dashcam as one example of a driving assistance device to which the embodiment is applied.

[0024] Figure 2B is a diagram schematically showing an appearance configuration of an output device to which the embodiment is applied.

[0025] Figure 3 is a block diagram showing a hardware configuration example of a dashcam to which the embodiment is applied.

[0026] Figure 4 is a block diagram showing a functional configuration example of a dashcam to which the embodiment is applied.

[0027] Figure 5 is a diagram for explaining a characteristic (relative movement speed) of a person to which the embodiment is applied.

[0028] Figure 6is a graph that describes the characteristic (distance from the vehicle-mounted camera) of the person involved in the embodiment.

[0029] Figure 7 is a flowchart that shows a series of actions of the driving assistance processing involved in the embodiment. DETAILED DESCRIPTION

[0030] Hereinafter, the embodiments will be described in detail with reference to the drawings. Note that the following embodiments do not limit the technical scope of the application in all respects. Also, not all combinations of features described in the embodiments are essential to the application. Two or more features from among the plurality of features described in the embodiments can be arbitrarily combined. Also, the same reference numerals are given to the same or similar components throughout the drawings, and repetitive description is omitted.

[0031] In the present embodiment, a case where the driving assistance device is a vehicle-mounted camera that is installed in the interior of a vehicle, takes pictures of the front, side, and rear of the vehicle, and records the appearance of the vehicle while it is running on a recording medium is described as an example. However, the driving assistance device can be a device other than a vehicle-mounted camera as long as it is installed in a vehicle and can take pictures of the outside of the vehicle. Also, in the following embodiments, a case where the moving body is a vehicle is described as an example. The vehicle is typically a four-wheeled vehicle, but the present embodiment can also be applied to other types of vehicles.

[0032] REFERENCE Figure 1 Examples of the installation position of the vehicle-mounted camera are described. The vehicle-mounted camera 101 is installed in the vehicle 100. The vehicle-mounted camera 101 is installed in a position on the front window glass of the vehicle 100 that does not obstruct the driver's view and in the vicinity of the interior rearview mirror of the vehicle 100 or the interior rearview mirror of the vehicle 100. The installation position of the vehicle-mounted camera 101 is not limited to this, and can be any position that enables the vehicle-mounted camera 101 to take pictures of the outside of the vehicle 100.

[0033] The vehicle-mounted camera 101 can transmit information that needs to be output to the driver to an external output device 102 by wireless or wired means, for example, and cause the transmitted warning to be output from the output device 102. Also, in the description of the present embodiment, a case where the output device 102 is an external device of the vehicle-mounted camera 101 is described as an example. However, the vehicle-mounted camera 101 can also be equipped with an output device (i.e., a display for display and a speaker for sound output).

[0034] Example of the external appearance of the vehicle-mounted camera

[0035] REFERENCE Figure 2AAn example of the external configuration of the car navigation device 101 will be described. The car navigation device 101 has a camera 202 on a front surface 201, and captures an image of the front direction (direction of movement) of the vehicle 100. The camera 202 is, for example, a camera having a wide-angle lens. In addition, the car navigation device 101 has a camera 211 on a back surface 210, and captures an image of the side and rear of the vehicle 100. The camera 211 is, for example, a camera having a fisheye lens. The driver of the vehicle 100 is included in the image captured by the camera 211. A top surface 203 constitutes the car navigation device 101.

[0036] Example of the external configuration of the output device

[0037] Reference Signs Figure 2B An example of the external configuration of the output device 102 will be described. The output device 102 is a device for outputting information to the occupant of the vehicle 100. The output device 102 includes a display device (for example, a display or a light emitting element) for providing visual information, and an acoustic device (for example, a speaker) for providing auditory information. The output device 102 causes the display device to display information by communicating with the car navigation device 101 in a wireless or wired manner.

[0038] The output device 102 has a display device on a top surface 220. The output device 102 can perform color display on a partial region (for example, 222) within a circle 221 representing the surroundings of the vehicle 100, according to an instruction from the car navigation device 101. The region 222 that is color displayed becomes a warning to be output to the driver or the like in the case where there is a risk in the corresponding direction. The output device 102 is, for example, provided on an instrument panel in the vicinity of the driver's seat of the vehicle 100. The output device 102 can also have a half mirror, and thereby reflect light output from the top surface 220 toward the driver.

[0039] Example of the hardware configuration of the car navigation device

[0040] Reference Signs Figure 3 An example of the hardware configuration of the car navigation device 101 will be described. The car navigation device 101 has, for example, a control unit 301, a storage device 302, a camera 303, an input device 304, and a communication device 305.

[0041] The control section 301 includes, for example, a processor such as a CPU (Central Processing Unit), and controls the operation of the entire vehicle-mounted camera 101. The control section 301 can also be realized by the CPU loading and executing a program stored in the storage 302, for example. The control section 301 can include, in addition to the CPU, a GPU (Graphics Processing Unit), a dedicated circuit for image processing, machine learning, and the like.

[0042] The storage 302 is a device that stores data and programs associated with the operation of the vehicle-mounted camera 101. For example, the storage 302 can also store a program that specifies the operation of the vehicle-mounted camera 101, temporary data at the time of executing the program, an image captured by the imaging device 303, an ambient sound picked up by the microphone, and the like. The storage 302 can be realized by a memory such as a RAM (Random Access Memory) or a ROM (Read Only Memory). Further, the storage 302 can include a secondary storage device such as an SSD (Solid State Drive).

[0043] The imaging device 303 includes a camera that captures the outside in front of the vehicle 100. The imaging device 303 can also be a wide-angle camera, for example. The camera is the above-described imaging device 202 that is located on the front side (front side of the vehicle 100) of the vehicle-mounted camera 101. In addition, the imaging device 303 includes a camera that can capture the rear and the side of the vehicle 100, that is, the above-described imaging device 211. The camera can also be a camera equipped with a fisheye lens. The camera that captures the rear is disposed on the rear side (rear side of the vehicle) of the vehicle-mounted camera. In the present embodiment, a case where the same camera captures the rear and the side of the vehicle is described as an example. However, the camera that captures the rear and the camera that captures the side can be independent cameras. The camera that captures the side can also be disposed on the side surface of the vehicle-mounted camera. Part of the cameras that capture multiple directions can also be independent from the vehicle-mounted camera.

[0044] The input device 304 is a device for inputting an instruction from an occupant of the vehicle 100. The input device 304 includes, for example, a button, a touch panel. In addition, the input device 304 can also include a microphone that inputs the voice of the occupant and the voice of the outside.

[0045] The communication device 305 is a device for enabling the driving recorder 101 to communicate with the output device 102 in a wired or wireless manner. The communication device 305 sends the warning information generated by the driving assistance processing in the control unit 301 to the output device 102. The communication device 305 can communicate with other devices in a wired manner or in a wireless manner. Wireless communication can also be communication based on protocols such as short-range communication (for example, Bluetooth (registered trademark)). The communication device 305 can also be configured to be able to communicate with the control ECU of the vehicle 100 in a wired or wireless manner. The communication device 305 can also be omitted.

[0046] <Example of the functional configuration of a driving recorder>

[0047] Reference Figure 4 , an example of a functional configuration implemented by the control unit 301 of the drive recorder 101 is described. Figure 4 Each of the components shown is realized by, for example, the CPU of the control unit 301 executing a computer program.

[0048] The image acquisition unit 401 acquires a front-view image from the camera 202, which captures images of the front of the dashcam 101. Furthermore, the image acquisition unit 401 acquires a rear-view image from the camera 211, which captures images of the rear of the dashcam 101. The rear-view image includes objects outside the vehicle and inside the vehicle. For example, the rear-view image includes pedestrians walking outside the vehicle 100 and passengers inside the vehicle.

[0049] The person recognition unit 402 recognizes a person included in the image (rear-view image) acquired by the image acquisition unit 401. The person recognition unit 402 may also use a neural network that inputs the rear-view image and outputs a person recognition result. If at least a portion of a person's body is included in the image, the person recognition unit 402 may recognize that portion as a person.

[0050] The characteristic estimation unit 403 estimates one or more characteristics of a person in the image identified by the person recognition unit 402. For example, the one or more characteristics may include at least one of the person's distance from the drive recorder 101, the person's relative speed relative to the vehicle 100, and the person's height. The characteristics of a person will be described later.

[0051] The pedestrian detection unit 404 determines whether one or more characteristics of the person estimated by the characteristic estimation unit 403 satisfy respective pre-set criteria, and detects all persons satisfying the criteria as pedestrians. The criteria that the characteristics should satisfy will be described later.

[0052] The output control section 405 causes the output device 102 to output a warning in a case where the detected pedestrian approaches a position in a predetermined range from the travel trajectory of the vehicle 100. For example, the control section 301 estimates a predicted trajectory with respect to the detected pedestrian, and causes the output device 102 to output a warning indicating the direction in which the pedestrian exists in a case where a warning is needed.

[0053] Next, the characteristics of the person involved in the present embodiment and the criteria that the characteristics should satisfy will be described with reference to Figure 5 and Figure 6

[0054] Figure 5 A specific example in which the characteristic of the person indicates the relative moving speed of the person with respect to the vehicle 100 will be described. For example, assume that the person 501 and the person 502 are riding in the vehicle 100, and the person 503 is passing outside the vehicle. The characteristic estimation section 403 can estimate the predicted trajectory and the moving speed of the vehicle from the changes in the front captured image (time series) or the rear captured image (time series). In addition, the characteristic estimation section 403 can estimate the moving speed of each of the person 501, the person 502, and the person 503 using the rear captured image (time series). For example, the characteristic estimation section 403 estimates the absolute value of the relative moving speed of the person 501 and the person 502 with respect to the vehicle 100 based on the changes in the region of the person 501 and the person 502 or the predicted trajectory of the person. In addition, the characteristic estimation section 403 estimates the absolute value of the relative moving speed of the person 503 with respect to the vehicle 100 based on the changes in the region of the person 503 or the predicted trajectory of the person 503. Furthermore, the moving speed of the person can be estimated using a known method, and is not limited to the above-described method, and can be estimated, for example, by inputting the rear captured image to a neural network.

[0055] In a case where the absolute value of the relative moving speed of the person with respect to the vehicle 100 is larger than a threshold value set in advance, the pedestrian detection section 404 determines that the person in the image satisfies the criterion of the relative moving speed of the person with respect to the vehicle 100. For example, the absolute value of the relative moving speed of the person 501 and the person 502 with respect to the vehicle 100 is generally small (i.e., smaller than the threshold value). In this case, for example, the pedestrian detection section 404 determines that the person 501 and the person 502 do not satisfy the criterion of the relative moving speed, and does not detect the person 501 and the person 502 as pedestrians. On the other hand, in a case where the absolute value of the relative moving speed of the person 503 with respect to the vehicle 100 exceeds the threshold value, the pedestrian detection section 404 determines that the person 503 satisfies the criterion of the relative moving speed and detects the person 503 as a pedestrian. In this way, it is possible to remove the person having a moving speed that is inappropriate as a pedestrian from the processing target by taking into account the relative moving speed of the person, and it is possible to detect pedestrians outside the vehicle with high accuracy.

[0056] Figure 6 ​A specific example in which the characteristic of the person indicates the distance of the person from the car recorder 101. As with Figure 5 For example, it is assumed that the person 501 and the person 502 are seated in the vehicle 100, and the person 503 is passing outside the vehicle.

[0057] The characteristic estimation unit 403 estimates the distance from the car recorder 101 to the person 501, the person 502, and the person 503 using the rearward captured image (time series). The distance from the car recorder 101 to the person can be estimated by a known method, for example, by inputting the rearward captured image to a neural network.

[0058] In a case where the distance of the person 501 and the person 502 from the car recorder 101 is below a threshold value (for example, a threshold value indicating the range of 601) corresponding to the inside of the vehicle 100, the pedestrian detection unit 404 determines that the person 501 and the person 502 do not satisfy the criterion of the distance. Therefore, the person 501 and the person 502 are not detected as pedestrians.

[0059] On the other hand, in a case where the distance of the person 503 from the car recorder 101 is greater than the threshold value corresponding to the inside of the vehicle 100 and is below a threshold value (for example, a threshold value indicating the range of 602) of a prescribed distance, the pedestrian detection unit 404 determines that the person 503 satisfies the criterion of the distance of the person from the car recorder. Therefore, the person 503 is detected as a pedestrian. In this way, it is possible to remove a person located at an inappropriate distance as a pedestrian from the processing target by taking into account the distance from the car recorder 100, and to detect a pedestrian outside the vehicle with high accuracy.

[0060] The characteristic of the person can also be the height of the person. In general, the size of the face of a person has less individual difference, and thus is a useful parameter when detecting a person from an image. However, in a case where the person is far from the imaging device, the size of the face in the image becomes very small. Therefore, in the present embodiment, the height of the person is used as the characteristic of the person in order to effectively evaluate using image information even in a case where the person is far from the imaging device. However, the size of the face of the person can also be further used as the characteristic of the person.

[0061] The characteristic estimation unit 403 estimates the height of the person based on the size of the person region in the image. The height of the person can be estimated by a known method, for example, by inputting the rearward captured image to a neural network.

[0062] In a case where the height of the person is included in a range set in accordance with the distance of the person from the car navigation device 101, the person detection unit 404 determines that the person satisfies the criterion of the height. Further, in a case where the height and the like of the person are estimated from the captured image, the farther the distance of the person becomes, the more the estimation error is likely to become large. Therefore, in the present embodiment, a range of the allowable estimation error of the height is set in advance for each distance from the car navigation device 101. By so doing, in the present embodiment, in a case where the estimated height exceeds the range of the allowable error, the person is not detected as a pedestrian. In this way, it is possible to remove a person having a height in a range that is inappropriate as a pedestrian from the processing target in consideration of the height of the person, and it is possible to detect a pedestrian outside the vehicle with high accuracy.

[0063] <Sequence of actions of driving assistance processing in the car navigation device>

[0064] Next, the sequence of actions of the driving assistance processing performed in the car navigation device 101 will be described with reference to Figure 7 to Fig. 8. Further, the present processing is realized by the CPU of the control unit 301 executing the computer program stored in the storage 302.

[0065] Further, in the following description, as a processing for detecting a pedestrian, a case where a person in an image satisfying a criterion set in advance is detected as a pedestrian will be described as an example. However, the processing for detecting a pedestrian can also include other arbitrary detection processing. In the present embodiment, by not detecting a person in an image in which a characteristic does not satisfy a criterion set in advance as a pedestrian, it is possible to reduce the false detection of a pedestrian, and as a result, it is possible to detect a pedestrian outside the vehicle with high accuracy.

[0066] In S701, the image acquisition unit 401 acquires a rearward captured image capturing the rear of the car navigation device 101. In the rearward captured image, a subject including the outside of the vehicle and the interior of the vehicle is included in the image. In S702, the person recognition unit 402 recognizes a person in the image. In S703, the characteristic estimation unit 403 estimates one or more characteristics of the person in the image. As described above, the characteristics of the person include at least any one of the distance of the person from the car navigation device 101, the relative moving speed of the person with respect to the vehicle 100, and the height of the person, for example.

[0067] In S704, the pedestrian detection unit 404 determines whether the estimated characteristics satisfy a predetermined criterion. Specifically, as described with reference to Figure 4 , Figure 5 and Figure 6As explained above, it is determined whether the estimated characteristics of the person, such as the height of the person, satisfy the respective predetermined criteria. In a case where the characteristics of the pedestrian satisfy the predetermined criteria (in a case where a plurality of characteristics are handled, all of the criteria are satisfied), the pedestrian detection unit 404 causes the process to proceed to S705, and in a case where the characteristics of the pedestrian do not satisfy the predetermined criteria, the series of processes ends. In S705, the pedestrian detection unit 404 detects the person as a pedestrian. For example, in the example of the above-described Figure 5 and Figure 6 the person 503 is detected as a pedestrian.

[0068] In S706, the output control unit 405 determines whether or not it is necessary to output a warning to the driver with respect to the detected pedestrian. For example, in a case where the detected pedestrian approaches a position within a predetermined range from the travel trajectory of the vehicle 100, the output control unit 405 determines that it is necessary to output a warning. In a case where it is determined that it is necessary to output a warning, the output control unit 405 causes the process to proceed to S707, and in a case where it is determined that it is not necessary to output a warning, the series of processes ends.

[0069] In S707, the output control unit 405 causes the output device 102 to output a warning indicating the direction in which the pedestrian exists (for example, the area 222 of FIG. 22). The output control unit 405 causes the output device 102 to output a warning indicating the direction in which the pedestrian exists, for example, on the basis of the trajectory that has been obtained with respect to the detected pedestrian. Figure 2B

[0070] Further, in the above-described embodiment, in order to detect a pedestrian existing in the surroundings of the vehicle, it is determined whether or not the characteristics of the person satisfy the criteria. The control unit 302 can also perform a process of removing a registered occupant from the processing target before performing the process of estimating the characteristics of the person by the characteristic estimation unit 403. For example, the control unit 302 (by user operation or the like) registers the face of the occupant in the drive recorder 101 in advance. Then, the control unit 302 can also determine a person having a feature of the face that coincides with the feature of the face registered in advance among the persons in the recognized image. The characteristic estimation unit 403 can estimate one or more characteristics of the person other than the determined person. By so doing, it is possible to omit unnecessary estimation processing with respect to the occupant, and it is possible to achieve high-speed processing and power saving.

[0071] In addition, in the above-described embodiment, a case where the warning is output by the output device 102 is exemplified, but the warning can also be output by at least any one of a smartphone, a tablet terminal, a head-mounted display, and a wearable device terminal. These devices cause a display device to display the warning by communicating with the drive recorder 101 in a wireless or wired manner.

[0072] <Summary of Embodiments> ​

[0073] 1. The driving assistance device (e.g., 101) of the above embodiment is installed in a moving body (e.g., 100),

[0074] The driving assistance device has:

[0075] a photographing mechanism (e.g., 401) that photographs an image including a subject of an interior of the moving body and an exterior of the moving body;

[0076] a recognition mechanism (e.g., 402) that recognizes a person in the image using the photographed image;

[0077] a presumption mechanism (e.g., 403) that presumes one or more characteristics of the recognized person in the image;

[0078] a detection mechanism (e.g., 404) that detects the person in the image as a pedestrian when the presumed one or more characteristics respectively satisfy a predetermined criterion;

[0079] an output control mechanism (e.g., 405) that causes a warning related to the presence of the detected pedestrian to be output from an output device,

[0080] the one or more characteristics include at least any one of a distance of the person from the driving assistance device, a relative moving speed of the person with respect to the moving body, and a height of the person.

[0081] According to this embodiment, in a case where a photographed image including an interior of a vehicle and an exterior of the vehicle is used, a pedestrian outside the vehicle can be detected with high accuracy.

[0082] 2. In the driving assistance device in the above embodiment,

[0083] in a case where a distance of the person in the image from the driving assistance device is greater than a threshold value corresponding to an interior of the moving body and is equal to or less than a predetermined far threshold value, the detection mechanism determines that the person in the image satisfies the criterion of the distance of the person from the driving assistance device.

[0084] According to this embodiment, a person located at an inappropriate distance as a pedestrian can be removed from the processing target in consideration of the distance from the vehicle event data recorder 100, and a pedestrian outside the vehicle can be detected with high accuracy.

[0085] 3. In the driving assistance device in the above embodiment,

[0086] In a case where an absolute value of the relative movement speed of the person with respect to the moving body is greater than a threshold value set in advance, the detection mechanism determines that the person in the image satisfies the criterion of the relative movement speed of the person with respect to the moving body.

[0087] According to this embodiment, the relative movement speed of the person is taken into consideration, and a person having a movement speed inappropriate as a pedestrian is removed from the processing target, so that a pedestrian outside the vehicle can be detected with high precision.

[0088] 4. The driving assistance device in the above embodiment,

[0089] In a case where the height of the person is included in a range set in accordance with the distance of the person from the driving assistance device, the detection mechanism determines that the person in the image satisfies the criterion of the height of the person.

[0090] According to this embodiment, the height of the person is taken into consideration, and a person having a height inappropriate as a pedestrian is removed from the processing target, so that a pedestrian outside the vehicle can be detected with high precision.

[0091] 5. The driving assistance device in the above embodiment,

[0092] The estimation mechanism estimates one or more characteristics of the person other than the person having the features of the face in agreement with the features of the face of the person registered in advance, among the persons in the recognized image.

[0093] According to this embodiment, unnecessary estimation processing for the occupant can be omitted, and the processing can be speeded up and power consumption can be reduced.

[0094] 6. The driving assistance device in the above embodiment,

[0095] The output control mechanism causes a warning indicating the direction in which the detected pedestrian is present to be output from the output device.

[0096] According to this embodiment, the driver can quickly and easily grasp where the object of the warning is located around.

[0097] 7. The driving assistance device in the above embodiment,

[0098] The output device is included in the driving assistance device.

[0099] According to this embodiment, the driving assistance can be performed with an easily set-up configuration by a single driving assistance device.

[0100] 8. The driving assistance device in the above embodiment,

[0101] The output device displays the warning on a display device of the output device by communicating with the driving assist device in a wireless or wired manner.

[0102] According to this embodiment, the output device can be arranged at a position that is easily perceived by the driver.

[0103] 9. The driving assist device in the above embodiment,

[0104] The output device is at least one of a smartphone, a tablet terminal, a head-mounted display, and a wearable device terminal.

[0105] According to this embodiment, a device used alone can be used as the output device, and the manufacturing cost of the driving assist device can be reduced.

[0106] 10. The driving assist method in the above embodiment is executed by a driving assist device (e.g., 101) installed on a moving body (e.g., 100),

[0107] The driving assist method includes the following steps:

[0108] An image including a subject inside a room of the moving body and outside the moving body is captured (e.g., S701);

[0109] A person in the captured image is recognized (e.g., S702);

[0110] One or more characteristics of the recognized person in the image are estimated (e.g., S703);

[0111] The person in the image whose estimated one or more characteristics satisfy a predetermined criterion is detected as a pedestrian (e.g., S704); and

[0112] A warning related to the presence of the detected pedestrian is output from an output device (e.g., S705),

[0113] The one or more characteristics include at least one of a distance of the person from the driving assist device, a relative moving speed of the person with respect to the moving body, and a height of the person.

[0114] According to this embodiment, in a case where a captured image including a room inside a vehicle and outside the vehicle is used, a pedestrian outside the vehicle can be detected with high accuracy.

[0115] 11. The program in the above embodiment is used to cause a computer to function as each mechanism of a driving assist device (e.g., 101) installed on a moving body (e.g., 100),

[0116] The driving assistance device includes:

[0117] a photographing mechanism (e.g., 401) that photographs an image including a subject inside the moving body and outside the moving body;

[0118] a recognition mechanism (e.g., 402) that recognizes a person in the photographed image;

[0119] an estimation mechanism (e.g., 403) that estimates one or more characteristics of the recognized person in the image;

[0120] a detection mechanism (e.g., 404) that detects the person in the image as a pedestrian when the estimated one or more characteristics respectively satisfy a predetermined criterion; and

[0121] an output control mechanism (e.g., 405) that causes a warning related to the presence of the detected pedestrian to be output from an output device,

[0122] the one or more characteristics include at least any one of a distance of the person from the driving assistance device, a relative moving speed of the person with respect to the moving body, and a height of the person.

[0123] According to this embodiment, a pedestrian outside a vehicle can be detected with high accuracy when using a photographed image including the inside of the vehicle and the outside of the vehicle.

[0124] 12. The storage medium (e.g., 302) of the above embodiment stores a program that causes a computer to function as each mechanism of a driving assistance device (e.g., 101) installed in a moving body (e.g., 100),

[0125] The driving assistance device includes:

[0126] a photographing mechanism (e.g., 401) that photographs an image including a subject inside the moving body and outside the moving body;

[0127] a recognition mechanism (e.g., 402) that recognizes a person in the photographed image;

[0128] an estimation mechanism (e.g., 403) that estimates one or more characteristics of the recognized person in the image;

[0129] a detection mechanism (e.g., 404) that detects the person in the image as a pedestrian when the estimated one or more characteristics respectively satisfy a predetermined criterion; and

[0130] an output control mechanism (e.g., 405) that causes a warning related to the detected presence of the pedestrian to be output from an output device,

[0131] The one or more characteristics include at least any one of a distance of the person from the driving assist device, a relative movement speed of the person with respect to the moving body, and a height of the person.

[0132] According to this embodiment, in a case where a captured image including the inside of the vehicle cabin and the outside of the vehicle is used, a pedestrian outside of the vehicle can be detected with high precision.

[0133] The application is not limited to the above-described embodiments, and various modifications and changes can be made within the scope of the gist of the application.

[0134] Explanation of Reference Signs

[0135] 100: vehicle; 101: drive recorder; 201: control section; 202: storage device; 203: imaging device; 205: output device.

Claims

1. A driving assistance device mounted on a mobile body, characterized by comprising: a photographing mechanism that photographs an image of a subject including an indoor of the mobile body and an outside of the mobile body; an identifying mechanism that identifies a person in the image using the photographed image; an inferring mechanism that infers one or more characteristics of the person in the identified image; a detecting mechanism that detects the person in the image whose inferred one or more characteristics satisfy a predetermined criterion, respectively, as a pedestrian; and an output control mechanism that causes a warning about an existence of the detected pedestrian to be output from an output device, the one or more characteristics including at least any one of a distance of the person from the driving assistance device, a relative moving speed of the person with respect to the mobile body, and a height of the person. the detecting mechanism determines that the person in the image satisfies the criterion of the distance of the person from the driving assistance device, in a case where the distance of the person from the driving assistance device is larger than a threshold value corresponding to the indoor of the mobile body and is equal to or smaller than a predetermined far threshold value. the detecting mechanism determines that the person in the image satisfies the criterion of the relative moving speed of the person with respect to the mobile body, in a case where an absolute value of the relative moving speed of the person with respect to the mobile body is larger than a predetermined threshold value.

2. The driving assist apparatus according to claim 1, characterized by the detecting mechanism determines that the person in the image satisfies the criterion of the height of the person, in a case where the height of the person is included in a range set in accordance with the distance of the person from the driving assistance device.

3. The driving assist apparatus according to claim 1, characterized by the inferring mechanism infers one or more characteristics of a person other than a person having a feature of a face in common with a feature of a previously registered face, among the person in the identified image.

4. The driving assist apparatus according to claim 1, characterized by the output control mechanism causes a warning indicating a direction in which the detected pedestrian exists to be output from the output device.

5. The driving assist apparatus according to claim 1, characterized by the output device is included in the driving assistance device.

6. The driving assist apparatus according to claim 1, characterized by the output device displays the warning on a display device of the output device by communicating with the driving assistance device in a wireless or wired manner.

7. The driving assist apparatus according to claim 1, characterized by the output device is at least any one of a smartphone, a tablet terminal, a head-mounted display, and a wearable device terminal.

8. The driving assist apparatus according to claim 1, characterized by 10.A driving assistance method executed by a driving assistance device mounted on a mobile body, characterized by comprising:

9. The driving assist apparatus according to claim 8, characterized by a step of photographing an image of a subject including an indoor of the mobile body and an outside of the mobile body; a step of identifying a person in the image using the photographed image; a step of inferring one or more characteristics of the person in the identified image; a step of detecting the person in the image whose inferred one or more characteristics satisfy a predetermined criterion, respectively, as a pedestrian; and a step of causing a warning about an existence of the detected pedestrian to be output from an output device, the one or more characteristics including at least any one of a distance of the person from the driving assistance device, a relative moving speed of the person with respect to the mobile body, and a height of the person. ​ ​ ​ ​ 11. A program for causing a computer to function as each means of a driving assist device mounted on a mobile body, characterized by comprising: the driving assist device comprises: an imaging means that images an object including an indoor of the mobile body and an outside of the mobile body; an identifying means that identifies a person in the image using the imaged image; an estimating means that estimates one or more characteristics of the person identified in the image; a detecting means that detects the person in the image whose one or more characteristics estimated satisfy a predetermined criterion as a pedestrian; and an output control means that causes a warning about an existence of the pedestrian detected to be output from an output device, the one or more characteristics include at least any one of a distance of the person from the driving assist device, a relative moving speed of the person with respect to the mobile body, and a height of the person.

12. A storage medium storing a program for causing a computer to function as each means of a driving assist device mounted on a mobile body, characterized by comprising: the driving assist device comprises: an imaging means that images an object including an indoor of the mobile body and an outside of the mobile body; an identifying means that identifies a person in the image using the imaged image; an estimating means that estimates one or more characteristics of the person identified in the image; a detecting means that detects the person in the image whose one or more characteristics estimated satisfy a predetermined criterion as a pedestrian; and an output control means that causes a warning about an existence of the pedestrian detected to be output from an output device, the one or more characteristics include at least any one of a distance of the person from the driving assist device, a relative moving speed of the person with respect to the mobile body, and a height of the person. ​ ​

Images