Boarding support device, boarding support method, and boarding support computer program

Abstract

To provide a boarding support device for supporting a user being a visually disabled person to prevent the user from colliding with a vehicle body when boarding a vehicle.SOLUTION: A boarding support device includes: a determination section 32 for detecting a behavior when a user being a visually disabled person is boarding a vehicle 10 based on an image generated by a camera 11 which is installed to photograph a boarding / alighting position of the vehicle 10, so as to determine whether the detected behavior satisfies a collision prediction condition which predicts the collision of the vehicle 10 with the user; and a warning processing section 33 for warning the user of the collision via a notification section 12 provided in the vehicle 10 when the collision prediction condition is satisfied.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to a boarding support device, a boarding support method, and a computer program for boarding support that assist visually impaired persons who are about to board a vehicle.

Background Art

[0002] Techniques have been proposed for smoothly guiding a visually impaired user to a dispatched autonomous vehicle (see Patent Document 1). When it is determined that the user is a visually impaired person, the dispatching system disclosed in Patent Document 1 executes voice guidance processing for visually impaired persons. The voice guidance processing includes generating guidance information including the traveling direction for the user to approach the autonomous vehicle based on the image recognition result, and transmitting the guidance information to the user by voice.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When the user who is about to board a vehicle is a visually impaired person, when the user is about to board, the user may hit a part of his or her body against the vehicle body. Therefore, it is desirable to assist the user so that the user does not collide with the vehicle body when the visually impaired user boards.

[0005] Therefore, an object of the present invention is to provide a boarding support device that can assist a user so that the user does not collide with the vehicle body when the visually impaired user boards.

Means for Solving the Problems

[0006] In one aspect of the present invention, a boarding assistance device is provided. This boarding assistance device detects the behavior of a visually impaired user when attempting to board a vehicle based on an image generated by a camera provided to be able to photograph the boarding and alighting positions of the vehicle, and determines whether the detected behavior satisfies a collision prediction condition in which it is predicted that the vehicle and the user will collide. It has a warning processing unit that warns the user of a collision via a notification unit provided in the vehicle when the collision prediction condition is satisfied.

[0007] In one embodiment, this boarding assistance device further has a condition adjustment unit that makes the collision prediction condition stricter as the number of times the user boards the vehicle without a collision warning increases.

[0008] In one embodiment, this boarding assistance device further has a condition adjustment unit that makes the collision prediction condition stricter when the ratio of the number of times a collision warning is given to the number of times the user boards the vehicle in a recent predetermined period is lower than a predetermined threshold.

[0009] In one embodiment, this boarding assistance device further has a condition adjustment unit that relaxes the collision prediction condition as the volume around the vehicle increases.

[0010] In one embodiment, the determination unit predicts the trajectory of a predetermined part by detecting the predetermined part of the user from each of a plurality of images generated in time series by the camera, and determines the remaining time until the predetermined part collides with the vehicle body other than the boarding and alighting opening of the vehicle based on the predicted trajectory of the predetermined part, and determines whether the collision prediction condition is satisfied based on the remaining time.

[0011] According to another embodiment, a boarding assistance method is provided. This boarding assistance method detects the behavior of a visually impaired user when attempting to board a vehicle based on an image generated by a camera provided to be able to photograph the boarding and alighting positions of the vehicle, and determines whether the detected behavior satisfies a collision prediction condition in which it is predicted that the vehicle and the user will collide. When the collision prediction condition is satisfied, it includes warning the user of the collision via a notification unit provided in the vehicle.

[0012] According to still another embodiment, a computer program for boarding assistance is provided. This computer program for boarding assistance detects the behavior of a visually impaired user when attempting to board a vehicle based on an image generated by a camera provided to be able to photograph the boarding and alighting positions of the vehicle, and determines whether the detected behavior satisfies a collision prediction condition in which it is predicted that the vehicle and the user will collide. When the collision prediction condition is satisfied, it includes instructions for causing a processor mounted on the vehicle to warn the user of the collision via a notification unit provided in the vehicle.

Advantages of the Invention

[0013] The boarding assistance device according to the present disclosure has an effect of being able to assist a user so that the user does not collide with the vehicle body when the visually impaired user boards the vehicle.

Brief Description of the Drawings

[0014]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Embodiments for Carrying Out the Invention

[0015] Hereinafter, a boarding support device, a boarding support method, and a boarding support computer program will be described with reference to the drawings. This boarding support device detects the behavior of a user, who is a visually impaired person, when attempting to board a vehicle based on an image generated by a camera provided so as to be able to photograph the boarding and alighting positions of the vehicle. Then, when the detected behavior satisfies a collision prediction condition in which it is predicted that the vehicle and the user will collide, the boarding support device warns the user of the collision via a notification unit provided in the vehicle.

[0016] FIG. 1 is a schematic configuration diagram of a vehicle equipped with a ride assistance device according to one embodiment. The vehicle 10 equipped with the ride assistance device can be a taxi, a bus, or a private car, and can be a vehicle controlled by automatic driving. For this purpose, the vehicle 10 includes an external sensor (not shown) for obtaining information around the vehicle, a positioning device (not shown), a storage device (not shown) for storing map information, a wireless communication terminal (not shown), and an electronic control unit (ECU, not shown). Further, the vehicle 10 may include a navigation device (not shown) for determining a driving route from the current position of the vehicle 10 to the destination. The external sensor, the positioning device, the storage device, the wireless communication terminal, and the navigation device are communicably connected to the ECU via an in-vehicle network (not shown) provided in the vehicle 10 and conforming to a predetermined standard. The external sensor can be, for example, a camera provided to photograph the surroundings of the vehicle, or a distance measuring sensor such as a radar or a LiDAR sensor for detecting the distance to an object existing around the vehicle. The positioning device is configured to measure the position of the host vehicle, and includes, for example, a receiver for receiving a global positioning system (GPS) signal and an arithmetic circuit for calculating the position of the vehicle from the GPS signal. The wireless communication terminal has a wireless communication function and is configured to be communicable with other devices located outside the vehicle 10 via a wireless base station (not shown). The ECU automatically controls the vehicle 10 so that the vehicle 10 travels along the driving route received via the wireless communication terminal or obtained by the navigation device. The ECU refers to the position of the vehicle 10 measured by the positioning device, the sensor signal representing the situation around the vehicle 10 obtained by the external sensor, and the map information for automatic driving control. Further, the ECU executes control of each part of the vehicle 10, such as control of opening and closing the door provided at the boarding and alighting opening 10a of the vehicle 10. Note that the boarding and alighting opening 10a is an example of the boarding and alighting position of the vehicle 10.

[0017] Vehicle 10 further includes a camera 11 provided to be able to photograph the boarding and alighting opening 10a of vehicle 10 and its surroundings, a notification device 12 capable of giving a predetermined voice notification to a user located around the boarding and alighting opening, and a boarding assistance device 13. The camera 11 and the notification device 12 are communicably connected to the boarding assistance device 13 via an in-vehicle network. Further, the boarding assistance device 13 may be communicably connected to an ECU and a wireless communication terminal via the in-vehicle network.

[0018] The camera 11 is attached, for example, above the boarding and alighting opening 10a of vehicle 10 and directed downward so that its shooting range includes the boarding and alighting opening 10a and its surroundings, particularly the area outside the vehicle near the boarding and alighting opening 10a. The camera 11 generates an image representing the boarding and alighting opening 10a and its surroundings at a predetermined shooting cycle (for example, 1 / 30 second to 1 / 10 second). The image obtained by the camera 11 may be a color image or a gray image. Note that a plurality of cameras 11 capable of photographing the boarding and alighting opening 10a and its surroundings may be provided on the vehicle 10. Further, when the vehicle 10 has a plurality of boarding and alighting openings, a camera 11 capable of photographing each boarding and alighting opening and its surroundings may be provided for each boarding and alighting opening. Each time the camera 11 generates an image, it outputs the generated image to the boarding assistance device 13 via the in-vehicle network.

[0019] The notification device 12 is, for example, a speaker or a buzzer, and outputs a voice representing a predetermined notification, for example, a notification indicating a warning that a user's body is likely to collide with the vehicle body of vehicle 10, in accordance with a notification signal from the boarding assistance device 13. For this purpose, the notification device 12 is attached around the boarding and alighting opening 10a so that a user located around the boarding and alighting opening 10a can hear the voice output when the boarding and alighting opening 10a is open.

[0020] The boarding assistance device 13 executes boarding assistance processing for a visually impaired user based on the image generated by the camera 11.

[0021] FIG. 2 is a hardware configuration diagram of the boarding support device 13. As shown in FIG. 2, the boarding support device 13 includes a communication interface 21, a memory 22, and a processor 23. The communication interface 21, the memory 22, and the processor 23 may each be configured as separate circuits, or may be integrally configured as one integrated circuit.

[0022] The communication interface 21 has an interface circuit for connecting the boarding support device 13 to the in-vehicle network. And each time the communication interface 21 receives an image from the camera 11, it passes the received image to the processor 23. Also, when the communication interface 21 receives a notification signal to be output to the notification device 12 from the processor 23, it outputs the notification signal to the notification device 12.

[0023] The memory 22 is an example of a storage unit, and includes, for example, a volatile semiconductor memory and a non-volatile semiconductor memory. And the memory 22 stores various algorithms and various data used in the boarding support process executed by the processor 23 of the boarding support device 13. For example, the memory 22 stores personal information and identification information of the user, parameters for specifying an identifier used for detecting each part of the user, and warning thresholds that define collision prediction conditions. Further, the memory 22 temporarily stores images received from the camera 11 and various data generated during the boarding support process.

[0024] The processor 23 includes one or more CPUs (Central Processing Units) and their peripheral circuits. The processor 23 may further include other arithmetic circuits such as a logical arithmetic unit, a numerical arithmetic unit, or a graphic processing unit. And when the processor 23 is notified that the door of the boarding and alighting opening 10a has been opened from the ECU, it executes the boarding support process.

[0025] FIG. 3 is a functional block diagram of the processor 23 related to the boarding support process. The processor 23 includes a user determination unit 31, a determination unit 32, a warning processing unit 33, and a condition adjustment unit 34. Each of these units included in the processor 23 is a functional module realized by, for example, a computer program operating on the processor 23. Alternatively, each of these units included in the processor 23 may be a dedicated arithmetic circuit provided in the processor 23.

[0026] The user determination unit 31 determines whether a user who intends to board the vehicle 10 is a visually impaired person. For example, when the vehicle 10 is a taxi, the vehicle 10 may receive a pick-up instruction including the user's scheduled boarding position and personal information about the user via a wireless communication terminal. Also, when the vehicle 10 is a private car, personal information about the user may be registered in advance and stored in the memory 22. In such a case, the user determination unit 31 refers to the personal information. Then, when the personal information indicates that the user is a visually impaired person, the user determination unit 31 determines that the user who intends to board is a visually impaired person.

[0027] Further, the user determination unit 31 may determine whether the user is a visually impaired person based on the image obtained by the camera 11. In this case, the user determination unit 31 inputs the image to an identifier that has been pre-trained to detect tools (e.g., white canes) or guide dogs held by visually impaired persons. When the identifier detects such a tool or guide dog from the image, the user determination unit 31 determines that the user is a visually impaired person. As such an identifier, one based on a so-called deep neural network (DNN) is used. For example, as such an identifier, a DNN having a convolutional neural network (CNN) type architecture such as Single Shot MultiBox Detector or Faster R-CNN, or a DNN having an attention mechanism such as Vision Transformer is used. Such an identifier is pre-trained according to a predetermined learning method such as the error backpropagation method to detect an object to be detected from an image using big data such as a large number of teacher images.

[0028] In personal information, if it is not indicated that the user is a visually impaired person, or if tools and guide dogs owned by visually impaired persons are not detected from the image, the user determination unit 31 determines that the user is not a visually impaired person.

[0029] The user determination unit 31 notifies the determination unit 32 of the determination result as to whether the user is a visually impaired person.

[0030] When the user is a visually impaired person, the determination unit 32 detects the behavior when the user attempts to board the vehicle 10. Then, the determination unit 32 determines whether the detected behavior satisfies a collision prediction condition in which it is predicted that the vehicle 10 and the user will collide.

[0031] In this embodiment, the determination unit 32 inputs each of a plurality of images obtained in time series by the camera 11 into a discriminator that has been pre-trained to detect one or more parts of a user that may hit the vehicle body around the boarding and alighting opening 10a, thereby detecting those parts for each image. Note that one or more parts of the user to be detected include, for example, the head, hands, and feet. Note that since a user may intentionally grasp a part of the vehicle body with their hand to support their own body, the hand of the user may not be included in the parts to be detected. Further, as the discriminator, the determination unit 32 can use a CNN-type architecture or a DNN having an attention mechanism, similar to that used in the user determination unit 31. Alternatively, as such a discriminator, a discriminator based on another machine learning algorithm such as adaBoost or a support vector machine may be used. Alternatively, the determination unit 32 may detect the parts of the user from each image based on a detection method other than machine learning, such as template matching. Then, for each part, the determination unit 32 detects, as the trajectory of that part, the temporal position change of that part based on the position of that part detected from each image. The determination unit 32 predicts the future trajectory of each detected part based on the trajectory of that part. At this time, the determination unit 32 may apply a prediction filter such as a Kalman Filter or extrapolate the trajectory of the detected part to predict the future trajectory of that part. The determination unit 32 determines, for each part, the time to collision (TTC), which is the collision margin time until the predicted trajectory of that part overlaps with any position on the vehicle body 10 other than the boarding and alighting opening 10a, that is, the vehicle body around the boarding and alighting opening 10a. The TTC of each part represents the behavior of the user. Note that the shape of the vehicle body and the positions of each part of the vehicle body in the image coordinate system viewed from the camera 11 are stored in advance. Further, for a part whose predicted trajectory does not overlap with the position of the vehicle body, the determination unit 32 sets the TTC to ∞.

[0032] Further, the determination unit 32 may obtain the collision margin time TTC for each part by sequentially inputting a plurality of images obtained in time series from the camera 11 to an identifier that has been pre-trained to predict the collision margin time for each part of the user based on the plurality of images obtained in time series. In this case, as such an identifier, for example, a DNN having a recursive structure such as a Recurrent Neural Network (RNN) or Long Short-Term Memory (LSTM) is used.

[0033] The determination unit 32 calculates, as a collision prediction evaluation value, the sum of the reciprocals of the TTCs calculated for each part multiplied by a predetermined weight coefficient. Alternatively, the determination unit 32 may use, as the collision prediction evaluation value, the maximum value among the reciprocals of the TTCs calculated for each part. Then, when the collision prediction evaluation value is equal to or greater than the warning threshold, the determination unit 32 determines that the collision prediction condition is satisfied. On the other hand, when the collision prediction evaluation value is less than the warning threshold, the determination unit 32 determines that the collision prediction condition is not satisfied. Note that when the user can be identified, that is, when the personal information of the user is stored in advance or when the personal information of the user can be received via the wireless communication terminal, the determination unit 32 may read and use, from the memory 22, a warning threshold associated with the identification information of the user that matches the identification information included in the personal information.

[0034] When the collision prediction condition is satisfied, the determination unit 32 notifies the warning processing unit 33 of this fact and the part where the collision margin time TTC is minimized. Further, after the collision prediction condition is once satisfied, when the warning continuation condition, which is stricter than the collision prediction condition, is no longer satisfied, the determination unit 32 may notify the warning processing unit 33 of this fact. In the present embodiment, when the collision prediction evaluation value becomes lower than the stop threshold, which is a value lower than the warning threshold, the determination unit 32 determines that the warning continuation condition is no longer satisfied.

[0035] Note that when a plurality of cameras 11 are provided around the boarding and alighting opening 10a, the determination unit 32 may execute the above-described determination process for each of the plurality of cameras 11 based on the images obtained by the cameras.

[0036] Furthermore, when the user who attempts to board the vehicle 10 has a visual impairment, the determination unit 32 may detect that the user has boarded the vehicle 10 based on the image obtained by the camera 11. For example, when the positions of the respective parts of the user detected from the image indicate that they are inside the vehicle 10, and when the boarding support device 13 is notified by the ECU that the door has been closed, the determination unit 32 determines that the user has boarded. Alternatively, when the ECU detects the boarding of the user based on sensor signals obtained by other sensors provided inside the vehicle compartment, such as a camera or a seating sensor for monitoring the interior of the vehicle compartment, when a signal indicating that the user has boarded is notified from the ECU to the boarding support device 13, the determination unit 32 determines that the user has boarded.

[0037] When the determination unit 32 can identify the user, it stores in the memory 22 the date and time (hereinafter referred to as the boarding date and time) when it is determined that the user has boarded, in association with the personal information of the user.

[0038] When the warning processing unit 33 is notified by the determination unit 32 that the collision prediction condition has been satisfied, it warns the user of the collision by voice via the notification device 12. At that time, the warning processing unit 33 causes the notification device 12 to output, as voice, a message for warning of the collision. Alternatively, the warning processing unit 33 may cause the notification device 12 to output a predetermined warning sound indicating that a collision is being warned. Further, the warning processing unit 33 may cause the notification device 12 to output, as voice, a message indicating the part where the time to collision TTC is minimized (for example, "Please be careful of your head"). Further, when it is assumed that there are other passengers other than the user who attempts to board, as in the case where the vehicle 10 is a bus, the warning processing unit 33 may notify, via the notification device 12, a voice message requesting the other passengers to assist the boarding of the user. Also, when the user is identified (for example, when the vehicle 10 is a private car, or when the vehicle 10 has received a pick-up instruction including personal information), when the warning processing unit 33 issues a collision warning, it stores, in association with the personal information of the user stored in the memory 22, the date and time when the warning was implemented (hereinafter simply referred to as the warning implementation date and time).

[0039] Note that when a predetermined time has elapsed since the warning process unit 33 started the collision warning, or when a predetermined time has elapsed since it is notified from the determination unit 32 that the warning continuation condition is no longer satisfied, the warning process unit 33 stops notifying the warning. Further, when the determination unit 32 detects that the user has boarded the vehicle 10, the warning process unit 33 may stop notifying the warning.

[0040] FIG. 4 is a diagram for explaining the outline of the boarding support process according to the present embodiment. In this example, from the behavior 401 of the user 400 from time t1 to time t2, it is predicted that the user 400 will collide with the vehicle body of the vehicle 10 around the boarding and alighting opening 10a of the vehicle 10, and the collision prediction value is greater than the warning threshold value. Therefore, a collision warning is given to the user 400 via the notification device 12.

[0041] When it is possible to identify the user, the condition adjustment unit 34 adjusts the collision prediction conditions according to the user. For example, when the vehicle 10 is the user's private car, the user is assumed to be a specific individual. Also, when the vehicle 10 receives a pick-up instruction including personal information, the condition adjustment unit 34 can identify the user. In such a case, the condition adjustment unit 34 refers to the warning execution date and time associated with the personal information and the boarding date and time. Then, as the number of consecutive times without warning that the user boards the vehicle 10 without receiving a collision warning increases, the condition adjustment unit 34 changes to make the collision prediction conditions stricter. That is, the condition adjustment unit 34 increases the warning threshold value as the number of consecutive times without warning increases. As a result, it becomes difficult to give a collision warning to a user who has become accustomed to boarding the vehicle 10, thus reducing the annoyance of the user. In this way, it becomes possible to set an appropriate warning threshold value according to the user. Note that the upper limit of the warning threshold value may be set in advance. That is, when the number of consecutive times without warning continues for a certain level or more, the condition adjustment unit 34 may set the warning threshold value to the upper limit and not increase it any further.

[0042] According to a modification example, the condition adjustment unit 34 may obtain the number of rides and the number of warning implementation times in a predetermined period by counting the ride date and time and the warning implementation date and time included in the most recent predetermined period. Then, when the ratio of the number of warning implementation times to the number of rides is less than a predetermined threshold, the condition adjustment unit 34 may make the collision prediction condition stricter.

[0043] In addition, when a collision warning is issued at the time of the user's ride at a certain point in time, the condition adjustment unit 34 may change the collision prediction condition so as to relax it, that is, to decrease the warning threshold. Thereby, it is possible to prevent the situation where an appropriate warning is not given to the user due to the collision prediction condition being set too strictly. Alternatively, the condition adjustment unit 34 may change the collision prediction condition so as to relax it as the number of days elapsed since the user last rode in the vehicle 10 increases. Thereby, even if the user forgets the feeling when getting into the vehicle 10, it is possible to appropriately issue a collision warning. Note that the lower limit of the warning threshold may be set in advance. That is, when the number of days elapsed since the last ride becomes longer than a certain level, the condition adjustment unit 34 may set the warning threshold to its lower limit and not decrease it any further.

[0044] The condition adjustment unit 34 stores the modified collision prediction condition in the memory 22 in association with the personal information of the user.

[0045] FIG. 5 is a diagram showing an example of a change in a warning threshold value, which is an example of collision prediction conditions. In FIG. 5, the horizontal axis represents the number of times the user has boarded the vehicle, and the vertical axis represents the warning threshold value. And graph 500 is a graph showing the change in the warning threshold value according to the number of times of boarding. In this example, in period P1, the user repeatedly boards vehicle 10 without warning. Therefore, as the number of times of boarding increases, the condition adjustment unit 34 increases the warning threshold value. That is, the collision prediction conditions become stricter. Also, when the user boards vehicle 10 for the Na-th time, a collision warning is issued. Therefore, when the user boards vehicle 10 next after the Na-th time, the condition adjustment unit 34 decreases the warning threshold value by a predetermined amount. In the subsequent period P2, again, the user repeatedly boards vehicle 10 without warning, so the condition adjustment unit 34 increases the warning threshold value as the number of consecutive times without warning increases.

[0046] FIG. 6 is an operation flowchart of the boarding support process. Each time the user attempts to board vehicle 10, for example, when the ECU notifies that the door at the boarding and alighting opening 10a has been opened, the processor 23 executes the boarding support process according to the operation flowchart shown below.

[0047] The user determination unit 31 of the processor 23 determines whether the user attempting to board vehicle 10 is a visually impaired person (step S101). If the user is not a visually impaired person (step S101-No), the processor 23 ends the boarding support process. On the other hand, if the user is a visually impaired person (step S101-Yes), the determination unit 32 of the processor 23 detects the user's behavior and determines whether the user's behavior satisfies the collision prediction conditions (step S102). When the user's behavior satisfies the collision prediction conditions (step S102-Yes), the warning processing unit 33 of the processor 23 warns the user of a collision with the vehicle body of vehicle 10 via the notification device 12 (step S103).

[0048] After step S103, or when the user's behavior does not meet the collision prediction conditions in step S102 (step S102 - No), the determination unit 32 determines whether the user has completed boarding the vehicle 10 (step S104). If the user has not completed boarding (step S104 - No), the processor 23 repeats the processing after step S102. On the other hand, if the user has completed boarding (step S104 - Yes), the condition adjustment unit 34 of the processor 23 changes the collision prediction conditions based on, for example, the number of consecutive times without warnings (step S105). Then the processor 23 ends the boarding support process.

[0049] As described above, this boarding support device detects the behavior of a user who is a visually impaired person when attempting to board a vehicle. And when the detected behavior meets the collision prediction conditions, the boarding support device warns the user of a collision via a notification unit provided in the vehicle. Therefore, this boarding support device can assist the user so that the user does not collide with the vehicle body when the user who is a visually impaired person boards the vehicle.

[0050] According to a modification, the collision prediction conditions in the case where the vehicle 10 is the user's private car may be set more strictly than the collision prediction conditions in the case where the vehicle 10 is a bus or a taxi. When the vehicle 10 is the user's private car, since the user is accustomed to boarding the vehicle 10, it is assumed that the possibility of the user's body colliding with the vehicle body of the vehicle 10 when boarding the vehicle 10 is low. Therefore, by setting the collision prediction conditions as described above, the annoyance felt by the user is reduced.

[0051] According to another modification example, the condition adjustment unit 34 may adjust the collision prediction conditions according to the situation around the vehicle 10 when the user gets into the vehicle 10. For example, the louder the noise around the vehicle, the more difficult it is for a visually impaired user to rely on important sound information for that user. As a result, it becomes difficult for the user to grasp the positional relationship between the vehicle 10 and himself / herself. Therefore, the louder the volume around the vehicle 10 when the user gets into the vehicle 10, the more the condition adjustment unit 34 relaxes the collision prediction conditions, that is, lowers the warning threshold. Alternatively, the warning processing unit 33 may increase the volume of the warning output from the notification device 12 as the volume around the vehicle 10 when the user gets into the vehicle 10 becomes louder. Note that the condition adjustment unit 34 (or the warning processing unit 33, hereinafter only the condition adjustment unit 34 will be described representatively) may measure the volume around the vehicle 10 based on the sound collected by a microphone provided in the vehicle 10 when the door of the boarding and alighting opening 10a is open. Alternatively, the condition adjustment unit 34 may determine whether an object that may generate a large noise is shown in an image generated by the camera 11 or another camera for photographing the surroundings of the vehicle 10. And when such an object is shown, the condition adjustment unit 34 may determine that the volume around the vehicle 10 is so large as to prevent the user from getting into the vehicle 10, and relax the collision prediction conditions more than in the normal state (that is, when the volume around the vehicle 10 does not prevent the user from getting into the vehicle 10). Such an object can be, for example, an emergency vehicle, a festival float, or rain with a predetermined rainfall or more. The condition adjustment unit 34 may determine whether such an object is shown in the image by inputting the image to an identifier that has been pre-learned to detect such an object. Note that the condition adjustment unit 34 can use a CNN or a DNN having an attention mechanism as such an identifier. Alternatively, the condition adjustment unit 34 may also determine that rain with a predetermined rainfall or more is falling when the operation mode of the wiper of the vehicle 10 is in a mode of operating at a predetermined speed or more, or when the sensor value by a rainfall sensor provided in the vehicle 10 indicates that it is a predetermined rainfall or more.When the condition adjustment unit 34 determines that rain with a rainfall of a predetermined amount or more is falling, the collision prediction conditions in this case may be relaxed compared to the collision prediction conditions during normal times.

[0052] In addition, when a user who is visually impaired rides in the vehicle together with a person who can assist the user's riding, such as a family member or a friend (hereinafter referred to as a supporter), since assistance for riding can be obtained, the user may feel that the collision warning is troublesome. Therefore, when the user rides in the vehicle 10 together with a supporter, the condition adjustment unit 34 may set the collision prediction conditions more strictly than the collision prediction conditions when the user rides in the vehicle 10 alone, that is, the warning threshold may be increased. Thereby, the annoyance that the user feels about the collision warning is reduced.

[0053] When the vehicle 10 is a taxi, if the number of passengers scheduled to ride included in the pick-up instruction is two or more, the condition adjustment unit 34 may determine that the user rides in the vehicle 10 together with a supporter. Alternatively, the condition adjustment unit 34 may determine whether a person represented together with the user in an image (hereinafter referred to as a past image) obtained by the camera 11 when the user has ridden in the vehicle 10 in the past and a person represented together with the user in an image (hereinafter referred to as a current image) obtained by the camera 11 when the user is about to ride in the vehicle 10 at the current time match. When the person represented together with the user in the past image and the person represented together with the user in the current image match, the condition adjustment unit 34 may determine that the user rides in the vehicle 10 together with a supporter. In this case, the condition adjustment unit 34 may detect the persons represented in each of the past image and the current image and determine whether the detected persons match. For this purpose, the condition adjustment unit 34 inputs the past image and the current image into a discriminator that has been pre-trained to detect the persons represented in the images, thereby detecting the persons represented in each image. Note that the condition adjustment unit 34 can use a CNN or a DNN having an attention mechanism as such a discriminator. Then, the condition adjustment unit 34 may determine whether the person represented in the past image and the person represented in the current image match according to a predetermined matching determination method.

[0054] A computer program that causes a computer to execute the processing executed by the processor 23 of the above-described boarding support device 13 may be recorded and distributed on a recording medium such as an optical recording medium or a magnetic recording medium.

[0055] As described above, those skilled in the art can make various changes according to the implemented form within the scope of the present invention.

Explanation of Reference Numerals

[0056] 10 Vehicle 11 Camera 12 Notification device 13 Boarding support device 21 Communication interface 22 Memory 23 Processor 31 User determination unit 32 Determination unit 33 Warning processing unit 34 Condition adjustment unit

Claims

1. Based on an image generated by a camera provided so as to be able to photograph the boarding and alighting position of a vehicle, a determination unit that detects the behavior of a user who is visually impaired when attempting to board the vehicle, and determines whether the detected behavior satisfies a collision prediction condition in which it is predicted that the vehicle and the user will collide; A warning processing unit that warns the user of a collision via a notification unit provided in the vehicle when the collision prediction condition is satisfied; A boarding support device having the above.

2. The boarding support device according to claim 1, further comprising a condition adjustment unit that makes the collision prediction condition stricter as the number of times the user boards the vehicle without the warning being implemented increases.

3. The boarding support device according to claim 1, further comprising a condition adjustment unit that makes the collision prediction condition stricter when the ratio of the number of times the warning is implemented to the number of times the user boards the vehicle in a recent predetermined period is lower than a predetermined threshold.

4. The boarding support device according to claim 1, further comprising a condition adjustment unit that relaxes the collision prediction condition as the volume around the vehicle increases.

5. The determination unit predicts the trajectory of a predetermined part of the user by detecting the predetermined part of the user from each of a plurality of images generated in time series by the camera, obtains a remaining time until the part collides with a vehicle body other than the boarding and alighting opening of the vehicle based on the predicted trajectory of the part, and determines whether the collision prediction condition is satisfied based on the remaining time. The boarding support device according to any one of claims 1 to 4.

6. Based on an image generated by a camera provided so as to be able to photograph the boarding and alighting position of a vehicle, the behavior of a user who is visually impaired when attempting to board the vehicle is detected, It is determined whether the detected behavior satisfies a collision prediction condition in which it is predicted that the vehicle and the user will collide, When the collision prediction condition is satisfied, warning the user of the collision via a notification unit provided in the vehicle A passenger assistance method including this.

7. Based on an image generated by a camera provided so as to be able to photograph the boarding and alighting positions of a vehicle, detecting the behavior of a user who is a visually impaired person when trying to board the vehicle, Determining whether or not the detected behavior satisfies a collision prediction condition in which it is predicted that the vehicle and the user will collide, When the collision prediction condition is satisfied, warning the user of the collision via a notification unit provided in the vehicle A computer program for passenger assistance for causing a processor mounted on the vehicle to execute this.

Images