Moving object, determination device and determination method

Abstract

To provide a moving object, a determination device and a determination method that can determine whether or not a change has occurred in an object associated with a user riding on the moving object until the user gets off the moving object.SOLUTION: The above-mentioned problem is solved by: acquiring an image of a get-on scene in which a user gets on a moving object as a first image; acquiring an image of a get-off scene in which the user gets off the moving object as a second image; specifying a target user who will get off the moving object; and determining whether there is a change in a target object associated with the target user based on the first image and the second image including the target user.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a moving object, a determination device, and a determination method. [Background technology]

[0002] A technology is known that receives first video data of the interior of a vehicle at a first time and second video data of the interior of the vehicle at a second time, and determines whether or not there has been a change inside the vehicle between the first and second times based on the first and second video data (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2019 / 180985 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the technology described in Patent Document 1 has a problem in that, for example, if a user in a car leaves something behind in a blind spot of the sensor, it is not possible to determine whether there has been a change in an object accompanying the user between the time the user gets out of the car and the time the user gets out of the car.

[0005] The problem that the present invention aims to solve is to provide a moving body, a determination device, and a determination method that can determine whether or not there has been a change in an object associated with a user between the time the user gets on the moving body and the time the user gets off the moving body. [Means for solving the problem]

[0006] The present invention solves the above problem by acquiring an image of a boarding scene in which a user boards a moving body as a first image, acquiring an image of a disembarking scene in which the user disembarks from the moving body as a second image, identifying a target user disembarking from the moving body, and determining whether there has been a change in a target object associated with the target user based on the first image and the second image including the target user. Effect of the Invention

[0007] According to the present invention, it is possible to determine whether or not there has been a change in an object associated with a user while the user is on a moving body until the user gets off the moving body. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing a block configuration of a determination system applied to a vehicle, which is a moving body, according to this embodiment. [Diagram 2] FIG. 2 is a diagram showing an example in which each device of the determination system in this embodiment is mounted on a vehicle. [Diagram 3] FIG. 3 is a diagram showing an example of the layout of standing areas at a boarding and alighting location. [Figure 4] FIG. 4 is a diagram showing the external appearance of a side of a vehicle equipped with a user display. [Diagram 5] FIG. 5 is a flowchart showing an example of a control procedure of the determination method according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described based on the drawings. The moving body according to this embodiment determines whether or not there is a change in a target object accompanying a user until the user gets off the moving body after getting on the moving body. In this embodiment, a vehicle is used as an example of the moving body, but the moving body may be any moving body that transports a user who gets on at a boarding location to a disembarking location, such as a train or a bullet train. In addition, an example of a vehicle that can run autonomously, that is, a driverless vehicle, is used as the vehicle, but the example is not limited to this, and may be a vehicle that can run manually by a driver. In addition, the vehicle is a vehicle that can be shared. That is, the vehicle is equipped with multiple seats on which multiple users can sit, and multiple users can ride together. The target object is an object that moves with the user, and is the user's belongings and / or a companion accompanying the user. Whether or not there is a change in the target object is, for example, whether or not there is a target object left behind by the user in the moving body.

[0010] In this embodiment, the change in the user's belongings is described as an example of a change in the user's belongings, for example, a change in the user's belongings in which the user leaves behind belongings that the user had when getting on the vehicle 2. However, in this embodiment, the change is not limited to this, and may be a change in the user's belongings in which the user has more belongings, for example, a change in the user's belongings in which the user has belongings that he did not have when getting on the vehicle when he gets off the vehicle. This is an example in which the user mistakenly gets off the vehicle while taking someone else's belongings. In addition to the change in the belongings, the change may also be a change in the accompanying person. This is, for example, a case in which the user leaves behind a companion user who accompanied the user when getting on the vehicle 2 in the vehicle 2, or a case in which a companion user who did not accompany the user when getting on the vehicle is with the user when getting off the vehicle. The accompanying user is, for example, the user's child. As described above, in this embodiment, the increase or decrease in the target object associated with the user when comparing the user when getting on and the user when getting off the vehicle is regarded as the change in the target object associated with the user.

[0011] FIG. 1 is a diagram showing a block configuration of a determination system applied to a vehicle 2 according to this embodiment. A determination device 10 and a determination system 1 including the determination device 10 are mounted on the vehicle 2. The vehicle 2 is a vehicle used in a vehicle dispatch system (not shown). The vehicle dispatch system is a system that provides a service of dispatching a vehicle 2 to a user to pick up or drop off the user, that is, a so-called ride hailing service. The vehicle dispatch system drives the vehicle 2 to a boarding location of the user according to a user's reservation for use, and drives the vehicle 2 to a disembarking location of the user after the user boards the vehicle. The user is a person who uses the vehicle 2 in the ride hailing service. In the determination system 1, the determination device 10 compares the user at the time of boarding with the user at the time of disembarking, and when it is determined that there is a change in a target object associated with the user when the user disembarks from the vehicle 2, it outputs notification information including a determination result to the user.

[0012] The vehicle 2 is an autonomous vehicle, and is an automobile equipped with a navigation device and a function of automatically controlling driving control (speed control and steering control). The vehicle 2 may be manned or unmanned. The vehicle does not necessarily have to be equipped with an automatic control function, and may be equipped with a driving assistance function and driven by a human. The person driving the vehicle 2 may be a passenger in the vehicle 2, or may be a remote operator who remotely operates the vehicle 2 outside the vehicle 2. The vehicle 2 is, for example, a bus or a taxi. The vehicle 2 is an automobile that allows passengers to share the vehicle. Each user can board the vehicle 2 at a different boarding location and disembark from the vehicle 2 at a different disembarking location. That is, in the vehicle 2, a user may share a ride with a user who gets on and off at a different boarding and disembarking location.

[0013] As shown in FIG. 1, in the determination system 1, the vehicle 2 includes an in-vehicle image acquisition unit 20, an outside-vehicle image acquisition unit 30, an authentication unit 40, an inside-vehicle output unit 50, an outside-vehicle output unit 60, a storage unit 70, a determination unit 101, a management unit 102, and an in-vehicle device 200. The determination unit 101 and the management unit 102 are provided in a controller 100 of the determination device 10 mounted on the vehicle 2, as described later. The in-vehicle device 200 includes a vehicle controller 210 and various devices 220. The determination device 10 and the in-vehicle device 200 are connected by a CAN (Controller Area Network) or other in-vehicle LAN to transmit and receive information between them. The communication between the determination device 10 and the in-vehicle image acquisition unit 20, the outside-vehicle image acquisition unit 30, and the authentication unit 40 may be either wired communication or wireless communication, and may be, for example, wired communication using a USB cable. In addition, communication between the determination device 10 and the in-vehicle output unit 50 and the out-vehicle output unit 60 may be either wired communication or wireless communication. For example, wireless communication may include wireless communication using Wi-Fi (registered trademark) or Bluetooth (registered trademark).

[0014] Here, an example in which the determination system 1 in this embodiment is mounted on a vehicle 2 will be described with reference to FIG. 2. FIG. 2 is a diagram showing an example in which each device of a vehicle dispatch system is mounted on a vehicle. In FIG. 2, the positions of each device are shown in an overhead view showing the interior of the vehicle 2. Note that FIG. 2 is an example, and the positions at which each device is disposed may be positions other than those shown in FIG. 2. In FIG. 2, the determination device 10 is provided at a position on the left rear side of the vehicle 2. The determination device 10 is connected to a wireless router 11 by an Ethernet (registered trademark) connection using a communication protocol such as TCP / IP, and is communicatively connected to a first in-vehicle display 51, a second in-vehicle display 52, a following vehicle exterior display 61, and a user exterior display 62 via the wireless router 11. The wireless router 11 is connected to each display by a Wi-Fi (registered trademark) connection using a communication protocol such as TCP / IP.

[0015] The first in-vehicle display 51 and the second in-vehicle display 52 are examples of the in-vehicle output unit 50 in Fig. 1. The first in-vehicle display 51 is a display arranged to output information to a user sitting in a seat in the second row. The second in-vehicle display 52 is a display arranged to output information to a user sitting in a seat in the third row. The following vehicle exterior display 61 and the user exterior display 62 are examples of the vehicle exterior output unit 60 in Fig. 1. The following vehicle exterior display 61 is arranged behind the vehicle 2 to output information to the vehicles following the vehicle 2.

[0016] The user exterior display 62 is a display for outputting information to a user when getting on and off, and is arranged near the door 14 of the vehicle 2, which serves as the entrance for getting on and off. For example, as shown in FIG. 2, when the left door 14 corresponding to the position of the second row seats of the vehicle 2 serves as the entrance for getting on and off, the user exterior display 62 is arranged at the left door 14. The user exterior display 62 is connected to the exterior microphone 63 and the exterior speaker 64 by wireless communication such as Bluetooth (registered trademark). The exterior microphone 63 acquires the voice of the user when getting on and off. The exterior speaker 64 outputs voice information to the user when getting on and off. The exterior speaker 64 is an example of the exterior output unit 60 of FIG. 1.

[0017] The determination device 10 is also connected to the in-vehicle image acquisition unit 20, the outside-vehicle image acquisition unit 30, and the QR code (registered trademark) reader 41 through USB connection. The outside-vehicle image acquisition unit 30 is disposed on the side of the vehicle 2 on the entrance / exit side. The QR code (registered trademark) reader 41 is an example of the authentication unit 40 in FIG. 1, and reads the QR code (registered trademark) presented by the user. The determination device 10 is also connected to the vehicle system 12 that controls the vehicle 2 through a CAN. The determination device 10 may also be connected to an in-vehicle display 53 for the administrator through a wireless router 11. The in-vehicle display 53 for the administrator is an example of the in-vehicle output unit 50, and is disposed in the passenger seat of the vehicle 2. The display for the administrator is not limited to an in-vehicle display, and may be disposed outside the vehicle.

[0018] Each device will be described below. The interior image acquisition unit 20 is a camera using an imaging element such as a CCD, and captures an image of the interior of the vehicle 2. For example, the interior of the vehicle including the user is captured. The number and location of the interior image acquisition units 20 are not particularly limited, but for example, one wide-angle interior image acquisition unit 20 is installed near the interior ceiling of the vehicle 2. The interior image acquisition unit 20 captures images so that images of the driver's seat and passenger seat in the front seats, and each seat row in the rear seats are included in the captured image area. The interior image acquisition unit 20 may also capture images of the interior of the vehicle including the entrance and exit of the vehicle 2.

[0019] In this embodiment, the vehicle interior image acquisition unit 20 acquires an image of the vehicle interior including the user when the user gets on and off the vehicle 2. Specifically, the vehicle interior image acquisition unit 20 captures an image of the vehicle interior including the user from the time when the vehicle 2 arrives at the boarding and alighting location until the vehicle 2 departs from the boarding and alighting location. The vehicle interior image acquisition unit 20 may acquire images at a certain period from the time of arrival to the time of departure, or may acquire images at a predetermined timing. The acquired image may be one image, or may be two or more images (videos) that are continuous in time series. The acquired image includes the user at the time of boarding or alighting, that is, the user performing the boarding or alighting action in the vehicle interior. The vehicle interior image acquisition unit 20 may also capture an image of the user after boarding or before alighting in the vehicle interior. For example, the vehicle interior image acquisition unit 20 captures an image of the user in the vehicle interior from the time when the user gets on the vehicle 2 at the boarding location until the vehicle 2 departs from the boarding location. For example, during the period from when the vehicle 2 arrives at the drop-off location until the user gets off the vehicle, the vehicle interior image acquisition unit 20 captures an image of the user inside the vehicle. The image captured by the vehicle interior image acquisition unit 20 is output to the determination device 10.

[0020] The outside-vehicle image acquisition unit 30 is a camera using an imaging element such as a CCD, and captures an image of the surrounding environment outside the cabin of the vehicle 2. For example, the surrounding environment including the user outside the cabin is captured. The number and location of the outside-vehicle image acquisition unit 30 are not particularly limited, but for example, the outside-vehicle image acquisition unit 30 is installed at a position where it can capture an image of the user when the user gets on and off, such as at or near the door of the boarding / alighting entrance. In addition, the outside-vehicle image acquisition unit 30 is arranged on the side of the vehicle 2 on the boarding / alighting entrance side, and captures an image of the user from the front and / or side when getting on or off the vehicle.

[0021] In this embodiment, the vehicle exterior image acquisition unit 30 acquires an image of the outside of the vehicle cabin including the user when the user gets on and off the vehicle 2. Specifically, the vehicle exterior image acquisition unit 30 captures an image of the outside of the vehicle cabin including the user from the time when the vehicle 2 arrives at the boarding and alighting location until the vehicle 2 departs from the boarding and alighting location. The vehicle exterior image acquisition unit 30 may acquire images at a certain period from the time of arrival to the time of departure, or may acquire images at a predetermined timing. The acquired image may be one image, or may be two or more images (videos) that are continuous in time series. The acquired image includes the user at the time of boarding or alighting, that is, the user performing the boarding or alighting operation outside the vehicle cabin. In addition, the vehicle exterior image acquisition unit 30 may capture an image of the user before boarding or the user after alighting outside the vehicle cabin. For example, the vehicle exterior image acquisition unit 30 captures an image of the user outside the vehicle cabin from the time when the vehicle 2 arrives at the boarding location until the user starts boarding. For example, the vehicle exterior image acquisition unit 30 captures an image of a user when the user authenticates boarding based on a usage reservation with the authentication unit 40. The vehicle exterior image acquisition unit 30 captures an image of the user outside the vehicle cabin from the time the user gets off the vehicle 2 at the alighting location until the vehicle 2 departs from the alighting location. For example, the vehicle exterior image acquisition unit 30 captures an image of the user when the user inputs user identification information for a usage fee settlement process or the like when getting off the vehicle. The image captured by the vehicle exterior image acquisition unit 30 is output to the determination device 10.

[0022] In the following description, the vehicle exterior image acquisition unit 30 will be described as an example of the "image acquisition unit" in the claims. The acquisition unit acquires images of the user and the user's belongings. The "image acquisition unit" is not limited to the vehicle exterior image acquisition unit 30, and the vehicle interior image acquisition unit 20 may also be the "image acquisition unit." The image acquisition unit that acquires the first image described below and the image acquisition unit that acquires the second image described below may be the same camera, or may be separate cameras.

[0023] In a boarding scene where the user boards the vehicle 2, the vehicle exterior image acquisition unit 30 acquires a first image. The first image is an image of a boarding scene, and is an image of the user and the user's belongings in the boarding scene. The boarding scene includes the time when the user boards the vehicle 2 at the boarding location. For example, the first image includes the user when he / she is performing a boarding action to board the vehicle 2 (during boarding). The boarding action is a series of actions from the user opening the door at the boarding entrance of the vehicle 2, getting into the vehicle interior, and sitting in a seat. The boarding action may include a series of actions from the user performing boarding authentication to unlocking the door and opening the door. In addition, the boarding scene includes the time before the user boards the vehicle 2 at the boarding location and the time after the user boards the vehicle 2 at the boarding location. That is, the first image includes the user before and after boarding at the boarding location. For example, the first image may be an image of the user outside the vehicle interior before the user starts the boarding action at the boarding location. In addition, the first image may be an image of the user in the vehicle interior after the user finishes the boarding action at the boarding location.

[0024] In a scene where the user gets off the vehicle 2, the vehicle exterior image acquisition unit 30 acquires a second image. The second image is an image of the scene where the user gets off the vehicle 2, and is an image of the user and the user's belongings in the scene where the user gets off. The scene where the user gets off includes the time when the user gets off the vehicle 2 at the get-off location. For example, the second image includes the user when he / she is performing the disembarking action to get off the vehicle 2 (during disembarking). The disembarking action is a series of actions from when the user stands up from the seat, opens the door of the vehicle 2 at the entrance and exit, and goes outside the vehicle cabin. The disembarking action may include a series of actions from when the user goes outside the vehicle cabin to when the user closes the door and locks the door. In addition, the scene where the user gets off includes before the user gets off the vehicle 2 at the get-off location, and after the user gets off the vehicle 2 at the get-off location. That is, the second image includes the user before and after disembarking at the get-off location. For example, the second image may be an image of the user in the vehicle cabin before the user starts the disembarking action at the get-off location. The second image may be an image captured of the user outside the vehicle after the user has finished disembarking at the disembarking location.

[0025] In the following description, the first image will be described as an image of the user captured before the user starts a boarding operation at the boarding location. For example, the vehicle exterior image acquisition unit 30 acquires a first image of the user captured when the user performs boarding authentication at the boarding location. At this time, the first image may be associated with the user's identification information acquired by the boarding authentication. In addition, the second image will be described as an image of the user captured after the user has completed the boarding operation at the disembarking location. For example, the vehicle exterior image acquisition unit 30 acquires a second image of the user captured after the user has left the vehicle at the disembarking location. At this time, if the user has input the user's identification information for payment processing, the second image may be associated with the input user's identification information.

[0026] In this embodiment, the vehicle outside image acquisition unit 30 acquires an image including the front of the user. That is, the vehicle outside image acquisition unit 30 is installed at a position where the vehicle outside image acquisition unit 30 can image the user from the front in the boarding and disembarking scenes, and images the user from the front. The vehicle outside image acquisition unit 30 may also acquire an image including the side of the user. For example, the vehicle outside image acquisition unit 30 is installed at a position where the vehicle outside image acquisition unit 30 can image the user from the side in the boarding and disembarking scenes, and images the user from the side. The position where the user can be imaged from the side is, for example, the position of the door mirror of the vehicle 2.

[0027] In addition, in this embodiment, the vehicle outside image acquisition unit 30 acquires an image including a user located within the standing position area. The standing position area is an area indicating a position where the user stands to acquire an image. The standing position area is represented on the road surface of a boarding and alighting place where the user boards and alights the moving body. The standing position area is a rectangular area. The standing position area is an area of ​​a size that allows multiple users to stand and is within the imaging range (angle of view) of the vehicle outside image acquisition unit 30. In other words, the standing position area is a size that fits within one image. The vehicle outside image acquisition unit 30 acquires a first image including the user located within the standing position area before the user boards the vehicle 2 in the boarding scene, and acquires a second image including the user located within the standing position area after the user alights from the vehicle 2 in the alighting scene.

[0028] The standing position area is an area defined by lines drawn on the road surface. After the vehicle 2 arrives at the boarding / alighting location, the vehicle exterior image acquisition unit 30 acquires an image of the surrounding environment outside the vehicle cabin and performs image recognition processing on the acquired image. The vehicle exterior image acquisition unit 30 recognizes the standing position area defined by the lines by recognizing the lines drawn on the road surface through the image recognition processing. The vehicle exterior image acquisition unit 30 acquires an image including the user located within the recognized standing position area.

[0029] In this embodiment, the standing area may be an area displayed by illumination from a lighting unit provided in the vehicle 2. After the vehicle 2 arrives at the boarding / alighting location, the lighting unit illuminates the road surface at the boarding / alighting location, and the standing area is displayed by the illuminated illumination. The vehicle exterior image acquisition unit 30 acquires an image including the user located within the standing area displayed by illumination from the lighting unit. The vehicle 2 may use the vehicle exterior output unit 60 to notify the user of information instructing the user to enter the standing area together with his / her belongings. In this embodiment, since the standing area is displayed on the road surface at the boarding / alighting location, the user can be guided to a position where the user can be imaged without being cut off, and therefore, one image in which the user is imaged without being cut off can be acquired.

[0030] In this embodiment, the standing area may be divided into a plurality of sections. Each section is divided into a size that allows one person to enter. The outside-vehicle image acquisition unit 30 recognizes one user for each section, and acquires an image including the user and the user's belongings for each recognized user. This makes it possible to recognize the user and the user's belongings for each user, even if there are multiple users at the boarding and disembarking location.

[0031] Furthermore, in this embodiment, the standing area may be arranged such that multiple sections are lined up along the side of the vehicle 2. When the vehicle 2 arrives at the boarding / alighting location, it pulls over to the side of the roadway on the sidewalk or shoulder side of the boarding / alighting location and stops. At this time, the vehicle 2 stops so that the side of the boarding / alighting entrance side of the vehicle 2 is along the roadway edge. The multiple sections of the standing area are arranged so as to be approximately parallel to the side of the vehicle 2 stopped at the boarding / alighting location, i.e., the roadway edge.

[0032] Here, the arrangement of the standing area in this embodiment will be described with reference to FIG. 3. FIG. 3 is a diagram showing an example of the arrangement of the standing area at the boarding / alighting place. FIG. 3 shows a scene in which the vehicle V1 arrives at the boarding / alighting place P and stops at the roadside RS. At the boarding / alighting place P, the standing area A is shown on the road surface, and a plurality of sections A1, A2, A3, and A4 are arranged so as to be lined up along the side of the stopped vehicle V1. Users U1 and U2 are standing in the sections A1 and A2, respectively. The external sensor SD mounted on the vehicle V1 acquires an image of the surrounding environment including the users U1 and U2 located within the standing area in the imaging range SA. As a result, since the users are lined up side by side along the side of the vehicle V1 as seen from the external sensor SD, it is possible to prevent the users at the boarding / alighting place from overlapping with each other and being hidden.

[0033] The authentication unit 40 is a device that authenticates a user who gets into the vehicle 2. The user inputs identification information for identifying the user when getting into the vehicle. The authentication unit 40 acquires the identification information included in the use reservation from the storage unit 70, and performs boarding authentication when the identification information included in the use reservation matches the input identification information. Specifically, the authentication unit 40 generates a match image by capturing an image of a presentation target presented by the user, and matches the match image with a reference image. The authentication unit 40 authenticates the user when the match image matches the reference image. The authentication unit 40 is, for example, a QR code (registered trademark) reader. The presentation target is, for example, a two-dimensional barcode such as a QR code (registered trademark). The presentation target may also be biometric information of the user. The biometric information includes the user's face, etc. The reference image is an image that is stored in advance and is an image to be matched with the match image. The reference image is, for example, a two-dimensional barcode such as a QR code (registered trademark) or a facial photograph of the user. For example, when a user makes a reservation for use, a reference image for identifying the user is stored in the reservation information.

[0034] The in-vehicle output unit 50 is equipped in the vehicle 2 and includes a display and a speaker capable of outputting information to a user in the vehicle cabin. The display shows image information to the user. The speaker outputs audio information to the user. The in-vehicle output unit 50 is installed in a position visible to passengers seated in the rear seats in the vehicle cabin. Furthermore, if there are multiple seat rows, an in-vehicle output unit 50 is installed in each seat row. In this embodiment, the in-vehicle output unit 50 outputs confirmation information to confirm whether the user has left anything behind when the user gets off the vehicle.

[0035] The vehicle exterior output unit 60 is a display and a speaker that are provided outside the vehicle cabin of the vehicle 2 and can output information to a user outside the vehicle cabin. The display shows image information to the user. The speaker outputs audio information to the user. The vehicle exterior output unit 60 is installed in a position that is visible to the user when getting on, for example, at the door at the boarding / alighting entrance. If the display is a touch panel, the vehicle exterior output unit 60 may also serve as an input device. The vehicle exterior output unit 60 may also include lighting units such as hazard lights and headlights. The vehicle exterior output unit 60 uses these lighting units to light up or flash when outputting information.

[0036] In this embodiment, the vehicle exterior output unit 60 displays the determination result determined by the determination unit 101. An image including the determination result is displayed on the vehicle exterior output unit 60. If it is determined that there is a change in the belongings, the image including the determination result is an image indicating that there is a change in the belongings. For example, the image indicates that the user has belongings left behind inside the vehicle 2. The image including the determination result may also include a first image and a second image. The vehicle exterior output unit 60 may also include a lighting unit that emits lighting that displays the standing position area. The vehicle exterior output unit 60 uses the lighting unit to emit lighting that displays the standing position area on the road surface of the boarding and disembarking location.

[0037] In this embodiment, the vehicle exterior output unit 60 will be described as an example of the "display unit" in the claims, but the present invention is not limited to this, and the vehicle interior output unit 50 may be the "display unit". Note that the determination device 10 of this embodiment may use, as an output medium for outputting information, a portable terminal device that can be carried by an occupant and that can output image information and audio information, in addition to the vehicle interior output unit 50 and the vehicle exterior output unit 60.

[0038] The memory unit 70 is a storage medium that stores various information. When a user reserves the use of the vehicle 2, the memory unit 70 stores the reservation. The reservation includes the user's desired boarding location, disembarking location, number of passengers, and identification information that identifies the user. The identification information is, for example, an ID number assigned to each user. The identification information may be, for example, a reference image. The determination device 10 acquires the user's reservation from the vehicle dispatch system and stores it in the memory unit 70. The memory unit 70 also stores images acquired by the vehicle exterior image acquisition unit 30. For example, the images stored in the memory unit 70 are a first image and a second image.

[0039] Next, the determination device 10 of this embodiment will be described. The determination device 10 acquires an image including a user and a target object associated with the user, and determines whether or not there is a change in the target object associated with the user based on the acquired image. The target object includes the user's belongings and a companion. The determination device 10 includes a controller 100. The controller 100 is a computer including a ROM 12 in which a program is stored for determining whether or not there is a change in the target object and notifying the user of the determination result, a CPU 11 as an operating circuit that functions as the determination device 10 by executing the program stored in the ROM 12, and a RAM 13 that functions as an accessible storage device. The controller 100 includes a determination unit 101 and a management unit 102. The controller 100 is a computer that executes each function by cooperation between software for implementing the above functions or executing each process and the above hardware. Below, each functional block of the controller 100 according to this embodiment will be described. In the present embodiment, the determination device 10 is described as an example of the "determination unit" and "management unit" in the claims, but the "determination unit" and "management unit" in the claims may be an in-vehicle device other than the determination device 10, or may be separate devices. For example, the "management unit" may be a device different from the determination device 10.

[0040] The determination unit 101 executes an image acquisition process, a user identification process, and a determination process. First, the determination unit 101 executes an image acquisition process to acquire a first image and a second image from the vehicle exterior image acquisition unit 30. In the image acquisition process, the determination unit 101 determines whether the vehicle 2 has arrived at the boarding location and ended its travel. When the determination unit 101 determines that the vehicle 2 has arrived at the boarding location and ended its travel, the determination unit 101 transmits a control instruction to the vehicle exterior image acquisition unit 30 to acquire a first image, and receives the first image acquired by the vehicle exterior image acquisition unit 30. The determination of whether the vehicle 2 has arrived at the boarding location and ended its travel may be performed based on the position information of the vehicle 2 and the position information of the boarding location, or may be performed based on the determination results from the door device 221, the locking device 222, and / or the seat belt device 223.

[0041] In the image acquisition process, the determination unit 101 determines whether the vehicle 2 has arrived at the disembarking location and ended its travel. When the determination unit 101 determines that the vehicle 2 has arrived at the disembarking location and ended its travel, the determination unit 101 transmits a control instruction to the vehicle exterior image acquisition unit 30 to acquire a second image, and receives the second image acquired by the vehicle exterior image acquisition unit 30. The determination of whether the vehicle 2 has arrived at the disembarking location and ended its travel may be performed based on the position information of the vehicle 2 and the position information of the disembarking location, or may be performed based on the determination results from the door device 221, the locking device 222, and / or the seat belt device 223. The determination unit 101 receives the first image and the second image each time the vehicle exterior image acquisition unit 30 acquires the first image and the second image, and stores each image in the storage unit 70. The control instruction to acquire the first image or the second image may include an instruction regarding the timing of acquisition. For example, the instruction may include an instruction to acquire images at a certain period in a boarding scene or an alighting scene, or an instruction to acquire images at a predetermined timing in a boarding scene or an alighting scene. The determination unit 101 may acquire a first image and a second image associated with the user's identification information.

[0042] The boarding location and the disembarking location may be the same location, or may be different locations. When the boarding location and the disembarking location are the same location, there are users disembarking and users boarding at the boarding and disembarking location. In such a case, the user disembarking will disembark first, and the user boarding will board after that. Therefore, the determination unit 101 may first acquire a second image of a disembarking scene including the disembarking action of the user disembarking at the boarding and disembarking location, and acquire a first image of a boarding scene including the boarding action of the user boarding after the user has disembarked. In addition, the determination unit 101 may directly acquire an image from the outside-of-vehicle image acquisition unit 30 without storing it in the storage unit 70, or may read out an image stored in the storage unit 70 from the storage unit 70.

[0043] The determination unit 101 executes a user identification process to identify a user who gets on or off the vehicle. In the user identification process, the determination unit 101 identifies a user who gets on the vehicle 2 based on the acquired first image. For example, the determination unit 101 executes a face recognition process on the first image to recognize the user who gets on the vehicle 2. When a user performs boarding authentication based on a usage reservation, the determination unit 101 may identify the user based on identification information of the usage reservation. The identification information may be identification information of the usage reservation or may be face information of the user.

[0044] In the user identification process, when the second image is acquired, the determination unit 101 identifies the user who gets off the vehicle 2 as a target user based on the acquired second image. The target user is a user who is the subject of determination as to whether or not there has been a change in the target object. The determination unit 101 executes face recognition processing on the second image to recognize the user who gets off the vehicle 2. The determination unit 101 may identify the user based on identification information presented by the user before the user performs a payment process for the usage fee. The identification information may be the identification information of the usage reservation or may be the face information of the user.

[0045] Furthermore, the determination unit 101 may recognize a plurality of users as one group based on the first image at the boarding scene. The determination unit 101 accesses the information of the user's use reservation stored in the storage unit 70 based on the boarding authentication by the user, and acquires information on the number of passengers included in the use reservation. After the boarding authentication by the user, the determination unit 101 counts the number of users boarding the vehicle 2 based on the first image acquired while the plurality of users are boarding the vehicle 2 until the number of users boarding the vehicle 2 reaches the number of passengers included in the use reservation. Then, the determination unit 101 recognizes the plurality of users counted until the number of passengers reaches the number as one group.

[0046] For example, if the number of passengers included in the usage reservation is four, the determination unit 101 executes image recognition processing on the first image acquired after the user's boarding authentication, and counts the users boarding the vehicle 2 one by one in order. Then, when the determination unit 101 counts four users boarding, it recognizes the four users who have boarded up to that point as one group. When the determination unit 101 recognizes a group, it stores in the storage unit 70, for each group, identification information for identifying the group, identification information for identifying all users, and the first image in association with each other.

[0047] The determination unit 101, in the disembarking scene, identifies a user who belongs to the group disembarking from the vehicle 2 as a target user based on the second image. For example, the determination unit 101 executes an image recognition process on the second image, and when it is determined that the recognized user is a user who belongs to the group, identifies the user as a target user.

[0048] The determination unit 101 executes a determination process for determining whether or not there is a change in a target object associated with a target user based on a first image and a second image including the target user. Hereinafter, the target object will be described as the user's belongings. In the determination process, when the determination unit 101 identifies a target user in a dismounting scene, the determination unit 101 acquires a first image and a second image including the target user from the first image and the second image stored in the storage unit 70 based on the identification information of the identified target user. For example, the determination unit 101 identifies a user who matches the characteristics of the target user identified from the second image from the first image, and acquires a first image including the identified user. Then, the determination unit 101 recognizes the user's belongings included in the first image and the second image including the target user, and compares the belongings of the first image and the second image, thereby determining whether or not there is a change in the user's belongings between the first image and the second image. The determination unit 101 executes an image recognition process for each of the first image and the second image to recognize the user's belongings. The belongings are, for example, items worn by the user or carried by the user. Furthermore, when a user included in an image is within the standing position area, the determination unit 101 may recognize an article within the standing position area as a possession of the user.

[0049] For example, when the belongings of the target user shown in the first image are not shown in the second image, that is, when the belongings of the target user are reduced, the determination unit 101 determines that the target user has left some belongings behind. This is an example of determining that there is a change in the belongings of the target user when the target user forgets and leaves belongings behind in the vehicle. For example, when a vehicle is ride-sharing-enabled, even if there is something left behind in the vehicle and it is not possible to identify whether the left-behind item belongs to the user who got off or another user who remains in the vehicle, the user's belongings are compared before and after getting on and off, so that a change in the user's belongings can be determined.

[0050] Also, for example, the belongings for which the presence or absence of a change is determined may be belongings of a predetermined size or more. That is, the determination unit 101 determines whether or not there is a change in belongings of a predetermined size or more based on the first image and the second image including the target user. The predetermined size is, for example, the size of a general bag or suitcase, and is a size that is set in advance. Also, the predetermined size may be the size of belongings that can accommodate other belongings such as a bag or suitcase included in the first image. When the target user puts small items that he or she had when getting on the train into a bag or the like and gets off the train, the target user has not left anything behind in the vehicle interior, so it is possible to eliminate the possibility of determining that there is a change in the belongings even in such a case.

[0051] Furthermore, when the determination unit 101 identifies a group of users, the determination unit 101 determines whether or not there is a change in the target object associated with the target users belonging to the identified group, based on the first image and the second image including the target users belonging to the group. Specifically, the determination unit 101 acquires the first image and the second image including the group to which the target users belong, from the first image and the second image stored in the storage unit 70, based on the identification information of the group. Then, the determination unit 101 identifies all the target users and the belongings of all the target users included in the first image and the second image including the group, respectively, and compares the target users and the belongings identified in the first image and the second image, respectively, to determine whether or not there is a change in all the target users and the belongings of all the target users between the first image and the second image.

[0052] The management unit 102 manages the use of the vehicle 2 by the user. When a reservation for use by the user is made, the management unit 102 acquires information of the user's reservation for use from the vehicle dispatch system and stores it in the memory unit 70. When the vehicle 2 arrives at the user's boarding location, the management unit 102 displays authentication guidance for the user to perform boarding authentication on the display of the external output unit 60. The authentication guidance is guidance information that guides the user through the procedure for performing boarding authentication by operating the authentication unit 40. At this time, when the user has performed boarding authentication on the authentication unit 40, the determination unit 101 may identify the boarding user based on the identification information.

[0053] Furthermore, when the vehicle 2 arrives at the user's drop-off location, the management unit 102 displays notification information prompting the user to check whether or not they have left anything behind via the in-vehicle output unit 50 before undoing the seat belt. Furthermore, the notification information prompting the user to check whether or not they have left anything behind may be output via the out-vehicle output unit 60 when the user gets off the vehicle and when the door is closed.

[0054] After the user gets off the vehicle 2, the management unit 102 causes the outside-vehicle output unit 60 to display the determination result determined by the determination unit 101. The determination result includes a determination result that there is a change in the target object associated with the target user. The determination result may include a first image and a second image. That is, the determination result includes an image for the user to confirm the change in the target object. The outside-vehicle output unit 60 displays an image including the target object of the target user in the boarding scene and an image including the target object of the target user in the disembarking scene. The determination result may be a determination result that there is no change in the target object associated with the target user.

[0055] Here, an example of a method for outputting a determination result will be described with reference to FIG. 4. FIG. 4 is a diagram showing the external appearance of the side of a vehicle equipped with a user display. In FIG. 4, a user display D1 is disposed at a door at the entrance of a vehicle V1. A display screen S1 is an example of a screen displayed on the user display D1. An image DB (first image) including a user before boarding and an image DA (second image) including a user after disembarking are displayed on the display screen S1. The image DB shows a user wearing a hat and carrying a suitcase. The image DA shows a user, but the user is not wearing a hat and is not carrying a suitcase. The user can confirm that there is a change in belongings between the image DB and the image DA by checking the display screen S1.

[0056] The management unit 102 determines whether or not the target user has confirmed the determination result displayed by the external output unit 60. For example, the management unit 102 displays a selection button indicating that the user has confirmed the determination result together with the determination result, and determines whether or not the user has confirmed the determination result based on whether or not the user has selected the selection button. When the management unit 102 determines that the target user has confirmed the determination result, it starts a settlement process for the usage fee of the vehicle 2. In the settlement process, the management unit 102 causes the external output unit 60 to display a settlement screen that accepts settlement of the usage fee.

[0057] The communication unit 120 transmits and receives information to and from devices and systems external to the vehicle 2. The communication unit 120 receives a reservation made by a user from the vehicle dispatch system. The reservation includes a boarding location, an alighting location, the number of passengers, and user identification information.

[0058] The various devices 220 of this embodiment include a door device 221 that detects whether the doors of the vehicle 2 are opened or closed and outputs the detection result, a locking device 222 that detects the locking state of the vehicle 2 and outputs a door lock signal and a door unlock signal, and a seat belt device 223 that detects the use state of the seat belt of the vehicle 2 and outputs the signal. These various devices 220 output the detected information to the determination device 10 via the vehicle controller 210 or directly.

[0059] The door device 221 judges whether the driver has got in the vehicle based on the open / close flag of each door. The door device 221 outputs the judgment result to the judgment device 10. The judgment result is used in the judgment process of starting or ending driving.

[0060] The locking device 222 detects door lock and door unlock signals based on inputs to the locking device 222, and outputs the detection results to the determination device 10. Each locking information (unlocking information) is used in a process of detecting door opening and closing operations. The locking device 222 also detects door opening and closing operations, and after detecting a key operation signal of an occupant obtained from a key operation unit, determines whether each door has been opened or closed based on the opening and closing signals of each door obtained from the door device 221.

[0061] The seat belt device 223 detects the use status of the seat belt and outputs the detection result to the determination device 10. The determination result is used in the determination process of the start of driving or the end of driving. The determination result includes whether the user is using the seat belt or not.

[0062] The vehicle controller 210 is an on-board computer such as an electronic control unit (ECU), and electronically controls a drive mechanism that governs the operation of the vehicle 2. The vehicle controller 210 is included in a vehicle system. The vehicle controller 210 controls a drive device, a braking device, and a steering device included in the drive mechanism to make the vehicle 2 travel along a target route. The drive mechanism includes an electric motor and / or an internal combustion engine as a driving source, a power transmission device including a drive shaft and an automatic transmission that transmits the output from these driving sources to the drive wheels, a drive device that controls the power transmission device, and a braking device that brakes the wheels. The vehicle controller 210 generates control signals for each of these drive mechanisms based on input signals from accelerator operation and brake operation, and executes driving control including acceleration and deceleration of the vehicle. By sending control information to the drive mechanism, driving control including acceleration and deceleration of the vehicle can be automatically performed.

[0063] The vehicle controller 210 controls the steering device so that the vehicle 2 travels while maintaining a predetermined lateral position with respect to the travel path (trajectory). In this specification, the travel lane / travel path is a path corresponding to the travel path of the host vehicle, and can be identified by road, uphill / downhill of the road, and lane of the road. The steering device includes a steering actuator. The steering actuator includes a motor attached to the steering column shaft, etc. The steering device executes steering control of the vehicle based on a control signal obtained from the vehicle controller 210 or an input signal by the driver's steering operation.

[0064] Next, a determination procedure in the determination device of this embodiment will be described with reference to the flowchart of Fig. 5. Fig. 5 is a flowchart showing an example of a control procedure of the determination method according to this embodiment. When a user who has made a reservation gets into the vehicle 2, the determination device 10 starts the control flow from step S101.

[0065] In step S101, the determination device 10 acquires a first image acquired by the vehicle exterior image acquisition unit 30. For example, the vehicle exterior image acquisition unit 30 acquires the first image before the user gets into the vehicle 2 at the boarding location and transmits the first image to the determination device 10. In step S102, the determination device 10 acquires a second image acquired by the vehicle exterior image acquisition unit 30. For example, the vehicle exterior image acquisition unit 30 acquires the second image after the user gets out of the vehicle 2 at the disembarking location and transmits the second image to the determination device 10.

[0066] In step S103, the determination device 10 identifies the user who gets off as a target user. In step S104, the determination device 10 determines whether or not there is a change in a target object associated with the target user based on the first image and the second image including the target user. In step S105, the determination device 10 outputs the determination result in step S104 via the outside-vehicle output unit 60.

[0067] As described above, the moving body of this embodiment includes an image acquisition unit that acquires an image including a user and a target object associated with the user, and a determination unit that determines whether or not there is a change in the target object based on the image acquired by the image acquisition unit, the image acquisition unit acquires an image of a boarding scene in which the user boards the moving body as a first image, and acquires an image of a disembarking scene in which the user disembarks from the moving body as a second image, and the determination unit identifies the user disembarking from the moving body as a target user, and determines whether or not there is a change in the target object associated with the target user based on the first image and the second image including the identified target user. This makes it possible to determine whether or not there is a change in the object associated with the user while the user is on the moving body and disembarks from the moving body.

[0068] The moving body of the present embodiment further includes a storage unit that stores a reservation for using the moving body, including the number of passengers, and the determination unit recognizes the number of passengers as one group based on the first image at the boarding scene, identifies the users belonging to the group who will get off the moving body as target users based on the second image at the disembarking scene, and determines whether or not there is a change in the target object associated with the target users belonging to the group based on the first image and the second image including the target users. This makes it possible to determine whether or not there is a change in all users belonging to the group and the objects associated with all users, even if the users are a group.

[0069] In addition, in the moving body of the present embodiment, the determination unit determines whether or not there is a change in a target object of a predetermined size or larger based on the first image and the second image including the target user. This makes it possible to determine whether or not there is a change in an object, excluding a small object that can fit in a bag.

[0070] The moving object of the present embodiment further includes a display unit that displays the result of the determination, thereby enabling the user to check whether or not there has been a change in the object associated with the user.

[0071] In addition, the moving body of the present embodiment is a vehicle capable of autonomous driving. As a result, even in a vehicle that does not require a driver, it is possible to automatically determine whether or not there has been a change in an object associated with the user.

[0072] The mobile body of the present embodiment further includes a management unit that manages the use of the mobile body by the user, and the management unit determines whether the target user has confirmed the judgment result displayed by the display unit, and when it is determined that the target user has confirmed the judgment result, starts a settlement process for the usage fee of the mobile body. This allows the user to determine whether there has been a change in the object associated with the user before the settlement process for the usage fee is performed.

[0073] In addition, in the moving object of the present embodiment, the image acquisition unit acquires an image including a side view of the user, which makes it possible to determine whether or not there has been a change in an object associated with the user included in an image when the user is imaged from the side instead of from the front.

[0074] In addition, in the mobile body of this embodiment, the image acquisition unit acquires an image including the user located within the standing position area, and the standing position area is an area indicating the position where the user stands to acquire the image, and is represented on the road surface at the location where the user gets on and off the mobile body. This makes it possible to accurately acquire an image including the user at the boarding and alighting location.

[0075] In addition, in the mobile body of this embodiment, the standing position area is an area defined by lines drawn on a road surface, and the image acquisition unit recognizes the standing position area and acquires an image including the user located within the recognized standing position area, thereby making it possible to accurately acquire an image including the user located at the boarding and disembarking location.

[0076] The moving body of the present embodiment further includes an illumination unit that irradiates a road surface with illumination to display a standing position area, and the image acquisition unit acquires an image including a user located within the standing position area displayed by the illumination, thereby enabling an image including a user located at a boarding / disembarking location to be acquired with high accuracy.

[0077] In addition, in the moving object of this embodiment, the standing position area is divided into a plurality of sections, so that even if there are a plurality of users, images including each user can be obtained while distinguishing between the users.

[0078] In addition, in the moving body of this embodiment, the standing position area is arranged such that a plurality of sections are lined up along the side of the moving body, so that even if there are a plurality of users, it is possible to obtain images including each user by distinguishing between the users.

[0079] In addition, the determination device and the determination method of the present embodiment acquire an image including the user and a target object associated with the user in a boarding scene in which the user boards a moving body as a first image, acquire an image including the user and a target object associated with the user in a disembarking scene in which the user disembarks from the moving body as a second image, identify the user disembarking from the moving body as a target user, and determine whether or not there has been a change in the target object associated with the target user based on the first image and the second image including the target user. This makes it possible to determine whether or not there has been a change in the object associated with the user from the time the user boarded the moving body to the time the user disembarked from the moving body.

[0080] The above-described embodiments are described for the purpose of facilitating understanding of the present invention, and are not described for the purpose of limiting the present invention. Therefore, the elements disclosed in the above embodiments are intended to include all design modifications and equivalents that fall within the technical scope of the present invention. [Explanation of symbols]

[0081] 2. Vehicle 10…Judgment device 100…Controller 101…Judgment section 102…Management Department 20…In-vehicle image acquisition unit 30…External image acquisition unit 50…In-vehicle output section 60…External output section 70...Storage section

Claims

1. An image acquisition unit that acquires an image including a user and a target object associated with the user; a determination unit that determines whether or not there is a change in the target object based on the image acquired by the image acquisition unit, The image acquisition unit includes: acquiring, as a first image, the image of a riding scene in which the user rides on a moving object; acquiring, as a second image, the image of a scene in which the user gets off the moving object; The determination unit is Identifying the user who gets off the moving object as a target user; A moving body that determines whether or not there is a change in the target object associated with the specified target user based on the first image and the second image including the specified target user.

2. The moving body according to claim 1 , A storage unit that stores a reservation for use of the moving object, including the number of passengers, The determination unit is In the riding scene, the users with the number of passengers are recognized as one group based on the first image; In the disembarking scene, the user belonging to the group who disembarks from the moving object is identified as the target user based on the second image; A moving body that determines whether there is a change in the target object associated with the target user belonging to the group based on the first image and the second image including the target user.

3. The moving body according to claim 1 or 2, The determination unit is A moving body that determines whether there is a change in the target object of a predetermined size or larger based on the first image and the second image including the target user.

4. The moving body according to claim 1 or 2, The moving object further includes a display unit that displays a result of the determination made by the determination unit.

5. The moving body according to claim 1 or 2, The moving body is a vehicle capable of autonomous driving.

6. The moving body according to claim 4, A management unit that manages the use of the mobile object by the user, The management unit determining whether the target user has confirmed the determination result displayed by the display unit; the mobile object, when it is determined that the target user has confirmed the determination result, starting a settlement process for the usage fee of the mobile object;

7. The moving body according to claim 1 or 2, The image acquisition unit is a moving object that acquires the image including a side view of the user.

8. The moving body according to claim 1 or 2, The image acquisition unit acquires the image including the user located within a standing position area, The standing position area is an area indicating the position where the user is standing to acquire the image, and is a moving body represented on the road surface at the location where the user gets on and off the moving body.

9. The moving body according to claim 8, The standing position area is an area defined by lines drawn on the road surface, The image acquisition unit includes: Recognizing the standing area, A moving object that acquires the image including the user located within the recognized standing position area.

10. The moving body according to claim 8, A lighting unit that illuminates the road surface with lighting to display the standing position area, The image acquisition unit is a moving body that acquires the image including the user located within the standing position area displayed by the lighting.

11. The moving body according to claim 8, The standing position area of ​​the moving body is divided into a plurality of sections.

12. The moving body according to claim 11, The standing area is a moving body in which the plurality of sections are arranged in a line along a side of the moving body.

13. A determination device including a controller that obtains an image including a user and a target object associated with the user, and determines whether or not there is a change in the target object based on the obtained image, The controller: Acquiring, as a first image, an image of a boarding scene in which the user boards a moving body, and acquiring, as a second image, an image of a disembarking scene in which the user disembarks from the moving body; Identifying the user getting off the moving object as a target user based on the second image; A determination device that determines whether or not there is a change in the target object associated with the target user based on the first image and the second image including the target user.

14. A method for determining whether or not there is a change in the target object based on an image acquired by a determination device, the method comprising: The determination device includes: Acquiring, as a first image, an image of a boarding scene in which the user boards a moving body, and acquiring, as a second image, an image of a disembarking scene in which the user disembarks from the moving body; Identifying the user who gets off the moving object as a target user; A method for determining whether or not there is a change in the target object associated with the target user based on the first image and the second image including the target user.

Images