Occupant management device, occupant management method, and storage medium

Abstract

The present application provides a kind of passenger management device, passenger management method and storage medium, mainly provide a kind of can make the same passenger who does not hold own user ID ride vehicle, the passenger management system of the same passenger who rides is managed by the user ID of user. Passenger management system has: passenger information acquisition unit, the passenger ID of each area of the passenger identified from the image of camera is given;User information acquisition unit, obtain the user ID of user;And passenger information management unit, make the user ID be associated with passenger ID. There is set with the same passenger information acquisition unit, obtain the number of same passengers associated with user ID and have been registered in advance. When the number of same passengers associated with user ID is 1 or more, passenger information management unit is based on the number of same passengers, the user ID is associated with the passenger ID identified before the user ID is associated with passenger ID or after the user ID is associated with passenger ID.

Description

Technical Field

[0001] The invention relates to an occupant management device, an occupant management method, and a storage medium. Background Technology

[0002] In vehicles used by multiple users, such as driverless passenger vehicles, users issue user IDs in advance, present their user IDs to the vehicle, and ride in the vehicle (see, for example, Patent Document 1).

[0003] Existing technical documents Patent documents Patent Document 1: Japanese Patent Application Publication No. 2018-106315 Summary of the Invention The problem the invention aims to solve When a user wants to ride in a vehicle with a companion, the companion may need to issue their own user ID and ride in the vehicle. On the other hand, if a companion without their own user ID has already ridden in the vehicle, the vehicle cannot manage the companion using their own user ID. Therefore, in vehicles used by multiple users, there is a need for a mechanism that allows companions without their own user IDs to ride in the vehicle and manages the vehicles of those companions.

[0004] The purpose of this invention is to solve problems related to colleagues.

[0005] means for solving problems Regarding the aforementioned issues, the present invention provides an occupant management device for managing occupant information of a vehicle. It is characterized by comprising: an occupant information acquisition unit that assigns an occupant ID to each region in an image captured inside or near the vehicle's boarding gate that identifies an occupant; a user information acquisition unit that acquires the user ID of a user who has applied to ride in the vehicle; an occupant information management unit that associates the user ID with the occupant ID; and a companion information acquisition unit that acquires the number of companions associated with the user ID and pre-registered. When the number of companions associated with the user ID is 1 or more, the occupant information management unit associates the user ID with occupant IDs identified before or after the association of the user ID with the occupant ID, based on the number of companions.

[0006] Furthermore, the present invention is a storage medium storing an occupant management program, characterized in that the occupant management program causes a computer processor to execute the following steps: an occupant information acquisition step, assigning an occupant ID to each area in an image captured inside or near the vehicle's boarding gate to identify an object; a user information acquisition step, acquiring a user ID from a user who has applied to ride in the vehicle; an occupant information management step, associating the user ID with the occupant ID; and a companion information acquisition step, acquiring the number of companions associated with the user ID and pre-registered with the user ID; in the occupant information management step, when the number of companions associated with the user ID is 1 or more, the occupant associates the user ID with occupant IDs identified before or after the association of the user ID with the occupant ID, based on the number of companions.

[0007] The effects of the invention Through the above-described structure, the passenger management device of this invention enables passengers who do not possess their own user ID to ride in the vehicle, and manages the passengers by using the user's user ID. Attached Figure Description

[0008] Figure 1 This is an overall structural diagram of the occupant management device in this embodiment.

[0009] Figure 2A To show Figure 1 A block diagram of the structure of the occupant management device.

[0010] Figure 2B for Figure 2A A variation of the above.

[0011] Figure 3 This is a diagram illustrating the reading status of a user ID obtained through a terminal communication device. Figure 3 (a) in the text represents the state where reading is possible. Figure 3 (b) indicates a state that cannot be read.

[0012] Figure 4 A diagram illustrating the status of user IDs being acquired by the occupant monitoring device.

[0013] Figure 5 (a) is a block diagram showing the internal structure of the occupant monitoring device. Figure 5 (b) in the example is the output of the monitoring unit. Figure 5 (c) in the figure shows the region detected in the image.

[0014] Figure 6A block diagram illustrating the internal structure of the storage section of the occupant monitoring device.

[0015] Figure 7 Here is an example of a record in the passenger information table.

[0016] Figure 8 A block diagram illustrating the internal structure of the management department.

[0017] Figure 9 A timing diagram illustrating the operation of the occupant management device.

[0018] Figure 10 A flowchart of user standby processing for the occupant monitoring device.

[0019] Figure 11 This is a flowchart of the notification process for whether the user ID can be obtained.

[0020] Figure 12 A flowchart for retrieving user IDs.

[0021] Figure 13 The flowchart shows the process of retrieving data from a temporary queue.

[0022] Figure 14 This is a flowchart illustrating the processing of the management department's determination section.

[0023] Figure 15 This is a flowchart of the logic for determining whether a passenger should board a vehicle.

[0024] Figure 16 (a) shows the vehicle when no one is inside. Figure 16 (b) shows the state of the detected new occupant.

[0025] Figure 17 This is a flowchart of the move determination logic.

[0026] Figure 18 This is a flowchart of the logic for determining when to disembark.

[0027] Figure 19 This is a flowchart of the violation determination logic.

[0028] Figure 20 This is a timing diagram illustrating the occupant management device's response to violations.

[0029] Explanation of reference numerals in the attached figures 1: Occupant management device; 2: Vehicle; 3: Occupant; 5: User; 12: Camera; 13: Camera; 14: Image; 15: Companion; 25: Management department; 33: Area; 35: Determination department; 41: Occupant information acquisition department; 42: User information acquisition department; 43: Occupant information management department; 44: Companion information acquisition department. Detailed Implementation

[0030] use Figures 1-20 This embodiment will be described in detail.

[0031]

Example 1

[0032] Figure 1 This is an overall structural diagram of the occupant management device 1 in this embodiment. The occupant management device 1 is a device that primarily uses a computer with a processor and storage to execute occupant management methods using an occupant management program to manage the occupants 3 of the vehicle 2, thus constituting the occupant management system of the vehicle 2. Figure 1 In vehicle 2, the upper side indicated by the arrow is the front of the direction of travel.

[0033] Vehicle 2, for example, is used for passenger service where multiple users 5 ride together, and has multiple seats 4 inside. Vehicle 2 has a reception unit 8 at the entrance / exit 6 to obtain the user ID of the user 5. The user ID obtained by the reception unit 8 is sent via a service control device 9 to a passenger monitoring device 11 that monitors passengers 3. The service control device 9 controls various services performed by the passenger management device 1.

[0034] The occupant monitoring device 11 is connected to a camera 12 installed inside the vehicle 2, and inputs images 14 of the vehicle interior captured by the camera 12. The occupant monitoring device 11 can also be connected to a camera 13 installed around the entrance / exit 6, so that it can input images 14 of the interior and exterior of the vehicle around the entrance / exit 6 captured by the camera 13.

[0035] Vehicle 2 is, for example, a vehicle that can be ridden by multiple users 5 who do not know each other. This embodiment is primarily envisioned as an unmanned passenger vehicle. However, even if vehicle 2 is a manned passenger vehicle, passengers 3 can still be managed.

[0036] Passenger 3 refers to a person currently riding in vehicle 2, a person preparing to ride in, or a person already riding in. In this embodiment, passenger 3 refers to the person detected by the passenger monitoring device 11 from the images 14 of cameras 12 and 13, including the user 5 using vehicle 2 and the companions 15 of user 5. User 5 holds their own user ID and is either serving a passenger preparing to use vehicle 2 or is currently using it. When user 5 issues a user ID to register the use of vehicle 2 in advance, the number of companions 15 is recorded. Thereafter, without distinguishing between companions 15 and user 5, they are recorded as passenger 3.

[0037] Passenger 15 is a person who shares vehicle 2 with user 5 and uses passenger services without holding a user ID. The number of passengers in passenger 15 can be limited or unlimited. For example, in a group of multiple people, one person who has been issued a user ID becomes user 5, and the remaining members register as passenger 15 when they are issued their user IDs. In the case of a family with children, for example, the parents become user 5 by issuing a user ID, and the children become passenger 15 by registering.

[0038] In this way, by allowing companion 15, the entire group or family does not need to issue separate user IDs, thus increasing the convenience of passenger services. In this embodiment, as long as user 5 and companion 15 travel together, the order of boarding or alighting is irrelevant. For example, companion 15 can board or alight before or after user 5. User 5 can board or alight between companions 15 and other members of the same group.

[0039] Seat 4 is an individual seat for each of the three occupants, and a fixed number are provided in the vehicle. The configuration of seat 4 is not limited. For example, Figure 1 The occupants 3 are a group consisting of two people: a user 5 seated in the middle row seat 4 and a companion 15 seated in the rear row seat 4, as well as other users 5 seated in the rear row seat 4. The entrance / exit 6 is an opening provided in the vehicle 2 for people to pass through.

[0040] The User ID is a license associated with User 5, allowing them to ride in Vehicle 2. The User ID is issued by User 5's mobile terminal 16, obtained through a public network line from the operator's server. The User ID and the number of companions 15 are stored on User 5's mobile terminal 16. The User ID and the number of companions 15 can be sent from the operator's server to User 5's mobile terminal 16 at any time based on User 5's actions.

[0041] User IDs can be provided through recording media capable of displaying or recording user IDs, such as paper tickets, magnetic tickets, magnetic cards, or IC cards issued or registered at ticket windows or ticket machines. When using recording media, the mobile terminal 16 is interpreted as the recording medium. The user ID is printed on the surface of the paper ticket. Printing is the transfer of visual information—text, numbers, graphics, symbols, codes (e.g., barcodes, QR codes, registered trademarks)—or a combination thereof, onto the recording medium. Magnetic tickets, magnetic cards, and IC cards store the user ID as data.

[0042] In this embodiment, the user ID and the number of companions 15 are described as data stored in the user 5's mobile terminal 16. The user 5's mobile terminal 16 can also be a portable multi-functional communication terminal such as a smartphone or tablet computer.

[0043] The receiving unit 8 is a terminal device that accepts ride requests from user 5 when boarding or alighting from vehicle 2, and reads the user ID held by user 5 and the number of companions 15. The receiving unit 8 only needs to be able to read the user ID and the number of companions 15; for example, it can be a terminal communication device 8a that performs near-field wireless communication with user 5's mobile terminal 16. The receiving unit 8 can also be an electromagnetic induction type encoding reader 8b that reads the encoding of the user ID recorded on the mobile terminal 16 and the recording medium. The receiving unit 8 can also be an optical reader that optically reads the surface of the display and recording medium of the mobile terminal 16.

[0044] The receiving unit 8 can be either contact-based or contactless. In this embodiment, as an example of the receiving unit 8, a contact-based terminal communication device 8a is disclosed. The receiving unit 8 can operate independently or via control signals from any one of the service control device 9, the occupant monitoring device 11, or a server at an external operating site. In this embodiment, the receiving unit 8 operates collaboratively with the service control device 9 and the occupant monitoring device 11.

[0045] Service control unit 9 connects to the receiving unit 8 and the occupant monitoring device 11, facilitating information transmission between them. Service control unit 9 can be, for example, a computer equipped with a CPU (Central Processing Unit) and memory. Service control unit 9 can be connected to the server at the operating site via a public network line. Thus, information is transmitted between the receiving unit 8, service control unit 9, occupant monitoring device 11, and the server at the operating site, and remote operation can be performed from the server at the operating site as needed.

[0046] The occupant monitoring device 11 is a key component of the occupant management device 1. The occupant monitoring device 11 can be, for example, a computer with a CPU, a GPU (Graphics Processing Unit), and storage. The occupant monitoring device 11 can be mounted on the vehicle 2, or it can be located outside the vehicle 2 (e.g., as a server at an operating site) to remotely manage occupant information. Alternatively, its functions can be divided and located both inside the vehicle 2 and outside. The CPU or GPU is also referred to as a processor. In this embodiment, the occupant monitoring device 11 is located on the vehicle 2. Details of the occupant monitoring device 11 will be described later.

[0047] Camera 12 captures images of the interior of the vehicle 2 and outputs images 14 captured inside the vehicle 2. One or more cameras 12 can be installed inside the vehicle to capture images of all seats 4. Camera 13 captures images of the exterior of the vehicle 2 and the perimeter of the entrance / exit 6 and outputs images 14 captured near the entrance / exit outside the vehicle. Camera 13 is configured to face outwards from the entrance / exit 6.

[0048] Image 14 from camera 12 is an image of the interior of the vehicle captured by camera 12, showing the situation of the occupants 3 inside the vehicle. Images 14 from multiple cameras 12 can be combined into one image for processing, or can be switched between images 14. Image 14 from camera 13 is an image of the vehicle from outside to the entrance / exit 6 captured by camera 13, mainly showing the situation of the user 5 and companion 15 passing through the entrance / exit 6. Image 14 from camera 13 can be combined with images 14 from camera 12, or can be switched between images 14 from camera 13 and images 14 from camera 12.

[0049] Figure 2A A block diagram showing the structure of the main parts of the occupant management device 1.

[0050] Vehicle 2 has a display 21 and a speaker 22. The display 21 and speaker 22 are located in one or more places inside the vehicle. The occupant display 21 and speaker 22 can be installed for each occupant 3, for example, in each seat 4.

[0051] like Figure 2B As shown in the variation, the same display 21a and speaker 22a can be installed in the service providing device 23. The service providing device 23 is mainly used to provide various services by the service control device 9 in passenger service. The service providing device 23 is connected to the passenger management device 1 via wired or wireless means. The service providing device 23 can be installed in each seat 4. The service providing device 23 can be temporarily lent to a fellow passenger 15 as a portable terminal. In this case, the service providing device 23 is lent to the fellow passenger 15, for example, before or during the journey, and retrieved upon disembarkation.

[0052] The display 21b and speaker 22b can be disposed in or around the reception section 8 for the occupant 3 to pass through the entrance section 6. In this embodiment, as Figure 3 As shown, the receiving unit 8 has a display 21b on its surface.

[0053] Displays 21, 21a, and 21b can be displays for displaying image 14 or displays for displaying visual information. Displays 21, 21a, and 21b can be, for example, liquid crystal panels, organic EL (Organic Electro-Luminescence) panels, or display units for visual information using light sources such as LEDs (Light Emitting Diodes). The visual information can be, for example, at least one of text, graphics, markings, and colors. The visual information can be displayed by illuminating the image with light. Speakers 22, 22a, and 22b can provide necessary guidance or warnings through sound, beeps, etc.

[0054] The occupant monitoring device 11 has a monitoring unit 24, a management unit 25, and a storage unit 26 as functional units.

[0055] The monitoring unit 24 is an image processing unit that monitors the occupant 3 by performing image recognition on the images 14 input from the cameras 12 and 13. The monitoring unit 24 outputs the image recognition result, i.e., the recognition result, to the management unit 25. The recognition result of the monitoring unit 24 is stored in the storage unit 26 and used by the management unit 25.

[0056] The management unit 25 manages the information of passenger 3. The management unit 25 uses the identification results from the monitoring unit 24 and information from the receiving unit 8 of the service control device 9 to manage the information of passenger 3. The management unit 25 may also use information from the server at the operating site. The management unit 25 issues guidance, warnings, and other notifications to passenger 3 directly or via the service control device 9. At this time, the management unit 25 can appropriately send notifications to the vehicle interior, the displays 21, 21b of the receiving unit 8, the speakers 22, 22b, the user 5's mobile terminal 16, and the service providing device 23 of the companion 15.

[0057] Storage unit 26 stores various data and information required by the occupant management device 1 for managing the occupants 3. Storage unit 26 is located in the occupant monitoring device 11 and is primarily used by the occupant monitoring device 11, but can also be used by any one of the service control device 9, the receiving unit 8, or the server at the operating site. Storage unit 26 may also be located outside the occupant monitoring device 11, using at least a portion of the storage device possessed by any one of the service control device 9, the receiving unit 8, or the server at the operating site.

[0058] Figure 2A The user 5's mobile terminal 16 has a computing unit 28 that executes an installed application 27, a communication unit 29 that communicates with the receiving unit 8 and the server of the operating site, a display 31, and a speaker 32.

[0059] Application 27, executed by the computing unit 28, communicates with the server at the operating site and issues a user ID, registering companion 15 with the server. Application 27 receives warnings from the occupant monitoring device 11 directly or via the service control device 9, displays the warning on the display 31, or causes the speaker 32 to emit a warning sound. In this embodiment, application 27 is used in an autonomous passenger vehicle. The computing unit 28 is composed of a computer with a CPU and storage.

[0060] When user 5 issues a user ID and registers the number of companions 15, communication unit 29 accesses the server of the operating site via a public line such as the Internet. Communication unit 29 enables receiving unit 8 to read the user ID and the number of companions 15. Communication unit 29 can perform short-range wireless communication with receiving unit 8, and may also have an antenna that receives the magnetic field generated by the electromagnetic induction-type code reader 8b.

[0061] Display 31 has a touch panel that accepts input from the user. Display 31 can display the user ID and the number of companions 15, which can be optically read by the optical reader of the receiving unit 8. Display 31 and speaker 32 can also be used in the same way as service providing device 23 in the provision of various services (e.g., guidance, warnings) of service control device 9.

[0062] Figure 2B The service providing device 23 may have the same computing unit 28a and communication unit 29a as the mobile terminal 16. The service providing device 23 is a terminal that does not have the function of issuing user IDs and registering the number of companions 15, nor the function of enabling the receiving unit 8 to read the user IDs and the number of companions 15. The service providing device 23 can receive warnings from the management unit 25 directly or via the service control device 9, either wired or wirelessly, and display the warnings on the display 21a and cause the speaker 22a to emit a warning sound.

[0063] like Figure 3 As shown, user 5 applies for a ride in vehicle 2 by presenting mobile terminal 16 to the reception unit 8. Presenting mobile terminal 16 to terminal communication device 8a, which is an example of reception unit 8, is done by bringing mobile terminal 16 into contact with or near terminal communication device 8a. Hereafter, the contact or proximity of mobile terminal 16 to terminal communication device 8a is sometimes referred to as "contact".

[0064] like Figure 4 As shown, the receiving unit 8 reads terminal information, the pre-issued user ID, and the number of companions 15 by using the user 5 to apply for contact with the mobile terminal 16 of the vehicle 2, and sends this information to the service control device 9. The terminal information is inherent to the mobile terminal 16 and is the information that uniquely identifies the contacting mobile terminal 16.

[0065] The service control device 9 associates the terminal information received from the receiving unit 8, corresponding to the mobile terminal 16 of user 5 that applied for ride in vehicle 2, with a pre-issued user ID and stores it in its internal storage device 9a. The service control device 9 then sends the user ID that applied for ride in the vehicle and the number of companions 15 to the occupant monitoring device 11.

[0066] In the occupant monitoring device 11, such as Figure 5 As shown in (a) to (c), the monitoring unit 24 performs image recognition processing on the occupants 3 shown in the images 14 of cameras 12 and 13 as identification targets. For each identified occupant 3, the monitoring unit 24 outputs region 33 along with region number # to the management unit 25. Region 33 defines the range of a rectangle representing the position of the occupant 3 within the image 14. Furthermore, the region number # is different from the occupant ID described later. When the image 14 is from multiple cameras 12, region 33 contains information determining which camera 12 corresponds to the image 14.

[0067] The monitoring unit 24 records the various actions of the occupant 3 shown in images 14 from cameras 12 and 13 as events. In addition to area number # and area 33, the monitoring unit 24 also records necessary event information such as... Figure 5 The output is sent to the management department 25 as in example (b) of the example. Event information can also include, for example, information about violations such as the smoking status of each occupant 3.

[0068] The management unit 25 attaches the passenger detection time (which serves as the boarding time) and passenger ID to the output from the monitoring unit 24, generates a temporary queue as passenger detection information, and stores it as "detection information" in the storage unit 26. Furthermore, the passenger detection time is either the time when the monitoring unit 24 receives the identification result used for passenger detection, or the time when the monitoring unit 24 detects passenger 3 and begins monitoring.

[0069] A temporary queue is a queue that is not yet in use. A queue is a data structure that arranges and maintains data in a line according to the order of arrival, and stores and retrieves data according to the first-in, first-out (FIFO) rule. In addition to boarding detection information, there is also disembarkation detection information, which is generated in a similar way to the boarding detection information and stored as "detection information" in the temporary queue.

[0070] like Figure 6As shown, the storage unit 26 has at least a storage area 26a for temporarily storing the aforementioned user IDs and a storage area 26b for temporarily storing the number of companions 15. The storage unit 26 has a storage area 26c for storing the aforementioned temporary queues and a storage area 26d for storing the passenger table. The storage unit 26 has a storage area 26e for storing the user ID flag (described later), a storage area 26f for storing the detection counter, and a storage area 26g for storing the ride counter.

[0071] As described below, the user ID flag is primarily used for standby processing in the reception unit 8. A detection counter counts the number of passengers 3 passing through the entrance / exit unit 6. The detection counter is mainly used for user ID acquisition processing during boarding and alighting. The boarding counter is used as the allocation value when assigning passenger IDs to users 5 and their companions 15.

[0072] like Figure 7 As shown, the passenger form has a data area for storing user ID, boarding time, passenger ID, and area 33. However, the passenger form may also have other data areas besides those mentioned above. The boarding time can be the time when the management unit 25 receives the user ID from the service control device 9, i.e., the time of requesting boarding, or, as an example, the time when the mobile terminal 16 contacts the receiving unit 8, i.e., the terminal communication device 8a. The passenger form generates records according to user IDs, and each record contains data on the passenger IDs of the total number of people in the group, including the user 5 and their companions 15. Figure 7 The passenger form contains two records: one for user ID 12345, which corresponds to user 5 who is accompanied by a companion 15; and the other for user ID 54321, which corresponds to a different user ID, which corresponds to another user 5 who is not accompanied by a companion 15.

[0073] like Figure 8 As shown, the management unit 25 of the occupant monitoring device 11 includes a form management unit 34 and a determination unit 35. The form management unit 34 generates and manages the records of the occupant form. The determination unit 35 uses the recognition results of the image recognition of the monitoring unit 24 to monitor the actions of the occupant 3 and performs determination processing to determine if an incident has occurred.

[0074] When generating the passenger form, the form management unit 34 performs the standby processing of user 5 as performed by the receiving unit 8, and sends a notification to the receiving unit 8 or the display 21b around the receiving unit 8 to request whether the user ID can be obtained. The form management unit 34 performs notification processing to the user 5's mobile terminal 16 to request whether the user ID can be obtained, and performs the acquisition processing to obtain the user ID from the management unit 25.

[0075] The determination process in Unit 35 includes passenger determination, movement determination, disembarkation determination, and violation determination. The generation and determination of the passenger form can be performed in parallel, and the processing results can be used interchangeably.

[0076] like Figure 8 As shown, the management unit 25 has a passenger information acquisition unit 41, a user information acquisition unit 42, a passenger information management unit 43, and a companion information acquisition unit 44 as functional units.

[0077] The occupant information acquisition unit 41 assigns an occupant ID to each area 33 in the image 14 captured inside or near the boarding gate of vehicle 2 that identifies an object. The user information acquisition unit 42 acquires the user ID of the user 5 who applied for vehicle access. The occupant information management unit 43 associates the user ID that applied for vehicle access with the occupant ID. The companion information acquisition unit 44 acquires the number of companions 15 that the user 5 is traveling with and that have been pre-registered with the user ID. When the number of companions 15 associated with the user ID is 1 or more, the occupant information management unit 43 associates the user ID that applied for vehicle access with the occupant IDs identified before or after the user ID was associated with the occupant ID.

[0078] As described above, the management unit 25 manages information on occupants 3 currently riding in vehicle 2, occupants 3 preparing to ride in, and occupants 3 already riding in. Images 14 from cameras 12 and 13 are images of the interior and exterior of the vehicle, including the area around the entrance / exit 6. Occupants 3 are the objects of image recognition. Area 33 is the rectangular area occupied by the objects of image recognition from the monitoring unit 24. The occupant information acquisition unit 41 assigns occupant IDs to the user 5 (occupant 3) and their companion 15.

[0079] The user information acquisition unit 42 acquires the user ID of the user 5 who applied to ride in the vehicle. The passenger information management unit 43 performs the core function of the management unit 25, which associates passenger IDs with user IDs. While the user information acquisition unit 42 acquires the user ID, the companion information acquisition unit 44 acquires the number of companions 15 from the user 5's mobile terminal 16 or recording medium via the receiving unit 8 or the service control device 9.

[0080] "When the number of companions 15 is 1 or more" means that there are 15 registered companions. "Associating passenger IDs based on the number of companions 15" means that the passenger IDs of users 5 and companions 15 belonging to the same group are assigned the same user ID. "Before or after the association of user ID and passenger ID" means regardless of the order of boarding.

[0081] Regarding the occupant information management unit 43, during periods when the number of occupants 3 identified based on the image 14 taken inside or near the boarding gate of the vehicle 2 is less than the number of companions 15 corresponding to the user ID plus the number of users 5, or greater than the number of companions 15 plus the number of users 5, even if the user information acquisition unit 42 acquires a user ID different from the user ID, it will not associate the different user ID with the occupants 3 identified based on the image 14 taken inside or near the boarding gate of the vehicle 2.

[0082] "The period during which the number of occupants 3 identified based on image 14 taken inside or near the boarding gate of vehicle 2 is less than the number of companions 15 corresponding to user ID plus the number of users 5, or greater than the number of companions 15 plus the number of users 5" refers to a period when the number of occupants 3 detected by image recognition is inconsistent with the total number of people in the group of users 5 obtained from the receiving unit 8.

[0083] A different user ID refers to a user ID that is different from that of user 5 who obtained the user ID first. "Not associating a different user ID with occupant 3 identified from image 14" means not associating the occupant ID with other users 5 until the association of all members of the group of user 5 who obtained the user ID first is completed. This restricts other users 5 from cutting in line to board the bus.

[0084] If the number of occupants 3 identified based on images 14 taken inside or near the boarding gate of vehicle 2 is less than the number of companions 15 corresponding to the user ID plus the number of users 5, or greater than the number of companions 15 plus the number of users 5, the user information acquisition unit 42 sends a notification to the user 5 with the different user ID if it acquires a user ID different from the user ID. The processing unit 46 sends a notification to the user 5 corresponding to the user ID that applied for vehicle boarding.

[0085] The management unit 25 performs notification processing to the terminals and sends notifications to the mobile terminals 16 of users such as user 5. First, the notification is sent to the mobile terminals 16 of other users 5 who hold different user IDs. Second, to draw attention, the notification may also be sent to the user 5 whose user ID was read first. The user information acquisition unit 42 or the passenger information management unit 43 sends notifications to other users 5 and their mobile terminals 16 via the service control device 9 that stores terminal information.

[0086] The management unit 25 also has a notification unit 45, which notifies the user 5 in a discernible manner whether the user information acquisition unit 42 can process the user ID. If the number of occupants 3 identified from the images 14 of cameras 12 and 13 is less than the number of companions 15 corresponding to the user ID plus the number of users 5, or is greater than the number of companions 15 plus the number of users 5, the notification unit 45 notifies that the processing of the user ID cannot be carried out.

[0087] The notification unit 45 processes the request to determine whether the user ID can be obtained, and sends a notification to the receiving unit 8 or a display 21b or speaker 22b located near the receiving unit 8. The display 21b then displays a message indicating that the user ID cannot be received. The speaker 22b may also provide audio guidance indicating that the request cannot be received. The notification unit 45 may also send notifications to the aforementioned user 5 and other users' mobile terminals 16.

[0088] In the above, user standby processing is mainly handled by the user information acquisition unit 42, which works in conjunction with the service control device 9, controlling the acceptance unit 8. The notification unit 45 sends a notification to the acceptance unit 8 or the display 21b surrounding the acceptance unit 8, indicating whether the user ID can be obtained. The management unit 25 or the notification unit 45 sends a notification to the mobile terminal 16 of the user 5 who wants to apply for a ride in vehicle 2, indicating whether the user ID can be obtained. Centered on the passenger information management unit 43, the form management unit 34 performs the process of obtaining the user ID, which associates the user ID with the passenger ID.

[0089] The management unit 25 also includes a processing unit 46 and a determination unit 35, which function as functional units. The processing unit 46 sends a notification to the user 5 corresponding to the user ID. If the determination unit 35 receives an event generated by the companion 15, it determines the user ID corresponding to the companion 15 whose event was detected. The processing unit 46 sends an event-related notification to the terminal of the user 5 with the determined user ID.

[0090] While the processing unit 46 does not need to notify all events, it may notify a few events that could potentially disrupt the operation of vehicle 2 or cause inconvenience to other occupants 3 (e.g., occupant 3's misconduct). From the perspective of usability and privacy, it is preferable to notify individuals. The events generated by user 5 are directly notified to user 5's terminal (e.g., mobile terminal 16). The events generated by companion 15 are also indirectly communicated to companion 15 by user 5 to their terminal. The processing unit 46 sends the notification via the service control device 9, which stores terminal information. Alternatively, the processing unit 46 may send the events generated by companion 15 directly or via the service control device 9 to the terminal of companion 15, which functions as a service providing device 23, based on the occupant ID determined by the determination unit 35. The processing unit 46 may also perform a notification process to the terminal (e.g., user 5's mobile terminal 16) regarding whether the user ID can be obtained.

[0091] The event refers to the actions of passenger 3 detected from the output of monitoring unit 24, including boarding, moving, disembarking, and violations. Violations include, for example, various behaviors that violate the operating rules of passenger vehicles.

[0092] Violations include, for example, smoking, violent behavior inside the vehicle, inappropriate seating in seat 4 or other seats, not wearing a seatbelt, and others. For example, occupant 3's abnormalities due to illness, fainting, or other reasons, while not considered violations, are events that can be detected from the output of monitoring unit 24, just like violations. For convenience, occupant 3's abnormalities can be handled as violations or separately. Regarding occupant 3's abnormalities, violations can be interpreted as occupant abnormalities.

[0093] like Figure 8 As shown, in order to process boarding, moving, disembarking, and violations as events generated by passenger 3, the management department 25 has a boarding determination department 51, a moving determination department 52, a disembarking determination department 53, and a violation determination department 54 in its determination department 35. The boarding determination department 51 performs the boarding determination process, the moving determination department 52 performs the moving determination process, the disembarking determination department 53 performs the disembarking determination process, and the violation determination department 54 performs the violation determination process.

[0094] Furthermore, the form management unit 34 and the determination unit 35 can have overlapping functions without completely differentiating them. For example, the passenger information acquisition unit 41 can be configured to function as at least a portion of the boarding determination unit 51, movement determination unit 52, disembarkation determination unit 53, and violation determination unit 54 that use the determination unit 35. If the event being notified is only a violation, the violation determination unit 54 can be used as the determination unit 35, and the boarding determination unit 51, movement determination unit 52, and disembarkation determination unit 53 can be used as the passenger information acquisition unit 41. The functions of the notification unit 45 and the processing unit 46 can be combined into one.

[0095] <Function> The function of this embodiment is as follows.

[0096] In the occupant management device 1 of this embodiment, a computer having a processor and storage uses an occupant management program to execute the following occupant management method. That is, the occupant management program causes the processor to execute the following steps: an occupant information acquisition step, assigning an occupant ID to each area 33 in an image 14 captured inside or near the vehicle's boarding gate where an object is identified; a user information acquisition step, acquiring a user ID from a user 5 who has applied for vehicle access; an occupant information management step, associating the user ID with the occupant ID; and a companion information acquisition step, acquiring the number of companions 15 associated with the user ID and pre-registered with the user 5. In the occupant information management step, when the number of companions 15 associated with the user ID is 1 or more, the user ID is associated with occupant IDs identified before or after the user ID is associated with the occupant ID, based on the number of companions 15.

[0097] Specifically, the occupant management device 1 uses Figure 9 The sequence diagram shows the operation as follows. First, at time T1, user 5 uses application 27 on mobile terminal 16 to issue a user ID in advance and register the number of companions 15. Application 27 stores the user IDs from the operating site and the number of companions 15 in mobile terminal 16.

[0098] On the day of use, at time T2, user 5 contacts mobile terminal 16 with receiving unit 8 at entrance / exit 6 of vehicle 2. At this time, receiving unit 8 enters standby mode under the control of user information acquisition unit 42. During standby mode, notification unit 45 informs user 5 of the status. Figure 3 Is it a state that can obtain the user ID as shown in (a) in the diagram, or is it like... Figure 3 The user ID cannot be obtained as shown in (b) of the diagram. Notifications from the notification unit 45 are made using the receiving unit 8 or a light source such as a display 21b or an LED around the receiving unit 8.

[0099] Through this display, user 5 can determine whether the contact is valid, i.e., whether the process has transitioned from "standby processing" to "user ID acquisition processing". At this time, the form management unit 34 or notification unit 45 of the management unit 25 can notify user 5 whether the contact is valid via the display 31 or speaker 32 of the mobile terminal 16. Preferably, the management unit 25 notifies user 5's mobile terminal 16 at least when the contact is invalid. Through notification, user 5 can be reliably informed that boarding is permitted when the contact is valid and not permitted when the contact is invalid.

[0100] On the other hand, at time T3, the occupant management device 1 and the occupant monitoring device 11 are activated when or after the vehicle 2 is powered on, and begin monitoring of the occupants 3. At this time, the receiving unit 8 performs standby processing. Details of the standby processing will be described later.

[0101] If user 5's mobile terminal 16 touches or approaches a terminal communication device 8a, which is an example of the receiving unit 8, the service control device 9 inputs the terminal information of the mobile terminal 16, the user ID, and the number of companions 15 obtained from the mobile terminal 16, associates the terminal information with the user ID, and stores it. The service control device 9 outputs the user ID and the number of companions 15 to the user information acquisition unit 42.

[0102] At time T4, if the occupant monitoring device 11 inputs the user ID and the number of companions 15 into the user information acquisition unit 42, the user information acquisition unit 42 stores the user ID and the requested travel time as the travel time into the occupant table. At time T5, the occupant information acquisition unit 41 detects travel from the output of the monitoring unit 24, which has processed the images 14 of the cameras 12 and 13. At time T6, the occupant information acquisition unit 41 stores the travel detection information, with the occupant ID attached to the output of the monitoring unit 24, i.e., area 33, in the temporary queue of the storage unit 26. At time T7, the occupant information management unit 43 retrieves the travel ID and area 33 from the temporary queue, records them into the occupant table, and then deletes them from the temporary queue.

[0103] In the above content, regarding the occupant monitoring device 11 and the user information acquisition unit 42, as follows... Figure 10 The flowchart shows how the receiving unit 8 handles the standby process for user 5. "Standby processing for user 5" is used to process multiple users 5 sequentially. The receiving unit 8 waits for user 5 to present its user ID to the receiving unit 8 during the standby process.

[0104] In step S1, processing begins by turning on the power to the vehicle 2 and the occupant monitoring device 11. By turning on the power, the receiving unit 8 waits for the display of the user ID, which is achieved through contact with the user's mobile terminal 16, etc.

[0105] First, in step S2, the user information acquisition unit 42 turns the user ID acquisition flag ON. This enables the processing unit 8 to acquire the user ID. Acquiring the user ID is equivalent to accepting boarding the vehicle 2 and obtaining boarding permission. The processing unit 8 manages this by using the user ID acquisition flag in a way that prevents the simultaneous acquisition of multiple user IDs. Furthermore, the initial value of the user ID acquisition flag is set to ON.

[0106] In step S3, the notification unit 45 executes the "notification process for whether the user ID can be obtained" (described later) in parallel. The notification process for whether the user ID can be obtained will be described later.

[0107] In step S4, the user information acquisition unit 42 determines whether the user 5's mobile terminal 16 has contacted the receiving unit 8, etc. If there is no contact (i.e., step S4 is No), the process returns to step S4 and repeats the determination until it is Yes. If there is contact (i.e., step S4 is Yes), the process proceeds to step S5.

[0108] In step S5, the user information acquisition unit 42 determines whether the user ID can be obtained. For example... Figure 3 As shown in (b), if the result cannot be obtained (i.e., step S5 is not successful), proceed to step S6.

[0109] In step S6, the management unit 25 notifies the mobile terminal 16 corresponding to a user ID that is different from the user ID currently being acquired.

[0110] In step S7, the management unit 25 notifies the mobile terminal 16 corresponding to the user ID currently being acquired. The notification in step S7 can be performed as needed. The notifications in steps S6 and S7 can be performed using any one of the user information acquisition unit 42, the passenger information management unit 43, or the notification unit 45. Processing returns to step S4, and the judgment in step S4 is repeated until a yes is found. Figure 3 As shown in (a), if it is possible to obtain the result in step S5, proceed to step S8.

[0111] In step S8, the management unit 25 turns the user ID acquisition flag off (OFF). As a result, the receiving unit 8 becomes unable to acquire the user ID.

[0112] In step S9, the user information acquisition unit 42 performs user ID acquisition processing in parallel. The user ID acquisition processing, described later, involves the occupant monitoring device 11 acquiring information such as the user ID that user 5 has presented to the receiving unit 8 via the service control device 9. By executing this process in parallel with the standby processing described above, the receiving unit 8 can wait for other users 5 to present different user IDs. The processing returns to step S4 and proceeds with the processing after step S4.

[0113] The above-mentioned "notification processing that can obtain user ID" is as follows: Figure 11 The process is as described in the flowchart.

[0114] Notification processing that can obtain user ID from Figure 10The "User Standby Processing" branch of step S3 begins. First, in step S11, it is determined whether the user ID can be obtained. Figure 3 As shown in (a), if the user ID acquisition flag is enabled (i.e., step S11 is true), in step S12, the process is switched to user ID acquisition processing, and the process returns to step S11. Figure 3 As shown in (b), if the user ID acquisition flag is off, i.e., step S11 is not, in step S13, it is notified not to transfer to the user ID acquisition process and return to step S11.

[0115] The notification may be made by either the reception unit 8 or one of the displays 21b or speakers 22b located around the reception unit 8. For example, the display 21b may... Figure 3 As shown in (a), the process is transferred to retrieve the user ID, such as... Figure 3 As shown in (b) above, the display does not transfer to the user ID retrieval process. However, the content displayed on display 21b is not limited to this. Figure 3 In case (a), the display 21b can use blue, green, other reassuring colors, or safe colors for display. Figure 3 In case (b), the display 21b may use yellow, red, black, other attention-grabbing colors, or warning colors for display.

[0116] Figure 10 Step S9, "Parallel execution of user ID acquisition processing," is in the entire table of Management Department 25, Management Department 34, as shown. Figure 12 The process is as shown in the flowchart. The user ID acquisition process involves obtaining the user ID for each user 5, and if there is a companion 15, associating the obtained user ID with the companion 15.

[0117] If processing begins, firstly, in step S21, the passenger information acquisition unit 41 or the passenger information management unit 43 sets the detection counter to 0. In step S22, the user information acquisition unit 42 obtains the user ID from the user 5's mobile terminal 16 via the receiving unit 8 and the service control device 9.

[0118] The user information acquisition unit 42 of the passenger monitoring device 11 records the user ID and the record associated with the requested travel time received from the service control device 9 into the passenger form, and temporarily stores the number of companions 15 in the storage unit 26.

[0119] When user 5's mobile terminal 16 comes into contact with the receiving unit 8 and the user ID can be obtained at the time of contact, the user information acquisition unit 42 acquires the user ID. If the user ID cannot be obtained even if user 5's mobile terminal 16 comes into contact with the receiving unit 8, the user information acquisition unit 42 does not acquire the user ID. However, before acquiring the user ID, i.e., before user 5 boards the vehicle, if passenger 3 is detected by the monitoring unit 24 and passenger information acquisition unit 41, the boarding detection information has already been stored in a temporary queue in the storage unit 26. Therefore, past detection information in the temporary queue is retrospectively reflected in the passenger table. Thus, user 5 and their companion 15 can board or alight in any order within the group.

[0120] In step S23, the passenger information management unit 43 determines whether the temporary queue is empty. If the temporary queue is empty (i.e., step S23 is not empty), proceed to step S28. If the temporary queue is not empty (i.e., step S23 is not empty), in step S24, the passenger information management unit 43 mainly performs the temporary queue retrieval process described later.

[0121] In step S25, the passenger information management unit 43 determines whether the detection counter is 0 or the number obtained by adding the number of companions 15 to the number of users 5 (1). A detection counter of 0 indicates that no passenger 3 has been detected, while a detection counter obtained by adding the number of companions 15 to 1 indicates that the entire group of users 5 and companions 15 has been detected.

[0122] After recording the information in the passenger table in step S22, the passenger information management unit 43 can begin acquiring other user IDs based on the image recognition results of the monitoring unit 24 and the number of companions 15 temporarily stored in the storage unit 26. The passenger information management unit 43 can only begin acquiring other user IDs when the detection counter is 0 or when the total number of people is equal to the sum of the number of users 5 and companions 15. That is, the passenger information management unit 43 can only begin acquiring other user IDs when no one is detected or when all members of user 5's group are detected.

[0123] If step S25 is negative, in step S26, the user information acquisition unit 42 or the passenger information management unit 43 turns the "user ID acquisition available" flag off, and proceeds to step S28. As a result, the receiving unit 8 becomes unable to acquire the user ID. If step S25 is positive, in step S27, the user information acquisition unit 42 or the passenger information management unit 43 turns the "user ID acquisition available" flag on, and proceeds to step S28. As a result, the receiving unit 8 becomes capable of acquiring the user ID.

[0124] In step S28, the user information acquisition unit 42 determines whether other user IDs have been acquired. If no other user IDs have been acquired (i.e., step S28 is negative), the process returns to step S23 and repeats the subsequent steps. If other user IDs have been acquired (i.e., step S28 is positive), in step S29, either the companion information acquisition unit 44 or the passenger information management unit 43 discards the number of companions 15. When acquiring other user IDs begins, the passenger information management unit 43 discards the number of companions 15 temporarily stored in the storage unit 26.

[0125] In step S30, if the detection counter is 0, the passenger information management unit 43 deletes the record. That is, if the number of passenger IDs in the passenger table becomes 0 due to disembarking, the passenger information management unit 43 deletes the record from the passenger table. The process ends.

[0126] Figure 12 Step S24, "Temporary queue retrieval processing", is as follows: Figure 13 The process proceeds as shown in the flowchart. If processing begins, firstly, in step S31, the occupant information management unit 43 retrieves the detection information recorded by the occupant information acquisition unit 41 from the temporary queue of the storage unit 26.

[0127] In step S32, after the passenger detection information, which serves as detection information, has been retrieved from the temporary queue, the passenger information management unit 43 determines the record in the passenger form that has the closest required passenger time to the passenger 5's passenger detection time. For example, the passenger information management unit 43 determines that in Figure 12 The user ID record obtained in step S22. The occupant information management unit 43 appends the occupant ID and region 33 to the determined record, and increments the detection counter by 1.

[0128] In step S33, after the disembarkation detection information, which serves as detection information, has been retrieved from the temporary queue, the passenger information management unit 43 deletes the passenger ID and the corresponding area 33 from the record of the acquired user ID in the passenger table and decrements the detection counter by 1. The temporary queue retrieval process ends.

[0129] The "incident determination and processing" performed by the determination unit 35 of the management unit 25 is conducted in the following manner. The determination and processing utilizes the image recognition results from the monitoring unit 24 to determine boarding, movement, disembarkation, and violation. The determination unit 35, together with the occupant information acquisition unit 41, performs the following... Figure 14 The following image recognition processing is performed as shown in the flowchart.

[0130] If processing begins upon power-on, the passenger determination unit 51 waits for the monitoring unit 24 to detect that passenger 3 has boarded the vehicle. In step S41, the passenger determination unit 51 executes the passenger determination logic. In step S42, the passenger determination unit 51 determines whether passenger boarding has been detected. If passenger boarding has not been detected (i.e., step S42 is negative), the process proceeds to step S44. If passenger boarding has been detected (i.e., step S42 is positive), in step S43, the passenger determination unit 51 stores the passenger detection information in a temporary queue and proceeds to step S44.

[0131] If a ride is detected by the ride determination logic described later, the ride determination unit 51 generates a passenger ID for each ride detection. The passenger ID is the value of the ride counter. The initial value of the ride counter is 0, and it increments by 1 each time a ride is detected. Unlike the detection counter, the ride counter does not decrease when a passenger gets off the vehicle, and it is shared and used by multiple users 5. The ride determination unit 51 associates each passenger ID, each region 33 in each detection, and each ride detection time in each detection, and stores this as ride detection information in a temporary queue.

[0132] In step S44, the movement determination unit 52 executes the movement determination logic. In step S45, the movement determination unit 52 determines whether movement is detected. If no movement is detected (i.e., step S45 is negative), the process proceeds to step S47. If movement is detected (i.e., step S45 is positive), the movement determination unit 52 updates region 33 in step S46 and proceeds to step S47.

[0133] According to the movement determination logic described later, the movement determination unit 52 detects the movement of occupant 3 within vehicle 2 and determines the occupant ID of the moving occupant 3. Based on the detected movement, the movement determination unit 52 updates the region 33 corresponding to the determined occupant ID.

[0134] In step S47, the disembarkation determination unit 53 executes the disembarkation determination logic. In step S48, the disembarkation determination unit 53 determines whether disembarkation has been detected. If no disembarkation is detected (i.e., step S48 is negative), the process proceeds to step S50. If disembarkation is detected (i.e., step S48 is positive), in step S49, the disembarkation determination unit 53 decrements the detection counter by 1 and deletes the disembarkation detection information. Alternatively, the disembarkation determination unit 53 stores the disembarkation detection information in a temporary queue. Processing then proceeds to step S50.

[0135] Decrementing the detection counter by 1 and deleting the alighting detection information indicates a temporary alighting at the start of passenger service due to boarding at the wrong time. Storing the alighting detection information in a temporary queue indicates the end of passenger service.

[0136] According to the disembarkation determination logic described later, the disembarkation determination unit 53 detects the disembarkation of passenger 3 and determines the passenger ID. At this time, if the user ID acquisition process is not being performed by the user information acquisition unit 42 and the passenger information management unit 43, the disembarkation determination unit 53 deletes the passenger ID and region 33 from the passenger table. The result of this deletion is that if the number of passenger IDs in the passenger table is 0, the passenger information management unit 43 deletes the record. If the user ID acquisition process is in progress, the passenger information management unit 43 stores the passenger ID as disembarkation detection information in a temporary queue.

[0137] In step S50, the violation determination unit 54 executes the violation determination logic. In step S51, the violation determination unit 54 determines whether a violation has been detected. If step S51 is negative, the process proceeds to step S41. If step S51 is positive, in step S52, the processing unit 46 and the service control device 9 notify each other of the violation, and the process proceeds to step S41.

[0138] According to the violation determination logic described later, the violation determination unit 54 detects a violation by occupant 3 inside vehicle 2 and determines the occupant ID of the violating occupant 3. Based on the detected violation, the processing unit 46 notifies the user 5's mobile terminal 16, which has a user ID corresponding to the determined occupant ID, via the service control device 9. Alternatively, the processing unit 46 may notify the service providing device 23, which lends the determined occupant ID, directly or via the service control device 9. Or, the processing unit 46 may also notify the user of the violation directly or via the service control device 9 to the display 21 or speaker 22 located in the seat 4 of the determined occupant ID.

[0139] More specifically, the "ride determination logic" of the ride determination unit 51 is as follows: Figure 15 The flowchart should be followed as described above. Figure 14 The process begins in step S41. First, in step S61, the passenger determination unit 51 receives the information of one passenger 3 output by the monitoring unit 24. The monitoring unit 24 performs image processing on the images 14 from cameras 12 and 13 and outputs the information of one passenger 3. The passenger determination logic is implemented for each passenger 3 output by the monitoring unit 24.

[0140] In step S62, the passenger determination unit 51 retrieves the passenger ID from the storage unit 26, which has a region 33 that overlaps with the region 33 of the input passenger 3.

[0141] In step S63, the ride determination unit 51 determines whether the search result is no match. If there is a match (i.e., step S63 is no), the ride determination unit 51 ends the process. If there is no match (i.e., step S63 is yes), the ride determination unit 51 determines whether to take the ride in step S64 and ends the process.

[0142] For example, if from such Figure 16 The state shown in (a) is that there is no one in the car. Figure 16 If a new passenger 3 not stored in storage unit 26 is detected as shown in (b), then the passenger determination unit 51 determines that the passenger is on the vehicle, issues a passenger ID, and stores area 33 in storage unit 26.

[0143] The "movement determination logic" of the movement determination unit 52 is as follows: Figure 17 The flowchart should be followed as described above. Figure 14 If the processing in step S44 begins, then in step S71, the motion determination unit 52 inputs the information of one occupant 3 output by the monitoring unit 24. The monitoring unit 24 performs image processing on the images 14 from cameras 12 and 13 and outputs the information of one occupant 3. The motion determination logic is implemented for each occupant 3 output by the monitoring unit 24. When using multiple cameras 12 and 13, the monitoring unit 24 can use existing technology to implement the tracking processing between cameras 12 and 13.

[0144] In step S72, the movement determination unit 52 retrieves the passenger ID from the storage unit 26 of the region 33 that overlaps with the region 33 of the input passenger 3.

[0145] In step S73, the movement determination unit 52 determines whether there is a match in the search results. If there is a match, i.e., step S73 is negative, the movement determination unit 52 ends the process. If there is no match, i.e., step S73 is positive, the movement determination unit 52 determines that the move has occurred in step S74, identifies the occupant ID of the region 33 with the largest overlap in the matched regions 33 as the moving occupant 3, and ends the process.

[0146] The "disembarkation determination logic" of disembarkation determination unit 53 is as follows: Figure 18 The flowchart should be followed as described above. Figure 14 When the process begins in step S47, firstly, in step S81, the disembarkation determination unit 53 inputs the information of one occupant 3 stored in the storage unit 26. The monitoring unit 24 performs image processing on the images 14 from cameras 12 and 13 and outputs the information of one occupant 3. The disembarkation determination logic is implemented for each occupant 3 output by the monitoring unit 24.

[0147] In step S82, the disembarkation determination unit 53 searches for the region 33 in the output of the monitoring unit 24 that overlaps with the region 33 corresponding to the passenger ID in the storage unit 26.

[0148] In step S83, the disembarkation determination unit 53 determines whether there is a matching region 33 in the search results. If there is a matching region, i.e., step S83 is negative, the disembarkation determination unit 53 ends the process. If there is no matching region, i.e., step S83 is positive, in step S84, the disembarkation determination unit 53 determines whether the passenger has disembarked. The passenger 3 in the input storage unit 26 is identified as the passenger 3 who has disembarked, and the process ends.

[0149] The "Logic of Violation Judgment" in Violation Judgment Department 54 is as follows: Figure 19 The flowchart should be followed as described above. Figure 14 The process begins in step S50. First, in step S91, the violation determination unit 54 inputs the information of one occupant 3 output by the monitoring unit 24. The monitoring unit 24 performs image processing on the images 14 from cameras 12 and 13 and outputs the information of one occupant 3. The violation determination logic is implemented for each occupant 3 output by the monitoring unit 24.

[0150] In step S92, the violation determination unit 54 determines whether the information of one passenger 3 contains information about a violation. If the violation determination unit 54 determines the violation in step S92, the process ends. If the violation determination unit 54 determines the violation in step S92, the violation determination unit 54 proceeds to step S93 and retrieves the passenger ID from the storage unit 26 in the region 33 that overlaps with the region 33 of the input passenger 3.

[0151] In step S94, the violation determination unit 54 determines whether there is a match in the search results. If there is no match, i.e., step S94 is negative, the violation determination unit 54 ends the process. If there is a match, i.e., step S94 is positive, in step S95, the violation determination unit 54 identifies the occupant ID of the region 33 with the largest overlap in the matched region 33 as the occupant 3 who committed the violation, and ends the process.

[0152] As a result of the violation determination logic, the management unit 25 of the occupant monitoring device 11, such as Figure 20 It operates as shown in the timing diagram. The occupant monitoring device 11 determines the occupant ID of the violation by "determining the violation occupant ID" at time T11, and sends the user ID and violation information to the service control device 9.

[0153] Based on the image recognition results from the monitoring unit 24, the determination unit 35 determines the occupant ID of the occupant 3 who committed the violation. The processing unit 46 sends the user ID corresponding to the determined occupant ID and the violation information to the service control device 9.

[0154] Service control device 9 determines the violating user ID by "determining the violating user ID" at time T12 and sends the violation information to mobile terminal 16.

[0155] The service control device 9 determines the mobile terminal 16 owned by user 5 based on the terminal information corresponding to the received user ID. The service control device 9 then notifies application 27 on the mobile terminal 16 owned by user 5 of the violation information.

[0156] Mobile terminal 16 issues a "warning of violation" at time T13. Application 27 of mobile terminal 16 uses at least one of the display 31 and speaker 32 of mobile terminal 16 to urge user ID or companion 15 to stop the violation. In addition, service control device 9 can also urge companion 15 holding service providing device 23 to stop the violation.

[0157] If the violation cannot be improved by warnings from mobile terminal 16, service control device 9 uses at least one of display 21 and speaker 22 in vehicle 2 to issue a warning about the violation and urge the violation to stop.

[0158] In addition, the present invention provides a storage medium storing an occupant management program, which causes a computer processor to execute the following steps: an occupant information acquisition step, assigning an occupant ID to each area in an image taken inside or near the vehicle entrance where an object is identified; a user information acquisition step, acquiring a user ID from a user who has applied to ride in the vehicle; an occupant information management step, associating the user ID with the occupant ID; and a companion information acquisition step, acquiring the number of companions associated with the user ID and pre-registered with the user ID; in the occupant information management step, when the number of companions associated with the user ID is 1 or more, the occupant associates the user ID with occupant IDs identified before or after the association of the user ID with the occupant ID, based on the number of companions.

[0159] <Effects> The effects of this embodiment are shown below.

[0160] like Figure 8As shown, the occupant management device 1, occupant management method, and occupant management procedure for managing information of occupants 3 in vehicle 2 according to this embodiment include: an occupant information acquisition unit 41, which assigns an occupant ID to each area 33 in an image 14 captured inside or near the vehicle entrance where an occupant is identified; a user information acquisition unit 42, which acquires the user ID of a user 5 who has applied for vehicle access; an occupant information management unit 43, which associates the user ID with the occupant ID; and a companion information acquisition unit 44, which acquires the number of companions 15 associated with the user ID and pre-registered with the user ID. When the number of companions 15 associated with the user ID is 1 or more, the occupant information management unit 43 associates the user ID with occupant IDs identified before or after the user ID is associated with the occupant ID, based on the number of companions 15.

[0161] Therefore, before and after user 5 applies for and boards the vehicle, companion 15 traveling with user 5 can board the vehicle without their own application. The passenger information management unit 43 can identify the identified passenger 3 as a member of the same group as user 5. The number of pre-registered companions 15 is associated with the same user ID as members of the same group. Therefore, for example, driverless passenger vehicles can enable companions 15 who do not possess their own user IDs to board, and companions 15 can be managed using user 5's user ID.

[0162] The occupant management device 1, occupant management method, and occupant management procedure of this embodiment include: a processing unit 46 that sends a notification to a user 5 corresponding to a user ID; and a determination unit 35 that, if an event generated by a companion 15 is obtained, determines a user ID corresponding to either the companion 15 or the user 5 himself / herself. The processing unit 46 sends an event-related notification to a terminal, such as a mobile terminal 16, corresponding to the determined user ID.

[0163] Therefore, if one of the passengers 15 violates the rules of vehicle 2, the terminal of user 5, which corresponds to the user ID of passenger 15, can be notified of the rule violation. Thus, for example, an autonomous passenger vehicle can manage passengers 15 who do not possess a user ID for riding in vehicle 2 using user IDs. The determination unit 35 uses the output of the monitoring unit 24 to track events generated by passenger 3, and continues to associate user 5 with passenger 15 using user IDs after boarding. Thus, the determination unit 35, for example, determines user 5 and passenger 15 who violated the rules of vehicle 2, and for passenger 15, determines the user 5 corresponding to the user ID of passenger 15. The processing unit 46 sends a notification to the mobile terminal 16 of user 5 corresponding to the determined user ID to stop the violation. Therefore, for example, an autonomous passenger vehicle can manage violations by passengers 15 who do not possess a user ID for riding in vehicle 2 using user IDs.

[0164] In this embodiment, when the number of passengers 3 identified by the passenger information management unit 43 based on the image 14 taken inside or near the boarding gate of the vehicle 2 is less than or greater than the number of companions 15 corresponding to the user ID plus the number of users 5, the user information acquisition unit 42 will not associate the different user IDs with the passengers 3 identified based on the image 14 taken inside or near the boarding gate. Therefore, even if a user 5 has finished riding or alighting, and before all companions 15 board or alight, a different user 5 corresponding to another user ID tries to cut in line, the management unit 25 will not associate them. Thus, the management unit 25 can prevent cutting in line and correctly associate passenger IDs with user IDs.

[0165] In the occupant management device 1, occupant management method, and occupant management procedure of this embodiment, when the number of occupants 3 identified based on images 14 taken inside or near the boarding gate is less than the number of companions 15 corresponding to the user ID plus the number of users 5, or greater than the number of companions 15 plus the number of users 5, if the user information acquisition unit 42 acquires a user ID different from the user ID, the processing unit 46 sends a notification to other users 5 corresponding to the different user ID or to the user 5 of the user ID. Therefore, the management unit 25 can inform other users 5 or the user 5 themselves who have cut in line during boarding or alighting in their own group and draw their attention to the situation.

[0166] The occupant management device 1, occupant management method, and occupant management procedure of this embodiment also include a notification unit 45. The notification unit 45 notifies the user 5 in a discriminative manner whether the user information acquisition unit 42 can acquire the user ID. The notification unit 45 notifies whether the user ID can be acquired when the number of occupants 3 identified based on the image 14 taken inside or near the boarding gate is less than the number of companions 15 corresponding to the user ID plus the number of users 5, or greater than the number of companions 15 plus the number of users 5. Therefore, the notification suppresses other users 5 holding mobile terminals 16 assigned different user IDs from attempting to cut in line with a group of users 5 whose user IDs have already been identified by the user information acquisition unit 42.

Claims

1. A passenger management device for managing passenger information in a vehicle, characterized in that, have: The occupant information acquisition unit assigns an occupant ID to each area in images captured inside or near the vehicle's boarding gate that identifies the target. The user information acquisition unit obtains the user ID of the user who applied for a terminal to ride in the vehicle. The passenger information management department associates the user ID that applied to ride in the vehicle with the passenger ID, and The companion information acquisition department acquires the number of companions of the user who are associated with the user ID and have been registered in advance; When the number of companions associated with the user ID is 1 or more, the passenger information management unit associates the user ID with the passenger ID identified before or after the user ID is associated with the passenger ID, based on the number of companions.

2. The occupant management device according to claim 1, characterized in that, The crew members have: The processing unit sends a notification to the user corresponding to the user ID, and If the determination unit obtains an event generated by the companion, it determines the user ID corresponding to the companion whose event was detected or the user himself. The processing unit sends a notification related to the event to the terminal corresponding to the determined user ID.

3. The occupant management device according to claim 1, characterized in that, During the period when the number of occupants identified based on images taken inside the vehicle or near the external boarding gate is less than the number of companions corresponding to the user ID plus the number of the user, or greater than the number of companions plus the number of the user. Even if the user information acquisition unit acquires a user ID that is different from the user ID, the occupant information management unit will not associate the different user ID with the occupant identified based on images taken inside the vehicle or near the vehicle's boarding gate.

4. The occupant management device according to claim 2, characterized in that, During the period when the number of occupants identified based on images taken inside the vehicle or near the external boarding gate is less than the number of companions corresponding to the user ID plus the number of the user, or greater than the number of companions plus the number of the user. If the user information acquisition unit acquires a user ID that is different from the user ID, the processing unit sends a notification to other users corresponding to the different user ID or to the user of the user ID.

5. The occupant management device according to claim 1, characterized in that, The occupant also has a notification unit that notifies the user information acquisition unit in a discriminative manner whether the user ID can be obtained. During the period when the number of occupants identified based on images taken inside the vehicle or near the external boarding gate is less than the number of companions corresponding to the user ID plus the number of the user, or greater than the number of companions plus the number of the user. The notification department indicates whether the user ID can be obtained.

6. A passenger management method for managing passenger information in a vehicle, characterized in that, Includes the following steps: In the occupant information acquisition step, each area in the image taken inside or near the vehicle's boarding gate that identifies the target is assigned an occupant ID. The user information acquisition step involves obtaining the user ID from the user who applied for a ride in the vehicle using their terminal. The passenger information management steps associate the user ID with the passenger ID, and The companion information acquisition step involves obtaining the number of companions associated with the user ID and who have been pre-registered. In the passenger information management step, when the number of companions associated with the user ID is 1 or more, the passenger associates the user ID with the passenger ID identified before or after the user ID is associated with the passenger ID, based on the number of companions.

7. A storage medium storing an occupant management program, characterized in that, The occupant management program causes the computer's processor to perform the following steps: In the occupant information acquisition step, each area in the image taken inside or near the vehicle's boarding gate that identifies the target is assigned an occupant ID. The user information acquisition step involves obtaining the user ID from the user who applied for a ride in the vehicle using their terminal. The passenger information management steps associate the user ID that applied to ride in the vehicle with the passenger ID, and The companion information acquisition step involves obtaining the number of companions associated with the user ID and who have been pre-registered. In the passenger information management step, when the number of companions associated with the user ID is 1 or more, the passenger associates the user ID with the passenger ID identified before or after the user ID is associated with the passenger ID, based on the number of companions.

Images