Information processing device and information processing method

Abstract

To provide an information processing apparatus and an information processing method configured to improve accuracy of passenger information.SOLUTION: An information processing apparatus includes an acquisition unit, a detection unit, a determination unit, and a storage processing unit. The acquisition unit acquires multiple captured images obtained by imaging persons getting on / off a vehicle, from cameras disposed at entrance doors of the vehicle. The detection unit detects person information on a person captured in each of the multiple captured images, on the basis of the captured images acquired by the acquisition unit. The determination unit determines whether the persons captured in the captured images are identical, on the basis of the person information detected by the detection unit. The storage processing unit consolidates the person information corresponding to the persons determined to be identical by the determination unit, to generate passenger information on the passengers on the vehicle, and causes a storage unit to store the generated passenger information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device and an information processing method. [Background technology]

[0002] Various techniques have been proposed for calculating the number of passengers who have boarded and alighted from vehicles such as buses (see, for example, Patent Document 1). In the conventional techniques, cameras are installed at the boarding and alighting entrances, and the number of passengers (number of passengers) is calculated using images of the passengers taken by the cameras. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2014-219913 Summary of the Invention [Problem to be solved by the invention]

[0004] There is a technology for detecting passenger information, such as facial features and age, based on the captured images of the passengers. Since various processes, such as the calculation of the number of passengers, are performed based on such passenger information, there has been a demand for improving the accuracy of the passenger information.

[0005] The present invention has been made in view of the above, and has an object to provide an information processing device and an information processing method that can improve the accuracy of passenger information. [Means for solving the problem]

[0006] In order to solve the above problems and achieve the object, the present invention provides an information processing device comprising an acquisition unit, a detection unit, a determination unit, and a storage processing unit. The acquisition unit acquires a plurality of captured images of people getting on and off a vehicle from a camera installed at the entrance of the vehicle. The detection unit detects personal information about the people captured in each of the captured images based on the plurality of captured images acquired by the acquisition unit. The determination unit determines whether the people captured in each of the plurality of captured images are the same person based on the personal information detected by the detection unit. The storage processing unit aggregates the personal information corresponding to the people determined to be the same person by the determination unit to generate passenger information about passengers of the vehicle, and stores the generated passenger information in the storage unit. [Effects of the Invention]

[0007] According to the present invention, the accuracy of passenger information can be improved. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an overview of an information processing method according to an embodiment. [Figure 2] FIG. 2 is a block diagram showing an example of the configuration of an information processing system equipped with a bus terminal device. [Figure 3] FIG. 3 is a block diagram showing an example of the configuration of the entrance terminal device. [Figure 4] FIG. 4 is a diagram illustrating an example of person information. [Figure 5] FIG. 5 is a diagram illustrating an example of passenger information. [Figure 6] FIG. 6 is a block diagram showing an example of the configuration of an exit terminal device. [Figure 7] FIG. 7 is a diagram illustrating an example of person information. [Figure 8] FIG. 8 is a diagram illustrating an example of the alighting passenger information. [Figure 9] FIG. 9 is a block diagram illustrating an example of the configuration of the management server. [Figure 10]FIG. 10 is a diagram illustrating an example of a processing sequence executed by the information processing system. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of an information processing device and an information processing method disclosed in the present application will be described in detail with reference to the accompanying drawings. Note that the present invention is not limited to the embodiments described below.

[0010] <Outline of information processing method by information processing device> First, an overview of an information processing method by an information processing device according to an embodiment will be described below with reference to Fig. 1. Fig. 1 is a diagram showing an overview of the information processing method according to an embodiment.

[0011] The information processing method according to the embodiment can perform various processes, such as calculating the number of passengers on a vehicle. Note that, although a bus 1 will be described below as an example of a vehicle, the type of vehicle is not limited to this. In other words, the vehicle may be any other type of vehicle, such as a railroad car, an airplane, or a ship.

[0012] 1, a bus 1 is equipped with a bus terminal device 10, which functions as an information processing device. Specifically, the bus terminal device 10 includes an entrance terminal device 10a and an exit terminal device 10b.

[0013] The entrance terminal device 10a is installed at the entrance 2a of the bus 1. The entrance terminal device 10a is equipped with a first camera 11a. For example, the first camera 11a is installed at the entrance 2a of the bus 1 and inside the bus 1 (inside the bus), and captures images of passengers who are boarding the bus 1. The entrance terminal device 10a can detect passengers by detecting facial information from the captured images.

[0014] Exit terminal device 10b is installed at exit 2b of bus 1. Exit terminal device 10b is equipped with second camera 11b. For example, second camera 11b is installed at exit 2b of bus 1 and inside bus 1 (inside the bus), and captures images of passengers getting off the bus 1. Exit terminal device 10b can detect passengers getting off by detecting facial information from the captured images. In the following description, first camera 11a and second camera 11b will be referred to as "camera 11" when there is no need to distinguish between them.

[0015] The bus terminal device 10 can perform various processes such as calculating the number of passengers based on the number of passengers detected by the entrance terminal device 10a and the number of passengers detected by the exit terminal device 10b. The calculation of the number of passengers will be described in detail later.

[0016] Incidentally, various processes such as calculating the number of passengers are performed based on passenger information detected from the captured image, for example. The passenger information is information about the passengers, such as the above-mentioned facial information (for example, facial feature points of the passengers), gender, age, etc.

[0017] However, in the conventional technology, the accuracy of passenger information may be reduced. Specifically, for example, when a passenger is photographed by the camera 11, depending on the passenger's posture, the photographing environment, etc., two pieces of passenger information may be generated for the same passenger.

[0018] To explain this by taking an example, as shown in the upper part of FIG. 1, it is assumed that captured images A1 to A3 are taken of the same passenger P in the order of captured images A1, A2, and A3. Captured image A1 is an image taken when passenger P faces forward toward camera 11 when boarding bus 1. Captured image A2 is an image taken after capturing captured image A1, when passenger P turns his head to the side, for example, to look for a seat or to look outside the bus. Captured image A3 is an image taken after capturing image A2, when passenger P faces forward again toward camera 11.

[0019] As in the captured image A2 described above, even when the same passenger P is captured, the face may be temporarily cut off or blurred. As a result, the bus terminal device 10 may mistakenly detect that the passenger P detected in the captured images A1 and A3 and the passenger P detected in the captured image A2 are different passengers, and generate passenger information for two people. If passenger information for two people is generated in this way despite the fact that it is the same passenger P, the accuracy of the passenger information may be reduced, such as the same passenger P being counted as two people. This reduction in the accuracy of the passenger information may also affect the accuracy of various processes, such as the calculation of the number of passengers described above.

[0020] Therefore, the bus terminal device (information processing device) 10 according to this embodiment is configured to improve the accuracy of passenger information.

[0021] Specifically, the entrance terminal device 10a of the bus terminal device 10 first acquires, from the first camera 11a, a plurality of captured images of people (passengers) getting on the bus 1 (step S1).

[0022] Next, the entrance terminal device 10a detects person information about the person (passenger) captured in each of the captured images based on the acquired captured images (step S2). The person information includes, for example, feature point information indicating facial feature points of the person detected based on the captured images, and attribute information about the person, such as gender, age, age group, and whether or not the person is wearing a mask, which will be described later.

[0023] Furthermore, the entrance terminal device 10a may assign a person ID to each piece of person information, in other words, to each detected person. The person ID is identification information that identifies a person. In the example of FIG. 1, the person captured in the captured images A1 and A3 is assigned a person ID "E01," and the person captured in the captured image A2 is assigned a person ID "E02." In other words, the entrance terminal device 10a is assumed to have detected two people (passengers) with person IDs "E01" and "E02" from the captured images A1 to A3.

[0024] Next, the entrance terminal device 10a determines whether the person captured in each of the multiple captured images is the same person based on the detected person information (step S3). For example, the entrance terminal device 10a compares the multiple pieces of person information detected from the multiple captured images. If the difference between the pieces of person information (in other words, the error) is within a predetermined allowable range, the entrance terminal device 10a determines that the person corresponding to the person information (i.e., the person captured in each of the multiple captured images) is the same person.

[0025] 1, it is assumed that the difference between the personal information corresponding to the person with the person ID "E01" and the personal information corresponding to the person with the person ID "E02" is within an allowable range, and the people captured in the captured images A1 to A3 are determined to be the same person. Note that the above-described determination method is merely an example and is not limiting.

[0026] Next, the entrance terminal device 10a aggregates the personal information corresponding to the persons determined to be the same person to generate passenger information (an example of passenger information), and stores the generated passenger information in the storage unit 30a, in other words, registers it in the storage unit 30a (step S4). Specifically, in the example of Fig. 1, the entrance terminal device 10a aggregates the personal information corresponding to the person with the personal ID "E01" and the personal information corresponding to the person with the personal ID "E02" to generate passenger information for one person, and stores it in the storage unit 30a.

[0027] As a result, in the entrance terminal device (bus terminal device) 10a, even if the same passenger P is detected as a different passenger because, for example, his / her face is temporarily cut off, it is possible to prevent the generation of multiple passenger information, thereby improving the accuracy of the passenger information (passenger information).

[0028] Next, the entrance terminal device 10a outputs the passenger information stored in the storage unit 30a to the exit terminal device 10b at an appropriate timing (step S5).

[0029] In the exit terminal device 10b, by using a configuration similar to that of the entrance terminal device 10a, it is possible to improve the accuracy of the alighting passenger information (an example of passenger information) related to alighting passengers. Specifically, the exit terminal device 10b acquires, from the second camera 11b, a plurality of captured images of people (alighting passengers) getting off the bus 1 (step S6).

[0030] Next, the exit terminal device 10b detects person information about the person (disembarking passenger) captured in each of the plurality of captured images based on the acquired plurality of captured images (step S7).

[0031] Next, the exit terminal device 10b determines whether the person captured in each of the multiple captured images is the same person based on the detected person information (step S8). Note that the determination method in the exit terminal device 10b is the same as the determination method in the entrance terminal device 10a, but is not limited to this, and for example, different determination methods may be used in the exit terminal device 10b and the entrance terminal device 10a.

[0032] Next, the exit terminal device 10b aggregates the personal information corresponding to the persons determined to be the same person to generate disembarking passenger information (an example of passenger information), and stores the generated disembarking passenger information in the memory unit 30b, in other words, registers it in the memory unit 30b (step S9).

[0033] As a result, in the exit terminal device (bus terminal device) 10b, even if the same passenger P is detected as a different passenger because, for example, his / her face is temporarily cut off, it is possible to prevent multiple disembarking passenger information from being generated, thereby improving the accuracy of the disembarking passenger information (passenger information).

[0034] Next, the exit terminal device 10b calculates the number of passengers on the bus 1 based on the boarding passenger information and alighting passenger information output from the entrance terminal device 10a (step S10). For example, the exit terminal device 10b adds one passenger to the number of passengers when new passenger feature information or attribute information is registered in the passenger information; more precisely, it increments the boarding / alighting counter 34b1 (see FIG. 6 described later) that counts the current number of passengers on the bus 1.

[0035] Then, when the feature point information and attribute information of the disembarking passengers included in the disembarking passenger information match or approximately match the feature point information and attribute information of the passengers, exit terminal device 10b presumes that the passengers with the feature point information and attribute information have disembarked from bus 1, and calculates the value obtained by subtracting the number of disembarking passengers from the number of passengers as the current number of passengers on bus 1. More precisely, when the feature point information and attribute information of the disembarking passengers match or approximately match the feature point information and attribute information of the passengers, exit terminal device 10b decrements boarding and alighting counter 34b1 (see FIG. 7).

[0036] As described above, the bus terminal device 10 according to this embodiment can improve the accuracy of passenger information, which is information on boarding and alighting passengers, and therefore can improve the accuracy of various processes that use passenger information, such as calculating the number of passengers. That is, it is possible to prevent, for example, two pieces of passenger information from being generated for the same passenger, resulting in the same passenger being counted twice (double counting of the same person), and therefore improve the accuracy of various processes, such as calculating the number of passengers.

[0037] <Configuration of information processing system> Next, the configuration of an information processing system including a bus terminal device (information processing device) 10 according to an embodiment will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the configuration of an information processing system 100 including the bus terminal device 10.

[0038] 2, the information processing system 100 includes the above-described bus terminal device 10, a management server 200, a bus operator terminal device 300, and a user terminal device 400. The bus terminal device 10, the management server 200, the bus operator terminal device 300, and the user terminal device 400 are communicatively connected via a communication network N such as the Internet.

[0039] As described above, the bus terminal device 10 includes an entrance terminal device 10a and an exit terminal device 10b. The entrance terminal device 10a and the exit terminal device 10b are connected to each other so as to be able to communicate with each other via short-range wireless communication such as Wi-Fi (registered trademark), but this is not limiting, and they may be connected to each other so as to be able to communicate with each other via a communication network N or the like in addition to or instead of short-range wireless communication. The detailed configuration of the entrance terminal device 10a will be described later using FIG. 3 and the like. The detailed configuration of the exit terminal device 10b will be described later using FIG. 6 and the like.

[0040] The management server 200 is a server device that manages passenger information including boarding and alighting information transmitted from the bus terminal device 10, and boarding and alighting information including the number of passengers on the bus 1 and the degree of congestion (described later). The management server 200 can provide the passenger information, boarding and alighting information, and the like to the bus operator terminal device 300. The management server 200 can also provide the boarding and alighting information, and the like to the user terminal device 400. The management server 200 is an example of an external device, and is configured as, for example, a cloud server, but is not limited to this. The detailed configuration of the management server 200 will be described later using FIG. 9.

[0041] The bus operator terminal device 300 is a terminal device used by a bus operator that operates the bus 1. The bus operator terminal device 300 may be, for example, a personal computer (PC), a smartphone, or a tablet terminal, but is not limited to these. Passenger information, boarding and alighting information, etc. are transmitted from the management server 200 to the bus operator terminal device 300 and displayed thereon. This enables the bus operator to create an operation plan for the bus 1 according to, for example, passenger information, boarding and alighting information, etc.

[0042] The user terminal device 400 is a terminal device used by a user using the bus 1, or more precisely, a user who is about to use the bus 1. The user terminal device 400 may be, for example, a smartphone, a tablet device, or a PC, but is not limited to these. The congestion level information of the bus 1 is transmitted from the management server 200 to the user terminal device 400 and displayed thereon. This allows the user to check, for example, the congestion level of the bus 1 before boarding. Note that, for the sake of simplicity, two user terminal devices 400 are shown in FIG. 2, but the number is not limited to this and may be one or three or more.

[0043] <Configuration of the boarding gate terminal device> Next, the configuration of the entrance terminal device 10a will be described with reference to Fig. 3 etc. Fig. 3 is a block diagram showing an example of the configuration of the entrance terminal device 10a. Note that in each block diagram including Fig. 3, only the components necessary for explaining the features of this embodiment are shown as functional blocks, and descriptions of general components are omitted.

[0044] In other words, each component shown in a block diagram such as Figure 3 is a functional concept, and does not necessarily have to be physically configured as shown. For example, the specific form of distribution and integration of each functional block is not limited to that shown, and all or part of it can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0045] 3, the entrance terminal device 10a includes a first camera 11a, a positioning unit 12a, a control unit 20a, and a storage unit 30a. Note that the entrance terminal device 10a may be, but is not limited to, a smartphone or a tablet terminal.

[0046] The first camera 11a is a camera equipped with, for example, a lens and an imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The first camera 11a is located at the boarding entrance 2a of the bus 1 (see FIG. 1) and is also located inside the bus 1 (inside the bus). More specifically, the first camera 11a is located at a position that allows it to capture an image from inside the bus that includes at least the faces of passengers getting on through the boarding entrance 2a. The first camera 11a captures an image of the passenger (hereinafter, may be referred to as a "passenger captured image") and outputs the captured passenger captured image to the control unit 20a.

[0047] The positioning unit 12a acquires position information (e.g., latitude and longitude) of the bus 1. For example, the positioning unit 12a receives radio waves transmitted from a GPS (Global Positioning System) satellite, acquires position information of the bus 1 (hereinafter, may be referred to as "bus position information") based on the received radio waves, and outputs the acquired bus position information to the control unit 20a. Note that the positioning unit 12a may output bus position information acquired when the bus 1 is stopped at a bus stop as the position information of the bus stop where the bus 1 is stopped.

[0048] The control unit 20a includes an acquisition unit 21a, a detection unit 22a, a judgment unit 23a, a registration unit 24a, and an output unit 25a, and includes, for example, a computer having a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), input / output ports, etc., and various circuits.

[0049] The CPU of the computer functions as the acquisition unit 21a, detection unit 22a, determination unit 23a, registration unit 24a, and output unit 25a of the control unit 20a, for example, by reading and executing a program stored in the ROM.

[0050] In addition, at least some or all of the acquisition unit 21a, detection unit 22a, judgment unit 23a, registration unit 24a and output unit 25a of the control unit 20a can be configured using hardware such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

[0051] The storage unit 30a is configured with a storage device such as a nonvolatile memory, a data flash, etc. The storage unit 30a stores person information 31a, passenger information 32a, various programs, and the like.

[0052] The person information 31a is information about the person captured in each of the multiple passenger captured images. Here, the person information 31a will be described with reference to FIG. 4. FIG. 4 is a diagram showing an example of the person information 31a. As shown in FIG. 4, the person information 31a includes items such as "passenger image," "person ID," "feature points," "gender," "age," "age group," "whether or not a mask is worn," and "accuracy," and each item is associated with each other.

[0053] "Passenger image" is information about a captured image of a passenger. In the example shown in FIG. 4, for convenience, the "passenger image" is described abstractly as "image D01," but specific information is stored in "image D01." Hereinafter, other information may also be described abstractly.

[0054] "Person ID" is identification information that identifies the person captured in the passenger image. "Feature points" is feature point information of the face of the person (passenger) captured in the passenger image, and is an example of face information. Feature point information includes, for example, the positions of feature points such as the eyes (inner corners, outer corners, etc.), nose, and mouth on the face of the passenger captured in the passenger image, but is not limited to these.

[0055] "Gender" is information indicating the gender of the passenger shown in the passenger image. "Age" is information indicating the age of the passenger shown in the passenger image, and "Age group" is information indicating the age group of the passenger. "Mask presence / absence" is information indicating whether the passenger shown in the passenger image is wearing a mask. Note that the above-mentioned information such as gender, age, age group, and mask presence / absence are examples of attribute information of a person (passenger).

[0056] The "accuracy" is information indicating the accuracy of the person information detected based on the corresponding captured image (here, the passenger captured image). The "accuracy" is calculated by, for example, the detection unit 22a, which will be described later.

[0057] In the example shown in Figure 4, in the person information detected based on the passenger image "Image D01," the data for the person (passenger) identified by person ID "E01" indicates that the feature point is "Feature Point F01," the gender is "Male," the age is "28," the age group is "Adult," whether or not wearing a mask is "No," and the accuracy is "Medium."

[0058] 4, it is assumed that the passenger images are captured in the order of "Image D01" to "Image D07." It is also assumed that the passenger images "Image D04" to "Image D06" are images in which the faces of people (passengers) are temporarily cut off or blurred (see captured image A2 in FIG. 1).

[0059] 4 shows an example in which a person with person ID "E01" is detected based on passenger images "Image D01" to "Image D03" out of multiple captured images, a person with person ID "E02" is detected based on passenger images "Image D04" to "Image D06", and a person with person ID "E01" is detected again based on passenger image "Image D07". As will be described later, the person with person ID "E01" and the person with person ID "E02" are the same person (passenger).

[0060] Furthermore, when there are multiple pieces of personal information corresponding to the person IDs "E01" and "E02," the content of the personal information may be determined by majority voting, averaging, or the like. As an example, for the personal information corresponding to the person ID "E02," the gender is detected as "male" twice and as "female" once. Such a detection error is caused by the face being cut off in the corresponding passenger images "image D04" to "image D06." In such a case, for example, the gender of the personal information corresponding to the person ID "E02" may be determined to be "male" by majority voting. As another example, the age of the personal information corresponding to the person IDs "E01" and "E02" may be determined by averaging. Note that the above-described methods of determining the content of personal information are merely examples and are not limiting.

[0061] Returning to the explanation of FIG. 3, passenger information 32a is information related to passengers. Here, passenger information 32a will be explained using FIG. 5. FIG. 5 is a diagram showing an example of passenger information 32a. As shown in FIG. 5, passenger information 32a includes items such as "passenger ID," "passenger image," "feature points," "gender," "age," "age group," "whether mask was worn," and "boarding stop," and each item is associated with another.

[0062] "Passenger ID" is identification information that identifies the passenger. "Passenger image," "feature points," "gender," "age group," and "whether or not a mask is worn" are the same as the above-mentioned personal information 31a, so explanations will be omitted here. "Boarding stop" is information that indicates the stop at which the passenger boarded.

[0063] In the example shown in Figure 5, the data for the passenger identified by passenger ID "P01" indicates that the passenger image is "Image D01," the feature point is "Feature point F01," the gender is "Male," the age is "30," the age group is "Adult," whether or not a mask is worn is "No," and the stop at which the passenger boarded is "Stop G01."

[0064] The passenger information for passenger ID "P01" corresponds to the person information for person ID "E01" when it is determined that the person with person ID "E01" and the person with person ID "E02" in person information 31a are the same person, which will be described later.

[0065] Returning to the explanation of FIG. 3, the acquisition unit 21a of the control unit 20a acquires, from the first camera 11a, a plurality of passenger captured images of people (passengers) riding on the bus 1. For example, the acquisition unit 21a acquires the passenger captured images when the bus 1 is stopped at a bus stop, but the timing of acquisition is not limited to this. The acquisition unit 21a outputs the acquired passenger captured images to the detection unit 22a. Note that the passenger captured images are an example of captured images.

[0066] The acquisition unit 21b acquires bus position information from the positioning unit 12a and outputs the acquired information to the registration unit 24a.

[0067] The detection unit 22a detects personal information about the person (passenger) captured in each of the multiple passenger captured images based on the multiple passenger captured images. Specifically, the detection unit 22a detects facial information (more specifically, facial feature point information) of the passenger from the passenger captured images. For example, the detection unit 22a analyzes the passenger captured images to extract feature points such as the eyes (inner corners, outer corners, etc.), nose, mouth, and contours, and detects the positions of the extracted feature points as feature point information.

[0068] Furthermore, the detection unit 22a analyzes the passenger captured images to detect attribute information of the passengers, such as their gender, age, age group, whether they are wearing a mask, etc. In other words, the detection unit 22a detects attribute information about the person (passenger) captured in each of the plurality of passenger captured images, based on the plurality of passenger captured images.

[0069] In addition, the detection unit 22a assigns a person ID to each person detected by detecting feature point information, etc., and stores the person ID, the detected feature point information, and attribute information together with the passenger image in the memory unit 30a as person information 31a.

[0070] Furthermore, the detection unit 22a can calculate the accuracy of the person information detected based on the captured image of the passenger. For example, the detection unit 22a can calculate the accuracy indicating the reliability of the person information based on various information such as the number of feature points detected based on the captured image of the passenger and the clarity of the captured image. For example, the detection unit 22a calculates the accuracy so that the accuracy increases as the number of detected feature points increases. The detection unit 22a adds information indicating the calculated accuracy to the person information 31a and stores it in the storage unit 30a. Note that the above-mentioned detection methods of the feature point information and attribute information and the calculation method of the accuracy of the person information can be set arbitrarily.

[0071] The determination unit 23a determines whether the person captured in each of the multiple passenger captured images is the same person based on the person information, etc. For example, the determination unit 23a reads the person information 31a from the storage unit 30a. Then, the determination unit 23a compares the multiple pieces of person information detected from the multiple captured images.

[0072] As an example, the determination unit 23a compares personal information corresponding to a person with a personal ID "E01" shown in Fig. 4 with personal information corresponding to a person with a personal ID "E02" and determines whether the difference (in other words, error) between the pieces of personal information is within a predetermined allowable range. Note that the above-mentioned predetermined allowable range is set so that the pieces of personal information are similar to each other and can be estimated to be the same person, but is not limited to this.

[0073] Specifically, the determination unit 23a performs a process of determining whether or not a person is the same person based on personal information such as feature point information and attribute information. For example, if the feature point information corresponding to a person captured in each of the multiple passenger captured images is within a predetermined tolerance range, the determination unit 23a can determine that the person is the same person. Furthermore, if the attribute information (e.g., gender, age, age group, presence or absence of a mask, etc.) corresponding to a person captured in each of the multiple passenger captured images is within a predetermined tolerance range, the determination unit 23a can determine that the person is the same person. Then, the determination unit 23a notifies the registration unit 24a of the determination result.

[0074] In this manner, in this embodiment, by using feature point information and attribute information, it is possible to accurately perform the process of determining whether the person is the same. Note that in the process of determining whether the person is the same, it is not necessary to use all of the various types of information included in the personal information, and a configuration may be used in which only some of the information is used, or even a configuration may be used in which other types of information is used.

[0075] The registration unit 24a stores (registers) the passenger information in the storage unit 30a. The registration unit 24a is an example of a storage processing unit.

[0076] For example, the registration unit 24a reads out the person information 31a from the storage unit 30a. Then, the registration unit 24a registers the passenger information based on the read out person information 31a and the determination result of the determination unit 23a. In detail, the registration unit 24a aggregates the person information corresponding to the persons determined to be the same person by the determination unit 23a to generate passenger information (passenger information), and registers the generated passenger information in the storage unit 30a.

[0077] As an example, the registration unit 24a aggregates the personal information corresponding to the person with the person ID "E01" shown in Figure 4 and the personal information corresponding to the person with the person ID "E02" to generate passenger information for one person and registers it in the memory unit 30a (see passenger ID "P01" in Figure 5).

[0078] As a result, in this embodiment, even if the same passenger is detected as a different passenger because, for example, their face is temporarily cut off, it is possible to prevent multiple passenger information from being generated, thereby improving the accuracy of the passenger information.

[0079] Furthermore, the registration unit 24a may generate and register passenger information (passenger information) using personal information selected according to the accuracy information from among the personal information corresponding to the persons determined to be the same person. For example, the registration unit 24a selects personal information with a relatively high accuracy, in other words, relatively probable personal information, from among the personal information corresponding to the persons determined to be the same person. In the example of FIG. 4, personal information with a "high" accuracy is selected. Then, the registration unit 24a may register the selected personal information as passenger information.

[0080] In this manner, in this embodiment, passenger information is registered using personal information selected according to the accuracy information, thereby further improving the accuracy of passenger information.

[0081] Furthermore, the registration unit 24a may generate and register passenger information (passenger information) using the most recently detected personal information among the personal information corresponding to persons determined to be the same person. In other words, the registration unit 24a may generate and register passenger information using the most recent personal information among the personal information corresponding to persons determined to be the same person. In the example of FIG. 4, the registration unit 24a may register the personal information corresponding to passenger image "D07" as passenger information.

[0082] In this way, in this embodiment, passenger information is registered using the last detected person information, so the passenger information is the most up-to-date information, thereby further improving the accuracy of the passenger information.

[0083] In addition, the method of selecting person information based on the above-mentioned accuracy information and the method of using the last detected person information may be combined as appropriate, or these methods may be configured to be used with priority assigned.

[0084] Furthermore, the registration unit 24a may add information about the bus stop where the passenger image was taken, that is, the bus stop where the passenger boarded, to the passenger information 32a based on the bus position information of the positioning unit 12a.

[0085] The output unit 25a accesses the storage unit 30a and outputs the passenger information 32a to the exit terminal device 10b. Note that the output unit 25a executes the output process between the time the bus 1 departs from a bus stop and the time the bus 1 arrives at the next bus stop, but the output timing is not limited to this.

[0086] <Configuration of Exit Terminal Device> Next, the configuration of the exit terminal device 10b will be described with reference to Fig. 6 etc. Fig. 6 is a block diagram showing an example of the configuration of the exit terminal device 10b.

[0087] 6, the exit terminal device 10b includes a second camera 11b, a positioning unit 12b, a control unit 20b, and a storage unit 30b. Note that the exit terminal device 10b may be a smartphone, a tablet terminal, or the like, but is not limited to these.

[0088] The second camera 11b is a camera equipped with, for example, a lens and an imaging element such as a CCD or CMOS. The second camera 11b is located at the exit 2b of the bus 1 (see FIG. 1) and is also located inside the bus 1 (inside the bus). Specifically, the second camera 11b is located at a position that allows it to capture an image from inside the bus that includes at least the face of a passenger (passenger) getting off at the exit 2b. The second camera 11b captures an image of the passenger (hereinafter, may be referred to as an "image of the passenger getting off") and outputs the captured image of the passenger to the control unit 20b.

[0089] The positioning unit 12b acquires the position information of the bus 1 based on radio waves transmitted from a GPS satellite, for example, and outputs the acquired position information to the control unit 20b. Note that the positioning unit 12b may output the position information of the bus 1 acquired when the bus 1 is stopped at a bus stop as the position information of the bus stop where the bus 1 is stopped.

[0090] The control unit 20b includes an acquisition unit 21b, a detection unit 22b, a judgment unit 23b, a registration unit 24b, a matching unit 25b, a calculation unit 26b, and a transmission unit 27b, and includes, for example, a computer having a CPU, ROM, RAM, input / output ports, etc., and various circuits.

[0091] The computer's CPU functions as the acquisition unit 21b, detection unit 22b, judgment unit 23b, registration unit 24b, matching unit 25b, calculation unit 26b and transmission unit 27b of the control unit 20b, for example, by reading and executing a program stored in the ROM.

[0092] In addition, at least some or all of the acquisition unit 21b, detection unit 22b, judgment unit 23b, registration unit 24b, matching unit 25b, calculation unit 26b and transmission unit 27b of the control unit 20b can be configured using hardware such as ASIC or FPGA.

[0093] In addition, since the acquisition unit 21b, detection unit 22b, judgment unit 23b and registration unit 24b of the control unit 20b have the same or similar configurations as the acquisition unit 21a, detection unit 22a, judgment unit 23a and registration unit 24a of the control unit 20a of the entrance terminal device 10a, duplicated explanations may be omitted below.

[0094] The storage unit 30b is a storage unit configured with a storage device such as a non-volatile memory, a data flash, etc. The storage unit 30b stores passenger information 31b, person information 32b, passenger information 33b, boarding and alighting information 34b, various programs, etc.

[0095] The passenger information 31b is the same as the passenger information 32a output from the entrance terminal device 10a, and therefore a description thereof will be omitted here (see FIG. 5).

[0096] The person information 32b is information about the person captured in each of the multiple disembarking passenger captured images. Here, the person information 32b will be described with reference to FIG. 7. FIG. 7 is a diagram showing an example of the person information 32b. As shown in FIG. 7, the person information 32b includes items such as "disembarking passenger image," "person ID," "feature points," "gender," "age," "age group," "whether or not a mask is worn," and "accuracy," and each item is associated with each other.

[0097] "Image of person getting off" is information on the captured image of the person getting off. "Person ID" is identification information that identifies the person captured in the captured image of the person getting off. "Feature points" is feature point information on the face of the person (person getting off) captured in the captured image of the person getting off. "Gender" is information indicating the gender of the person getting off captured in the captured image of the person getting off. "Age" is information indicating the age of the person getting off captured in the captured image of the person getting off, and "Age group" is information indicating the age group of the person getting off. "Mask presence / absence" is information indicating whether the person getting off captured in the captured image of the person getting off is wearing a mask. "Accuracy" is information indicating the accuracy of the person information detected based on the corresponding captured image (here, the captured image of the person getting off).

[0098] In the example shown in Figure 7, in the person information detected based on the disembarking person image "Image D21," the data for the person (disembarking person) identified by person ID "E21" indicates that the feature point is "Feature Point F01," the gender is "Male," the age is "28," the age group is "Adult," whether or not the person is wearing a mask is "No," and the accuracy is "Medium."

[0099] In FIG. 7, it is assumed that the alighting passenger images "Image D22" to "Image D26" are images in which the faces of the people (alighting passengers) are temporarily cut off or blurred. FIG. 7 shows an example in which a person with person ID "E21" is detected based on alighting passenger images "Image D21" to "Image D23" from among the multiple captured images, a person with person ID "E22" is detected based on alighting passenger images "Image D24" to "Image D26", and a person with person ID "E21" is detected again based on alighting passenger image "Image D27". As will be described later, the person with person ID "E21" and the person with person ID "E22" are the same person (passenger).

[0100] Returning to the explanation of FIG. 6, the alighting passenger information 33b is information relating to alighting passengers. Here, the alighting passenger information 33b will be explained using FIG. 8. FIG. 8 is a diagram showing an example of the alighting passenger information 33b. As shown in FIG. 8, the alighting passenger information 33b includes items such as "alighting passenger ID," "alighting passenger image," "feature points," "gender," "age," "age group," "whether mask worn," "route name," "stop where passenger disembarked," and "matching result," and each item is associated with each other.

[0101] "Disembarking passenger ID" is identification information that identifies the disembarking passenger. "Disembarking passenger image," "feature points," "gender," "age group," and "whether or not a mask is worn" are the same as those in the personal information 32b described above, and therefore will not be described here. "Disembarking stop" is information that indicates the stop at which the disembarking passenger boarded the bus. "Matching result" is information that indicates the matching result when the boarding passenger information 31b is matched with the disembarking passenger information 33b by the matching unit 25b described later. The "matching result" includes, for example, information about the boarding passenger ID of the boarding passenger information 31b that matches the disembarking passenger information 33b, but is not limited to this.

[0102] In the example shown in Figure 8, the data for the passenger identified by passenger ID "Q01" indicates that the passenger image is "Image D21," the feature point is "Feature point F01," the gender is "Male," the age is "30," the age group is "Adult," whether or not the mask was worn is "No," the stop where the passenger disembarked is "Stop G11," and the matching result is "Matches passenger ID [P01]."

[0103] Returning to the explanation of Figure 6, boarding and alighting information 34b is information relating to the number of passengers getting on and off bus 1. For example, boarding and alighting information 34b includes boarding and alighting counter 34b1 and congestion level information 34b2. Boarding and alighting counter 34b1 is a counter that counts the current number of passengers on bus 1. In other words, boarding and alighting counter 34b1 can also be said to be passenger number information that indicates the number of passengers.

[0104] The congestion level information 34b2 is information indicating the congestion level of the bus 1, more specifically, the current congestion level of the bus 1. The congestion level is, for example, an index value indicating the degree of congestion on the bus 1. As an example, the congestion level is a value indicated on several levels according to the number of passengers relative to the capacity of the bus 1, but is not limited to this.

[0105] Acquisition unit 21b of control unit 20b acquires, from second camera 11b, a plurality of disembarking passenger captured images of people (disembarking passengers) getting off bus 1, and outputs the images to detection unit 22a. Note that the disembarking passenger captured images are an example of a captured image.

[0106] The acquisition unit 21b acquires bus position information from the positioning unit 12b, and outputs the acquired information to the registration unit 24b.

[0107] The detection unit 22b detects personal information about the person (disembarking person) captured in each of the multiple disembarking person captured images based on the multiple disembarking person captured images. Specifically, the detection unit 22b detects feature point information and attribute information of the disembarking person from the disembarking person captured images. The detection unit 22b also assigns a personal ID to each person detected by detecting the feature point information, etc., and stores the personal ID, the detected feature point information, and the attribute information together with the disembarking person captured images as personal information 32b in the storage unit 30b. The detection unit 22b also calculates the accuracy of the personal information detected based on the disembarking person captured images, adds it to the personal information 32b, and stores it in the storage unit 30b.

[0108] The determination unit 23b determines whether the people captured in the multiple captured images of people getting off are the same person based on the person information, etc. For example, if the feature point information or attribute information corresponding to the people captured in the multiple captured images of people getting off are within a predetermined tolerance range, the determination unit 23b determines that the people are the same person. Then, the determination unit 23b notifies the registration unit 24b of the determination result.

[0109] The registration unit 24b stores (registers) the alighting passenger information (passenger information) in the storage unit 30b. The registration unit 24b is an example of a storage processing unit.

[0110] For example, the registration unit 24b reads out the person information 32b from the storage unit 30b. Then, the registration unit 24b registers the alighting passenger information based on the read out person information 32b and the determination result of the determination unit 23b. In detail, the registration unit 24b aggregates the person information corresponding to the persons determined to be the same person by the determination unit 23b to generate alighting passenger information (passenger information), and registers the generated alighting passenger information in the storage unit 30b.

[0111] As an example, the registration unit 24b aggregates the personal information corresponding to the person with the person ID "E21" shown in Figure 7 and the personal information corresponding to the person with the person ID "E22" to generate information on one person who gets off the train, and registers this information in the memory unit 30b (see the person ID "Q01" in Figure 8).

[0112] As a result, in this embodiment, even if the same passenger is detected as a different passenger because, for example, their face is temporarily cut off, it is possible to prevent multiple pieces of disembarking passenger information from being generated, thereby improving the accuracy of the disembarking passenger information.

[0113] In addition, in the registration unit 24b, similar to the registration unit 24a, for example, the registration unit 24b may select person information according to the accuracy information to generate the disembarking person information, or may generate the disembarking person information using the person information detected last.

[0114] Furthermore, the registration unit 24b may add information about the bus stop where the disembarking passenger image was captured, that is, the bus stop where the disembarking passenger got off, to the disembarking passenger information 33b based on the bus position information of the positioning unit 12b.

[0115] The matching unit 25b matches the boarding passenger information 31b with the alighting passenger information 33b. For example, the matching unit 25b accesses the boarding passenger information 31b and the alighting passenger information 33b in the storage unit 30b and performs a comparison to determine whether the feature point information and attribute information of the alighting passenger match or approximately match with the feature point information and attribute information of the boarding passenger. If the feature point information, etc. of the alighting passenger matches or approximately matches with the feature point information, etc. of the boarding passenger, the matching unit 25b registers the comparison result in the "matching result" of the alighting passenger information 33b (see FIG. 8). Here, the matching unit 25b indicates that the feature point information and attribute information of the alighting passenger with the alighting passenger ID "Q01" match the feature point information and attribute information of the information of the passenger ID "P01."

[0116] Calculation unit 26b calculates boarding and alighting information including the current number of passengers and the degree of congestion on bus 1. For example, calculation unit 26b calculates the number of passengers on bus 1 based on boarding passenger information 31b and alighting passenger information 33b. More specifically, when new feature point information or attribute information of a passenger is registered in passenger information 31b, calculation unit 26b adds one to the number of passengers, more specifically, increments boarding and alighting counter 34b1.

[0117] Furthermore, when information about a passenger disembarking that matches or nearly matches the characteristic point information or attribute information of a passenger is registered in the disembarking passenger information 33b, the calculation unit 26b estimates that the passenger with such characteristic point information has disembarked from the bus 1, and calculates the value obtained by subtracting one disembarking passenger from the number of passengers as the current number of passengers on the bus 1; more specifically, it decrements the boarding / disembarking counter 34b1.

[0118] Calculation unit 26b calculates the current congestion level of bus 1 based on information from boarding / alighting counter 34b1 (i.e., information about the current number of passengers on bus 1) and bus 1 capacity information stored in advance in storage unit 30b. Calculation unit 26b then stores information indicating the calculated congestion level in storage unit 30b as congestion level information 34b2. Note that calculation unit 26b executes the calculation process between the time bus 1 departs from a bus stop and the time it arrives at the next bus stop, but the timing of the calculation is not limited to this.

[0119] The transmitter 27b transmits to the management server 200 passenger information including boarding passenger information 31b and alighting passenger information 33b, and boarding and alighting information including information on the number of passengers on the bus 1, which is the value of the boarding and alighting counter 34b1, and congestion level information 34b2.

[0120] The transmitting unit 27b performs the transmission process between the time when the bus 1 departs from the bus stop and the calculation process etc. is completed by the calculation unit 26b and the time when the bus 1 arrives at the next bus stop, but the timing of transmission is not limited to this.

[0121] <Management Server Configuration> Next, the configuration of the management server 200 will be described with reference to Fig. 9. Fig. 9 is a block diagram showing an example of the configuration of the management server 200.

[0122] 9, the management server 200 includes a communication unit 201, a control unit 210, and a storage unit 220. The communication unit 201 is a communication interface that is connected to the communication network N so as to enable two-way communication, and transmits and receives information to and from the bus terminal device 10, the bus operator terminal device 300, the user terminal device 400, and the like.

[0123] The control unit 210 includes an acquisition unit 211 and a provision unit 212, and includes, for example, a computer having a CPU, ROM, RAM, input / output ports, and various other circuits. The CPU of the computer functions as the acquisition unit 211 and provision unit 212 of the control unit 210, for example, by reading and executing a program stored in the ROM. Furthermore, at least some or all of the acquisition unit 211 and provision unit 212 of the control unit 210 can be configured using hardware such as an ASIC or FPGA.

[0124] The storage unit 220 is configured with a storage device such as a nonvolatile memory, a data flash, etc. The storage unit 220 stores passenger information 221, boarding and alighting information 222, various programs, and the like.

[0125] The passenger information 221 is information (e.g., boarding information and alighting information) about passengers on the bus 1 transmitted from the exit terminal device 10b. The boarding and alighting information 222 is information about the number of passengers who boarded and alighted on the bus 1 (e.g., information about the number of passengers on the bus 1 and congestion level information).

[0126] The acquisition unit 211 of the control unit 210 acquires the information transmitted from the exit terminal device 10b, and stores the acquired information in the storage unit 220 as passenger information 221 and boarding / alighting information 222.

[0127] The providing unit 212 provides various types of information to the user terminal device 400 and the bus operator terminal device 300. For example, the providing unit 212 can provide congestion level information and passenger number information included in the boarding and alighting information 222 to, for example, the user terminal device 400 and the bus operator terminal device 300 via the communication unit 201. Furthermore, the providing unit 212 can provide passenger information 221 to, for example, the bus operator terminal device 300 via the communication unit 201.

[0128] <Control process of information processing system according to embodiment> Next, a processing procedure executed by the information processing system 100 including the bus terminal device 10 according to the embodiment will be described with reference to Fig. 10. Fig. 10 is a diagram showing an example of a processing sequence executed by the information processing system 100 including the bus terminal device 10 according to the embodiment.

[0129] 10, first, the entrance terminal device 10a acquires a plurality of passenger captured images from the first camera 11a (step S100). Next, the entrance terminal device 10a detects person information about the person (passenger) captured in each of the plurality of passenger captured images based on the plurality of passenger captured images (step S101).

[0130] Next, the entrance terminal device 10a executes a process of determining whether or not the person captured in each of the plurality of passenger captured images is the same person, based on the person information (step S102).

[0131] Next, the entrance terminal device 10a aggregates the personal information corresponding to the persons determined to be the same person to generate passenger information, and registers the generated passenger information in the storage unit 30a (step S103).The entrance terminal device 10a then outputs the passenger information to the exit terminal device 10b (step S104).

[0132] The exit terminal device 10b acquires a plurality of images of disembarking passengers from the second camera 11b (step S105). Next, the exit terminal device 10b detects person information about the person (disembarking passenger) captured in each of the plurality of images of disembarking passengers based on the plurality of images of disembarking passengers (step S106).

[0133] Next, the exit terminal device 10b executes a process of determining whether or not the person captured in each of the plurality of exit passenger captured images is the same person, based on the person information (step S107).

[0134] Next, the exit terminal device 10b aggregates the personal information corresponding to the persons determined to be the same person to generate disembarking passenger information, and registers the generated disembarking passenger information in the storage unit 30b (step S108). Here, it is assumed that the passenger information output from the entrance terminal device 10a in the previous process has been registered in the storage unit 30b.

[0135] Next, the exit terminal device 10b compares the boarding passenger information with the alighting passenger information (step S109). For example, the exit terminal device 10b compares whether the feature information and attribute information of the alighting passenger in the alighting passenger information matches or substantially matches the feature information and attribute information of the boarding passenger in the boarding passenger information.

[0136] The exit terminal device 10b calculates boarding and alighting information (i.e., the number of passengers and the degree of congestion on the bus 1) based on the boarding passenger information, alighting passenger information, and the collation result (step S110). When new passenger information is output from the entrance terminal device 10a, the exit terminal device 10b updates the passenger information in the storage unit 30b (step S111).

[0137] Then, the exit terminal device 10b transmits passenger information and boarding / alighting information to the management server 200 (step S112). The management server 200 provides, for example, congestion level information and the number of passengers included in the boarding / alighting information to the bus operator terminal device 300 and the user terminal device 400 (step S113).

[0138] As described above, bus terminal device 10 (an example of an information processing device) according to the embodiment includes acquisition units 21a and 21b, detection units 22a and 22b, determination units 23a and 23b, and registration units (an example of a storage processing unit) 24a and 24b. Acquisition units 21a and 21b acquire multiple captured images of people getting on and off bus 1 (an example of a vehicle) from camera 11 installed at the entrance of bus 1. Detection units 22a and 22b detect personal information about the people captured in each of the multiple captured images based on the multiple captured images acquired by acquisition units 21a and 21b. Determination units 23a and 23b determine whether the people captured in each of the multiple captured images are the same person based on the personal information detected by detection units 22a and 22b. The registration units 24a and 24b aggregate personal information corresponding to persons determined to be the same person by the determination units 23a and 23b to generate passenger information (boarding information, alighting information) regarding passengers on the bus 1, and store the generated passenger information in the storage units 30a and 30b. This makes it possible to improve the accuracy of the passenger information in this embodiment.

[0139] In the above description, the entrance terminal device 10a, the exit terminal device 10b, and the management server 200 each perform various processes, but the devices that perform the various processes are not limited to those described above.

[0140] That is, a part or all of the processing performed in the entrance terminal device 10a may be performed in the exit terminal device 10b or the management server 200. Also, a part or all of the processing performed in the exit terminal device 10b may be performed in the entrance terminal device 10a or the management server 200. Also, a part or all of the processing performed in the management server 200 may be performed in the entrance terminal device 10a or the exit terminal device 10b.

[0141] In the above embodiment, the bus 1 has two entrances, the entrance 2a and the exit 2b, but this is not limiting. For example, the bus terminal device (information processing device) 10 according to this embodiment can be applied to a bus with one entrance or three or more entrances.

[0142] Further advantages and modifications will readily occur to those skilled in the art. Therefore, the invention in its broader aspects is not limited to the specific details and representative embodiments shown and described above. Accordingly, various modifications may be made without departing from the spirit or scope of the general inventive concept as defined by the appended claims and their equivalents. [Explanation of symbols]

[0143] 10 Bus terminal equipment 10a Boarding terminal device 10b Exit terminal equipment 21a,21b Acquisition part 22a, 22b Detector 23a, 23b Judgment section 24a, 24b Registration Department 30a,30b Storage section 100 Information Processing Systems 200 Management Server

Claims

1. A plurality of captured images of people getting on and off the vehicle are acquired from a camera installed at the entrance of the vehicle; Detecting personal information about a person captured in each of the plurality of captured images based on the acquired plurality of captured images; determining whether the person captured in each of the plurality of captured images is the same person based on the detected person information; a control unit that aggregates the personal information corresponding to the persons determined to be the same person to generate passenger information regarding the passengers of the vehicle, and stores the generated passenger information in a storage unit; Equipped with The control unit calculating accuracy information indicating accuracy of the person information detected based on the captured image; generating the passenger information using the personal information selected in accordance with the accuracy information from the personal information corresponding to the person determined to be the same person, and storing the generated passenger information in the storage unit; Information processing device.

2. The control unit Detecting attribute information about a person captured in each of the plurality of captured images as the person information based on the plurality of captured images; determining whether the person captured in each of the plurality of captured images is the same person based on the attribute information; The information processing device according to claim 1 .

3. The control unit generating the passenger information using the last detected personal information among the personal information corresponding to the person determined to be the same person, and storing the generated passenger information in the storage unit; 3. The information processing device according to claim 1.

4. an acquiring step of acquiring a plurality of captured images of people getting on and off the vehicle from a camera provided at a boarding / alighting entrance of the vehicle; a detection step of detecting, based on the plurality of captured images acquired in the acquisition step, personal information about a person captured in each of the plurality of captured images; a determination step of determining whether or not the person captured in each of the plurality of captured images is the same person based on the person information detected in the detection step; a storage processing step of aggregating the personal information corresponding to the persons determined to be the same person in the determination step to generate passenger information regarding the passengers of the vehicle, and storing the generated passenger information in a storage unit; Including, The detecting step calculating accuracy information indicating accuracy of the person information detected based on the captured image; The storage processing step includes: generating the passenger information using the personal information selected in accordance with the accuracy information from the personal information corresponding to the persons determined to be the same person in the determination step, and storing the generated passenger information in the storage unit; Information processing methods.

Images