Ticket gate system, ticket system, and passenger image identification method
The ticket gate system uses location data and image analysis to accurately identify passengers with mobile device malfunctions or invalid tickets, addressing the challenges of crowd differentiation and ensuring efficient fare collection.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- HITACHI LTD
- Filing Date
- 2022-11-28
- Publication Date
- 2026-06-04
Smart Images
Figure 0007870237000001 
Figure 0007870237000002 
Figure 0007870237000003
Abstract
Description
Technical Field
[0001] The present invention relates to a ticket gate system, a passenger ticket system, and a method for identifying a passenger image.
Background Art
[0002] Patent Document 1 discloses collecting a passenger's movement route by positioning a mobile device possessed by the passenger to construct a journey. As a result, a transportation operator can bill a passenger without issuing a dedicated ticket medium or installing a ticket gate, and the passenger can do without managing the dedicated ticket medium.
[0003] Patent Document 2 discloses a technique for easily identifying a person in a video captured from multiple viewpoints by combining a sensor with ambiguous person identification, specifically, the reception intensity of a radio signal received by a mobile terminal possessed by a person, and monitoring the person's behavior. Training data is generated by combining the position of the same person reflected in each image acquired in advance with the sensor data detecting the person acquired at the same time. An estimated value of the sensor data at the position of the person reflected in each video acquired in a continuous time series is calculated based on the training data, and the identification information of the person is specified based on the time series change of the similarity between the estimated value and the acquired sensor data.
Prior Art Documents
Patent Documents
[0004]
Patent Document 1
Patent Document 2
Summary of the Invention
Problems to be Solved by the Invention
[0005] While enabling passengers to use public transport through ticketing applications installed on their mobile devices can increase passenger convenience, it also necessitates dealing with gate malfunctions such as mobile device failures or invalid ticketing applications (hereinafter referred to as possessing invalid tickets).
[0006] The inventors considered using images (video) from cameras installed within the station to identify passengers carrying mobile devices experiencing such gate malfunctions, thereby realizing the gate function. For example, Patent Document 2 discloses a technology for identifying images of people carrying mobile devices. However, to realize the gate function of public transportation, it is necessary to identify passengers carrying mobile devices experiencing gate malfunctions from among a large number of passengers moving in roughly the same direction within a relatively narrow area (passageway). With the technology of Patent Document 2, it is difficult to identify people who are too close together and behaving similarly to be distinguishable by the identification accuracy of the sensor data. [Means for solving the problem]
[0007] A ticket gate system according to one embodiment of the present invention includes: a terminal tracking unit that stores location data indicating the location of a user terminal; a travel itinerary construction unit that identifies the entry station and exit station of a passenger carrying a user terminal from the location data stored in the terminal tracking unit and constructs the travel itinerary data of the passenger; a person tracking unit that identifies and tracks passengers passing through a predetermined ticket gate area from images taken of the ticket gate area within the station premises; and a group of passengers that passed through the ticket gate area in close proximity to each other from the images, and for each group, registers as group data the passenger images of the passengers included in the group and the user IDs corresponding to the user terminals that passed through the ticket gate area during the period in which the group passed through the ticket gate area, which are identified based on the location data stored in the terminal tracking unit. The system includes a group creation unit, and, if it is determined that a ticket gate anomaly has occurred in the group data, a related group data which is created from images of a ticket gate area within a station other than the station where the passenger images of the first group data where the ticket gate anomaly occurred were taken, and an identification unit which, based on the image similarity between the passenger images of the first group data and the passenger images of the related group data, assigns the passenger images to the user ID of the first group data or the related group data, and identifies the passenger images of the passengers who possess the user terminal that caused the ticket gate anomaly. The terminal tracking unit, itinerary construction unit, person tracking unit, group creation unit, and identification unit each execute programs stored in a storage device using a processor. Function . [Effects of the Invention]
[0008] This system provides a ticket gate system that can identify the passenger image of a passenger who possesses a user terminal experiencing a ticket gate malfunction from among a large number of passenger images.
[0009] Further details regarding the problems disclosed in this application, and their solutions, will be made clear in the section on embodiments for carrying out the invention and in the drawings. [Brief explanation of the drawing]
[0010] [Figure 1] This diagram shows a station where the ticket gate system is in operation. [Figure 2A] This is an example of an abnormal ticket inspection for the "invalid ticket type". [Figure 2B] This is a diagram plotting passenger images in the feature space. [Figure 2C] This is a diagram plotting passenger images in the feature space. [Figure 3] This is an example of an abnormal ticket inspection for the "battery dead type". [Figure 4A] This is a functional block diagram of the ticket inspection system. [Figure 4B] This is a block diagram of the information processing device. [Figure 5] This is a flowchart of the ticket inspection function. [Figure 6] This is a flowchart for group creation. [Figure 7] This is a flowchart for the abnormal ticket inspection response process of the "battery dead type". [Figure 8] This is a flowchart for the abnormal ticket inspection response process of the "invalid ticket type". [Figure 9] This is a flowchart of the clustering process. [Figure 10A] This is an example of the data structure of group data (image). [Figure 10B] This is an example of the data structure of group data (user ID). [Figure 11A] This is an example of user assignment information (initial value). [Figure 11B] This is an example of image assignment information (initial value). [Figure 12A] This is an example of user assignment information (processing result). [Figure 12B] This is an example of image assignment information (processing result). [Figure 13] This is an example of the data structure of journey data. [Figure 14] This is an example of the display screen on the operation terminal. [Figure 15] This is an example of the inquiry screen on the user terminal.
Modes for Carrying Out the Invention
[0011] Figure 1 shows the station 1 to which the ticket gate system of this embodiment is applied. Passengers are carrying mobile devices 2 such as smartphones and smartwatches. A ticket application for using the railway is installed on the mobile device 2. Numerous wireless base stations 3 are installed within the station premises or inside the train cars, and the status of the mobile device 2 is constantly monitored by one of the wireless base stations 3 communicating with the mobile device 2. As a result, the itinerary, such as the mobile device 2 entering station X and exiting station Y, is recorded, and the fare corresponding to the itinerary is paid from the fare payment method (e.g., credit card) linked to the ticket application of the holder of the mobile device 2 (passenger).
[0012] By using mobile devices 2 and wireless base stations 3, passengers' locations can be seamlessly tracked and their itineraries understood throughout their journey, eliminating ticket gates at stations. Since ticket gates can hinder the flow of people, eliminating them has the advantage of allowing for smoother entry and exit to and from stations, even with high passenger volumes.
[0013] However, a malfunction in mobile device 2 may prevent communication between mobile device 2 and the wireless base station 3, making it impossible to confirm the itinerary, or the ticket application may be invalid, preventing fare payment. Therefore, some kind of ticket gate function is needed that allows direct access to the passenger holding the mobile device 2 that caused the ticket gate malfunction, instead of a physical ticket gate.
[0014] In this embodiment, a camera 4 is installed in the passageway between the platform and the station entrance, and the ticket gate function is realized by using image information from camera 4 in combination with other methods. Hereinafter, the area captured by camera 4 will be referred to as the ticket gate area. With recent wireless positioning technology, it is possible to distinguish objects with an identification accuracy of approximately 50 cm. Therefore, by comparing the image information captured by camera 4 with the location information of mobile device 2 that has passed through the ticket gate area, it is possible to link images of mobile device 2 and passengers to the extent of the positioning accuracy.
[0015] However, although positioning accuracy has improved, if mobile devices 2 pass through the ticket gate area at a distance below the limit of positioning accuracy due to reasons such as congestion in the station passageway or reduced positioning accuracy due to multipath or obstruction, it becomes impossible to link the passenger images captured by camera 4 to the mobile devices 2 on a one-to-one basis. Even in such cases, it is desirable to be able to identify the passenger image of the mobile device that caused the ticket gate malfunction from the passenger images in the captured images. In the ticket gate system of this embodiment, by analyzing image data captured by camera 4 at a different station together with the position data of the mobile device 2 that passed through the ticket gate area, it becomes possible to identify the owner of a mobile device 2 that passed through the ticket gate area at a certain station at a distance below the limit of positioning accuracy using images.
[0016] Figure 4A shows a functional block diagram of the ticket gate system 10. The ticket gate system 10 functions as a ticketing system by coordinating with terminals and other systems via a network. Figure 4A shows the ticket gate system 10 along with other elements that make up the ticketing system.
[0017] Specifically, the ticket gate system 10 is implemented by an information processing device that primarily includes a processor (CPU) 51, memory 52, storage device 53, input / output device 54, communication device 55, and bus 56, as shown in Figure 4B. The processor 51 functions as a functional unit that provides predetermined functions by executing processing according to a program loaded into the memory 52. The storage device 53 stores data and programs used by the functional unit. The input / output device 54 includes input devices such as a keyboard and pointing device, and output devices such as a display. The communication device 55 enables communication with other information processing devices via a network. These are connected to each other via the bus 56 so that they can communicate with one another. It should be noted that all functions of the ticket gate system 10 do not need to be implemented by a single information processing device; they may be implemented by multiple information processing devices.
[0018] In Figure 4A, the user terminal 32 and the station camera 35 correspond to the mobile device 2 and camera 4 shown in Figure 1, respectively. The location information of the user terminal 32 is calculated by the positioning unit 33. The positioning method of the positioning unit 33 is not particularly limited; it may perform positioning by receiving signals from beacons 5 installed at station 1, or it may perform positioning based on the received signal strength of transmission and reception signals with multiple wireless base stations 3. In addition to received signal strength, positioning may also be performed based on the angle of arrival and estimated arrival time. The positioning unit 33 calculates the location information of the user terminal 32 at a predetermined frequency, and the calculated user terminal location data is stored in the terminal tracking unit 12 of the ticket gate system 10. The itinerary construction unit 13 constructs the itinerary of the passenger carrying the user terminal 32 by identifying the entry station and exit station of the user terminal 32 based on the location data stored in the terminal tracking unit 12. The constructed itinerary data is stored in the itinerary database 14. Although this example shows positioning performed by the user terminal 32, a configuration in which positioning is performed on the base station side is also possible. In this case, the location information of the user terminal 32 is sent from the base station to the terminal tracking unit 12 along with information that identifies the user terminal, and is stored there.
[0019] Figure 13 shows an example of the data structure of the itinerary data 100. The itinerary data 100 includes user ID 101, date of use 102, entry time 103, entry station 104, exit time 105, exit station 106, anomaly details 107, response result 108, settlement date 109, and settlement method 110. User ID 101 is an ID (identification information) uniquely assigned to the user terminal 32. The entry station 104 and exit station 106 are identified based on the location data of the user terminal 32 and registered in the itinerary data 100 along with the date and time. If no gate anomaly occurs, "no anomaly" is registered in the anomaly details 107, and the result of the settlement process performed by the fare settlement system 42 (see Figure 4A) is registered in the settlement date 109 and settlement method 110. On the other hand, if a gate anomaly occurs, the details are registered in the anomaly details 107, and the response result is registered in the response result 108. The itinerary data 100 is updated as needed based on the location data of the user terminal stored in the terminal tracking unit 12. The itinerary data 100 is also updated based on information identified by the identification unit 11, which will be described later.
[0020] Returning to the explanation of Figure 4A, the images (moving images) from the station camera 35 are sent from the station camera 35 to the person tracking unit 15 of the ticket gate system 10, where the person tracking unit 15 extracts the passengers shown in the images. This process can utilize known techniques to identify and track people in an image based on the image's features. As will be described in detail later, the group creation unit 16 stores the image data of multiple passengers who passed through the ticket gate area in close proximity to each other, along with the user IDs of the user terminals 32 that passed through the ticket gate area during the time the images were taken, as group data in the group database 17. The identification unit 11 uses the group data stored in the group database 17 to identify the passenger who possessed the user terminal 32 that caused the ticket gate anomaly, and updates the itinerary data 100 according to the result of this identification.
[0021] Figure 5 shows a flowchart of the ticket gate function performed by the ticket gate system 10. The left side of the flowchart displays the functional parts of the ticket gate system 10 that perform each step.
[0022] First, the identification unit 11 checks whether new group data has been stored in the group database 17 by the group creation unit 16 (S01). If new group data has been stored, the following processing is performed. Details of the group data and how it is created will be described later.
[0023] Compare the number of user IDs \(n\) and the number of passenger image data \(m\) in the newly registered group data (S02). If \(n < m\), identify the passenger image data corresponding to the user ID that is the cause of the discrepancy (S03). Such a ticket gate abnormality is hereinafter referred to as a "battery depletion type" ticket gate abnormality based on its typical cause. On the other hand, if \(n = m\), check whether there is a user ID with an invalid ticket (S04). If \(n = m\) and there is no user ID with an invalid ticket, it is normal and analysis of the group data is unnecessary. On the other hand, if \(n = m\) and there is a user ID with an invalid ticket, identify the passenger image data corresponding to the user ID with the invalid ticket (S05). Such a ticket gate abnormality is hereinafter referred to as an "invalid ticket type" ticket gate abnormality. Based on the results of identifying the user ID and passenger image by step S03 or step S05 for the ticket gate abnormality, update the journey data 100 in the journey database 14 (S06).
[0024] Subsequently, the notification unit 18 makes the following responses according to the type of ticket gate abnormality identified by the identification unit 11. In the case of a "battery depletion type" ticket gate abnormality, after the user terminal is restored, inquire the user terminal whether the complementation performed by the identification unit 11 is correct, and finally determine the entry and exit stations of the journey data 100 based on the response result (S07). The settlement request unit 20 makes a fare claim to the fare settlement system 42 based on the determined journey data, and updates the journey data 100 based on the result (S08).
[0025] On the other hand, in the case of an "invalid ticket type" ticket gate abnormality, display the analysis result on the operation terminal 31 that can be confirmed by the station staff (S09). The station staff identifies the passenger holding the user terminal with an invalid ticket from the passenger image information displayed on the operation terminal 31 based on the image and performs settlement processing. The station staff updates the journey data 100 via the notification unit 18 so as to reflect the result of the settlement processing (S10).
[0026] Fig. 6 shows the group creation flowchart implemented by the group creation unit 16. Figs. 10A and B show examples of group data created by this flow.
[0027] First, an empty new group is created (S11). When a passenger image is detected in the ticket gate area (S12), it is determined whether the difference in the passage time of the last passenger image in that group is greater than or equal to a predetermined value (S13). If it is less than the predetermined value, the passenger image is added to the group; if it is greater than or equal to the predetermined value, the passenger image is not added to the group, and the group is defined. Here, the predetermined value for determining whether or not to add to a group is determined based on the positioning accuracy of the user terminal 32. As a result, images of passengers carrying user terminals 32 that cannot be identified from location information are included in the group. Steps S11 to S14 create group data (images) as shown in Figure 10A. Group 61 has a group ID that identifies the group created in steps S11 to S14, and image 62 has a passenger image ID that identifies the passenger image. For example, it is shown that group I contains four passenger images consisting of IMG1 to IMG4.
[0028] Once a group is defined, the user ID and location data of the user terminals acquired within the range of the group's start time (the time the first passenger in the group passed through the ticket gate area) and end time (the time the last passenger in the group passed through the ticket gate area) (referred to as the passage period) are obtained from the terminal tracking unit 12 (S15). Next, the time-series features of the user terminal location data are calculated, and user terminals with a similarity of a predetermined value or higher are designated as nearby users (S16). In other words, nearby users are users that cannot be distinguished based on location data. Next, the time-series features of the user terminal location estimated from the image data are compared with the time-series features of the user terminal location data, and passenger images with a similarity of a predetermined value or higher are designated as assignable images (S17).
[0029] Steps S15-S17 create group data (user ID) as shown in Figure 10B. Group 66 has the same group ID as group 61 in Figure 10A, user ID 67 is the user ID identified in step S15, nearby user 68 is the user ID of the nearby user identified in step S16, and assignable image 69 is the passenger image ID identified as an assignable image in step S17. The examples of group data in Figures 10A and 10B correspond to the example shown in Figure 2A, so the processing in steps S15-S17 will be explained in accordance with the examples in Figures 10B and 2A.
[0030] In Figures 10B and 2A, user IDs B through E are identified as user IDs for Group I during the passage period. In Figure 2A, the time-series features of each user ID obtained in step S16 are displayed as circles surrounding the alphabet representing the user ID. The time-series features are based on the change in the user ID's position data during the passage period. The closer the user IDs are to each other when passing through the ticket gate area during Group I's passage period, the more similar the time-series features will be. Conversely, the further apart the user IDs are when passing through the ticket gate area, the more different the time-series features will be. Therefore, nearby users who are user terminal owners and cannot be distinguished by position data are identified based on the similarity of the time-series features. In the example of Group I, for instance, the time-series features of user ID B are similar to those of user ID C (the circles overlap), so user ID C is a nearby user of user ID B. In contrast, the time-series features of user ID B are not similar to those of user IDs D and E (the circles do not overlap), therefore user IDs D and E are not neighboring users of user ID B. Similarly, neighboring users can be determined for each user ID included in the group.
[0031] On the other hand, since the user terminal location can also be estimated from the image data, in step S17, the estimated time-series features of the user terminal location are calculated using the location information obtained from the image, similar to step S16. This makes it possible to associate passenger image data with location data. Specifically, in the example of group I, the time-series features of user ID B are compared with the estimated time-series features of passenger image IDs IMG1 to 4, and passenger image IDs with estimated time-series features similar to those of user ID B are identified as assignable images. As shown in Figure 10B, in the example of Figure 2A, passenger image IDs IMG1 to 2 are identified as assignable images for user ID B.
[0032] The group data 60 and 65 obtained in this way are registered in the group database 17 (S18), and the group registration process by the group creation unit 16 is completed. The group creation unit 16 executes the flowchart in Figure 6 each time a passenger image is detected from the images from the station camera 35, and performs the creation of group data and registration in the group database 17.
[0033] Next, we will explain the process that the identification unit 11 executes based on the group data registered in the group database 17 when a ticket gate malfunction occurs.
[0034] Figure 7 shows the flowchart for the ticket gate malfunction response process for the "battery dead type" (step S03 in the flowchart of Figure 5). Figure 3 also shows an example of a ticket gate malfunction for the "battery dead type".
[0035] First, a list of potential related groups that are reachable from or can reach the group where the discrepancy occurs is created based on the location, time, and train schedule of the group where the discrepancy occurs (S21). The train schedule acquisition unit 19 (see Figure 4A) acquires the train schedule from the train management system 41. In the example in Figure 3, the group where the discrepancy occurs is group i. Since group i is the departure station, the potential related groups would be any group that entered a station so that it can reach group i at the time of departure from station Y. Note that, in order to reduce the computational load of subsequent processing, it is not necessary to extract all reachable groups at once. For example, the user's railway usage section can be estimated based on past usage records, and potential related groups can be preferentially extracted from sections with a usage rate above a certain level, and the sections from which potential related groups are extracted can be gradually expanded.
[0036] Next, the abnormality details 107 (see Figure 13) of the travel data 100 for the user IDs included in each related group candidate are obtained, and groups containing user IDs with content other than "no abnormality" are extracted as related groups (S22). Since the user terminal 32 is constantly monitored for its status, if the battery runs out and location information cannot be obtained along the way, this fact is registered in the abnormality details 107 of the travel data 100. Therefore, from the abnormality details registered in the travel data 100, the group ID of the related group containing the user ID that is a candidate for a mismatched user can be extracted from the related group candidates extracted in step S21. In the example in Figure 3, group ii (user ID C is a candidate for a mismatched user) and group iii (user ID G is a candidate for a mismatched user) are extracted as related groups.
[0037] If there is only one candidate for a mismatched user, the single candidate is identified as the mismatched user (S23,24), and the abnormality content 107 of the mismatched user's itinerary data 100 is updated to "Itinerary terminated midway" (S28).
[0038] In contrast, if there are multiple candidates for mismatched users as shown in Figure 3, the mismatched user is identified from the candidates based on the similarity of the passenger images (S23-S27). To evaluate the similarity of the passenger images, several predetermined image features are calculated for each passenger image, and each passenger image is plotted in a feature space spanned by these multiple image features. The similarity of the passenger images is determined by the distance between the passenger images in the feature space. Therefore, for each candidate for mismatched user, the user ID and the user IDs of neighboring users are obtained as clustering targets for determining the similarity of the passenger images (S25), clustering is performed on the passenger images to be clustered, and the user IDs are assigned to the passenger images (S26). Details of this will be described later. As a result, user IDs that are assigned to passenger images in the group where the mismatch occurred (group i in the example of Figure 3) and are not included in that group are identified as mismatched users (S27). Subsequently, the abnormal content 107 of the mismatched user's itinerary data 100 is updated to "itinerary terminated midway" (S28).
[0039] Figure 8 shows a flowchart of the gate error handling process for "invalid ticket type" (step S05 in the flowchart of Figure 5). Figure 2A also shows an example of a gate error for "invalid ticket type".
[0040] First, the group data is monitored, and the system waits until there are two or more groups that contain users holding invalid tickets (S31). In the example in Figure 2A, user ID C in group I is the user ID of the user holding the invalid ticket, and when this user exits from station Y, the group creation unit 16 creates group II, which is the second group that contains users holding invalid tickets.
[0041] Subsequently, all groups containing users holding invalid tickets are identified from the group database 17, and groups containing the user ID of the user holding the invalid ticket or the user IDs of nearby users are designated as related groups (S32). In the example in Figure 2A, group III, which includes user ID A, a nearby user of user ID C in group II, is included as a related group and is targeted for clustering.
[0042] Subsequently, clustering is performed on the passenger images to be clustered, and user IDs are assigned to the passenger images (S33). The details of this are the same as step S26 in the flowchart in Figure 7, and will be described later. For users who have been identified as having invalid tickets, the abnormality details 107 in the itinerary data 100 are updated to "invalid ticket" (S34).
[0043] Figure 9 shows the flowchart for the clustering process (step S26 in the flowchart of Figure 7, and step S33 in the flowchart of Figure 8). In the clustering process, since the passenger image corresponding to a given user ID cannot be directly identified, the similarity of passenger images is used to first identify the passenger images of nearby users, and then the passenger image corresponding to the given user ID is identified by process of elimination.
[0044] First, all passenger images to be clustered are obtained from the group data stored in the group database 17 (see Figures 10A and 10B) (S41), and user assignment information and image assignment information are initialized (S42-S43). The clustering targets are the group data where the ticket gate anomaly occurred and its related group data. Furthermore, it is preferable to narrow down the processing to the user ID corresponding to the user terminal that caused the ticket gate anomaly, the user IDs of nearby users, and the passenger images that can be assigned to them. Figure 11A shows the initialized user assignment information 70, and Figure 11B shows the initialized image assignment information 75. Note that Figures 11A and 11B are based on the example shown in Figure 2A.
[0045] In the user assignment information 70 shown in Figure 11A, the user ID 71 is registered as the user ID of the clustering target obtained in step S25 of the flowchart in Figure 7 or step S32 of the flowchart in Figure 8. The completion status 72 is registered as the assignment status of passenger images in clustering, and the assigned image 73 is registered as the passenger image ID assigned to that user ID. After initialization, the completion status 72 is registered as "incomplete," and the assigned image 73 is empty.
[0046] In the image assignment information 75 shown in Figure 11B, the passenger image ID 76 is registered as the passenger image ID, the image feature 77 is registered as the plotted position of the passenger image in the feature space, the assignment candidate user ID label 78 is registered as a list of user IDs that are candidates for assignment to the passenger image ID, and the assignment count 79 is registered as the number of times a user ID has been assigned to the passenger image ID. During initialization, the contents of the passenger image ID 76, image feature 77, and assignment candidate user ID label 78 are registered based on the group data stored in the group database 17. The assignment candidate user ID label 78 is registered in the format "group-user ID". For example, "IB" indicates "user ID B of group I". The initial value of the assignment count is 0.
[0047] Figure 2B shows the plot of passenger images corresponding to image feature 77 in the feature space. The plot position of image feature 77 in the image assignment information 75 is represented as (image feature 1, image feature 2).
[0048] Returning to the explanation of Figure 9, select one user ID whose completion status 72 in the user assignment information 70 is incomplete (S44). If the selected user ID is included in two or more groups, calculate clusters with an image similarity of a certain value or higher for passenger images with an unassigned user ID and an assignment count of less than one (S48). If the selected user ID is included in only one group (No in S45) or if multiple clusters with two or more elements are obtained in the clusters calculated in step S48 (No in S49), set the passenger image ID that includes the selected user ID in the assignment candidate user ID label 78 as the assignment image for the selected user ID (S46), and set the completion status of the user assignment information 70 to provisional assignment.
[0049] In the example in Figure 2A, user IDs D, F, and G belong to only one group (No in S45). Since there are no other passenger images of the same user corresponding to these user IDs, there are no passenger images that would normally be clustered. However, image features will have similar values if they belong to the same person, but they can also be similar even if they belong to different people. In other words, the false negative rate is low, but the false positive rate is high. For this reason, all possible passenger image IDs based on location information are tentatively assigned to these user IDs, and the corresponding passenger image IDs are identified by removing passenger image IDs that have been confirmed to be assigned to other user IDs. For example, passenger image IDs IMG2,3,4 are mechanically tentatively assigned to image 73 for user ID D.
[0050] Figure 2C shows the clustering results superimposed on the plotted locations of the passenger images in Figure 2B. In this example, four clusters, clusters 81 to 84, are obtained. If there are multiple clusters containing passenger images that may correspond to the selected user ID, it is not possible to narrow down the passenger images corresponding to the selected user ID. Therefore, in this case as well, a provisional passenger image ID that can be assigned (No in S49) is assigned, and the identifiable user ID is identified in advance.
[0051] The user ID can be assigned to an image only if a single cluster with two or more elements is found for the selected user ID. In this case, the user ID is assigned to a passenger image based on the similarity information of the passenger images.
[0052] Therefore, the passenger image ID of the cluster is set to the assigned image 73 of the user assignment information 70 for the selected user ID (S50). If the cluster contains multiple passenger images of the same group, the reciprocal of the number of passenger images of the same group included in the cluster is added to the assignment count 79 of the image assignment information 75 for the passenger images included in the cluster (S51). After that, the user IDs whose assignments have been confirmed are removed from the assignment candidate user ID labels 78 of the image assignment information 75 (S52), and the completion status 72 of the selected user IDs in the user assignment information 70 is set to completed, and the completion status 72 of the provisionally assigned user IDs is set to incomplete (S53), and the process from step S44 to step S54 is repeated until all users are no longer incomplete. The updated user assignment information 70 and image assignment information 75 by this flow are shown in Figures 12A and 12B, respectively.
[0053] In the case of Figure 2C, it will be as follows.
[0054] If the selected user ID is E, the assignable passenger image IDs are IMG3 and IMG4 from group I, and IMG5 and IMG6 from group II. Therefore, passenger image IDs IMG4 and IMG5, which constitute cluster 83, are identified as user ID E.
[0055] If the selected user ID is A, it can be determined that cluster 81, which contains passenger image ID IMG10 from group III, is the cluster containing the passenger image of user ID A. However, it cannot be determined which of the passenger image IDs IMG7 or IMG8, which belong to the same group II, is the passenger image of user ID A. In this case, in step S51, the number of assignments for passenger image IDs IMG7 and IMG8 is set to 0.5. Next, if the selected user ID is B, the cluster containing the passenger image of user ID B must contain either passenger image ID IMG7 or IMG8 from group II. Since cluster 81 is already determined, it is determined that cluster 82 is the cluster containing the passenger image of user ID A. In this case as well, it cannot be determined which of the passenger image IDs IMG7 or IMG8, which belong to the same group II, is the passenger image of user ID B. Therefore, in step S51, 0.5 is added to the number of assignments for passenger image IDs IMG7 and IMG8, resulting in a total number of assignments of 1. In the end, in this case, it is determined that user ID A is either passenger image ID IMG7 or IMG8.
[0056] As a result, although the passenger image for user ID C could initially be passenger image IDs IMG5, 6, or 7, user ID E was assigned to passenger image ID IMG5, and user ID A was assigned to passenger image ID IMG7 or IMG8. This revealed that cluster 84, which includes passenger image ID IMG6, is the passenger image for user ID C. In other words, the passenger images for user IDs with invalid tickets are identified as passenger image ID IMG2 in group I and passenger image ID IMG6 in group II.
[0057] As described above, the user ID of the user terminal that caused the ticket gate malfunction and / or an image of the holder are identified. The results of the actions taken based on this identification are finally reflected in the itinerary data 100 shown in Figure 13. Record 91 is an example in which the results of the station staff's response to a ticket gate malfunction of the "invalid ticket type" are registered (steps S9 and S10 in the flowchart of Figure 5). Record 92 is an example in which, in a ticket gate malfunction of the "battery dead type," the exit station was estimated (in this case, "system restored" is displayed), and after confirmation with the user, the completion of the payment is registered (steps S7 and S8 in the flowchart of Figure 5).
[0058] Figure 14 shows an example of a display screen on the operational terminal 31 to prompt station staff to take action in the event of a ticket gate malfunction. The display screen 120 notifies station staff that a ticket gate malfunction has occurred and prompts them to take direct action. By enabling station staff to directly interact with users when a ticket gate malfunction occurs, it is expected that malicious fare evasion will be deterred.
[0059] The display screen 120 shows the details of the ticket gate error (in this case, possession of an invalid ticket) 121, the user ID 122, the entry and exit stations and their passage times 123, 125, and images 124, 126 taken at the entry and exit stations. For example, in the case of Figure 2A, image 124 is an image from Group I, and image 126 is an image from Group II. Furthermore, images 124 and 126 highlight the passenger images of users who possessed invalid tickets.
[0060] Figure 15 shows an example of an inquiry screen that the notification unit 18 displays to the result confirmation unit 34 of the user terminal 32 to confirm the itinerary when a "battery dead" type ticket gate malfunction occurs. The inquiry screen 130 displays the user's itinerary 131 restored by the ticket gate system 10. If the restored itinerary is correct, the user presses the confirmation button 132; if it is incorrect, they press the correction button 133 and enter the correct itinerary. In this way, by allowing the user to process the itinerary restored by the ticket gate system 10 on the system, counter service is unnecessary, reducing the burden on both the user and the operator.
[0061] Although embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above, and various modifications are included. For example, the embodiments described above are detailed explanations of the configuration in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the configurations described. Furthermore, it is possible to add, delete, or replace some of the configurations of each embodiment with other configurations.
[0062] Furthermore, each of the above-described configurations, functions, processing units, and processing means may be implemented in hardware, either partially or entirely, by designing them as integrated circuits, for example. They can also be implemented by software program code that implements each of the functions shown in the embodiments.
[0063] In the embodiments described above, the control lines and information lines shown are those deemed necessary for illustrative purposes, and not all control lines and information lines are necessarily shown in the actual product. All components may be interconnected. Furthermore, although various types of information have been illustrated in a table format above, this information may be managed in a format other than a table. [Explanation of symbols]
[0064] 1: Station, 2: Mobile device, 3: Wireless base station, 4: Camera, 5: Beacon, 10: Ticket gate system, 11: Identification unit, 12: Terminal tracking unit, 13: Itinerary construction unit, 14: Itinerary database, 15: Person tracking unit, 16: Group creation unit, 17: Group database, 18: Notification unit, 19: Operation schedule acquisition unit, 20: Settlement request unit, 31: Operation terminal, 32: User terminal, 33: Positioning unit, 34: Result confirmation unit, 35: Station premises camera, 41: Operation management system, 42: Fare settlement system, 51: Processor, 52: Memory, 53: Storage device, 54: Input / output device, 55: Communication device, 56: Bus, 60: Group data (image), 61: Group, 62: Image, 65: Group data (User ID), 66: Group, 67: User ID, 68: Nearby user, 69: Assignable Image, 70: User assignment information, 71: User ID, 72: Completion status, 73: Assigned image, 75: Image assignment information, 76: Passenger image ID, 77: Image features, 78: Assignment candidate user ID label, 79: Number of assignments, 81, 82, 83, 84: Clusters, 91, 92: Records, 100: Itinerary data, 101: User ID, 102: Date of use, 103: Entry time, 104: Entry station, 105: Exit Time, 106: Exit station, 107: Details of the anomaly, 108: Result of the response, 109: Settlement date, 110: Settlement method, 120: Display screen, 121: Details of the ticket gate anomaly, 122: User ID, 123: Entry station and time of passage, 124: Image at the entry station, 125: Exit station and time of passage, 126: Image at the exit station, 130: Inquiry screen, 131: Itinerary, 132: Confirm button, 133: Correction button.
Claims
1. A terminal tracking unit that stores location data indicating the location of the user terminal, A travel itinerary construction unit identifies the entry station and exit station of a passenger carrying the user terminal from the location data stored in the terminal tracking unit and constructs the travel itinerary data of the passenger. A person tracking unit identifies and tracks passengers passing through a designated ticket gate area within a station premises from images captured of that area, A group creation unit creates groups of passengers who passed through the ticket gate area in close proximity to each other from the aforementioned images, and for each group, registers as group data the passenger images of the passengers included in the group and the user IDs corresponding to the user terminals that passed through the ticket gate area during the period in which the group passed through the ticket gate area, which are identified based on the location data stored in the terminal tracking unit. If it is determined that a ticket gate anomaly has occurred in the group data, the identification unit identifies related group data which is group data created from images of the ticket gate area within a station other than the station where the passenger images of the first group data in which the ticket gate anomaly occurred were captured, assigns the passenger images to the user ID of the first group data or the related group data based on the image similarity between the passenger images of the first group data and the passenger images of the related group data, and identifies the passenger images of the passengers who possess the user terminal that caused the ticket gate anomaly. The ticket gate system is characterized in that the terminal tracking unit, the itinerary construction unit, the person tracking unit, the group creation unit, and the identification unit each function by executing a program stored in a storage device using a processor.
2. In claim 1, The ticket gate system is characterized in that the group creation unit calculates a first time-series feature for the period during which the group passed through the ticket gate area from the location data of the user terminal corresponding to the user ID registered as group data, and a second time-series feature for the period during which the group passed through the ticket gate area from the location information of the passenger image estimated from the image, and if the similarity between the first time-series feature and the second time-series feature is greater than or equal to a predetermined value, the passenger image for which the second time-series feature was calculated is identified as an assignable image for the user ID for which the first time-series feature was calculated.
3. In claim 1, The ticket gate anomaly in the first group data mentioned above is the possession of an invalid ticket. The ticket gate system is characterized in that the identifying unit identifies, in addition to the first group data, group data including a user ID possessing an invalid ticket or a user ID corresponding to a user terminal that is indistinguishable from the user terminal corresponding to the user ID possessing the invalid ticket based on the location data, as the related group data.
4. In claim 1, The ticket gate anomaly in the first group data is a mismatch between the number of passenger images and the number of user IDs in the first group data. The ticket gate system is characterized in that the identifying unit identifies as the related group data group data a group data created from images of the ticket gate area within a station other than the station where the passenger images of the first group data were taken, which are passengers of the first group data, and which include a user ID corresponding to a user terminal where an abnormality has occurred in the travel itinerary data.
5. In claim 4, The ticket gate system is characterized in that the specified unit updates the itinerary data with the station where the passenger image of the passenger holding the user terminal that caused the ticket gate malfunction was captured as the exit station.
6. In claim 1, The ticket gate system is characterized in that the group creation unit groups together passengers who pass through the ticket gate area within a time difference of less than a predetermined value from the passenger who passed through the ticket gate area immediately before, and the time difference for grouping together is determined based on the positioning accuracy of the user terminal's location data.
7. Ticket gate system, It has a fare settlement system that is connected to the aforementioned ticket gate system and settles fares, The aforementioned ticket gate system is A terminal tracking unit that stores location data indicating the location of the user terminal, A travel itinerary construction unit identifies the entry station and exit station of a passenger carrying the user terminal from the location data stored in the terminal tracking unit and constructs the travel itinerary data of the passenger. A person tracking unit identifies and tracks passengers passing through a designated ticket gate area within a station premises from images captured of that area, A group creation unit creates groups of passengers who passed through the ticket gate area in close proximity to each other from the aforementioned images, and for each group, registers as group data the passenger images of the passengers included in the group and the user IDs corresponding to the user terminals that passed through the ticket gate area during the period in which the group passed through the ticket gate area, which are identified based on the location data stored in the terminal tracking unit. If it is determined that a ticket gate anomaly has occurred in the group data, the identification unit identifies related group data which is group data created from images of the ticket gate area within a station other than the station where the passenger images of the first group data in which the ticket gate anomaly occurred were captured, assigns the passenger images to the user ID of the first group data or the related group data based on the image similarity between the passenger images of the first group data and the passenger images of the related group data, and identifies the passenger images of the passengers who possess the user terminal that caused the ticket gate anomaly. A settlement request unit that requests fares from the fare settlement system based on passenger itinerary data, It has, A ticketing system characterized in that the terminal tracking unit, the itinerary construction unit, the person tracking unit, the group creation unit, the identification unit, and the settlement request unit each function by executing a program stored in a storage device using a processor.
8. In claim 7, The station staff has an operational terminal that can be used for verification. The ticketing system is characterized in that the operating terminal displays the passenger image of the passenger who possesses the user terminal that caused the ticket gate abnormality identified by the specific unit, in a manner that makes it distinguishable from other passenger images.
9. A method for identifying passenger images performed by a ticket gate system comprising a terminal tracking unit, a travel itinerary construction unit, a person tracking unit, a group creation unit, and a identification unit, The terminal tracking unit stores location data indicating the location of the user terminal. The itinerary construction unit identifies the entry station and exit station of the passenger carrying the user terminal from the location data stored in the terminal tracking unit and constructs the passenger's itinerary data. The aforementioned person tracking unit identifies and tracks passengers passing through a designated ticket gate area from images taken of that area within the station premises. The group creation unit creates groups of passengers who passed through the ticket gate area in close proximity to each other from the images, and for each group, it registers the passenger images of the passengers included in the group and the user IDs corresponding to the user terminals that passed through the ticket gate area during the period in which the group passed through the ticket gate area, which are identified based on the location data stored in the terminal tracking unit, as group data. If the identification unit determines that a ticket gate anomaly has occurred in the group data, it identifies related group data which is group data created from images of the ticket gate area within a station other than the station where the passenger images of the first group data in which the ticket gate anomaly occurred were captured, and assigns the passenger images to the user ID of the first group data or the related group data based on the image similarity between the passenger images of the first group data and the passenger images of the related group data, and identifies the passenger images of the passengers who possess the user terminal that caused the ticket gate anomaly. A method for identifying passenger images, characterized by the features described above.
10. In claim 9, The passenger image identification method is characterized in that the group creation unit calculates a first time-series feature for the period during which the group passed through the ticket gate area from the location data of the user terminal corresponding to the user ID registered as group data, and a second time-series feature for the period during which the group passed through the ticket gate area from the location information of the passenger image estimated from the image, and if the similarity between the first time-series feature and the second time-series feature is greater than or equal to a predetermined value, the passenger image for which the second time-series feature was calculated is identified as an assignable image for the user ID for which the first time-series feature was calculated.