Information processing device, information processing method, and program

Abstract

[Problem] To provide an information processing device, information processing method, and program capable of accurately distinguishing a person reflected in a captured image. [Solution] An information processing device provided with a control unit that executes a control for associating a person in a captured image with a user on the basis of a comparison between movement information acquired from information detected by an information processing terminal and movement information for one person or more acquired from a captured image of a user carrying said information processing terminal, and a comparison between status information for the user acquired from the information processing terminal and status information for the one person or more acquired from the captured image.

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In recent years, there is a demand for indoor positioning technology for use in marketing research, personalizing customer experiences, and the like. For example, there is a technology that uses inertial sensors such as an acceleration sensor, an angular velocity sensor, and a geomagnetic sensor to obtain relative movement information both indoors and outdoors. For example, the following Patent Document 1 discloses an indoor positioning method that calculates position information using a movable camera installed indoors and a mobile terminal equipped with a PDR (Pedestrian Dead Reckoning) function carried by a pedestrian. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6340862 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the above-mentioned Patent Document 1, in order to cover a wide range, a movable camera is rotated in the direction of position information obtained from a mobile terminal of a pedestrian to capture images, but if the position information contains an error, it is difficult to control it correctly. Also, no consideration is given to matching people in the field of view with the user.

[0005] In view of this, the present disclosure proposes an information processing device, an information processing method, and a program capable of accurately identifying a person appearing in a captured image. [Means for solving the problem]

[0006] According to the present disclosure, an information processing device is proposed that includes a control unit that performs control to match a person appearing in an captured image with the user based on a comparison between movement information obtained from information detected by an information processing terminal and movement information of one or more people obtained from an captured image of a user carrying the information processing terminal, and a comparison between status information of the user obtained from the information processing terminal and status information of the one or more people obtained from the captured image.

[0007] According to the present disclosure, an information processing method is proposed, which includes a processor controlling matching of people appearing in a captured image with the user based on a comparison between movement information obtained from information detected by an information processing terminal and movement information of one or more people obtained from a captured image of a user carrying the information processing terminal, and a comparison between status information of the user obtained from the information processing terminal and status information of the one or more people obtained from the captured image.

[0008] According to the present disclosure, a program is proposed to cause a computer to function as a control unit that controls matching of people appearing in an captured image with the user based on a comparison between movement information obtained from information detected by an information processing terminal and movement information of one or more people obtained from an captured image of a user carrying the information processing terminal, and a comparison between status information of the user obtained from the information processing terminal and status information of the one or more people obtained from the captured image. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an overview of an information processing system according to an embodiment of the present disclosure. [Diagram 2] 11A and 11B are diagrams illustrating movement vector matching according to the present embodiment. [Diagram 3] FIG. 13 is a diagram illustrating a case where a determination cannot be made only by comparing movement vectors. [Figure 4] 1 is a diagram illustrating an example of a configuration of an information processing system according to an embodiment of the present invention. [Diagram 5] FIG. 2 is a block diagram showing an example of the configuration of an environment camera according to the present embodiment. [Figure 6] FIG. 2 is a block diagram showing an example of the configuration of a user terminal according to the present embodiment. [Figure 7] FIG. 2 is a block diagram showing an example of a configuration of a server according to the present embodiment. [Figure 8] FIG. 2 is a sequence diagram showing an example of the flow of operation processing of the information processing system according to the present embodiment. [Figure 9] 10 is a flowchart showing an example of the flow of operational processing of the environment camera according to the present embodiment. [Figure 10] 10 is a flowchart showing an example of the flow of operational processing of a server according to the present embodiment. [Figure 11] 13 is a diagram showing an example of a screen for notifying a user of authentication information registration according to the present embodiment. FIG. [Figure 12] FIG. 13 is a diagram showing an example of a deletion screen for authentication information of a user according to the present embodiment. [Figure 13] 11 is a diagram showing an example of a setting screen for automatically registering user positioning and authentication information according to the present embodiment. FIG. [Figure 14] 10 is a flowchart showing an example of the flow of a matching process according to the present embodiment. [Figure 15] 11 is a diagram illustrating the timing of a user's walking steps obtained from an image captured by an environmental camera according to this embodiment and the timing of the user's walking steps obtained from sensing data of a user terminal. FIG. [Figure 16] 10 is a flowchart showing an example of a flow of a walking step extraction process in a user terminal according to the present embodiment. [Figure 17] 10 is a flowchart showing an example of the flow of a walking step extraction process in the environmental camera according to the present embodiment. [Figure 18] 10 is a flowchart showing an example of a flow of a walking step matching process in the server according to the present embodiment. [Figure 19] 11A and 11B are diagrams illustrating the matching of the mutual positional relationship according to the present embodiment. [Figure 20] 11A and 11B are diagrams illustrating another example of the comparison of the mutual positional relationship according to the present embodiment. [Figure 21] 11A and 11B are diagrams illustrating another example of the comparison of the mutual positional relationship according to the present embodiment. [Figure 22] FIG. 11 is a diagram illustrating the matching of behavioral information according to the present embodiment. [Figure 23] 1A and 1B are diagrams illustrating movement pattern matching according to the present embodiment. [Figure 24] 11 is a diagram illustrating recognized actions and services provided to a user according to an application example of this embodiment. FIG. [Diagram 25] 11 is a diagram illustrating a payment process using authentication information according to an application example of this embodiment. FIG. [Figure 26] FIG. 11 is a sequence diagram showing an example of the flow of a payment process using authentication information according to an application example of this embodiment. [Figure 27] 11 is a diagram illustrating a payment process using location information and a PIN number according to an application example of this embodiment. FIG. [Figure 28] FIG. 13 is a diagram showing an example of a location information share screen according to an application example of this embodiment. [Figure 29] 13 is a diagram showing an example of an operation screen for selecting an arbitrary user from an image of inside a facility and notifying the user of information according to an application example of this embodiment. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configurations are denoted by the same reference numerals, and redundant description will be omitted.

[0011] The description will be given in the following order: 1. Overview of information processing system according to one embodiment of the present disclosure 2. Configuration Example 3. Example of operation processing 3-1. Operational processing of information processing systems 3-2. Operation process of the environment camera 20 3-3. Operation process of server 10 4. Matching process and calculation of match confidence score 4-1. Matching process flow 4-2. Matching movement vectors and calculating matching confidence scores 4-3. Matching walking steps and calculating matching confidence score 4-4. Matching of positional relationships and calculation of matching confidence score 4-5. Matching behavioral information and calculating match reliability score 4-6.Matching movement patterns and calculating match reliability scores 4-7. Aggregation of Matching Confidence Score 5. Application Examples 6. Summary

[0012] <<1. Overview of Information Processing System According to One Embodiment of the Present Disclosure>> FIG. 1 is a diagram for explaining an overview of an information processing system according to an embodiment of the present disclosure. In this embodiment, it is possible to detect the movement trajectory of a user who visits a facility such as a shopping mall with high accuracy using data from a sensor provided on an information processing terminal carried by the user and images from one or more environmental cameras installed in the facility. In addition, by identifying a user in the facility by matching the movement trajectory, it is possible to detect the movement trajectory of a visitor in the facility while maintaining the anonymity of the user. The movement trajectory is a movement route, and more specifically, includes position information (e.g., global coordinates) and time information.

[0013] 1, for example, a user terminal 30A (an example of an information processing terminal) carried by a user A calculates a movement vector of the user A as a movement trajectory based on sensing data, for example, by Pedestrian Dead Reckoning (PDR) (an example of indoor positioning). Note that in this example, the alphabet A is added after the reference symbol to indicate that the user terminal 30 is carried by the user A, but the configuration of the user terminal 30A is the same as that of the user terminal 30 described later.

[0014] On the other hand, the environmental camera 20 installed in the facility calculates the movement vector of the user within the angle of view by, for example, detecting the face or body from the captured image. In this embodiment, for example, the environmental camera 20 may not intentionally capture the target user, but may assume a state in which the user is reflected in the image captured by the environmental camera 20. Alternatively, it is assumed that the environmental camera 20 tracks and captures a specific face, or a camera of a self-propelled robot tracks and captures a specific person, and so on. In this embodiment, the camera installed in the facility continuously captures the surrounding environment, so it is called an "environmental camera." The "environmental camera" is also called a surveillance camera. The environmental camera is not limited to a fixed camera, and may be a camera mounted on a moving object such as a self-propelled robot or a drone. A wide-angle lens may be used for the camera.

[0015] Then, by comparing the movement vectors acquired by the user terminal 30A and the environmental camera 20, it becomes possible to associate the person appearing in the captured image with the user terminal 30A carried by the person. This allows, for example, a user ID (an example of an identification character string for identifying a user, e.g., a numerical value) acquired from the user terminal 30A to be associated with authentication information (information that can identify a person from their appearance) such as a face image 400 detected from the captured image. Then, it becomes possible to provide various services, such as a payment process, using the authentication information. User A's own authentication information is automatically registered simply by walking around the facility, so that the user A does not have to take the trouble of registering it himself. The automatically registered authentication information is deleted when the user A leaves the facility, so that the privacy of the user A can be protected.

[0016] In addition, the PDR used as an indoor positioning means may accumulate errors over time due to the nature of calculating relative movement vectors. In this embodiment, the absolute position of the user terminal (i.e., the user) indoors can be detected with high accuracy by combining the results of analyzing the video from the environmental camera.

[0017] In addition, facilities may include a variety of locations other than shopping malls, such as supermarkets, convenience stores, bookstores, restaurants, apparel stores, retail stores, spas (hot bathing facilities), libraries, community centers, schools, stations, hotels, hospitals, indoor / outdoor event venues, movie theaters, resort facilities, sports facilities, amusement parks, theme parks, parking lots, parks, or shrines and temples.

[0018] (Matching movement vectors) Here, the comparison of the movement vector will be described. First, the user terminal 30 calculates the movement information (specifically, the movement trajectory) of the user (carrying the user terminal 30) based on sensing data detected by a motion sensor such as an acceleration sensor, an angular velocity sensor, and a magnetic sensor. The motion sensor may include an IMU (Inertial Measurement Unit) that detects three-axis acceleration and angular velocity. More specifically, the movement vector is calculated by, for example, a PDR. The PDR is a positioning means that can obtain relative movement information regardless of whether indoors or outdoors, based on acceleration, angular velocity, and magnetism. On the other hand, the environmental camera 20 performs face detection, body detection, feature point detection, posture detection, etc. (i.e., moving body detection) from the captured video for each frame, and calculates the relative movement trajectory from the change in position (i.e., movement) of the same person (same moving body) captured in each frame. If multiple people are captured in the captured image, the movement trajectory of each person is calculated.

[0019] Then, by comparing the movement trajectories, it becomes possible to link one or more people captured within the angle of view to the user terminal 30 carried by the person. FIG. 2 is a diagram for explaining the matching of movement vectors according to this embodiment. In the matching of movement vectors, as shown in FIG. 2, for example, when there are a movement vector calculated using PDR in the user terminal 30 and movement vectors A, B, and C calculated from the captured image in the environmental camera 20, the movement vector most similar to the movement vector derived from the PDR is determined. The similarity may be determined, for example, using cosine similarity. Also, it is not limited to cosine similarity, and known techniques can be applied to the matching.

[0020] However, as shown in FIG. 3, when multiple people move in the same direction at the same time within the angle of view of the environmental camera 20, the calculated movement vectors A and B will be almost the same, and it may be difficult to determine which one is correct by comparison alone.

[0021] Therefore, in the present disclosure, in addition to matching of the movement vector, matching of the user's state information is performed to enable accurate judgment. Matching of state information is, for example, matching of walking steps, matching of behavior information, matching of mutual positional relationship, etc. Details will be described later.

[0022] Furthermore, the system according to the present disclosure realizes a positioning function that can correctly match individuals (individual user terminals) even if multiple users are present within the same field of view. In addition, by using this system, it is possible to automatically register authentication information (for example, face information, skeletal information (height and other characteristics related to body structure), clothing information, etc.), and various services can be provided to users.

[0023] The information processing system according to an embodiment of the present disclosure has been described above. Next, the configuration of the information processing system according to the present embodiment and each device included in the information processing system will be described with reference to the drawings.

[0024] <<2. Configuration Example>> <2-1. System configuration example> FIG. 4 is a diagram showing an example of the configuration of an information processing system according to this embodiment. As shown in FIG. 4, the information processing system according to this embodiment includes an environmental camera 20, a user terminal 30, and a server 10. For example, a plurality of environmental cameras 20 are installed in a facility, and the installation locations are known to the server 10. Each environmental camera 20 has a camera ID (an example of an identification character string for identifying each camera). The user terminal 30 is an information processing terminal carried by a user who visits the facility. The user terminal 30 may be, for example, a smartphone, a mobile phone terminal, a tablet terminal, or a wearable device attached to the user's body. The user may register in advance with this system using the user terminal 30, and the user ID may be known to the server 10. Registration in this system may be performed by installing a predetermined application in the user terminal 30, or may be performed on a predetermined website.

[0025] The server 10 receives the camera ID and the analysis result of the captured image (movement vector of each person in the angle of view, status information, etc.) from the environmental camera 20 via a network 40 (for example, a dedicated line within the facility). The server 10 also receives the user ID and the analysis result of the sensing data (movement vector based on the PDR, status information, etc.) from the user terminal 30 via a network 41 (for example, Wi-Fi (registered trademark), Bluetooth (registered trademark), Internet line). The server 10 also compares the movement vector and the status information to accurately obtain the user's movement route within the facility. The server 10 can notify the user terminal 30 of appropriate information according to the user's position and behavior, or grant a coupon, thereby making it possible to carry out promotions. In addition, the marketing effect can be measured based on the user's movement route and behavior within the facility. In addition, the server 10 can automatically register authentication information such as a face image extracted from the captured image in association with the user ID. This allows the server 10 to provide various services within the facility through face authentication, etc. For example, if the user's credit card information has been registered in advance, payment can be made within the facility using facial recognition (without using the user terminal 30).

[0026] 4 is merely an example, and the present embodiment is not limited to this. For example, the server 10 may be composed of a plurality of devices. Furthermore, the network 40 and the network 41 may be the same network.

[0027] Next, the configuration of each device included in the information processing system according to this embodiment will be specifically described.

[0028] <2-2. Example of the configuration of the environmental camera 20> 5 is a block diagram showing an example of the configuration of the environmental camera 20 according to this embodiment. As shown in FIG. 5, the environmental camera 20 has a control unit 200, a beacon transmission unit 210, an imaging unit 220, a distance measurement unit 230, a communication unit 240, and a storage unit 250.

[0029] (Control unit 200) The control unit 200 functions as an arithmetic processing unit and a control unit, and controls the overall operation of the environment camera 20 in accordance with various programs. The control unit 200 is realized by electronic circuits such as a CPU (Central Processing Unit) and a microprocessor. The control unit 200 may also include a ROM (Read Only Memory) that stores the programs to be used, arithmetic parameters, etc., and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.

[0030] In addition, the control unit 200 according to this embodiment also functions as a human body region extraction unit 201, a moving body extraction unit 202, a movement vector calculation unit 203, a walking step extraction unit 204, a positional relationship extraction unit 205, a behavior information extraction unit 206, and an authentication information extraction unit 207.

[0031] The human body region extraction unit 201 extracts a region in which a human body is shown for each captured image (each frame of a moving image) acquired from the imaging unit 220. Next, the moving body extraction unit 202 compares a plurality of frames, extracts a moving body based on the human body region data, and assigns a moving body number. For example, the moving body extraction unit 202 compares each frame arranged in time series based on face and body feature points detected from the human body region, extracts a moving person (moving body), and assigns a moving body number. Such moving body extraction can be performed for every t seconds of video. The t seconds may be, for example, a time period corresponding to the transmission interval of a beacon periodically transmitted from the beacon transmission unit 210. More specifically, it may be, for example, a time period equal to the transmission interval, a time period half the transmission interval, or a time period including a plurality of transmission intervals.

[0032] The motion vector calculation unit 203 calculates the motion vector of each moving object based on the captured image. For example, the motion vector calculation unit 203 extracts the change in the position of each moving object in the video for t seconds, converts the trajectory connecting these into global coordinates, and calculates it as a motion vector. The motion vector calculation unit 203 outputs motion vector information for each moving object number.

[0033] The walking step extraction unit 204 analyzes the timing of the walking steps of each moving object in the video for t seconds (for example, the timing of the foot touching the ground), and outputs walking step information for each moving object number.

[0034] The positional relationship extraction unit 205 analyzes the positional relationship (mutual positional relationship) of multiple moving objects captured in a captured image. The positional relationship extraction unit 205 may analyze the positional relationship of the multiple moving objects by analyzing the captured image, or may analyze the positional relationship of the multiple moving objects by referring to distance information indicating the distance between each moving object and the environmental camera 20 acquired by the distance measurement unit 230. The positional relationship extraction unit 205 may also analyze changes in the positional relationship in t seconds of video. The positional relationship extraction unit 205 outputs mutual positional relationship information (with surrounding moving objects) for each moving object number.

[0035] The behavior information extraction unit 206 extracts behavior information of multiple moving objects captured in a captured image (also referred to as behavior recognition). The behavior information extraction unit 206 analyzes the captured image and extracts behaviors such as standing still, walking, running, stairs, elevators, and escalators. A known technique may be applied to extract the behavior information. The behavior information extraction unit 206 outputs the behavior information for each moving object number.

[0036] The authentication information extraction unit 207 extracts information that can be used to authenticate a user (information that allows a user to be identified from appearance; referred to as authentication information in this specification) from the captured image. For example, information on the face, body, bone structure, clothing, or walking style may be extracted as authentication information. The authentication information may be quantified as features. The authentication information extraction unit 207 outputs authentication information for each mobile unit number.

[0037] The control unit 200 controls the communication unit 240 to transmit the above-described movement vector information, walking step information, mutual positional relationship information, and behavior information for each moving object number, together with the camera ID and the image capture time, to the server 10. The timing of transmission is not particularly limited. The control unit 200 may also transmit the captured image to the server 10.

[0038] (Beacon transmission unit 210) The beacon transmitter 210 has a function of transmitting a beacon, which is an example of a signal for triggering the user terminal 30. Here, as an example, a BLE (Bluetooth Low Energy) beacon is transmitted. There are no particular limitations on the timing of transmitting the beacon, but for example, the beacon transmitter 210 transmits continuously at a predetermined interval. In addition, it is desirable that the beacon reaches at least the user terminal 30 located within the angle of view of the environmental camera 20.

[0039] (Imaging unit 220) The imaging unit 220 has a lens system including an imaging lens, an aperture, a zoom lens, and a focus lens, a drive system for performing focusing and zooming operations on the lens system, a solid-state imaging element array for photoelectrically converting imaging light obtained by the lens system to generate an imaging signal, etc. The solid-state imaging element array may be realized by, for example, a CCD (Charge Coupled Device) sensor array or a CMOS (Complementary Metal Oxide Semiconductor) sensor array.

[0040] (Distance measurement section 230) The distance measuring unit 230 has a function of acquiring distance information indicating the distance to the user terminal 30. The distance measuring method is not particularly limited, and may be, for example, distance measurement using RSSI of Bluetooth or distance measurement using UWB (Ultra-Wide Band). The distance measuring unit 230 may calculate the distance, or the distance measuring unit 230 may receive distance information calculated on the user terminal 30 side. The distance measuring unit 230 may also be a depth sensor or the like capable of acquiring the distance to the moving body (user).

[0041] (Communication unit 240) The communication unit 240 is a communication module for transmitting and receiving data to and from other devices. For example, the communication unit 240 connects to the server 10 via the network 40 and transmits and receives data thereto.

[0042] (Storage unit 250) The storage unit 250 stores programs and the like for the control unit 200 to execute various processes. The storage unit 250 is configured with a storage device including a storage medium, a recording device for recording data in the storage medium, a reading device for reading data from the storage medium, and a deleting device for deleting data recorded in the storage medium. The storage unit 250 may be realized with a ROM (Read Only Memory) for storing programs and arithmetic parameters used in the processing of the control unit 200, and a RAM (Random Access Memory) for temporarily storing parameters that change as appropriate.

[0043] The configuration of the environmental camera 20 according to this embodiment has been specifically described above. Note that the configuration of the environmental camera 20 according to this embodiment is not limited to the example shown in FIG. 5. For example, the environmental camera 20 does not need to have the distance measuring unit 230. In addition, the environmental camera 20 may have at least one of the walking step extraction unit 204, the positional relationship extraction unit 205, and the behavior information extraction unit 206.

[0044] Also, at least one of the various processes performed by the control unit 200 of the environmental camera 20 may be configured to be performed by the server 10. For example, the control unit 200 may perform processes up to the process by the moving object extraction unit 202, and transmit the processing results and the captured image to the server 10, and the server 10 may perform processes by the movement vector calculation unit 203, the walking step extraction unit 204, the positional relationship extraction unit 205, the behavior information extraction unit 206, and the authentication information extraction unit 207.

[0045] <2-3. Example of the configuration of the user terminal 30> Fig. 6 is a block diagram showing an example of the configuration of a user terminal 30 according to this embodiment. As shown in Fig. 6, the user terminal 30 has a control unit 300, a satellite positioning unit 310, a beacon receiving unit 320, a distance measuring unit 330, a motion sensor unit 340, a communication unit 350, a display unit 360, an operation input unit 370, and a storage unit 380.

[0046] (Control unit 300) The control unit 300 functions as an arithmetic processing unit and a control unit, and controls the overall operation of the user terminal 30 in accordance with various programs. The control unit 300 is realized by electronic circuits such as a CPU (Central Processing Unit) and a microprocessor. The control unit 300 may also include a ROM (Read Only Memory) that stores the programs to be used, arithmetic parameters, etc., and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.

[0047] The control unit 300 according to this embodiment also functions as a movement vector calculation unit 301, a walking step extraction unit 302, and a behavior information extraction unit 303.

[0048] The movement vector calculation unit 301 calculates a movement vector of a user (a person carrying the user terminal 30) based on sensing data from the motion sensor unit 340. More specifically, the movement vector calculation unit 301 may calculate the movement vector by PDR (an example of indoor positioning) that estimates a movement trajectory from the number of steps, stride length, and direction of travel based on the acceleration, angular velocity, and magnetic data obtained by the motion sensor unit 340.

[0049] The walking step extraction unit 302 extracts the timing of the walking steps of the user (the person carrying the user terminal 30) based on the sensing data of the motion sensor unit 340. The timing of the walking steps is the time when the foot touches the ground, and for example, a walking cycle may be obtained from the three-axis acceleration obtained by the motion sensor unit 340, and the maximum value of the acceleration in each walking cycle may be detected as the landing timing. Details will be described later.

[0050] The behavior information extraction unit 303 extracts behaviors of the user (person carrying the user terminal 30), such as standing still, walking, running, stairs, elevators, and escalators, based on the sensing data of the motion sensor unit 340. For example, the behavior information extraction unit 303 can recognize the user's behavior based on the acceleration, angular velocity, magnetism, air pressure, and the like obtained by the motion sensor unit 340. Known technologies can be applied as the behavior recognition algorithm.

[0051] When the control unit 300 determines that the user has entered a facility of interest from the position information acquired by the satellite positioning unit 310, it continuously calculates the movement vector, extracts the walking steps, and extracts the behavior information, and controls the transmission to the server 10. Such control may be performed by an application installed in advance in the user terminal 30 for use of this system. The position information of the facility of interest, the contents of the data to be transmitted, etc. may be stored in the storage unit 380.

[0052] (Satellite positioning unit 310) The satellite positioning unit 310 receives GPS (Global Positioning System) and other GNSS (Global Navigation Satellite System) signals, and detects the location where the user terminal 30 is located based on the received signals. The satellite positioning unit 310 is an example of an outdoor positioning unit. In addition to the GNSS, the position may be detected by, for example, Wi-Fi, Bluetooth, or the like. The satellite positioning unit 310 outputs the detected position information to the control unit 300. Based on the position information, the control unit 300 can grasp that the user has entered a target facility.

[0053] (Beacon receiving unit 320) The beacon receiving unit 320 includes: Environmental Camera 20The control unit 300 has a function of receiving a beacon transmitted from the environmental camera 20. The beacon is an example of a signal that triggers the control unit 300 to start a predetermined control. The reception of the beacon specifies a time period to be checked by the server 10, which will be described later. The beacon may include the camera ID (e.g., MAC address) of the environmental camera 20 that transmitted the beacon. For example, the control unit 300 may obtain data on the distance to the environmental camera 20 and the distance to other user terminals in the vicinity by the distance measurement unit 330 in response to the reception of the beacon, and may perform control to transmit the data to the server 10 together with the camera ID, the reception time of the beacon, and the user ID contained in the received beacon. Note that when the control unit 300 loses reception of the beacon, it may perform control to transmit to the server 10 the distance information (continuously) obtained during that time together with the start and end times of reception of the beacon, the camera ID, and the user ID. Even when the control unit 300 is acquiring distance information and controlling transmission in response to receiving a beacon, it continues to calculate movement vectors within the facility, extract walking steps, and extract behavioral information, and transmits this to the server 10 together with time information and the user ID.

[0054] (Distance measurement section 330) The distance measuring unit 330 measures the distance between the environment camera 20 and the user terminal 30, and also the distance between the user terminal 30 and other user terminals located in the vicinity of the user terminal 30. The distance measuring method is not particularly limited, and may be, for example, distance measuring using RSSI of Bluetooth or distance measuring using UWB (Ultra-Wide Band). The distance between the environment camera 20 and the user terminal 30 may be calculated by the distance measuring unit 330, or the distance measuring unit 330 may receive distance information calculated on the environment camera 20 side. Based on such distance information between the environment camera 20 and other user terminals, the server 10 can grasp the relative positions between the user and the environment camera 20, or between the user and other users.

[0055] (Motion sensor unit 340) The motion sensor unit 340 is a sensor for capturing human movement. For example, the motion sensor unit 340 includes an acceleration sensor, an angular velocity sensor, a geomagnetic sensor, and an air pressure sensor. The motion sensor unit 340 may also have a sensor capable of detecting a total of nine axes, including a three-axis gyro sensor, a three-axis acceleration sensor, and a three-axis geomagnetic sensor. The motion sensor unit 340 outputs data (sensing data) detected by each sensor to the control unit 300.

[0056] (Communication unit 350) The communication unit 350 is a wireless communication module for transmitting and receiving data to and from other devices. For example, the communication unit 350 is wirelessly connected to a network 41 and transmits and receives data to and from the server 10 via the network 41. The wireless communication method is not particularly limited, but any communication method such as wireless LAN (Local Area Network), Bluetooth (registered trademark), Wi-Fi (registered trademark), LTE (Long Term Evolution), 3G (third generation mobile communication method), 4G (fourth generation mobile communication method), and 5G (fifth generation mobile communication method) can be used.

[0057] (Display section 360) The display unit 360 has a function of displaying various operation screens, notification screens, and the like. The display unit 360 is realized by a display device such as a liquid crystal display (LCD) or an organic electroluminescence (EL) display. When the user terminal 30 is a head mounted display (HMD), the display unit 360 may be a transmissive display. The display unit 360 may be a laser scanning display such as a retinal direct imaging display. The display unit 360 may include an imaging optical system that enlarges and projects the display screen to form an enlarged virtual image having a predetermined angle of view on the user's pupil.

[0058] The display unit 360 can display notification contents (such as registration notification or deletion notification of authentication information) received by the user terminal 30 from the server 10, coupon information, product information, advertisements, etc. The server 10 can provide information from the facility the user has entered.

[0059] (Operation input unit 370) The operation input unit 370 receives an operation instruction from a user and outputs the operation content to the control unit 300. The operation input unit 370 may be, for example, a button, a switch, a touch sensor, or the like. The operation input unit 370 may also receive an operation instruction by detecting the user's voice, movement, gestures, or the like.

[0060] (Storage unit 380) The storage unit 380 stores programs and the like for the control unit 300 to execute various processes. The storage unit 380 is configured with a storage device including a storage medium, a recording device for recording data in the storage medium, a reading device for reading data from the storage medium, and a deleting device for deleting data recorded in the storage medium. The storage unit 380 may be realized with a ROM (Read Only Memory) for storing programs and arithmetic parameters used in the processing of the control unit 300, and a RAM (Random Access Memory) for temporarily storing parameters that change as appropriate.

[0061] The configuration of the user terminal 30 according to this embodiment has been specifically described above. 6 For example, the user terminal 30 may have at least one of a walking step extraction unit 302 and a behavior information extraction unit 303.

[0062] Also, at least one of the various processes performed by the control unit 300 of the user terminal 30 may be configured to be performed by the server 10. For example, the control unit 300 may transmit data (sensing data) detected by the motion sensor unit 340 together with the detection time to the server 10, and the server 10 may perform the processes of the movement vector calculation unit 301, the walking step extraction unit 302, and the behavior information extraction unit 303.

[0063] Furthermore, the user terminal 30 may further include sensors such as a camera, an infrared sensor, a biosensor (pulse, heart rate, sweat, blood pressure, body temperature, respiration, electromyogram, and brain wave), a microphone, etc. For example, the behavioral information extraction unit 303 may extract behavioral information (behavior recognition) using sensing data from these sensors.

[0064] The user terminal 30 may be realized by a plurality of devices. For example, the user terminal 30 may include a display device (corresponding to at least the display unit 360) realized by an HMD or the like, and an information processing terminal (corresponding to at least the control unit 300) realized by a smartphone, a tablet terminal, a wearable device, or the like.

[0065] Although the display unit 360 has been given as an example of an information presenting unit, the present embodiment is not limited thereto, and various notifications to the user (notification of registration of authentication information, notification of deletion, notification of advantageous information such as coupons that can be used at the facility, advertisement of the facility, etc.) may be performed by audio output from an audio output unit (not shown). The audio output unit may be provided in the user terminal 30, or may be configured separately from the user terminal 30. Also, audio may be output from a speaker provided in the facility. The speaker provided in the facility may be a directional speaker. In the case of a directional speaker, audio can be output toward the position of the user to notify the user of predetermined information. Also, information presentation may be performed by combining display output and audio output. Also, whether or not authentication information has been registered, notification of deletion of authentication information, etc. may be performed by a light-emitting unit or a vibration unit provided in the user terminal 30. Various notifications can be performed by the light-emitting color and light-emitting pattern (blinking, etc.) of the light-emitting unit, the vibration pattern, etc.

[0066] <2-4. Example of Server 10 Configuration> 7 is a block diagram showing an example of the configuration of the server 10 according to the present embodiment. As shown in FIG.

[0067] The control unit 100 functions as an arithmetic processing unit and a control unit, and controls the overall operation of the server 10 in accordance with various programs. The control unit 100 is realized by electronic circuits such as a CPU (Central Processing Unit) and a microprocessor. The control unit 100 may also include a ROM (Read Only Memory) that stores the programs to be used, arithmetic parameters, etc., and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.

[0068] The control unit 100 stores each data received from the environmental camera 20 and the user terminal 30 via the communication unit 110 in the storage unit 120. Specifically, the control unit 100 performs control to store information received from the environmental camera 20 (camera ID, time information, captured image (video), various information obtained by analyzing the captured image (movement vectors of each moving object shown in the image, walking steps, behavior information, mutual positional relationship, authentication information, etc.)) in a captured image DB (database) 121. In addition, the control unit 100 performs control to store information received from the user terminal 30 (user ID, time information, movement vector information (e.g. PDR position information), walking step information, behavior information, etc.) in a route history DB 122.

[0069] The control unit 100 also functions as a movement vector matching unit 101, a walking step matching unit 102, a positional relationship matching unit 103, a behavioral information matching unit 104, a movement pattern matching unit 105, an association unit 106, an information registration unit 107, and a route update unit 108.

[0070] The movement vector collation unit 101 compares the movement vector obtained from the user terminal 30 with the movement vector of the moving object obtained from the environmental camera 20. Specifically, the movement vector collation unit 101 compares the movement vector at the time when the user terminal 30 receives the beacon from the environmental camera 20 (for example, t seconds from the start to the end of reception) with the movement vector of each moving object acquired from the captured image at that time captured by the environmental camera 20 that transmitted the beacon. As described with reference to FIG. 2, the movement vector collation may be performed by calculating the similarity between the movement vector of the user terminal 30 and the movement vector of each moving object, and determining the most similar movement vector. It can be said that the moving object showing the most similar movement vector is the user carrying the user terminal 30.

[0071] The walking step collation unit 102 compares the walking steps obtained from the user terminal 30 with the walking steps of the moving object obtained from the environmental camera 20. Specifically, the walking step collation unit 102 compares the walking steps at the time when the user terminal 30 receives a beacon from the environmental camera 20 (for example, t seconds from the start to the end of reception) with the walking steps of each moving object acquired from the captured image at the time captured by the environmental camera 20 that transmitted the beacon. Details of the walking step collation process will be described later. Also, the walking steps are an example of "state information" in this embodiment. As described with reference to FIG. 3, when multiple moving objects (people) move in the same direction at the same time within the same angle of view, it is difficult to distinguish the user only by the above-mentioned movement vector collation. Therefore, in this embodiment, the accuracy of the discrimination can be improved by discriminating the user by taking into account not only the movement vector collation but also the state information (for example, walking steps).

[0072] The positional relationship matching unit 103 matches the positional relationship obtained from the user terminal 30 (the mutual positional relationship with the environmental camera 20 or other surrounding user terminals) with the positional relationship of the moving body obtained from the environmental camera 20 (the distance to each moving body or the mutual positional relationship between each moving body and the moving bodies around it). The mutual positional relationship may include position information (e.g., global coordinates) and distance information. More specifically, the positional relationship matching unit 103 matches the positional relationship at the time when the user terminal 30 receives a beacon from the environmental camera 20 (e.g., t seconds from the start to the end of reception) with the positional relationship of each moving body acquired from the captured image at that time captured by the environmental camera 20 that transmitted the beacon. Details of the positional relationship matching process will be described later. The positional relationship is an example of "state information" in this embodiment. As described with reference to FIG. 3, when multiple moving bodies (people) move in the same direction at the same time within the same angle of view, it becomes difficult to identify the user only by matching the movement vectors. Therefore, in this embodiment, the user is identified not only by matching the movement vectors but also by matching state information (for example, positional relationship), thereby making it possible to improve the accuracy of identification.

[0073] The behavior information matching unit 104 matches the behavior information obtained from the user terminal 30 with the behavior information of the moving object obtained from the environmental camera 20. Specifically, Behavioral information matching unit 104 The user terminal 30 compares the behavior information at the time when the user terminal 30 receives the beacon from the environmental camera 20 (for example, t seconds from the start to the end of reception) with the behavior information of each moving object acquired from the captured image at that time captured by the environmental camera 20 that transmitted the beacon. Details of the behavior information comparison process will be described later. In addition, the behavior information is an example of "status information" in this embodiment. As described with reference to FIG. 3, when multiple moving objects (people) move in the same direction at the same time within the same angle of view, it is difficult to identify the user only by comparing the movement vectors. Therefore, in this embodiment, the accuracy of the identification can be improved by not only comparing the movement vectors but also comparing the status information (for example, behavior information) to identify the user.

[0074] The movement pattern matching unit 105 matches the user's movement pattern based on the information obtained from the user terminal 30 with the movement pattern of each moving object based on the information (integrated information) obtained from the multiple environmental cameras 20. The movement pattern is, for example, a movement pattern that fits a preset rule (such as a route set using map information within a facility).

[0075] The movement pattern matching unit 105 extracts a movement pattern by integrating a movement vector at the time when the user terminal 30 receives a beacon from the first environmental camera (for example, t seconds from the start of reception to the end) and a movement vector at the time when the user terminal 30 receives a beacon from a second environmental camera adjacent (located nearby) to the first environmental camera (for example, t seconds from the start of reception to the end). The movement pattern matching unit 105 also extracts a movement pattern of each moving object by integrating the movement vectors of each moving object obtained from the captured images captured by the adjacent first and second environmental cameras. Note that, for identification of each moving object extracted from the captured images captured by the adjacent first and second environmental cameras (searching for the same person), for example, external features such as face information, skeletal information, clothing information, or walking style information of each moving object are used.

[0076] Details of the movement pattern matching process will be described later. The movement pattern is an example of "state information" in this embodiment. As described with reference to FIG. 3, when multiple moving objects (people) move in the same direction at the same time within the same angle of view, it becomes difficult to identify the user only by matching the movement vectors. Therefore, in this embodiment, the user is identified not only by matching the movement vectors but also by matching state information (e.g., movement patterns), thereby making it possible to improve the accuracy of the identification.

[0077] Based on the above-mentioned matching result, the matching unit 106 matches the person (moving object) captured in the captured image acquired by the environmental camera 20 with the user who owns the user terminal 30. For example, the matching unit 106 may calculate a matching reliability score indicating the reliability of each match and perform the matching by referring to the calculated matching reliability score. Specifically, the matching unit 106 adds up the matching reliability scores for each moving object, determines the moving object with the largest sum as the user, and matches the moving object with the user.

[0078] Furthermore, the matching unit 106 may match users based on at least one of the matching results, or may match users based on at least two or more of the matching results. For example, the matching unit 106 may match users based on matching movement vectors and matching state information (e.g., at least one of walking steps, positional relationship, behavior information, and movement pattern). State information includes information that can be recognized from the outside (i.e., information that can be observed from the outside, such as movement and position). The state information is stored in the user terminal. 30 The state information may be extracted from various sensing data. The state information may be extracted from the captured image of the environmental camera 20 by image analysis. In this embodiment, walking steps, positional relationships (coordinate positions, distance information, etc.), behavior information, and movement patterns are used as examples of state information, but the state information according to the present disclosure is not limited to these. In addition, the association unit 106 may perform association based on matching of one or more pieces of state information when the user cannot be identified (the target moving object cannot be specified) by only matching the movement vector. In addition, the association unit 106 may integrate each matching result and determine the moving object that has been matched most (successfully matched) as the user.

[0079] The information registration unit 107 registers authentication information (such as face information) of a person (moving object) associated with a user in the authentication information DB 123 as authentication information of the user. Specifically, the information registration unit 107 registers a user ID and authentication information. The authentication information to be registered is assumed to be authentication information acquired by the environmental camera 20 where matching was successful. If authentication information cannot be acquired by the environmental camera 20 where matching was successful due to occlusion or the like, an environmental camera 20 that the user has passed through in the past may be estimated from the data of the user ID in the route history DB 122, and authentication information acquired by the environmental camera 20 may be registered. Similarly, for a user who has been successfully matched at a certain timing, authentication information acquired from an image of an environmental camera 20 that the user has passed through after that may be registered. The information registration unit 107 may also notify the user terminal 30 that the authentication information has been registered.

[0080] The camera dedicated to acquiring authentication information and the environmental camera for matching movement vectors, etc. may be separate. The camera dedicated to acquiring authentication information may be installed in the same place as the environmental camera for matching, or may be installed in one or more places (for example, the entrance of the facility, etc.) where more visitors' faces can be photographed. For example, authentication information may be acquired at the time of entry, and when matching using movement vectors, etc. is completed, the authentication information acquired at the time of entry can be registered in association with the user ID. The performance specifications of the camera for acquiring authentication information may be different from those of the environmental camera for matching movement vectors, such as narrowing the angle of view or using a high-magnification, high-resolution camera. For example, the time period during which the user passed near the camera dedicated to acquiring authentication information may be identified from the route history of the user ID for which matching was successful, or the time period during which the user passed based on the time period during which a beacon was received from the camera dedicated to acquiring authentication information may be identified, and authentication information acquired from the camera dedicated to acquiring authentication information during that time period (more accurate authentication information) may be registered in the authentication information DB 123. If multiple pieces of authentication information are acquired from a camera dedicated to acquiring authentication information during the time period in which the user passed by, the correct authentication information may be acquired by matching the authentication information of the user acquired from the verification camera.

[0081] In addition, the camera dedicated to obtaining authentication information may be installed on a drone that can measure its position accurately using UWB, etc. The drone flies around the building and photographs visitors.

[0082] On the other hand, if the matching is not successful, the information registration unit 107 registers in the undetermined information DB 124 the user ID and camera ID for which matching is not successful (that is, for which correspondence cannot be established).

[0083] Furthermore, the information registration unit 107 may update the authentication information stored in the authentication information DB 123 based on the latest data. Furthermore, the information registration unit 107 may increase the authentication information stored in the authentication information DB 123 based on the latest data.

[0084] The route update unit 108 performs data update by merging the user's past movement route stored in the route history DB 122 with the newly acquired movement route, and by performing position correction processing after the user is associated. As described above, PDR position information (i.e., movement vector) is continuously transmitted to the server 10 from the user terminal 30 of the user who has entered the facility. The route update unit 108 merges the acquired PDR position information and generates the user's movement route within the facility. Also, as described above, due to the nature of PDR, which is the calculation of a relative movement vector, errors may accumulate over time. Therefore, in this embodiment, when the association unit 106 identifies (associates) the user, the route update unit 108 corrects the movement route stored in the route history DB 122 based on the movement vector (movement vector of the associated moving object) obtained from the environmental camera 20. This makes it possible to generate a more accurate movement route.

[0085] After the user's authentication information (face information, etc.) is automatically registered, the route update unit 108 can identify the user based on the authentication information from the captured image obtained from the environmental camera 20 and correct the travel route.

[0086] Furthermore, the control unit 100 according to this embodiment may further perform control to notify the user of various information from the facility. For example, the control unit 100 may perform control to transmit appropriate information from the communication unit 110 to the user terminal 30 according to the user's movement route and behavior. The information may be transmitted to a terminal address registered in advance.

[0087] (Communication unit 110) The communication unit 110 is a communication module for transmitting and receiving data to and from other devices. For example, the communication unit 110 connects to one or more environmental cameras 20 (cameras installed at various locations within a facility) via a network 40 to transmit and receive data. The communication unit 110 also connects to one or more user terminals 30 (user terminals carried by each user moving within the facility) via a network 41 to transmit and receive data.

[0088] (Storage unit 120) The storage unit 120 stores programs and the like for the control unit 100 to execute various processes. The storage unit 120 is configured with a storage device including a storage medium, a recording device for recording data in the storage medium, a reading device for reading data from the storage medium, and a deleting device for deleting data recorded in the storage medium. The storage unit 120 may be realized with a ROM (Read Only Memory) for storing programs and arithmetic parameters used in the processing of the control unit 100, and a RAM (Random Access Memory) for temporarily storing parameters that change as appropriate.

[0089] The storage unit 120 according to this embodiment stores a captured image DB 121, a route history DB 122, an authentication information DB 123, an undetermined information DB 124, and an in-facility map information DB 125. The captured image DB 121 stores data obtained from the environmental cameras 20 (camera ID, time information, captured images, movement vector information, walking step information, behavior information, authentication information, etc.). The route history DB 122 stores data obtained from the user terminal 30 (user ID, time information, movement vector information, walking step information, behavior information, distance information, received camera ID, etc.). The authentication information DB 123 stores user IDs and authentication information in association with each other. The undetermined information DB 124 stores user IDs and camera IDs that have not been successfully matched. The in-facility map information DB 125 stores map information within the facility. The map information may include installation location information of each environmental camera 20.

[0090] The configuration of the server 10 according to the present embodiment has been specifically described above. Note that the configuration of the server 10 according to the present embodiment is not limited to the example shown in Fig. 7. For example, the server 10 may be realized by a plurality of devices.

[0091] In addition, the server 10 may have each function of the control unit 200 of the environment camera 20 and each function of the control unit 300 of the user terminal 30.

[0092] <<3. Operation processing example >> Next, the operation process according to this embodiment will be specifically described.

[0093] <3-1. Operational processing of information processing system> FIG. 8 is a sequence diagram showing an example of the flow of operation processing of the information processing system according to this embodiment.

[0094] As shown in FIG. 8, first, the user terminal 30 acquires location information based on satellite positioning from the satellite positioning unit 310 (step S103).

[0095] Next, when it is determined that the user has entered the target facility based on the position information (step S106 / Yes), the control unit 300 activates the motion sensor unit 340 (step S109). position The information may be stored in advance in the storage unit 380. Note that such an activation trigger for the motion sensor unit 340 is just an example, and it is also assumed that the motion sensor unit 340 is already activated. The control unit 300 may also activate the motion sensor unit 340 in response to a signal (such as a BLE beacon) received from a device installed at the entrance of the target facility or the like.

[0096] Next, the control unit 300 calculates a movement vector and extracts walking steps and behavior information based on the sensing data obtained from the motion sensor unit 340 (step S112). The calculation of the movement vector and the extraction of the walking steps and behavior information can be performed continuously while moving within the facility.

[0097] Next, the control unit 300 performs control to transmit the movement vector (PDR position information), walking steps, and behavior information together with the acquisition time and the user ID to the server 10 (step S115). The timing of transmission to the server 10 is not particularly limited, but may be, for example, at regular time intervals.

[0098] Next, the server 10 stores the information received from the user terminal 30 in the route history DB 122 (step S118).

[0099] Meanwhile, one or more environmental cameras 20 installed in the facility continuously capture images of the surroundings, and are controlled to periodically transmit a predetermined beacon (step S121).

[0100] Next, the environmental camera 20 transmits the information acquired from the captured image to the server 10 (step S124). Specifically, the environmental camera 20 performs control to transmit to the server 10 the movement vector, walking step, positional relationship, behavior information, etc. of each moving object acquired from the captured image, and authentication information, together with the captured image, time information, camera ID, etc.

[0101] Next, the server 10 stores the information received from the environmental camera in the captured image DB 121 (step S126).

[0102] Next, in response to receiving the beacon from the environmental camera 20, the user terminal 30 performs control to transmit to the server 10 time information indicating the time period during which the beacon was received (for example, the start and end times of reception of the beacon. Since the user is moving, the beacon may be lost when the user moves away from the environmental camera 20), the camera ID, the user ID, and the distance measurement information contained in the beacon (step S127). The distance measurement information is information acquired by the user terminal 30 through the distance measurement unit 330 using reception of the beacon from the environmental camera 20 as a trigger. The distance measurement information is, for example, information on the distance between the user terminal 30 and the environmental camera 20, or information on the distance between the user terminal 30 and other user terminals located in the vicinity.

[0103] Next, the control unit 100 of the server 10 performs various checks based on the information received from the environmental camera 20 and the information received from the user terminal 30, identifies (associates) the user, and registers the authentication information (of the associated user) in the authentication information DB 123 (step S130). The server 10 can perform user authentication within the facility using the authentication information. The server 10 can also provide various services such as payment processing by user authentication. The device that provides these services and performs user authentication may be an information processing device or system separate from the server 10.

[0104] Next, the information registration unit 107 corrects and updates the route history (movement route history) stored in the user's route history DB 122 based on the user's movement vector acquired from the captured image of the environmental camera 20 (step S133). This corrects errors in the movement route based on the PDR, and improves the accuracy of the movement route. The server 10 can provide the user with appropriate information based on the user's position and movement route within the facility, predetermined actions at predetermined places, etc.

[0105] Then, when the user terminal 30 detects that the user has left the target facility (step S136 / Yes), it requests the server 10 to delete the data associated with the user's user ID (step S139). The fact that the user has left the target facility may be determined based on, for example, a signal (such as a BLE beacon) received from a device provided at the facility exit, or based on location information acquired by the satellite positioning unit 310.

[0106] When the server 10 receives the request, it deletes all information linked to the user ID from the captured image DB 121, the route history DB 122, the authentication information DB 123, and the undetermined information DB 124 (step S142). Note that, for example, if the user approves, the server 10 deletes the link with the user ID from the route history DB 122 and leaves the information of the travel route as a history while maintaining anonymity, which can be used for measuring marketing effectiveness, etc.

[0107] In addition, as an example, the process in which the user terminal 30 automatically requests the deletion of data linked to the user ID when the user leaves the target facility has been described here, but the present embodiment is not limited to this. For example, the user may manually request the deletion using the user terminal 30.

[0108] An example of the overall flow of the information processing system according to the present embodiment has been described above. Next, each operation process of the information processing system according to the present embodiment will be specifically described in order.

[0109] <3-2. Operation process of the environmental camera 20> Fig. 9 is a flow chart showing an example of the flow of the operation process of the environmental camera 20 according to this embodiment. As shown in Fig. 9, first, the control unit 200 of the environmental camera 20 acquires images (video) captured by the imaging unit 220 for t seconds (step S203). The captured video may be stored in the storage unit 250 and may be retrieved by the control unit 200 as needed.

[0110] Next, the human body region extraction unit 201 extracts people (human body regions) from each frame (each still image constituting a video for t seconds) (step S206).

[0111] Next, the moving object extraction unit 202 extracts a moving object based on a comparison of each frame (step S209). Specifically, the moving object extraction unit 202 compares each frame arranged in time series, recognizes the movement of the extracted person, and extracts it as a moving object.

[0112] Next, the moving object extraction unit 202 assigns a moving object number to the extracted moving object (step S212).

[0113] Next, the motion vector calculation unit 203 calculates the motion vector of each moving object based on the captured image (step S215).

[0114] Next, the walking step extraction unit 204 extracts the walking steps of each moving object based on the captured image (step S218).

[0115] Next, the positional relationship extraction unit 205 extracts the relative positions of the moving objects based on the captured images (step S 221 For example, the positional relationship extraction unit 205 may grasp the perspective relationship from the size of each moving body shown in the captured image, and extract the relative positional relationship between each moving body and the relative positional relationship between each moving body and the environmental camera 20. Furthermore, the positional relationship extraction unit 205 may extract the positional relationship between each moving body by performing trilateration based on the distance between each moving body and the environmental camera 20.

[0116] Next, the authentication information extraction unit 207 extracts authentication information of each moving object from the captured image (step S224).

[0117] and, Environmental Camera 20 The server 10 transmits the movement vector, walking steps, relative positional relationship, and authentication information of each moving body to which the moving body number has been assigned, together with the camera ID and time information (step S227). These pieces of information are stored in the captured image DB 121 of the server 10. The server 10 compares the newly stored information with the relative positional relationship of each moving body in the video of the previous time period acquired from the environmental camera 20 of the camera ID, which is already stored in the captured image DB 121, and performs a process of linking the moving body numbers of moving bodies considered to be the same moving body.

[0118] The above describes an example of the operation process of the environment camera 20. Note that the operation process of the environment camera 20 does not necessarily have to be performed in the order shown in Fig. 9. For example, the processes shown in steps S215 to S224 may be performed in a different order or in parallel.

[0119] <3-3. Operational process of server 10> Next, an operational process of the server 10 according to this embodiment will be described below. Fig. 10 is a flow chart showing an example of the flow of the operational process of the server 10 according to this embodiment.

[0120] As shown in Fig. 10, first, the server 10 receives information acquired by the user terminal 30 and transmitted to the server 10, triggered by reception of a beacon transmitted from the environmental camera 20 (step S303). When the user terminal 30 detects a beacon from the environmental camera 20, as shown in step S127 in Fig. 8, it stores the time when it started detecting the beacon and the camera ID (MAC address included in the beacon). Next, the user terminal 30 acquires the distance between the user terminal 30 and the environmental camera 20 and the distance to other user terminals present in the vicinity by the motion sensor unit 340. Then, the user terminal 30 regards the time when the beacon is lost as the time when the detection of the beacon ends, and transmits the time when the detection of the beacon started and the time when the detection of the beacon ended, the distance measurement data, the camera ID, and the user ID to the server 10.

[0121] Next, the control unit 100 of the server 10 performs a matching process to match the information obtained from the user terminal 30 with the information obtained from the environmental camera 20 (step S306). By matching, it is possible to associate the moving object shown in the captured image with the user (i.e., to identify the user). The information obtained from the user terminal 30 is the information received in the above step S303 and the information received from the user terminal 30 and stored in the route history DB 122 as shown in step S118 of FIG. 8. The information obtained from the environmental camera 20 is the information received from the environmental camera 20 and stored in the captured image DB 121 as shown in step S126 of FIG. 8. When the user passes within the angle of view of the environmental camera 20, it is possible to match the information obtained from the captured image with the information obtained from the user terminal 30 held by the user, and it is possible to correct the user's movement route within the facility and register authentication information.

[0122] The timing of performing the matching process is not particularly limited. In addition, the control unit 100 may calculate a matching reliability score indicating the reliability of each match (the likelihood that it is the user) for each match in the matching process, and may calculate a matching reliability score by adding these scores together (referred to as a "combined matching reliability score" in this specification) for each moving object. The addition may be, for example, a method of taking a weighted average of each matching reliability score. Details of the matching process and the calculation of the matching reliability score will be described later.

[0123] Next, the association unit 106 determines whether the combined matching confidence score exceeds a threshold value (step S309).

[0124] Next, when the combined matching reliability score exceeds the threshold (step S309 / Yes), the association unit 106 identifies the moving object having the maximum combined matching reliability score as the target moving object (i.e., identifies the moving object as the user) (step S312). This allows association between the moving object shown in the captured image and the user (specifically, the user ID obtained from the user terminal 30).

[0125] Next, the route update unit 108 extracts data (movement route history) of the user ID associated with the moving object from the route history DB 122 (step S315).

[0126] Next, the route update unit 108 corrects the route history of the user according to the movement vector acquired from the environmental camera 20 (step S318). Specifically, by correcting the movement route of the user who moved within the angle of view of the environmental camera 20 according to the movement vector (derived from the image) acquired from the environmental camera 20, it becomes possible to correct an error in the movement vector (derived from the PDR) acquired from the user terminal 30.

[0127] Furthermore, the information registration unit 107 associates the user ID associated with the moving object with the authentication information of the moving object acquired from the captured image, and registers the associated user ID and the authentication information of the moving object acquired from the captured image in the authentication information DB 123 (step S321). This makes it possible to automatically register authentication information such as a user's face image (facial features).

[0128] Then, the information registration unit 107 notifies the user terminal 30 that the authentication information has been registered (step S324). FIG. 11 shows an example of a screen for notifying the user of the registration of authentication information according to this embodiment. 。 As shown in Fig. 11, a notification screen 361 is displayed on the display unit 360 of the user terminal 30. The notification screen 361 displays the name of the facility (e.g., "S-Mall") and displays a message that information about the user has been registered in the system. For example, the user's approval may be obtained by tapping an OK button on the notification screen 361. The server 10 may delete the authentication information if the user's approval cannot be obtained.

[0129] Also, as a function of the application, the user may be allowed to arbitrarily delete authentication information. FIG. 12 is a diagram showing an example of a deletion screen for authentication information. As shown in FIG. 12, the deletion screen 362 displays registered authentication information. When the delete button on the deletion screen 362 is tapped, the user terminal 30 transmits a deletion request to the server 10, and the server 10 deletes the authentication information. Also, as a function of the application, a function for disabling positioning within a facility and automatic registration of authentication information may be provided. Also, as an opt-in type function, only users who have consented and turned on the setting when using the application may be subject to positioning and automatic registration of authentication information. FIG. 13 is a diagram showing an example of a setting screen for positioning and automatic registration of authentication information. As shown in FIG. 13, a setting button may be displayed on the setting screen 363 so that the setting can be turned on / off.

[0130] On the other hand, if the combined matching reliability score does not exceed the threshold (step S309 / No), the user ID and the camera ID are stored in the undetermined information DB 124 as the user could not be identified (coincidence could not be made) (step S327). Note that if correlation is made later (a large number of environmental cameras 20 are installed in the facility, and correlation can be made based on information obtained from the other environmental cameras 20), the user ID and the camera ID that have been correlated may be deleted from the undetermined information DB 124.

[0131] An example of the operation process of the server 10 according to the present embodiment has been described above. Note that the operation process of the server 10 does not necessarily have to be performed in the order shown in Fig. 10. For example, the processes shown in steps S315 and S318 and the processes shown in steps S321 and S324 may be performed in a different order or in parallel.

[0132] <<4. Matching process and calculation of matching confidence score>> Next, the matching process and the calculation of the matching confidence score performed in step S306 in FIG. 10 will be described.

[0133] < 4 -1. Matching process flow> Fig. 14 is a flowchart showing an example of the flow of the matching process according to this embodiment. As shown in Fig. 14, first, the control unit 100 extracts data for the relevant time period (user ID, movement vector, walking steps, etc. received from the user terminal 30) from the route history DB 122 (step S403). The relevant time period is the time from the start time to the end time of beacon detection by the user terminal 30, which is included in the information received from the user terminal 30 in step S303 of Fig. 10.

[0134] Next, the control unit 100 extracts data (captured images, movement vectors, walking steps, etc. acquired from the environmental camera 20) for the relevant time period for the relevant camera ID from the captured image DB 121 (step S406). The relevant camera ID is the camera ID included in the beacon detected by the user terminal 30, which is included in the information received from the user terminal 30 in step S303 of FIG.

[0135] Next, the control unit 100 performs a comparison of the movement vector (step S409), the walking step (step S412), the behavior information (step S415), the positional relationship (step S418), and the movement pattern (step S421) based on the extracted data. In each comparison process, a comparison reliability score is calculated for each moving object.

[0136] Then, the association unit 106 sums up the matching reliability scores calculated in each matching process for each moving object (step S424). The summing method is not particularly limited, but may be, for example, a method of calculating a weighted average of each matching reliability score.

[0137] An example of the flow of the matching process has been described above. The matching process does not necessarily have to be performed in the order shown in FIG. 14. For example, the matching processes shown in steps S409 to S421 may be performed in a different order or in parallel. All of the matching processes shown in steps S409 to S421 may be performed, or at least any of the matching processes may be performed. When performing each matching process in sequence, the control unit 100 may skip other subsequent matching processes if the matching is successful (i.e., if one moving object corresponding to the user has been identified).

[0138] The control unit 100 may also perform at least two types of matching: "matching of movement vectors" and "matching of at least one of walking steps, behavior information, positional relationship, and movement pattern." When the control unit 100 is unable to identify (specify) the user by matching only the movement vectors, it may further perform matching of at least one of walking steps, behavior information, positional relationship, and movement pattern. The association unit 106 may also integrate the matching results and specify the moving object that has been matched most frequently (successfully matched) as the user.

[0139] Next, each of the collation processes will be described.

[0140] <4-2. Matching of movement vectors and calculation of matching confidence scores> In the comparison of the movement vectors, for example, cosine similarity is calculated as described with reference to FIG. 2. In this case, the movement vector comparison unit 101 calculates the movement vector (mv cam ) and the motion vector (mv acc ) may be used as a matching confidence score (Pmv) of the moving vector.

[0141]

number

[0142] <4-3. Walking step matching and calculation of matching reliability score> In matching the walking steps, for example, the time consistency between the timing of the walking steps (foot contact) obtained from an image captured by the environmental camera 20 and the timing of the walking steps (foot contact) obtained from the sensing data of the user terminal 30 is calculated.

[0143] Here, calculation of the timing consistency of walking steps will be described with reference to Figs.

[0144] FIG. 15 is a diagram illustrating the timing of the walking steps of the user A obtained from the captured image of the environmental camera 20 and the timing of the walking steps of the user A obtained from the sensing data of the user terminal 30A. The user terminal 30A analyzes the three-axis acceleration detected by the motion sensor unit 340, for example, to determine the foot contact timing (t acc0 , t acc1 , t acc2 ) is detected from the captured image. Here, for example, it is assumed that the user terminal 30A is in either the left or right trouser pocket of the user A, and the landing timing of one foot is detected from the peak of the acceleration waveform. Note that the three-axis acceleration waveform shown in FIG. 15 is merely an example, and the present embodiment is not limited to this. Meanwhile, the environmental camera 20 detects, for example, the area of ​​the user A's feet (shoes) and the area of ​​the head from the captured image, and detects the landing timing (t cam0 , t cam1 , t cam2 ) is detected. The environmental camera 20 may detect the foot landing timing from the waist position of the user A by performing bone detection (skeleton recognition) from the captured image. The environmental camera 20 may detect the landing timing of the right foot and the landing timing of the left foot, respectively, or may detect the landing timing of both feet and divide it to calculate the landing timing of one foot. The walking step matching unit 102 calculates the time coincidence (time difference) of each landing timing (landing timing of one foot) to match the walking steps. Note that in this embodiment, the landing timing of one foot is matched as an example, but the present disclosure is not limited to this, and the walking step matching unit 102 may match the landing timing of both feet. For example, when the user terminal 30 is in the user's backpack or handbag, the landing timing of both feet may appear as a peak of acceleration. The user terminal 30 can also determine its own holding position (whether it is a holding position where a difference between the left and right occurs or a holding position where no difference between the left and right occurs) by machine learning of the motion sensor unit 340.

[0145] Fig. 16 is a flowchart showing an example of the flow of a walking step extraction process in the user terminal 30. As shown in Fig. 16, first, the walking step extraction unit 302 acquires three-axis acceleration from the motion sensor unit 340 (step S503).

[0146] Next, the walking step extraction unit 302 calculates a norm from the three-axis acceleration (step S506), and performs FFT (Fast Fourier transform) on the calculation result. r Transform) is then performed (step S509).

[0147] Next, the walking step extraction unit 302 acquires the maximum power frequency as the walking cycle from the FFT processing result (step S512).

[0148] Next, the walking step extraction unit 302 removes components other than those in the maximum power frequency band (step S515).

[0149] Next, the walking step extraction unit 302 performs IFFT (Inverse Fast Fourier Transform) on the data from which components other than the maximum power frequency band have been removed. r Transform) is then performed (step S518).

[0150] Then, the walking step extraction unit 302 extracts the timing of one-foot ground contact from the data subjected to IFFT by performing peak detection in accordance with the walking cycle (step S521).

[0151] The above describes the extraction of walking steps based on acceleration data. Note that the extraction of walking steps described with reference to Fig. 16 is just one example, and the method of extracting walking steps according to this embodiment is not limited to this.

[0152] Fig. 17 is a flowchart showing an example of the flow of the walking step extraction process in the environmental camera 20. As shown in Fig. 17, first, the environmental camera 20 captures an image using the imaging unit 220 (step S533).

[0153] Next, the walking step extraction unit 204 detects the foot (or shoe) area from the captured image (step S536).

[0154] Next, if the foot (shoe) region is detected (step S536 / Yes), the walking step extraction unit 204 detects the landing timing of the right foot and the landing timing of the left foot from the state of the foot (shoe) (step S542, step S545). Generally, when a person walks, he or she lands on the ground from the heel and steps out with the toe, so the walking step extraction unit 204 may detect the landing timing of the heel. Depending on the shooting conditions, for example, the frame in which the absolute value of the movement vector of the heel part becomes 0 for the first time in each walking cycle may be set as the landing timing. Alternatively, the first frame in which the vector from the heel to the toe becomes parallel to the ground from a state facing the sky may be set as the landing timing. Note that in this embodiment, as an example, in order to compare the landing timing of one foot, the walking step extraction unit 204 detects the landing timing of the right foot and the left foot, respectively.

[0155] On the other hand, if the foot (shoe) region cannot be detected (step S536 / No), the walking step extraction unit 204 detects the head region (step S548).

[0156] If the head region is detected (step S551 / Yes), the walking step extraction unit 204 detects the ground contact timings of both feet according to the head position (step S554) and divides the ground contact timing sequence into one foot each (step S557). In general, the body moves up and down when walking and sinks the most when touching the ground, so the walking step extraction unit 204 determines the frame in each walking cycle in which the head position (e.g., the position of the top of the head) reaches the bottom as the ground contact timing. However, since it is not clear which foot, the left or right, has made the ground contact, the walking step extraction unit 204 needs to divide the ground contact timings by allocating them alternately along the time series in order to obtain sequence data for one foot.

[0157] Furthermore, in addition to the head position, the walking step extraction unit 204 can also improve accuracy by performing a process to select the frame with the maximum head pitch angle, taking advantage of the fact that humans perform a movement called "compensatory head rotation" in which the head tilts up and down as it sinks down to prevent the viewpoint from shaking when walking.

[0158] On the other hand, if the head region cannot be detected (step S551 / No), the walking step extraction unit 204 outputs an error.

[0159] In this embodiment, the foot or head detection is used as an example to detect the landing timing, but the present embodiment is not limited to this. For example, when the skeleton can be recognized from the captured image, the walking step extraction unit 204 may detect the timing when the angle of the lower limbs is maximum and the direction of the vertical movement vector of the left and right legs is switched as the landing timing. The walking step extraction unit 204 may also detect the landing timing based on the waist position. The waist position sinks the most at the timing of landing in the walking cycle. When the face can be recognized from the captured image, the walking step extraction unit 204 may detect the lower limit of each cycle of the vertical movement vector of the face parts as the landing timing.

[0160] The above describes the extraction of walking steps based on a captured image. Note that the extraction of walking steps described with reference to Fig. 17 is just one example, and the method of extracting walking steps according to this embodiment is not limited to this.

[0161] FIG. 18 is a flowchart showing an example of the flow of the walking step matching process in the server 10. As shown in FIG. 18, first, the walking step matching unit 102 matches the step time between the one-foot ground contact timing extracted by the user terminal 30 and each of the left and right data extracted by the environmental camera 20 (see steps S542 and S545 above) or each of the divided data (see step S557 above) (step S573). For example, the walking step matching unit 102, as shown in FIG. 15, matches the step time between the one-foot ground contact timing (t cam0 = Tx, t cam1 = Ty, tcam2 =Tz) and the time of one-foot contact timing (t acc0 =Tx+0.02,t acc1 = Ty, t acc2 = Tz-0.1) does not exceed the threshold, the walking step timing may be determined to match. Note that, when data on the ground contact timing of the right foot and the left foot is obtained from the environmental camera 20, the walking step comparison unit 102 calculates the time difference between each and adopts the one with the shorter total time difference.

[0162] More specifically, for example, the walking step comparison unit 102 may determine that the comparison is successful when the following formula 2 is satisfied. In this embodiment, a moving object having a movement vector that satisfies the following formula 2 may be determined (specified) to be the user.

[0163]

number

[0164] Next, the walking step matching unit 102 calculates a matching reliability score (P ws For example, the walking step comparison unit 102 calculates the walking steps (landing timing t cam ) and the walking steps (ground contact timing t acc ) time lag (T diffws ) is calculated, and the time lag is within the allowable range (Threshold ws ) is the matching confidence score for the walking steps. ws ) may be calculated.

[0165]

number

[0166]

number

[0167] Although the method of matching walking steps and calculating a matching reliability score has been described above, the above content is an example, and the method of matching walking steps and calculating a matching reliability score according to this embodiment is not limited to this. Also, here, as an example, a walking step indicating the timing when the foot touches the ground is used, but this embodiment is not limited to this, and some characteristics related to walking, such as the walking style, walking rhythm, and walking posture, may be used as state information for matching.

[0168] <4-4. Matching of positional relationships and calculation of matching reliability score> In matching the positional relationship, the positional relationship matching unit 103 compares the relative positional relationship of each moving object within the angle of view of the environmental camera 20 with the relative positional relationship of the user terminal 30 calculated based on distance information acquired by the user terminal 30, and determines (identifies) a user from one or more moving objects captured in the captured image based on the comparison of the relative positional relationships.

[0169] Here, the mutual positional relationship refers to the positional relationship between the user terminal 30 and another user terminal, the positional relationship between the user terminal 30 and the environmental camera 20, or the positional relationship between the user terminal 30, another user terminal, and the environmental camera 20. The mutual positional relationship may be a distance or may be converted into global coordinates.

[0170] The positional relationship checking unit 103 may calculate the relative positional relationship between the three parties based on distance information (distance information from the other user terminal and distance information from the environmental camera 20) acquired by the user terminal 30 through the distance measuring unit 330. For example, the positional relationship checking unit 103 may calculate the distance (d A-cam ) and the distance between user A and user B (d A-B The relative position is calculated based on the distance (d B-cam) may be acquired from the user terminal 30B. Alternatively, the positional relationship collation unit 103 may use distance information (which may be acquired by the distance measurement unit 230) acquired by the environment camera 20 performing UWB communication or the like with the user terminal 30A and the user terminal 30B, respectively.

[0171] On the other hand, the positional relationship matching unit 103 calculates the positional relationship of each moving body based on the captured image obtained from the environmental camera 20. For example, as shown in Fig. 19, the positional relationship matching unit 103 analyzes the captured image 222 to estimate the perspective relationship (positional relationship) of each moving body (moving body M1 and moving body M2) shown in the captured image 222. At this time, the positional relationship matching unit 103 may refer to data from a depth sensor or the like provided in the environmental camera 20.

[0172] Then, the positional relationship matching unit 103 matches the positional relationship via the distance measuring unit 330 obtained from the user terminal 30 with the positional relationship via the image obtained from the environmental camera 20, and associates the user A (the person who holds the user terminal 30A) with the moving object. For example, the positional relationship matching unit 103 may calculate a matching reliability score for each moving object based on the consistency of the mutual positional relationship. For example, the positional relationship matching unit 103 may determine the matching reliability score based on how close the deviation of the mutual positional relationship is to 0 within an allowable range.

[0173] Although the above describes a comparison that takes into account the positional relationship between users, the present embodiment is not limited to this. For example, the server 10 may use the distance information (d rf ) and distance information (d cam The positional relationship comparison unit 103 of the server 10 compares these distances (d rf and each d cam ) and the difference is within the tolerance range (Tolerance rel) according to the number of moving objects that fit within the frame. Next, the positional relationship matching unit 103 takes the average of the matching confidence scores for each frame as the matching confidence score as the final matching result of the mutual positional relationship. Such a matching confidence score is calculated for each moving object. The moving object with the highest matching confidence score among the moving objects captured in the video is the moving object to be searched for (i.e., the user). A specific description will be given below with reference to FIG. 20.

[0174] FIG. 20 is a diagram illustrating another example of the comparison of the mutual positional relationship according to this embodiment. FIG. 20 is a diagram illustrating the arrangement of the environmental camera 20 and the moving object M at a certain timing (frame k=i). The positional relationship comparison unit 103 compares the distance (d cam k:ID ) is the distance (d rfk ) and Tolerance rel Based on the number (Nk) of mobile objects (mobile object numbers: IDs) present in the range, a matching reliability score (P rel In the example shown in FIG. 20, the moving bodies M10, M11, and M12 calculate the distances (d rfk ) and Tolerance rel ) and the number of moving objects is "3" (the target carrying the user terminal 30 is M12).

[0175] The positional relationship matching unit 103 counts the number of moving objects within the allowable range, for example, as shown in the following formulas 5 and 6. In the following formula 6, the subscript "k:ID" indicates data in the kth frame whose moving object number is ID.

[0176]

number

[0177]

number

[0178] Tolerance rel ), there is a high probability of misidentifying the target moving object (identifying the user). rel k Conversely, the Tolerance rel ), the reliability score for that frame is 1.0. For example, as shown in the following formula 7, the positional relationship matching unit 103 calculates the matching reliability score (P rel k ) is calculated.

[0179]

number

[0180] In the example shown in FIG. 20 (at a certain timing (frame k=i)), for example, Threshold rel If =10, Tolerance rel The number of moving objects in is 3 (Ni=3), and the matching confidence score of each moving object (M10, M11, M12) in the frame is P rel i =1-log 10 (3) is calculated to be approximately 0.52.

[0181] Then, the average of the matching confidence scores of each moving object calculated for each frame is taken as the final matching confidence score of each moving object (P rel The positional relationship matching unit 103 calculates the final matching reliability score (P rel ) is calculated.

[0182]

number

[0183] Note that the frame length (time interval between frames) may be set to, for example, 1 second since the interval is short when matched with the frame rate of the environment camera 20.

[0184] It is also possible to verify the positional relationships by finding an optimal solution using distance information between three people over multiple frames. Hereinafter, this will be described with reference to FIG.

[0185] FIG. 21 is a diagram for explaining another example of the comparison of mutual positional relationships according to this embodiment. In this example, distance information between the user terminal 30 and other user terminals measured by the user terminal 30 side is used. As shown in FIG. 21, while the user terminal 30A is within the angle of view of the environmental camera 20 (while receiving a beacon), the user terminal 30A collects distance information (distance measurement data; distance measurement data) between the user terminal 30A and other nearby user terminals 30B and 30C. At0-Bt0 , d At0-Ct0 , d At1-Bt1 , d At1-Ct1 ) at regular intervals. Next, when the user terminal 30A goes out of the angle of view (when the beacon is lost), the user terminal 30A transmits the measured distance data to the server 10.

[0186] The positional relationship matching unit 103 of the server 10 estimates the distances between moving objects in the image obtained from the environmental camera 20 (the distances between moving objects M1, M2, and M3) from the image of each frame, and performs a full search to find the combination that minimizes the error with the distance information measured by the user terminal 30A.

[0187] The positional relationship matching unit 103 determines a moving object as a search target (a moving object corresponding to the user) when the cumulative error between the distance estimated from the video and the distance measured by the user terminal 30A is the smallest and the error is below a threshold value.

[0188] A matching confidence score can be calculated based on the difference between the cumulative error and a threshold. For example, P rel =1-diff / threshold, or if the threshold is exceeded, P rel = 0. If N (N ≥ 2) optimal solutions are found as a result of exhaustive search, P rel Divide by N and output the result as the matching confidence score.

[0189] Although the method of matching the positional relationship and calculating the matching reliability score has been described above, the above content is merely an example, and the method of matching the positional relationship and calculating the matching reliability score according to this embodiment is not limited to this.

[0190] <4-5. Matching behavioral information and calculating match reliability score> In matching the behavioral information, the time at which a specific behavior (e.g., elevator, escalator, stairs, etc.) immediately before or after the walking section detected by the user terminal 30 switches between walking and the time at which a specific behavior of the moving object captured by the environmental camera 20 switches between walking and the time at which the moving object ...

[0191] For example, as shown in FIG. 22, since user A does not walk while riding the escalator, user A who rode the escalator and user B who simply stood still on the spot are the same and cannot be distinguished from each other when only the movement vector derived from the PDR is viewed. In this embodiment, when a specific behavior is detected before and after a walking section, the time when the behavior switches from walking to the walking can be compared, making it possible to identify the user. A specific behavior can be identified from a captured image or in-building map information. For example, when an escalator is included in the angle of view of the environmental camera 20 shown in FIG. 22, user A who got off the escalator and started walking can be detected from the captured image. In addition, user B who stood still on the spot for a while and then started walking can also be detected from the captured image shown in FIG. 22. In addition, the behavior information matching unit 104 can grasp the locations of escalators, elevators, stairs, etc. from not only the captured image but also the in-building map information, so that it is possible to detect a specific behavior from the in-building map information and the movement trajectory detected by the user terminal 30.

[0192] The behavior information matching unit 104 calculates the time lag (T diff act ) is within the tolerance range (Tolerance act ), the matching confidence score (P act) should be set to 1.0, otherwise it may be set to 0.0.

[0193]

number

[0194] Although the method of matching behavioral information and calculating a matching reliability score has been described above, the above content is merely an example, and the method of matching behavioral information and calculating a matching reliability score according to this embodiment is not limited to this.

[0195] <4-6. Matching of movement patterns and calculation of matching reliability score> In matching the movement pattern, a determination is made as to whether or not the movement pattern matches a predetermined rule (movement pattern) based on the movement vector (PDR position information) acquired by the user terminal 30 and the integration of information from multiple environmental cameras 20 (Chain of multi cameras) from the placement of the environmental cameras 20 and the map information of the facility.

[0196] When a travel pattern that matches the rule is found, the travel pattern matching unit 105 calculates a matching reliability score (P ch ) is set to 1.0, otherwise the matching confidence score is set to 0.0. A specific description will be given below with reference to FIG.

[0197] Fig. 23 is a diagram for explaining the matching of movement patterns according to this embodiment. In Fig. 23, environmental cameras 20R, 20L, and 20C are arranged. The server 10 has the arrangement of each of these environmental cameras 20 and map information.

[0198] (Rule 1: Course A) For example, after the user terminal 30 detects a left turn of 90°±45° in the Crossing Area shown in FIG. 23 by PDR, a moving object that enters the field of view of the environmental camera 20C is highly likely to have passed through course A. The movement pattern matching unit 105 judges whether or not the moving object to be matched captured by the environmental camera 20C is also captured by the environmental camera 20L (identity (same person) can be searched for based on external characteristics of each moving object such as face, clothing, walking style, and skeleton). If the moving object to be matched captured by the environmental camera 20C is also captured by the environmental camera 20L, the moving object is the search object (corresponding to the user), and the matching reliability score is set to 1.0. On the other hand, if the movement pattern matching unit 105 does not find a moving object captured by the environmental camera 20C in the image of the environmental camera 20L, the matching reliability score is set to 0.0.

[0199] (Rule 2: Course B) Furthermore, a moving object that enters the angle of view of environmental camera 20C after it is detected by the PDR of user terminal 30 that it is moving straight without turning within the Crossing Area is highly likely to have taken course B. If the moving object to be matched that is captured by environmental camera 20C is captured in the image of environmental camera 20L or environmental camera 20R, movement pattern matching unit 105 sets the matching reliability score to 0.0, and if it is not captured in either, the matching reliability score to 1.0.

[0200] (Rule 3: Course C) Furthermore, a moving object that enters the field of view of the environmental camera 20C after a 90°±45° right turn is detected by the user terminal 30 via PDR in the Crossing Area is highly likely to have taken Course C. If a moving object to be matched that is captured by the environmental camera 20C is also captured by the environmental camera 20R, the movement pattern matching unit 105 determines that the moving object is a search target and sets the matching reliability score to 1.0. On the other hand, if the moving object captured by the environmental camera 20C is not found in the image of the environmental camera 20R, the movement pattern matching unit 105 sets the matching reliability score to 0.0.

[0201] The above describes the method of matching movement patterns (matching based on the integration of information from multiple cameras) and calculating the matching reliability score, but the above is just an example, and the method of matching movement patterns and calculating the matching reliability score according to this embodiment is not limited to this.

[0202] <4-7. Aggregation of verification confidence scores> The matching unit 106 sums up the matching reliability scores output from each matching unit for each moving object, and performs final user discrimination (association between moving object and user) based on each summed matching reliability score. The summation of the matching reliability scores may be calculated by taking a weighted average of each score, for example, as shown in the following formula 10.

[0203]

number

[0204] Here, the subscript w is a weighting coefficient for the matching confidence score calculated by each matching unit. This weighting may be set heuristically to a fixed value based on, for example, characteristics based on the floor shape of the target facility and characteristics based on the arrangement of the environmental cameras, or it may be adjusted by machine learning by collecting learning data at the target facility.

[0205] <<5. Application Examples>> Next, various application examples will be described, such as utilization of authentication information registered according to the present embodiment, and information notification services according to the movement route and behavior of a specified user. Note that, as an example, the server 10 is described here as providing various services, but the present disclosure is not limited to this, and various services may be provided by a server separate from the server 10.

[0206] <5-1. First application example> When the server 10 detects a person performing a predetermined action in a predetermined location from the video captured by the environmental camera 20, the server 10 notifies the person of predetermined information. For example, when the environmental camera 20 detects a person area from the video and detects a predetermined action, it converts the coordinates of the person in the video into global coordinates and notifies the server 10 of the global coordinates and the detection time. The environmental camera 20 may also transmit the content of the detected action to the server 10 (for example, when multiple predetermined actions are defined, it also notifies which action was detected).

[0207] The server 10 extracts the user ID of a person who was at the coordinates at the time from the route history DB 122 based on the time information and coordinate information received from the environmental camera 20. As described above, the route history DB 122 accumulates the history of each user's movement route (location information) at any time.

[0208] Then, the server 10 performs control to transmit predetermined information to the user terminal 30 associated with the user ID. The association between the user ID and the user terminal 30 may be performed, for example, when a user creates an account to use the system, by registering the address or the like of the user terminal 30 in advance associated with the user ID.

[0209] When user association and movement route accumulation are performed in real time, the server 10 can instantly notify the user terminal 30 of appropriate information. FIG. 24 illustrates recognized actions and services provided to the user according to an application example of this embodiment. As shown in FIG. 24, in response to detection of actions such as gazing at a product displayed on a shelf in a store, picking up a product, or handing a product to a store clerk in front of a cash register (an example of a checkout location), the server 10 transmits coupons for the product and product information such as the material and place of origin to the user terminal 30. In addition, appropriate services can be provided according to various other actions.

[0210] The server 10 may also send special coupons to users who buy the same products at the same store every day. The server 10 may also provide a gamification element for searching for products that can earn coupons. The server 10 may also recommend a dinner recipe based on a combination of products that the user has bought, and may also send advertisements for products that are in short supply.

[0211] In addition, the server 10 may distribute coupons to users who stay for a long time at a restaurant such as a cafe, to induce them to make additional orders.

[0212] In the case of proximity determination using BLE beacons, there is a possibility that the signal may be sent to the wrong user device when there are multiple customers, but in this embodiment, by using PDR (i.e., positioning based on motion sensor data), users can be detected precisely. In addition, since PDR and cameras are combined, excessive environmental cameras that cover the entire floor are not necessarily required, so the store can easily set up and the maintenance costs can be reduced.

[0213] <5-2. Second application example> After sending coupons for directing users to the actual store, the server 10 can measure the marketing effect based on data on the behavior of users who actually visited the store, products that piqued their interest, and products that were purchased. It also becomes possible to precisely collect data on products that were reached in the actual store but not purchased, products that were passed by, and other products that were picked up and compared.

[0214] When using only camera footage, the user must be identified using some kind of authentication method before real-world behavior data can be collected and linked to an account on the web or in an app. However, in this embodiment, the location can be detected using the PDR of the user's device, making it easy to link the data.

[0215] <5-3.Third application example> Next, a case where the authentication information is used in a payment process will be described with reference to FIGS.

[0216] Fig. 25 is a diagram for explaining a payment process using authentication information according to an application example of this embodiment. As shown in Fig. 25, a user registers credit card information in advance in the server 10. The server 10 stores the user ID and the credit card information in association with each other. Note that credit card information is used here as an example, but is not particularly limited as long as it is information that can be used for payment. For example, electronic money information, account information, mobile phone number, or other information that can be used for payment may be used.

[0217] Thereafter, within the facility visited by the user, as described in the above embodiment, the user's current location is tracked by highly accurate position estimation (obtaining a movement path) that combines indoor positioning (obtaining a movement vector derived from the PDR) in the user terminal 30 and a movement vector derived from the image captured by the environmental camera 20.

[0218] Next, when a customer arrives at a predetermined location where payment is made, such as in front of the cash register, the system (server 10) determines the customer from the location information and identifies the user. Then, a camera (tablet terminal 50) placed in front of the cash register photographs the user's face for face authentication, and if authentication is successful, payment can be made using pre-registered credit card information. At this time, the user can make payment empty-handed without taking out the user terminal 30.

[0219] Such payment processing using authentication information will be specifically described below with reference to Fig. 26. Fig. 26 is a sequence diagram showing an example of the flow of payment processing according to an application example of this embodiment.

[0220] 26, first, the server 10 registers the user ID and authentication information in the authentication information DB 123 (step S603), and notifies the user terminal 30 that the authentication information has been registered (step S606). The registration process of the authentication information and the notification to the user are as described above.

[0221] Next, a camera for monitoring merchandise installed in the store analyzes the captured image, detects that the merchandise has been acquired by the customer, and notifies the server 10 (step S609). Note that the camera is not limited to a camera for monitoring merchandise, and the above-mentioned environmental camera 20 may be used. Also, a sensor device other than a camera may be used (or used in combination).

[0222] Next, the server 10 recognizes that the user whose authentication information is registered has acquired the product (step S612). "Acquiring a product" means, for example, putting the product into a cart such as a shopping basket. The server 10 stores information about the acquired product in association with the user ID.

[0223] Next, the server 10 recognizes that the user whose authentication information is registered is standing in front of the cash register (step S615). For example, the server 10 estimates the person standing in front of the cash register based on the user's movement route acquired in real time.

[0224] Next, the server 10 checks the authentication information of the user standing in front of the cash register (step S618 ) Specifically, the facial image of the user automatically registered in the authentication information DB 123 is compared with the facial image of the person standing in front of the cash register. A camera for facial recognition is installed in front of the cash register, and the face of the person standing in front of the cash register is photographed. The comparison of the authentication information (personal authentication) may be performed by the cash register terminal or by the server 10.

[0225] Next, if the authentication information is successfully verified (e.g., facial recognition) (step S618 / Yes), the cash register terminal displays a payment approval notice to the user on a touch panel display or the like provided for the user (step S621). For example, the cash register terminal displays a message such as "Do you want to make the payment? YES / NO." To make the payment, the user taps the "YES" button.

[0226] Next, if the payment is approved (step S624 / Yes), the cash register terminal obtains credit card information associated with the user from the server 10 and performs the payment process (step S627). The user's credit card information may be registered in advance in association with a user ID when the user creates an account to use this system, for example. This allows the user to perform the payment process using authentication information that is automatically registered within the facility and associated with the user's user ID, without having to take out the user terminal 30 or credit card at the cash register. The payment process using the credit card information may be performed by the server 10.

[0227] On the other hand, if the authentication information fails to be verified (step S618 / No), the cash register terminal displays a notification to the user urging them to authenticate using an alternative means (step S631). For example, the cash register terminal may display a message such as "When making a payment, please authenticate using your smartphone." The user may launch the application of this system and perform authentication using biometric authentication (fingerprint authentication, etc.) or by inputting a passcode, etc., on the user terminal 30.

[0228] Next, if authentication has been performed using the alternative means (step S634 / Yes), the cash register terminal acquires credit card information associated with the user from the server 10 and performs the payment process (step S627).

[0229] On the other hand, if authentication by the alternative means is not performed (step S634 / No) or if the payment is not approved (step S624 / No), the cashier terminal displays a message indicating that authentication by the system has resulted in an error (step S637). In this case, the user can pay with cash, a credit card, or electronic money.

[0230] In the above application example, the empty-handed payment process using authentication information has been described, but the present embodiment is not limited to this. For example, it is also possible to perform the empty-handed payment process using location information and a PIN number. The following description will be given with reference to FIG. 27.

[0231] FIG. 27 is a diagram for explaining a payment process using location information and a PIN number. As shown in FIG. 27, a user registers credit card information and a PIN number for payment (for example, a four-digit number) in advance in the server 10. The server 10 stores the user ID, credit card information, and PIN number in association with each other. Here again, credit card information is used as an example, but there is no particular limitation as long as the information can be used for payment. For example, electronic money information, account information, mobile phone number, or other information that can be used for payment can be used.

[0232] Thereafter, within the facility visited by the user, as described in the above embodiment, the user's current location is tracked by highly accurate position estimation (obtaining a movement path) that combines indoor positioning (obtaining a movement vector derived from the PDR) in the user terminal 30 and a movement vector derived from the image captured by the environmental camera 20.

[0233] Next, when a customer arrives at the payment location (in front of the register), the system (server 10) determines the location information and identifies the user. Then, a screen for inputting a PIN for payment is displayed on the tablet terminal 50 placed in front of the register. If the PIN input from the tablet terminal 50 matches the PIN linked to the identified user ID (obtained from the server 10) (i.e., authentication is successful), the register terminal is able to make payment using credit card information preregistered in the server 10. At this time, the user is able to make payment empty-handed without taking out the user terminal 30.

[0234] The authentication process and the payment process using credit card information may be performed by the server 10.

[0235] Also, to take into consideration the privacy of location information, a system may be used in which the user approaches the cash register and taps the smartphone (user terminal 30) placed in the user's pocket or the like with a finger several times to transmit location information to the store. When the user terminal 30 detects that the user has tapped the smartphone several times with a sensor, the user terminal 30 controls the transmission of PDR location information, etc. to the server 10.

[0236] In addition, in order to reduce anxiety when the user loses the user terminal 30, the server 10 may perform gait authentication to estimate the user's identity based on the way the user walks.

[0237] <5-4. Fourth application example> As another example of the empty-handed payment process described above, a system may be realized that executes payment processing when a user takes a product off a shelf. The server 10 executes payment processing using credit card information or the like associated with the user ID of the user who has taken the product, utilizing the association between the location information (movement route) acquired based on the information obtained from the user terminal 30 and the information obtained from the environmental camera 20 and the user terminal 30 (user ID).

[0238] The fact that an item has been taken from the shelf can be detected, for example, by a weight sensor installed on the shelf or an environmental camera 20 installed in the vicinity. The person in front of the shelf has already been captured by the environmental camera 20 and the PDR. When the server 10 confirms that only one item has changed using the weight sensor, it performs a payment process using credit card information or the like linked to the user ID of the person in front of the shelf. The server 10 then notifies the user terminal 30 of the payment result.

[0239] <5-5. Fifth application example> As an application example of this embodiment, it is also possible to automatically grasp the stocking status of each store clerk (who stocked what products and how many of them). This makes it easier to manage products in the store and record stocking. It is also possible to notify the wearable device used by the store clerk of advice (such as how to arrange products) according to the products they are stocking.

[0240] <5-6. Sixth Application Example> As an application example of this embodiment, an unmanned hotel check-in may be performed. A user makes a hotel reservation and registers his / her identity information using an application of this system previously installed on the user terminal 30 (or on a website).

[0241] The server 10 identifies (associates) the user based on information obtained from the user terminal 30 and information obtained from the environmental camera from the time the user enters the building in which the hotel is located until the time the user reaches the front desk, and automatically registers authentication information (e.g., a facial image) obtained from an image captured by the environmental camera 20.

[0242] Next, the server 10 identifies the user who has visited the front desk by indoor positioning, and displays the room number of the user on the digital signage that the user approached. For example, a screen such as "Welcome, Mr. XX. Please come to your room. The room number is '4122'" is displayed. When the user (guest) confirms the room number and goes to the room, the server 10 performs face authentication using a camera installed on the door of the room. Specifically, the server 10 performs identity verification (face authentication) by comparing a face image taken by the camera installed on the door of the room with a face image in the automatically registered authentication information. If the authentication is successful, the server 10 uses the preregistered information of the user as guest information for check-in and performs check-in processing. Then, the server 10 unlocks the door of the room when the check-in processing is completed.

[0243] After that, the server 10 can automatically call elevators and unlock / open doors in the building based on the location information and authentication information. The server 10 can also automatically settle payments for fitness services such as restaurants and pools in the building by verifying the user's identity using the location information and authentication information.

[0244] <5-7. Seventh application example> This embodiment can also be applied to meeting navigation within a building. Specifically, position information within a building can be shared among multiple user IDs so that it can be used for meeting up.

[0245] A user registers other user IDs on an application in advance in order to use location information within the building for meetings. For example, (before entering the building) a QR code (registered trademark) indicating the friend's user ID is displayed on an application installed on the friend's user terminal, and the user reads the QR code on his or her own user terminal 30 to register the friend's user ID. The server 10 registers the registered friend's user ID as a sharer of the user's location information. Note that the user and the friend may each give permission to disclose location information on the application.

[0246] Next, when the user wants to reunite with a friend after having been apart from the friend, the user opens an application on the user terminal 30 and selects the registered user ID of the friend. The user terminal 30 displays current location information of the selected user ID. FIG. 28 shows an example of a location information share screen according to an application example of this embodiment. As shown in FIG. 28, the location information share screen 365 displays, for example, on a floor map, a display P indicating the location of the friend and information indicating the time when the friend was at that location.

[0247] When the user and the friend leave the facility, the server 10 may delete each other's disclosure permission (delete the user ID registered as a location information sharing partner).

[0248] <5-8. Eighth Application Example> This embodiment can also be applied to a photo spot service within a building. Specifically, this is a service in which photos taken with a camera installed at a photo spot within a facility are delivered to the subject's smartphone or the like (user terminal 30).

[0249] People who visit the facility are captured by the environmental camera 20 installed in the facility and the PDR (an example of indoor positioning) in the user terminal 30. Next, when the user enters a photo spot and strikes a pose, the camera installed in the photo spot automatically starts taking pictures. Next, the taken pictures are automatically uploaded to the server 10, and the server 10 generates a download URL that is valid for a certain period of time for each user. Then, when the user finishes taking pictures and leaves the photo spot, the server 10 notifies the user's smartphone or the like (user terminal 30) of the download URL.

[0250] <5-9. Ninth Application Example> This embodiment can also be applied to a system in which a facility staff member or a business marketer notifies any user of information based on video of visitors within the facility.

[0251] Specifically, a person in charge of the facility selects an arbitrary user from a video of visitors in the facility. Here, FIG. 29 shows an example of an operation screen for selecting an arbitrary user from a video in the facility and notifying the user of information. As shown in FIG. 29, the person in charge 70 performs an operation of tapping an arbitrary user 612 in a video captured by the environmental camera 20 and displayed on the display unit 610 of the tablet terminal 60 for the administrator. When the selected user 612 (moving object) is associated with the user, the display unit 610 displays acquired information 614 (e.g., user ID, whether or not a face image has been acquired, whether or not bone information has been acquired, etc.). Next, the person in charge 70 selects information to be notified to the selected user 612 from a notification content selection item 616. Examples of the notification content selection item 616 include a message notification, an event guide, a coupon notification, a map guide notification, and a warning. Then, the server 10 transmits predetermined information (selected by the person in charge of the facility) to the user terminal 30 of the selected user.

[0252] In this way, facility staff or business marketers can directly select users from the images captured by the environmental camera 20 and notify them of any information (messages, event information, coupons, map guides, warnings, etc.). Since staff can visually check the state of visitors and notify them of appropriate information individually, customer satisfaction can be improved and marketing effects can be expected. For example, coupon information can be distributed to people in the target demographic of a store who are walking in front of the store to encourage them to enter the store. In addition, customers who appear to be lost on the floor can be immediately assisted by notifying them of floor guide information. In addition, visitors who are engaging in prohibited activities or violating manners can be directly warned without being conspicuous.

[0253] <5-10. Tenth Application Example> This embodiment can also be applied to games that use the real world.

[0254] For example, in an alternate reality game played in a facility, actions in the real world may be used as triggers for events in the game. An alternate reality game is an experiential game in which the everyday world is incorporated as part of the game, intersecting reality and virtuality.

[0255] Specifically, for example, when a user performs a certain action at a certain place in a facility, the user can receive an item in the game or a present can be mailed to his / her home. People who visit the facility are captured by the environmental camera 20 installed in the facility and the PDR (an example of indoor positioning) in the user terminal 30. In addition, the user's behavior information can be acquired by the user terminal 30 and the environmental camera 20, so that not only the location information but also the action can be used as a trigger. This makes it possible to provide more real-world oriented content.

[0256] The alternate reality game can be played not only indoors but also outdoors. For example, when a user reproduces a specific pose performed by a character in an anime at a specific outdoor location (such as a shrine or park) that appears in the anime, a camera installed at the location automatically captures an image, and the captured image is transmitted to the user terminal 30.

[0257] <5-11. 11th Application Example> This embodiment is also applicable to outdoor events, outdoor attractions, and the like.

[0258] Using the environmental camera 20 installed at a specific outdoor location and the PDR (or satellite positioning) in the user terminal 30, people who visit the specific location can be captured, and user behavior information can also be obtained by the user terminal 30 and the environmental camera 20.

[0259] Using the environmental camera 20 and the user terminal 30, the server 10 can automatically grasp the shape of the queue and the situation of the people waiting in line, and can provide appropriate information to each user individually depending on the situation of the queue, etc.

[0260] For example, if the column Prescribed If the server 10 is not in the shape of Prescribed Inform employees about any areas that are not in the shape of the original. Prescribed The user terminals 30 of those who are not in line may be notified to urge them to correct the queue.

[0261] The server 10 may also notify people in line of the waiting time and the like according to their location in line. The server 10 may also grasp the shape of the line and the situation of the people in line and notify the store of the situation.

[0262] In addition, the server 10 may distribute menu information to people in line, prepare the selected items (or items that match the preferences of the users in line) in advance, transport them to the orderer's location using a self-propelled robot or the like, and complete the payment process on the spot when the orderer acquires the items.

[0263] <5-12. 12th Application Example> This embodiment can also be applied to hospital operations.

[0264] For example, the server 10 can manage the behavior of hospital staff in the hospital (location information and work content) and entry information to the hospital and each room in the hospital using the user terminal 30 carried by the staff and the environmental camera 20 in the hospital. If the face information of the staff cannot be acquired because of a mask, the server 10 may enable entry to the hospital and each room mainly based on location information. The server 10 can also manage whether or not the staff is heading to a patient who requires emergency treatment.

[0265] The server 10 can also manage the status of patients in the hospital using a user terminal 30 carried by the patient and an environmental camera 20 in the hospital. For example, the server 10 can determine where a dementia patient is in the hospital. The server 10 can also determine the location of the patient based on location information even in a place where no camera is installed. The server 10 can also determine the examination information of the patient by face recognition.

[0266] In addition, since it is assumed that there may be a situation in which a user does not have a smartphone or a mobile phone terminal in a hospital, the device shape of the user terminal 30 may be, for example, a wristband type or a clip type (wearable device). In addition, when a wearable device is used, if it is in a shape that contacts the skin, various types of biometric information such as heart rate, vein, sweat amount, electromyogram, and brain wave can be acquired by the user terminal 30.

[0267] <<6. Summary>> As described above, in the information processing system according to the embodiment of the present disclosure, it is possible to accurately identify people appearing in captured images, i.e., to match people appearing in captured images with users, regardless of whether they are indoors or outdoors.

[0268] Although the preferred embodiment of the present disclosure has been described in detail above with reference to the accompanying drawings, the present technology is not limited to such examples. It is clear that a person having ordinary knowledge in the technical field of the present disclosure can conceive of various modified or amended examples within the scope of the technical ideas described in the claims, and it is understood that these also naturally belong to the technical scope of the present disclosure.

[0269] For example, a computer program can be created to cause hardware such as a CPU, ROM, and RAM built into the above-mentioned server 10, environmental camera 20, or user terminal 30 to perform the functions of the server 10, environmental camera 20, or user terminal 30. Also provided is a computer-readable storage medium storing the computer program.

[0270] In addition, the effects described in this specification are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that are apparent to a person skilled in the art from the description of this specification, in addition to or in place of the above effects.

[0271] The present technology can also be configured as follows. (1) An information processing device comprising: a control unit that performs control to match a person appearing in a captured image with the user based on a comparison between movement information obtained from information detected by an information processing terminal and movement information of one or more people obtained from a captured image of a user carrying the information processing terminal, and a comparison between status information of the user obtained from the information processing terminal and status information of the one or more people obtained from the captured image. (2) The information processing device according to (1), wherein the state information includes information that can be recognized from the outside. (3) The information processing device according to (1) or (2), wherein the user status information is extracted from sensing data acquired by a sensor provided in the information processing terminal. (4) The information processing device according to any one of (1) to (3), wherein the person's state information is extracted from the captured image by image analysis. (5) The information processing device according to any one of (1) to (4), wherein the state information is characteristic information relating to walking. (6) The information processing device according to any one of (1) to (5), wherein the state information is information relating to a position. (7) The information processing device according to any one of (1) to (6), wherein the state information is information relating to distance. (8) The information processing device according to any one of (1) to (7), wherein the state information includes a positional relationship between the user and other users around the user, and a positional relationship between the user and a camera that captures the captured image. (9) The information processing device according to any one of (1) to (8), wherein the state information is behavior information. (10) The information processing device according to any one of (1) to (9), wherein the state information is a movement pattern estimated according to a preset rule. (11) The information processing device according to (10), wherein the control unit compares the movement pattern based on movement information of the information processing terminal and external characteristics of each person obtained from captured images obtained from multiple cameras. (12) The information processing device according to any one of (1) to (11), wherein the control unit controls storage in a memory unit, in accordance with the association, of an identification string of the user registered in advance and authentication information of the associated person extracted from a captured image of the associated person. (13) The information processing device according to (12), wherein the control unit performs control to notify the information processing terminal that the authentication information has been stored. (14) The information processing device according to any one of (1) to (13), wherein the control unit calculates a matching reliability score for each person appearing in the captured image when matching the movement information and the status information, and associates the person having the largest sum of the matching reliability scores for each person with the user. (15) The information processing device according to any one of (1) to (14), wherein the movement information of the information processing terminal includes indoor positioning information. (16) The information processing device according to any one of (1) to (15), wherein the control unit stores movement trajectory information, which is movement information of the information processing terminal, in a memory unit as a route history of the user, and when the association is performed, controls updating the route history of the user based on the movement trajectory information obtained from the captured image of the associated person. (17) The information processing device according to any one of (1) to (16), wherein the control unit performs control to notify the information processing terminal of appropriate information in accordance with the action of the user for which the association has been established. (18) The information processing device according to any one of (1) to (17), wherein the control unit identifies a person who is present at a specified location where settlement is to be performed by referring to the route history of each user stored in a memory unit, and uses authentication information of the identified user to compare it with authentication information obtained from an image capturing the person who is present at the specified location where settlement is to be performed, thereby performing personal authentication for payment approval. (19) The processor: An information processing method including controlling matching of people appearing in an image captured with the user based on comparing movement information obtained from information detected by an information processing terminal with movement information of one or more people obtained from an image captured by a user carrying the information processing terminal, and comparing status information of the user obtained from the information processing terminal with status information of the one or more people obtained from the captured image. (20) Computer, A program for functioning as a control unit that performs control to match people appearing in an image captured with the user based on comparing movement information obtained from information detected by an information processing terminal with movement information of one or more people obtained from an image captured by a user carrying the information processing terminal, and comparing status information of the user obtained from the information processing terminal with status information of the one or more people obtained from the captured image. [Explanation of symbols]

[0272] 10 Server 100 Control section 101 Movement vector matching unit 102 Walking step verification unit 103 Positional relationship matching unit 104 Behavioral Information Matching Unit 105 Movement Pattern Matching Unit 106 Mapping section 107 Information Registration Department 108 Route Update Unit 110 Communications Department 120 Storage section 121 Captured Image DB 122 Route History DB 123 Authentication information DB 124 Undetermined Information DB 125 Map Information DB 20 Environmental Camera 30 User terminals

Claims

1. a control unit that performs control to associate a person appearing in a captured image with the user based on a comparison between movement information acquired from information detected by an information processing terminal and movement information of one or more persons acquired from a captured image of a user carrying the information processing terminal, and a comparison between status information of the user acquired from the information processing terminal and status information of the one or more persons acquired from the captured image, The control unit identifies a person who is in a predetermined place where the settlement is to be performed based on the user's route history stored in the storage unit, and compares authentication information of the identified user with authentication information acquired from an image capturing the person who is in the predetermined place where the settlement is to be performed, thereby performing personal authentication for approval of the settlement. Information processing device.

2. The information processing apparatus according to claim 1 , wherein the state information includes information that can be recognized from the outside.

3. The information processing device according to claim 1 , wherein the user status information is extracted from sensing data acquired by a sensor provided in the information processing terminal.

4. The information processing apparatus according to claim 1 , wherein the person's state information is extracted from the captured image by image analysis.

5. The information processing device according to claim 1 , wherein the state information is characteristic information relating to walking.

6. The information processing apparatus according to claim 1 , wherein the state information is information relating to a position.

7. The information processing device according to claim 1 , wherein the state information is information relating to a distance.

8. The information processing apparatus according to claim 1 , wherein the state information includes a positional relationship between the user and other users around the user, and a positional relationship between the user and a camera that captures the captured image.

9. The information processing apparatus according to claim 1 , wherein the state information is behavior information.

10. The information processing device according to claim 1 , wherein the state information is a movement pattern estimated according to a preset rule.

11. The information processing device according to claim 10 , wherein the control unit performs matching of the movement pattern based on movement information of the information processing terminal and external features of each person obtained from captured images obtained by a plurality of cameras.

12. 2. The information processing device according to claim 1, wherein the control unit controls to store in a memory unit, in accordance with the association, a pre-registered identification character string for the user and authentication information for the associated person extracted from a captured image of the associated person.

13. The information processing device according to claim 12 , wherein the control unit performs control to notify the information processing terminal that the authentication information has been stored.

14. The information processing device according to claim 1, wherein the control unit calculates a matching reliability score for each person appearing in the captured image when matching the movement information and the status information, and associates the person with the largest sum of the matching reliability scores for each person with the user.

15. The information processing device according to claim 1 , wherein the movement information of the information processing terminal includes indoor positioning information.

16. 2. The information processing device according to claim 1, wherein the control unit stores movement trajectory information, which is movement information of the information processing terminal, in a memory unit as a route history of the user, and when the association is performed, controls updating the route history of the user based on the movement trajectory information obtained from the captured image of the associated person.

17. The information processing device according to claim 1 , wherein the control unit performs control to notify the information processing terminal of appropriate information in accordance with the action of the user for which the association has been established.

18. The processor: a comparison between movement information acquired from information detected by an information processing terminal and movement information of one or more persons acquired from a captured image of a user carrying the information processing terminal, and a comparison between status information of the user acquired from the information processing terminal and status information of the one or more persons acquired from the captured image, and based on this comparison, a control is performed to associate a person appearing in the captured image with the user; Further, the method includes identifying a person who is in a predetermined place where the settlement is performed based on the user's route history stored in the storage unit, comparing authentication information of the identified user with authentication information obtained from an image capturing the person who is in the predetermined place where the settlement is performed, and performing personal authentication for approval of the settlement. Information processing methods.

19. Computer, functioning as a control unit that performs control to associate a person appearing in a captured image with the user based on a comparison between movement information acquired from information detected by an information processing terminal and movement information of one or more people acquired from a captured image of a user carrying the information processing terminal, and a comparison between status information of the user acquired from the information processing terminal and status information of the one or more people acquired from the captured image; The control unit identifies a person who is in a predetermined place where the settlement is to be performed based on the user's route history stored in the storage unit, and compares authentication information of the identified user with authentication information acquired from an image capturing the person who is in the predetermined place where the settlement is to be performed, thereby performing personal authentication for approval of the settlement. program.

Images