People search system

Abstract

To provide a system which offers improved person identification accuracy when identifying a person being searched for from among people in a camera image.SOLUTION: The present disclosure relates to a system for searching for a person. The system comprises a storage device for storing data containing one or more preregistered persons in association with a device owned by each of the one or more persons, and at least one computer configured to be communicable with a camera capable of capturing an image of persons. The at least one computer is configured to acquire person state quantities representing state quantities of a search target candidate included in an image captured by the camera upon specification of a search target candidate, acquire device state quantities representing state quantities acquired from a target device associated with the search target, and determine whether the search target candidate is the search target or not based on the degree of match between the person state quantities and the device state quantities.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to a system for searching for a particular person. [Background technology]

[0002] Patent Document 1 discloses an object recognition system that monitors the behavior of moving objects within a shooting range based on a video signal input from a camera, and compares terminal behavior data transmitted from a mobile terminal with estimation result history data for each moving object. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2016-212675 A [Patent Document 2] JP 2018-042190 A Summary of the Invention [Problem to be solved by the invention]

[0004] Consider searching for a specified person using camera images. When identifying the person you are looking for from among the people in the camera images, the challenge is how to improve the accuracy of identifying the person. [Means for solving the problem]

[0005] The present disclosure relates to a system for locating a person. The system includes a storage device that stores data linking one or more pre-registered persons with devices owned by each of the one or more persons, and at least one computer configured to communicate with a camera capable of capturing images of the persons. The at least one computer is designated as a search target, Acquire a person state quantity, which is a state quantity related to a search target candidate included in an image captured by the camera; A device state quantity is obtained, which is a state quantity obtained from a target device associated with the search target candidate; Based on the degree of agreement between the person state amount and the device state amount, it is determined whether the search target candidate is the search target person. Effect of the Invention

[0006] The system of the present disclosure includes a storage device that stores data linking one or more people registered in advance with devices owned by each of the one or more people. Then, upon receiving a designation of a person to be searched for, the system acquires state quantities related to the person in an image captured by a camera and state quantities related to the devices linked to the designated person to be searched for. The system then identifies the person to be searched for based on the degree of agreement between these. In this way, the accuracy of identifying a person can be improved. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a conceptual diagram for explaining an overview of a system according to an embodiment of the present invention. [Diagram 2] 1 is a block diagram showing an example of a configuration of a system according to an embodiment of the present invention. [Diagram 3] 4 is a flowchart showing an example of processing by the system according to the present embodiment. [Figure 4] 11 is a table showing examples of combinations of person state quantities and device state quantities. [Diagram 5] FIG. 11 is a conceptual diagram for explaining first and second modified examples. [Figure 6] FIG. 13 is a conceptual diagram for explaining a third modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0009] 1. System Description The present disclosure relates to a system for searching for a person. The system according to the embodiment of the present disclosure receives a designation of a person to be searched for and searches for the person among people included in an image captured by a camera. Fig. 1 is a conceptual diagram for explaining an example of a flow until a system 10 according to the embodiment of the present disclosure searches for a person.

[0010] This sequence of steps begins with input of a search request including a designation of a person P1 to be searched for into the system 10. Hereinafter, in this specification, the person designated as the person to be searched for by this search request is referred to as the search target. A search request including a designation of the search target is issued, for example, by a user of the system 10. For example, it is assumed that a parent who is a user of the system 10 uses the system 10 to search for their lost child.

[0011] The system 10 has a monitoring area in which a person can be searched for. In the monitoring area, one or more cameras are installed to capture images of people in the area. It is assumed that each person in the monitoring area carries at least one device. Information identifying the device carried by each person is registered in advance in the system 10 together with information identifying each person. The devices are communication terminals capable of communicating with the system 10, and each device is assigned a unique code. Examples of such devices include wearable terminals such as smart watches and smartphones.

[0012] First, the system 10 identifies a "target device." The target device is a device carried by the person to be searched. The system 10 can identify that the target device carried by the person P1 is device D1 by referring to the linking data 121 that links people and devices. The linking data 121 is created based on person registration information and device registration information that are registered in the system 10 when the person enters the monitored area.

[0013] The system 10 then acquires a time series change in the motion of the target device, specifically, a time series change in the angular velocity and acceleration of the target device. The time series change in the motion of the target device can be expressed as a waveform. The system 10 can calculate a waveform representing the motion of the target device by continuously acquiring data detected by a gyro sensor and an acceleration sensor built into the device.

[0014] The system 10 also acquires an image of a person who may be person P1 from the camera 20. In the example of Fig. 1, the image captured by the camera 20 includes two unidentified persons X and Y. The identities of the persons X and Y are unknown at this point in time, but they may be the search target person P1. Hereinafter, in this specification, the persons captured by the camera 20, that is, the persons who may be the search target, are referred to as search target candidates.

[0015] For the search target candidates X and Y, the system 10 acquires time series changes in the body movements of each person. The time series changes in the body movements of each person can be represented by a waveform. The system 10 continuously acquires images captured by the camera 20, and calculates waveforms representing the body movements of the people X and Y by performing image analysis on the images.

[0016] The body movements of search target candidates X and Y are obtained, for example, as follows: A person recognized by image recognition is represented by a skeletal model with joint positions estimated, and a point of interest is set within the skeletal model. The point of interest is set, for example, at a position where a device is thought to be located, such as the waist position of the search target candidate. The movements of the point of interest in the X and Y directions are measured and obtained as changes in the movement of each person.

[0017] Alternatively, since the movement of the torso part and the movement of the head of the search target candidate are linked, the system 10 may measure the movement of the torso part of the search target candidate in the X and Y directions of the head of the search target candidate recognized in the image as the movement of the torso part of the search target candidate.

[0018] The system 10 compares the time series changes in the movement of the target device D1 with the time series changes in the movements of the search target candidates X and Y. Then, based on the degree of agreement between them, it is determined whether the search target candidate X or Y is the search target person P1.

[0019] Since the target device is a device possessed by the person to be searched for, it is highly likely that the person to be searched for carries the target device on his / her body or in his / her pocket or bag. In other words, it is highly likely that the target device moves in accordance with the body movements of the person to be searched for. Therefore, if the movement of the candidate searched for is similar to the movement of the target device, the system 10 determines that the candidate searched for is the person to be searched for.

[0020] In the example of Fig. 1, the degree of match between the waveform representing the movement of person X and the waveform representing the movement of device D1 is large, and the degree of match between the waveform representing the movement of person Y and the waveform representing the movement of device D1 is small. Therefore, the system 10 can determine that person X is the search target P1. In this way, the search target specified in the search request can be found.

[0021] 2 is a block diagram showing an example of the configuration of the system 10. The system 10 includes at least one computer 11 (hereinafter simply referred to as computer 11) and at least one storage device 12 (hereinafter simply referred to as storage device 12).

[0022] The storage device 12 stores association data 121 and one or more programs 122 (hereinafter simply referred to as programs 122). The association data 121 is data that associates one or more people registered in advance with the devices 30 owned by each of the one or more people. The association data 121 is used to identify the target device.

[0023] The program 122 is a computer program executed by the computer 11 and is composed of a plurality of instructions. Various processes are realized by the system 10 through cooperation between the computer 11 executing the program 122 and the storage device 12. The program 122 may be recorded on a computer-readable recording medium.

[0024] The camera 20 is installed at any location. For example, the camera 20 may be an infrastructure camera installed so as to be able to capture images of people within a predetermined area. The camera 20 may be a thermal camera that detects temperature. The camera 20 may also include multiple cameras. The computer 11 is configured so as to be able to communicate with the camera 20 via a wireless network or a wired network. The camera 20 continuously captures images within the imaging range at regular intervals, and the computer 11 acquires the continuously captured images as time-series data via the communication network.

[0025] The device 30 (device D) is a device that can be a target device, that is, a device that is registered in a manner associated with a person in the association data 121. There may be a plurality of devices 30. The computer 11 is configured to be able to communicate with each of the devices 30 via a wireless network.

[0026] Each device 30 has a built-in sensor 31. The sensor 31 includes, for example, a gyro sensor, an acceleration sensor, a GPS sensor, a magnetic sensor, a temperature sensor, a heart rate sensor, a microphone, etc. The computer 11 communicates with the device 30 to acquire information detected by the sensor 31, such as the movement of the device 30 represented by angular velocity, acceleration, etc., and the position of the device 30.

[0027] 3 is a flowchart showing an example of the flow of processing executed by the computer 11. The processing in each step is realized by the computer 11 executing the program 122.

[0028] In step S100, the computer 11 receives a search request including a designation of a person to be searched for. The search request may be a request from a user of the system 10, or may be a request from a system external to the system 10.

[0029] As an example of a situation in which an external system issues a search request, it is assumed that a system that tracks people issues a search request to system 10 when it loses sight of the person being tracked. Alternatively, the external system may be a system that identifies people based on appearance data. A system that identifies people based on appearance data acquires an image captured by camera 20, and compares the appearance of one or more people contained therein with preregistered appearance data to identify the target person. At this time, if the image contains multiple people with similar appearances and therefore it becomes impossible to identify a person from the appearance data alone, it is assumed that a search request is issued to system 10.

[0030] In step S200, the computer 11 identifies the target device. The computer 11 can identify the target device from the linking data 121 stored in the storage device 12 and information about the search target person specified in the search request received in step S100.

[0031] In step S300, computer 11 acquires a "person state quantity." A person state quantity is a state quantity related to a search target candidate. A time series change in the movement of a search target candidate is one example of a person state quantity. Computer 11 can acquire the person state quantity by image analysis of an image captured by camera 20, or as information itself detected by camera 20 such as a thermal camera.

[0032] Moreover, the search target candidates can be determined as follows. The computer 11 acquires an image from the camera 20, and extracts people included in the image by image analysis. Then, for example, all people extracted by image analysis may be set as search target candidates. Alternatively, the computer 11 may narrow down the search target candidates based on location information. For example, the computer 11 communicates with the target device and acquires location information detected by the GPS sensor of the target device. If the person to be searched for has the target device, the acquired location information is equal to the location of the person to be searched for. Therefore, the computer 11 may set only people close to the location of the target device among the people extracted from the image as search target candidates.

[0033] As another example, when a search request specifying a search target is issued from an external system and candidates for the search target are determined in the external system, the search target candidates may be specified by the external system. However, even in this case, the people that can be the search target candidates are people included in the images captured by the camera 20.

[0034] In step S400, the computer 11 acquires a device state quantity. The device state quantity is a state quantity acquired from the target device. A time-series change in the movement of the target device is one example of the device state quantity. The device state quantity may be a state quantity related to such a target device itself, or a state quantity related to the person being searched for acquired through the target device. Examples of the state quantity related to the person being searched for acquired through the target device include the heart rate and body temperature of the person being searched for. When the target device is a wearable terminal, the heart rate and body temperature of the person being searched for can be acquired from a heart rate sensor, a temperature sensor, or the like included in the sensor 31.

[0035] The order in which the person state amount and the device state amount are acquired is arbitrary, and the person state amount may be acquired after the device state amount is acquired.

[0036] In step S500, the computer 11 compares the acquired person state quantity with the device state quantity. Then, based on the degree of match between the person state quantity and the device state quantity, it is determined whether the search target candidate is a search target person. For example, a waveform representing the movement of the search target candidate and a waveform representing the movement of the target device may be compared, and the determination may be made based on the degree of match between the amplitude and rhythm pattern of the waveform.

[0037] The system 10 determines that the search target candidate is the search target person if the degree of match between the person state amount and the device state amount is large, and determines that the search target candidate is not the search target person if the degree of match is small. A specific method of determination in step S500 will be described later.

[0038] 2. Person and device states As the person state amount and the device state amount, various state amounts acquired from the camera 20 and the sensor 31 of the device 30 can be used. However, each state amount is selected to have some correspondence relationship with each other so that the degree of coincidence can be compared. Figure 4 shows an example of a combination of the person state amount and the device state amount.

[0039] For example, a direction represented by north, south, east, and west, that is, an orientation in a horizontal plane, may be used as the state quantity. In other words, the person state quantity may be the body orientation of the search target candidate in the horizontal plane, and the device state quantity may be the orientation of the device in the horizontal plane. Alternatively, when the search target candidate holds a device in his / her hand and the orientation of the device held by the search target candidate can be acquired from an image captured by the camera 20, the orientation of the device held by the search target candidate may be used as the person state quantity instead of the body orientation of the search target candidate. The computer 11 can acquire the body orientation of the search target candidate and the orientation of the device held by the search target candidate from the image captured by the camera 20, and the orientation of the device from a magnetic sensor built into the target device.

[0040] Alternatively, the person state quantity may be the content of the conversation of the search target candidate, and the device state quantity may be the content of the conversation of the search target person. The computer 11 can obtain the content of the conversation of the search target candidate by acquiring the mouth movement of the search target candidate from the image captured by the camera 20 and analyzing the acquired mouth movement. Also, the content of the conversation of the search target person can be obtained by analyzing the voice acquired by the microphone of the target device.

[0041] Alternatively, the person state quantity may be the pulse rate of the search target candidate, and the device state quantity may be the heart rate of the search target candidate. The pulse rate of the search target candidate is acquired by analyzing the movement of the blood vessels of the search target candidate from an image captured by the camera 20. When the target device is a wearable terminal such as a smart watch, the heart rate of the search target candidate is acquired from a heart rate sensor built into the target device.

[0042] Alternatively, the person state amount may be the moving speed of the search target candidate, and the device state amount may be the moving speed of the target device. The moving speed of the search target candidate is acquired by image analysis of the image captured by the camera 20. The moving speed of the target device is acquired by combining information detected by, for example, an acceleration sensor, a GPS sensor, or the like of the target device.

[0043] Alternatively, the person state quantity may be the surface temperature of the body of the search target candidate, and the device state quantity may be the body temperature of the search target candidate. The surface temperature of the body of the search target candidate can be acquired when the camera 20 is a thermal camera. The body temperature of the search target candidate can be acquired from a temperature sensor built into the target device when the target device is a wearable device.

[0044] 3. How to identify search targets The method of determining the search target person in step S500 will be described below with three examples.

[0045] The first example is a method for determining when there are multiple search target candidates. In step S500, the computer 11 compares the person state amount and the device state amount for each of the multiple search target candidates. Then, among the multiple search target candidates, the person whose person state amount and device state amount match the most is determined to be the search target. In the first example, one person who is most likely to be the search target is determined to be the search target.

[0046] In the second example, a discrimination is performed using a threshold value. There may be one or more search target candidates. In step S500, the computer 11 calculates the degree of agreement between the person state amount and the device state amount for each search target candidate as some value. For example, when the person state amount is a waveform representing the movement of the search target candidate and the device state amount is a waveform representing the movement of the target device, the ratio of overlap between the two waveforms may be calculated as a value representing the degree of agreement.

[0047] Then, the computer 11 compares the degree of match between the person state amount and the device state amount for each search target candidate with a threshold, and determines that the search target candidate with a degree of match equal to or greater than the threshold is the search target. The threshold is set in advance and stored in the storage device 12. In the second determination method, all search target candidates with a degree of match equal to or greater than the threshold, that is, search target candidates that are highly likely to be the search target, are determined as the search target. At this time, there may be a plurality of search target candidates determined to be the search target, or there may be no search target candidates determined to be the search target. In that case, the processes from step S300 to step S500 may be repeated until the search target is narrowed down to one person, or until the search target is found.

[0048] The third example is a method for determining whether there are multiple search target candidates, and is a combination of the first and second examples. The computer 11 first calculates a value indicating the degree of match between the person state amount and the device state amount for each of the multiple search target candidates, and selects the search target candidates whose degree of match is equal to or greater than a threshold. Then, among the selected search target candidates, the person with the highest degree of match is determined to be the search target. The threshold in the third example may be set lower than or equal to the threshold in the second example.

[0049] In the third example, among the multiple search target candidates, a person whose degree of match is less than a threshold is excluded from the candidates of the search target. Therefore, it is possible to prevent a person whose degree of match is less than the threshold and who is unlikely to be the search target from being determined to be the search target. Note that in the third example, there may be cases where there is no search target candidate that is determined to be the search target. In that case, the process from step S300 to step S500 may be repeated until the search target is found.

[0050] 4. First and second modified examples The first and second modified examples will be described. These are modified examples that take into consideration the position of a device carried by a search target candidate. These modified examples are applicable to the case where the person state amount is a state amount that represents a time series change in the movement of at least a part of the body of the search target candidate or the movement of belongings, and the device state amount is a state amount that represents a time series change in the movement of the target device.

[0051] 5 is a diagram for explaining the first and second modified examples. In FIG. 5, three search target candidates A, B, and C captured by the camera 20 are shown.

[0052] The search target candidate A is walking with a smartphone in his / her hand, and if the search target candidate A is a person to be searched for, the smartphone held by the search target candidate A in his / her hand is likely to be the target device. Therefore, if the search target candidate A is a person to be searched for, the movement of the hand of the search target candidate A is considered to be closer to the movement of the target device than the movement of the torso part of the search target candidate A. Therefore, if the search target candidate is holding a device in his / her hand, the computer 11 may acquire a state quantity representing a time-series change in the movement of the hand of the search target candidate as a person state quantity, and may compare, for example, the period of the swing of the hand of the search target candidate with the period of a waveform representing the movement of the target device.

[0053] Search target candidate B does not hold a device in his / her hand, and it is unknown whether or not he / she has a device, and if so, where he / she has the device. However, if he / she does have a device, it is highly likely that he / she has the device in a position close to the center of gravity of the body, such as a pocket. Therefore, if search target candidate B is a person to be searched for, it is considered that the movement of the torso part of search target candidate B will be close to the movement of the target device. Therefore, when search target candidate B does not hold a device in his / her hand, computer 11 may acquire, as a person state quantity, a state quantity that represents a time series change in the movement of the torso part of the search target candidate, for example, a position corresponding to the center of gravity of the body.

[0054] The search target candidate C is not holding a device in his / her hand. Instead, the search target candidate C is holding a bag in his / her hand. It is unclear whether the search target candidate C has a device in his / her hand, and if so, where he / she has it, but it is highly likely that the device is stored in the bag. Therefore, if the search target candidate C is a person to be searched for, there is a high possibility that the movement of the bag held by the search target candidate C will be close to the movement of the target device. Therefore, in such a case where the search target candidate is holding an item such as a bag, the computer 11 may acquire, as the person state amount, a state amount that represents a time series change in the movement of the item held by the search target candidate.

[0055] In the first modification, computer 11 detects the position of a device carried by a search target candidate based on an image captured by camera 20. Then, when the position of the device is detected, a person state quantity corresponding to the position of the device is acquired. That is, when a search target candidate holds a device in his / her hand like search target candidate A, a state quantity representing a time series change in the movement of the hand of the search target candidate is acquired as a person state quantity. When a search target candidate does not hold a device in his / her hand like search target candidate B or C, a state quantity representing a time series change in the movement of the torso part of the search target candidate and the movement of the carried item is acquired as a person state quantity.

[0056] In the second modified example, the computer 11 does not determine the position of the device carried by the search target candidate. Instead, the computer 11 acquires state quantities representing time-series changes for each of the movement of the hand of the search target candidate, the movement of the torso of the search target candidate, and the movement of the belongings, such as a bag, of the search target candidate. Then, each of these is compared with the device state quantity, and the one that matches the device state quantity most closely is used as the person state quantity of the search target candidate. In this way, even if the position of the device carried by the search target candidate is unknown, if any of the movements at each position of the hand, torso, belongings, etc. of the search target candidate matches the movement of the target device, the search target candidate can be determined to be the search target person, and the search target person can be identified with higher accuracy.

[0057] 5.Third Variation The state amount of the device may be acquired from a device other than the target device. FIG. 6 is a diagram for explaining a third modified example. The target device is a device D3 associated with a search target person P3 in the association data 121. However, the computer 11 acquires state amounts not only from the target device D3 but also from other devices D4 and D5 registered in the association data 121. The computer 11 also compares the state amounts acquired from each device other than the target device with the person state amounts acquired from each of the search target candidates Q and R. As a result, when the person state amount of a certain search target candidate is more consistent with the state amount acquired from a device other than the target device, for example, the device D4, than the device state amount, the computer 11 determines that the search target candidate is highly likely to be the person P4 who owns the device D4.

[0058] Since the person state amount and the device state amount are obtained by different methods, they may not be completely consistent. However, by obtaining state amounts from devices other than the target device in this way and comparing them with the person state amount to determine the person to be searched for, it is possible to specify the person to be searched for with higher accuracy.

[0059] 6.Effects As described above, according to the system of the present embodiment, the person who has requested to be found is identified based on the degree of agreement between the person state amount and the device state amount, based on the association data that associates the person with the device. This makes it possible to accurately find the person for whom the search request has been made.

[0060] Also, as described above, a search request may be issued from an external system. After the external system narrows down the candidates for the person to be searched for, the system according to the present embodiment can be used to identify the person with even greater accuracy. In this way, the system according to the present embodiment can improve the accuracy of identifying a specific person. [Explanation of symbols]

[0061] 10 system, 11 computer, 12 storage device, 20 camera, 30 device, 31 sensor, 121 data, 122 program

Claims

1. 1. A system for finding a person, comprising: a storage device that stores data linking one or more persons registered in advance with a device owned by each of the one or more persons; A camera capable of capturing an image of a person and at least one computer configured to be able to communicate with the camera, The at least one computer Upon being designated as a search target, acquiring a person state quantity which is a state quantity related to a search target candidate included in an image captured by the camera; acquiring a device state quantity, which is a state quantity acquired from a target device linked to the search target person; Based on the degree of agreement between the person state amount and the device state amount, it is determined whether the search target candidate is the search target person. system.

2. the person state amount is a state amount that represents a time series change in at least one of a body movement of at least a part of the search target candidate and a movement of an item carried by the search target candidate, the device state quantity is a time series change in angular velocity and acceleration of the device, The degree of match is a degree of match between a waveform representing the movement of the search target candidate calculated from the person state amount and a waveform representing the movement of the target device calculated from the device state amount. The system of claim 1 .

3. The at least one computer Detecting a location of a device carried by the search target candidate based on an image captured by the camera; Based on the location of the device, When the search target candidate is holding a device in his / her hand, a state quantity representing a time-series change in a movement of the hand of the search target candidate is acquired as the person state quantity; When the search target candidate does not hold a device in his / her hand, a state quantity representing a time series change in the movement of the center of gravity of the body of the search target candidate is acquired as the person state quantity. The system of claim 2.

4. the person state quantity is a state quantity that represents a time series change in each of a hand movement of the search target candidate, a movement of a center of gravity position of the body of the search target candidate, and a movement of an item carried by the search target candidate, The degree of match is determined by comparing each time series change included in the person state amount with the device state amount. The system of claim 2.

5. the search target candidates are a plurality of people, The at least one computer calculating a degree of match between the person state amount and the device state amount for each of the plurality of search target candidates; The search target candidate having the highest degree of match is determined to be the search target person. The system of claim 1 .

Images