Target management system

Abstract

The present invention provides a technique capable of improving determination accuracy even in cases where it is not clear about whether or not a subject is the same as a specific target. A target management system is provided with one or more storage devices that store registration images for each of one or more targets, and one or more processors. The one or more processors acquire a subject image, which is an image of a subject shown on the camera, execute a determination process for determining whether the subject is the same as one or more targets on the basis of the registered image and the subject image, and if the result of the determination process satisfies a specific condition, determine whether the subject is the same as the one or more targets. A determination request requesting to determine who the subject is, and the subject image are provided to the user terminal, and a determination response indicating that the subject is the same as a specific target among the one or more targets is received from the user terminal in response to the determination request. The association between the subject and the specific target in the determination process is enhanced.

Description

Technical Field

[0001] The present disclosure relates to a technology for managing targets. Background Art

[0002] In Patent Document 1, a monitoring device for monitoring the processing content of a biometric authentication device is disclosed. The monitoring device displays the processing results of, for example, the facial images of passers-by who are not authenticated as registered persons in the authentication process based on facial images. By visually observing the processing results by a monitor, the monitor can immediately confirm whether the passer-by is a registered person.

[0003] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2009-093371

[0004] There is a known technology for determining whether a photographed object is the same as a specific target based on an image captured by a camera. Consider cases where it is not clear whether the photographed object is the same as a specific target. Summary of the Invention

[0005] One object of the present disclosure is to provide a technology that can improve the determination accuracy even when it is not clear whether a photographed object is the same as a specific target.

[0006] The first aspect relates to a target management system.

[0007] The target management system includes:

[0008] One or more storage devices that store respective registration images of one or more targets; and

[0009] One or more processors,

[0010] The one or more processors are configured to:

[0011] Acquire an image of the photographed object reflected in the camera, that is, the photographed object image,

[0012] Execute a determination process for determining whether the photographed object is the same as a certain target among the one or more targets based on the registration image and the photographed object image,

[0013] When the result of the determination process satisfies a specific condition, provide a determination request for requesting determination of who the photographed object is, together with the photographed object image, to a user terminal,

[0014] When a determination response indicating that the photographed object is the same as a specific target among the one or more targets is received from the user terminal in response to the determination request, strengthen the association between the photographed object and the specific target in the determination process.

[0015] According to the present disclosure, even when it is impossible to determine whether the photographed object is the same as a certain one of the targets through the determination process, by requesting a user who can reliably distinguish the photographed object from the targets via a determination request to make a determination, a correct determination can be made. In addition, since the association between the photographed object image and a specific target is strengthened by accumulating determination responses, the accuracy of the determination process gradually improves. Along with this, the provision frequency of the determination request gradually decreases. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a schematic diagram showing the overall view of the target management system.

[0017] Figure 2 is a schematic diagram for explaining an example of the determination process.

[0018] Figure 3 is a block diagram showing a configuration example of the management device.

[0019] Figure 4 is a flowchart showing an example of the processing path of the target management system.

[0020] Figure 5 is a flowchart corresponding to the second modification example.

[0021] Figure 6 is a schematic diagram showing an example of the display screen of the user terminal when the photographed object image and the determination request are provided together.

[0022] DESCRIPTION OF REFERENCE NUMERALS:

[0023] 1... target management system; 20... camera; 30... user terminal; 40... determination processing unit; 41... feature quantity extraction unit; 42... similarity calculation unit; 43... threshold determination unit; 100... management device; 110... control device; 120... processor; 130... storage device; 140... communication device; FEA-SJ... photographed object feature quantity; FEA-Ti... target feature quantity; IMG-SJ... photographed object image; IMG-Ti... registered image; REQ... determination request; RES... determination response; SJ... photographed object; SML-i... image similarity; Ti... target. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0024] Embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0025] 1. Target Management System

[0026] 1-1. Overall View of the Target Management System

[0027] Figure 1This is a schematic diagram showing the overall picture of the target management system 1. The target management system 1 includes a camera 20, a user terminal 30, and a management device 100.

[0028] The camera 20 takes pictures of the entire set area. The area where the camera 20 is set is preferably a place where one or more targets Ti (hereinafter simply referred to as target Ti) as management objects appear regularly or frequently. The user terminal 30 is an information terminal that the user U can operate. The user terminal 30 has a user interface for prompting information to the user U or accepting input from the user U. The user terminal 30 and the management device 100 can communicate with each other via a wireless or wired communication network. As a specific example of the user terminal 30, it can be devices such as a smart phone, a tablet computer, a PC, etc. As the user interface, a touch panel, a PC monitor, etc. are exemplified. The management device 100 manages the target management system 1. Typically, the management device 100 is a management server on the cloud. The management device 100 can also be composed of multiple servers performing distributed processing.

[0029] In addition, the camera 20 is an example of a camera used in the target management system 1. In fact, multiple cameras can be set throughout the entire living area of the target Ti (for example, the entire street). In this case, each camera communicates with the management device 100 via a communication network, and its captured data is centrally managed by the management device 100.

[0030] The structure of the target management system 1 will be briefly described. The management device 100 obtains the video image VID from the camera 20 and extracts the image of the object SJ reflected in the video image VID (object image IMG - SJ). The management device 100 performs a determination process to determine whether the object SJ is the same as a certain one of the targets Ti. In the determination process, the object image IMG - SJ is compared with the image of the target Ti (referred to as the registered image IMG - Ti) pre-registered in the management device 100. When the result of the determination process meets a specific condition, the management device 100 provides the object image IMG - SJ and a determination request REQ to the user terminal 30. The determination request REQ is a request for the user U to determine who the object SJ is. The user U sends a determination response RES via the user terminal 30, and the determination response RES is a response to the determination request REQ. When the content of the determination response RES indicates that the object SJ is the same as a specific target Ts among the targets Ti, the management device 100 strengthens the association between the object SJ and the specific target Ts in the determination process.

[0031] As one of the environments where the target management system 1 is utilized, an example can be given where a school-going child is supervised by the child's parents. In this case, the target Ti is the school-going child, and the user U is the parent of the target Ti. The camera 20 is preferably installed at the school gate or the like. The registration image IMG-Ti is registered in the management device by the user U via the user terminal 30. The user U does not necessarily have to be the parent of the target Ti, but is preferably a person who can reliably determine whether the subject SJ is the same as one of the targets Ti by viewing the subject image IMG-SJ.

[0032] 1-2. Determination process

[0033] Figure 2 It is a schematic diagram for explaining an example of the determination process. The main body of the determination process is the determination processing unit 40. The determination process includes processes such as feature quantity extraction, similarity calculation, and threshold determination.

[0034] The feature quantity extraction unit 41 extracts the subject feature quantity FEA-SJ from the subject image IMG-SJ using a machine learning model. Similarly, the feature quantity extraction unit 41 extracts the target feature quantity FEA-Ti from the registration image IMG-Ti. The machine learning model is pre-generated by machine learning such as deep learning. The specific forms of the subject feature quantity FEA-SJ and the target feature quantity FEA-Ti are feature vectors representing the features included in each image.

[0035] The similarity calculation unit 42 calculates the image similarity SML-i by comparing the subject feature quantity FEA-SJ and the target feature quantity FEA-Ti. Specifically, the closer the subject feature quantity FEA-SJ is to the target feature quantity FEA-Ti, the higher the image similarity SML-i is estimated. Conversely, the farther the subject feature quantity FEA-SJ is from the target feature quantity FEA-Ti, the lower the image similarity SML-i is estimated.

[0036] The threshold determination unit 43 compares the image similarity SML-i with a specified threshold. For example, when the image similarity SML-i is less than the first specified value TH1, the threshold determination unit 43 determines that the subject SJ is different from the target Ti. On the other hand, when the image similarity SML-i is greater than the second specified value TH2, the threshold determination unit 43 determines that the subject SJ is the same as the target Ti. Usually, the second specified value TH2 is set to be higher than the first specified value TH1.

[0037] There is also considered a case where it is unclear whether the subject SJ is the same as the target Ti (that is, a case where the reliability of the determination process is low). For example, a case can be cited where the target Ti is a pair of twins and it is unclear which of the target Ti as twins the subject SJ is. Such a case is defined as a "specific condition". As an example of a specific condition, a case can be cited where the image similarity SML-i is within a constant range (for example, not less than a first specified value TH1 and not more than a second specified value TH2).

[0038] In a case where the result of the determination process satisfies the specific condition, it is unclear whether the subject SJ is the same as the target Ti. Therefore, in a case where the result of the determination process satisfies the specific condition, the management device 100 provides the determination request REQ to the user terminal 30. The determination request REQ requests the user U to determine who the subject SJ is. In other words, the management device 100 issues the determination request REQ when it is impossible to determine with high reliability whether the subject SJ is the same as the target Ti based on the result of the determination process performed by the management device 100 itself.

[0039] In a case where the determination response RES indicates that the subject SJ is the same as the specific target Ts, the feature amount extraction unit 41 updates the machine learning model. Specifically, the feature amount extraction unit 41 updates the machine learning model in such a way that the "subject feature amount FEA-SJ extracted from the subject image IMG-SJ" is closer to the "specific target feature amount FEA-Ts extracted from the registration image of the specific target Ts (referred to as the specific registration image IMG-Ts)". In other words, the feature amount extraction unit 41 updates the machine learning model in such a way that the specific image similarity SML-s between the current subject SJ and the specific target Ts is further increased. That is, the association between the current subject SJ and the specific target Ts in the determination process is strengthened.

[0040] Regarding the subject feature quantity FEA-SJ and the target feature quantity FEA-Ti used in feature quantity extraction, facial images are mainly utilized. Facial images can be said to contain the feature quantities (inherent feature quantities) unique to an individual, and thus are effective in the determination process. On the other hand, the feature quantity extraction unit 41 may also have a machine learning model that extracts temporary feature quantities (clothing, hairstyle) in addition to extracting inherent feature quantities. The temporary feature quantities are used in the determination process only for a certain period. For example, the feature quantity extraction unit 41 stores the temporary feature quantities of the target Ti extracted for the first time on a certain day in the management device 100. The similarity calculation unit 42 compares the "temporary feature quantities extracted for the second time or later on this day" with the "stored temporary feature quantities" to calculate the "temporary similarity". The similarity calculation unit 42 combines the "similarity based on inherent feature quantities" and the "temporary similarity" to calculate the "comprehensive similarity". In this case, the threshold determination unit 43 uses the comprehensive similarity for threshold determination. After this day has passed, the feature quantity extraction unit 41 resets the "temporary feature quantities". In addition, the period during which the temporary feature quantities are stored and used in the determination process is not limited to 1 day, and may be other periods such as 1 week or 1 month.

[0041] 2. Structure and Processing Path of the Management Device

[0042] Figure 3 It is a block diagram showing a configuration example of the management device 100.

[0043] The management device 100 includes a control device 110 and a communication device 140. The communication device 140 communicates with the user terminal 30 and the camera 20. The control device 110 controls the management device 100. The control device 110 includes one or more processors 120 (hereinafter simply referred to as the processor 120) and one or more storage devices 130 (hereinafter simply referred to as the storage device 130). The processor 120 performs various processes. For example, the processor 120 includes a CPU (Central Processing Unit). The processor 120 can also be referred to as a processing circuitry. The storage device 130 stores various information required for the processes performed by the processor 120. Examples of the storage device 130 include a volatile memory, a non-volatile memory, an HDD (Hard Disk Drive), an SSD (Solid State Drive), etc.

[0044] The management program PROG is a computer program executed by the processor 120. By executing the management program PROG by the processor 120, the functions of the control device 110 can also be realized. The management program PROG is stored in the storage device 130. Alternatively, the management program PROG can also be recorded on a computer-readable recording medium. The management program PROG can also be provided via a network.

[0045] The control device 110 performs data transmission and reception with the camera 20 and the user terminal 30 via the communication device 140. In addition, the control device 110 functions as the determination processing unit 40 that executes the above determination processing.

[0046] The storage device 130 stores the registered image IMG-Ti and the target feature amount FEA-Ti. When the above temporary feature amount is used in the determination processing, the temporary feature amount is stored in the storage device 130.

[0047] Figure 4 It is a flowchart showing an example of the processing path of the target management system 1.

[0048] In step S10, the processor 120 acquires the captured object image IMG-SJ and estimates the image similarity SML-i between the captured object image IMG-SJ and the registered image IMG-Ti.

[0049] In step S11, the processor 120 compares the image similarity SML-i with a threshold value. For example, when the image similarity SML-i is less than the first specified value TH1, the processor 120 determines that the captured object SJ is different from the target Ti. On the other hand, when the image similarity SML-i is greater than the second specified value TH2, the processor 120 determines that the captured object SJ is the same as the target Ti. Step S10 and step S11 correspond to the determination processing. However, when the image similarity SML-i is equal to or greater than the first specified value TH1 and equal to or less than the second specified value TH2, it is not clear whether the captured object SJ is the same as the target Ti, that is, a specific condition is satisfied. In this case, the process proceeds to step S12.

[0050] In step S12, the processor 120 provides the captured object image IMG-SJ and the determination request REQ to the user terminal 30. The process proceeds to step S13.

[0051] In step S13, the processor 120 determines the content of the next process according to the content of the determination response RES for the determination request REQ sent from the user terminal 30. When the content of the determination response RES is "the subject SJ is the same as the specific target Ts" (step S13: Yes), the process proceeds to step S14. On the other hand, when the determination response RES indicates that "the subject SJ is not the same as the specific target Ts" (step S13: No), the process ends.

[0052] In step S14, the processor 120 strengthens the association between the subject SJ and the specific target Ts. After that, the process ends.

[0053] 3. Effects

[0054] Even when the management device 100 cannot determine whether the subject SJ is the same as the target Ti, the target management system 1 in the present disclosure can request the user U who can reliably distinguish the subject SJ from the target Ti via the determination request REQ for determination, so that correct determination can be performed. In addition, since the association between the subject image IMG - SJ and the specific target Ts is strengthened by accumulating the determination response RES, the accuracy of the determination process performed by the management device 100 itself is gradually improved. Along with this, the provision frequency of the determination request REQ is gradually reduced. In other words, the target management system 1 can efficiently learn the characteristics of the target Ti via the determination request REQ and the determination response RES, so that the accuracy of the determination process can be gradually improved.

[0055] The above-mentioned temporary feature quantities (such as clothing, hairstyle, etc.) are particularly effective immediately after starting to use the target management system 1. That is, even in the stage where the feature learning for the target Ti has not been advanced, the determination accuracy can be temporarily improved by using the temporary feature quantities.

[0056] 4. Variants

[0057] 4 - 1. The First Variant

[0058] As another example of providing specific conditions for the determination request REQ, the following example can be cited: in the determination process, for the target Ti, the image similarity SML-i equal to or greater than the specified value is calculated. After that, within a specified time, for the same target Ti, the image similarity SML-i equal to or greater than the specified value is calculated again. Depending on the installation location of the camera 20, it is unlikely that the same person will appear at the location multiple times within the specified time. For example, the camera 20 is installed at the school gate. During the school arrival time period, for a certain subject, the image similarity SML-i (equal to or greater than the specified value) with the target Ti is calculated. After that, within the same time period, for a certain subject, the image similarity SML-i (equal to or greater than the specified value) with the same target Ti is calculated. Since the same child (student) usually cannot go to school twice in a short period, in this case, it is very likely that one of the two determination processes performed by the management device 100 is incorrect. In this case, by sending the determination request REQ and using the determination response RES of the user U, the management device 100 can obtain feedback on the determination result obtained by its own determination.

[0059] 4-2. Second Modified Example

[0060] In this section, the case where there are multiple targets Ti is described. Here, the target Ti is distinguished as the first target T1 and the second target T2 for explanation. The registered images of the first target T1 and the second target T2 are respectively referred to as the first registered image IMG-T1 and the second registered image IMG-T2. Figure 5 It is a flowchart corresponding to the second modified example.

[0061] In step S20, the processor 120 calculates the first image similarity SML-1 between the subject image IMG-SJ and the first registered image IMG-T1. Similarly, the processor 120 also calculates the second image similarity SML-2 between the subject image IMG-SJ and the second registered image IMG-T2.

[0062] In step S21, the processor 120 compares the magnitude of the first image similarity SML-1 with the first specified value TH1 and the second specified value TH2. Similarly, the processor 120 compares the magnitude of the second image similarity SML-2 with the first specified value TH1 and the second specified value TH2. When at least one of the first image similarity SML-1 or the second image similarity SML-2 is equal to or greater than the first specified value TH1 and less than or equal to the second specified value TH2 (step S21: Yes), the process proceeds to Figure 4 step S12. In other cases (step S21: No), the process proceeds to step S22.

[0063] In step S22, the processor 120 compares the first image similarity SML-1 and the second image similarity SML-2 with the second specified value TH2. When both the first image similarity SML-1 and the second image similarity SML-2 are greater than the second specified value TH2 (step S22: Yes), the process proceeds to Figure 4 step S12. In other cases (step S22: No), the process ends.

[0064] The situation where it becomes "Yes" in step S21 is a situation where "the image similarity SML-i of a certain one of the targets Ti is within a certain range, and it is not clear whether the subject SJ is the same as the target Ti". On the other hand, the situation where it becomes "Yes" in step S22 is a situation where "the image similarity SML-i of any of the targets Ti is high, and any of the targets Ti can be determined to be the same as the subject SJ". In this situation, it is reasonable to proceed to step S12 and provide the subject image IMG-SJ and the determination request REQ to the user terminal 30.

[0065] As described above, even if there are multiple targets Ti, the target management system 1 can be applied. Especially when the appearances of the targets Ti are extremely similar to each other (for example, when the targets Ti are twins or other multiples, or siblings with extremely similar appearances), it is considered that in most cases, the management device 100 alone cannot make a correct determination. On the other hand, if it is the user U (typically the parent of the target Ti), then the extremely similar targets Ti can be reliably distinguished from each other. Therefore, the feature of the target management system 1 that utilizes the assistance based on the user U via the determination request REQ can be particularly utilized.

[0066] 5. Example of determination request display

[0067] Figure 6 is a schematic diagram showing an example of the display screen of the user terminal 30 when the subject image IMG-SJ and the determination request REQ are provided together.

[0068] Figure 6Example of a display screen when only the first target T1 (here, "Satoshi") is registered in (A). As the content of the determination request REQ, three items, namely, "Satoshi", "others", and "unknown", are displayed. When "Satoshi" is selected as the determination response RES, the feature quantity extraction unit 41 strengthens the association between the subject SJ and "Satoshi" (the first target T1). That is, "Satoshi" (the first target T1) is regarded as the specific target Ts. When "others" or "unknown" is selected as the determination response RES, the feature quantity extraction unit 41 does not update the machine learning model. In addition, when "others" is selected, the feature quantity extraction unit 41 may also use other machine learning models to learn the feature quantity FEA-SJ of the subject included in the subject image IMG-SJ as a parameter representing others (a being different from the first target T1).

[0069] Figure 6 Example of a display screen when a second target ("Hiroshi") is registered in addition to the first target ("Satoshi") in (B). As the content of the determination request REQ, four options, namely, "Satoshi", "Hiroshi", "others", and "unknown", are displayed. When "Satoshi" or "Hiroshi" is selected as the determination response RES, the feature quantity extraction unit 41 strengthens the association between the subject SJ and the selected one (i.e., the specific target Ts). When "others" or "unknown" is selected, the subsequent processing is the same as that in Figure 6 the example of (A).

[0070] Hereinafter, methods other than the above method that can be considered are given. In addition, the illustration is omitted.

[0071] The management device 100 may also request the user U to provide an image that satisfies a specified condition via the user terminal 30. For example, when the accuracy of the determination process for an image taken from a specific angle is particularly low, the management device 100 requests the user U to provide an image of the target Ti taken from the specific angle. In this case, a message urging the provision of an image of the target Ti taken from the specific angle is displayed on the user terminal 30 together with the determination request REQ. The image provided through this image request is an image that improves an important factor with low determination accuracy so far. Therefore, the feature quantity extraction unit 41 efficiently improves the determination accuracy by learning the feature quantity included in this image. On the contrary, the user U may also request the management device 100 to provide an image that satisfies a specified condition. For example, when the user U cannot make a determination response RES based on an image taken from the angle of the camera 20, the user U may also request the management device 100 to provide an image taken from another angle by a camera other than the camera 20. In this case, a column for the user U to input requirements is provided in the display screen of the determination request REQ.

[0072] Moreover, when the user U determines that the subject SJ is the same as the specific target Ts, a determination response RES may also be made on the basis of clearly indicating a specific part of the subject image IMG-SJ. For example, the user U clearly indicates the part of the face to be focused on for determination by clicking or single-clicking. The feature quantity extraction unit 41 uses the features related to the clearly indicated part to establish the association between the subject SJ and the specific target Ts. In this case, since the information on the features to be focused on and the content of the determination response RES to the determination request REQ are also accumulated together, the association between the subject SJ and the specific target Ts can be promoted more efficiently.

Claims

1. A target management system, wherein, the target management system includes: one or more storage devices storing respective registration images of one or more targets; and one or more processors, the one or more processors are configured to: acquire an image of an object imaged by a camera, that is, an object image, perform a determination process of determining whether the object is the same as a certain target among the one or more targets based on the registration image and the object image, when the result of the determination process satisfies a specific condition, provide a determination request for determining who the object is and the object image to a user terminal together, when receiving a determination response indicating that the object is the same as a specific target among the one or more targets in response to the determination request from the user terminal, strengthen the association between the object and the specific target in the determination process.

2. The target management system according to claim 1, wherein, the one or more targets include a first target, in the determination process, the one or more processors calculate a first image similarity between the registration image of the first target and the object image, the specific condition is that the first image similarity is equal to or greater than a first specified value and equal to or less than a second specified value.

3. The target management system according to claim 1, wherein, when receiving the determination response indicating that the object is the same as the specific target from the user terminal, the one or more processors are configured to: temporarily use a temporary feature amount of the object included in the object image as a temporary feature amount representing the specific target in the determination process, in the determination process, use an inherent feature amount of the object included in the object image as an inherent feature amount representing the specific target for a longer time than the temporary feature amount.

4. The target management system according to claim 1, wherein, the one or more targets include a first target and a second target, in the determination process, the one or more processors calculate a first image similarity between the registration image of the first target and the object image and a second image similarity between the registration image of the second target and the object image, the specific condition is that at least one of the first image similarity and the second image similarity is equal to or greater than a first specified value and equal to or less than a second specified value, or both the first image similarity and the second image similarity are greater than the second specified value.

5. The target management system according to claim 4, wherein, the first target and the second target are in a relationship of being twins or siblings.

Images