Target managing system

Abstract

To provide a technology capable of improving the determination precision even under a case in which it is unclear whether an object is consistent with a certain target or not.SOLUTION: A target managing system includes: one or a plurality of storage devices that store respective registered images of one or a plurality of targets; and one or a plurality of processors. The one or a plurality of processors are configured to: obtain an object image that is an image of an object picked up by a camera; execute a determination process for determining, based on the registered image and on the object image, whether the object is consistent with the one target or any one of the plurality of targets; provide, to a user terminal, a determination request for requesting a determination on who is the object together with the object image when the determination result satisfies a certain condition; and enhance the association between the object and the certain target in the determination process when receiving, from the user terminal relative to the determination request, a determination response indicating that the object is consistent with the one object or the certain target among the plurality of targets.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present disclosure relates to a technology for managing targets.

Background Art

[0002] Patent Document 1 discloses a monitoring device that monitors the processing content of a biometric authentication device. The monitoring device displays processing results such as the face image of a passerby who has not been authenticated as a registered person in the face image-based authentication process. By visually checking the processing results by a monitor, the monitor can immediately confirm whether the passerby is a registered person.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] A technology for determining whether a subject is the same as a specific target based on an image captured by a camera is known. There may be cases where it is not clear whether the subject is the same as a specific target.

[0005] One object of the present disclosure is to provide a technology capable of improving the determination accuracy even in cases where it is not clear whether a subject is the same as a specific target.

Means for Solving the Problems

[0006] A first aspect relates to a target management system. The target management system includes one or more storage devices that store registration images of each of one or more targets, one or more processors and is provided with. One or more processors obtain a subject image that is an image of a subject captured by a camera, execute a determination process to determine whether the subject is identical to any one of one or more targets based on a registered image and the subject image, when the result of the determination process satisfies a specific condition, provide a determination request for requesting a determination as to who the subject is to a user terminal together with the subject image, when a determination response indicating that the subject is identical to a specific target among one or more targets is received from the user terminal in response to the determination request, strengthen the association between the subject and the specific target in the determination process.

Advantages of the Invention

[0007] According to the present disclosure, even when it is impossible to determine whether the subject and the target are identical by the determination process, by requesting the user who can surely distinguish the subject and the target through the determination request, a correct determination becomes possible. Further, since the determination response is accumulated, the association between the subject image and the specific target is strengthened, so that the accuracy of the determination process is gradually improved. Along with this, the frequency of providing the determination request gradually decreases.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Modes for Carrying Out the Invention

[0009] With reference to the accompanying drawings, embodiments of the present disclosure will be described.

[0010] 1. Target Management System 1-1. Overview of the Target Management System FIG. 1 is a schematic diagram showing the overall configuration of a target management system 1. The target management system 1 includes a camera 20, a user terminal 30, and a management device 100.

[0011] The camera 20 is photographing the entire installed area. The area where the camera 20 is installed is preferably a place where one or more targets Ti (hereinafter simply referred to as target Ti) to be managed appear regularly or frequently. The user terminal 30 is an information terminal operable by the user U. The user terminal 30 includes a user interface that presents information to the user U and receives inputs from the user U. The user terminal 30 and the management device 100 can communicate with each other via a wireless or wired communication network. Specific examples of the user terminal 30 include devices such as smartphones, tablets, and PCs. Examples of the user interface include a touch panel and a PC display. The management device 100 manages the target management system 1. Typically, the management device 100 is a management server on the cloud. The management device 100 may be composed of a plurality of servers that perform distributed processing.

[0012] Note that the camera 20 is an example of a camera used in the target management system 1. In reality, a plurality of cameras may be installed over the entire living area (e.g., the entire town) of the target Ti. In this case, each camera communicates with the management device 100 via a communication network, and the imaging data thereof is intensively managed by the management device 100.

[0013] The mechanism of the target management system 1 will be briefly described. The management device 100 acquires the video VID from the camera 20 and extracts the image of the subject SJ (subject image IMG-SJ) shown in the video VID. The management device 100 executes a determination process to determine whether the subject SJ is the same as any of the targets Ti. In the determination process, the subject image IMG-SJ is compared with the image of the target Ti (referred to as the registered image IMG-Ti) registered in the management device 100 in advance. When the result of the determination process satisfies a specific condition, the management device 100 provides the subject image IMG-SJ and the determination request REQ to the user terminal 30. The determination request REQ requests the user U to determine who the subject SJ is. The user U transmits a determination response RES, which is a response to the determination request REQ, via the user terminal 30. When the content of the determination response RES indicates that the subject SJ is the same as a specific target Ts among the subject SJ and the target Ti, the management device 100 strengthens the association between the subject SJ and the specific target Ts in the determination process.

[0014] As an example of the environment in which the target management system 1 is used, there is an example where a child going to school is watched over by the child's parent. In this case, the target Ti is the child going to school, and the user U is the parent of the target Ti. The camera 20 is preferably installed at the school gate or the like. The registered image IMG-Ti is registered in the management device via the user terminal 30 by the user U. The user U does not necessarily have to be the parent of the target Ti, but it is desirable to have an entity that can surely determine whether the subject SJ is the same as any of the target Ti by looking at the subject image IMG-SJ.

[0015] 1-2. Determination Process FIG. 2 is a schematic diagram for explaining an example of the determination process. The main body of the determination process is the determination processing unit 40. The determination process includes steps such as feature amount extraction, similarity calculation, and threshold value determination.

[0016] The feature extraction unit 41 extracts the subject feature amount FEA-SJ from the subject image IMG-SJ using a machine learning model. Similarly, the feature extraction unit 41 extracts the target feature amount FEA-Ti from the registered image IMG-Ti. The machine learning model is generated in advance through machine learning such as deep learning. The specific forms of the subject feature amount FEA-SJ and the target feature amount FEA-Ti are feature vectors representing the features included in the respective images.

[0017] The similarity calculation unit 42 calculates the image similarity SML-i by comparing the subject feature amount FEA-SJ and the target feature amount FEA-Ti. Specifically, the closer the subject feature amount FEA-SJ and the target feature amount FEA-Ti are, the higher the calculated image similarity SML-i. Conversely, the farther the subject feature amount FEA-SJ and the target feature amount FEA-Ti are, the lower the calculated image similarity SML-i.

[0018] The threshold determination unit 43 compares the image similarity SML-i with a predetermined threshold. For example, when the image similarity SML-i is less than the first predetermined value TH1, the threshold determination unit 43 determines that the subject SJ is not the same as the target Ti. On the other hand, when the image similarity SML-i is greater than the second predetermined value TH2, the threshold determination unit 43 determines that the subject SJ is the same as the target Ti. Usually, the second predetermined value TH2 is set higher than the first predetermined value TH1.

[0019] There may also be a case where it is not clear whether the subject SJ is the same as the target Ti (that is, a case where the reliability of the determination process is not high). For example, a case where the target Ti is a twin and it is not clear which of the twins the subject SJ is can be cited. Such a case is defined as a "specific condition". As an example of the specific condition, the image similarity SML-i being within a certain range (for example, not less than the first predetermined value TH1 and not more than the second predetermined value TH2) can be cited.

[0020] When the result of the determination process satisfies a specific condition, it is not clear whether the subject SJ is the same as the target Ti. Therefore, when the result of the determination process satisfies the specific condition, the management device 100 provides a determination request REQ to the user terminal 30. The determination request REQ requests the user U to determine who the subject SJ is. In other words, the management device 100 issues the determination request REQ when it cannot determine with high confidence whether the subject SJ is the same as the target Ti based on the result of the determination process performed by the management device 100 itself.

[0021] When the determination response RES indicates that the subject SJ and the specific target Ts are the same, the feature extraction unit 41 updates the machine learning model. Specifically, the feature extraction unit 41 updates the machine learning model so that the "subject feature amount FEA-SJ extracted from the subject image IMG-SJ" and the "specific target feature amount FEA-Ts extracted from the registered image of the specific target Ts (referred to as the specific registered image IMG-Ts)" become closer. In other words, the feature extraction unit 41 updates the machine learning model so that the specific image similarity SML-s between the current subject SJ and the specific target Ts further increases. That is, the association between the current subject SJ and the specific target Ts in the determination process is strengthened.

[0022] For the subject feature amount FEA-SJ and the target feature amount FEA-Ti used for feature amount extraction, face images are mainly used. Since it can be said that face images contain feature amounts (unique feature amounts) unique to individuals, they are effective in the determination process. On the other hand, the feature amount extraction unit 41 may be provided with a machine learning model that extracts temporary feature amounts (clothing and hairstyles) separately from the extraction of unique feature amounts. The temporary feature amounts are used for the determination process only for a certain period. For example, the feature amount extraction unit 41 holds the temporary feature amount of the target Ti extracted for the first time on a certain day in the management device 100. The similarity calculation unit 42 calculates the "temporary similarity" by comparing the "temporary feature amounts extracted after the second time on that day" with the "held temporary feature amount". The similarity calculation unit 42 calculates the "integrated similarity" by combining the "similarity based on unique feature amounts" and the "temporary similarity". In this case, the threshold determination unit 43 uses the integrated similarity for threshold determination. The feature amount extraction unit 41 resets the "temporary feature amount" when that day has passed. Note that the period during which the temporary feature amount is held and used for the determination process is not limited to one day, and may be other periods such as one week or one month.

[0023] 2. Configuration and Processing Route of Management Device FIG. 3 is a block diagram showing a configuration example of the management device 100.

[0024] The management device 100 includes a control device 110 and a communication device 140. The communication device 140 communicates with the user terminal 30 and the camera 20. The control device 110 controls the management device 100. The control device 110 includes one or more processors 120 (hereinafter simply referred to as the processor 120) and one or more storage devices 130 (hereinafter simply referred to as the storage device 130). The processor 120 executes various processes. For example, the processor 120 includes a CPU (Central Processing Unit). The processor 120 can also be referred to as processing circuitry. The storage device 130 stores various information necessary for the processes by the processor 120. Examples of the storage device 130 include a volatile memory, a non-volatile memory, an HDD (Hard Disk Drive), an SSD (Solid State Drive), etc.

[0025] The management program PROG is a computer program executed by the processor 120. By the processor 120 executing the management program PROG, the functions of the control device 110 may be realized. The management program PROG is stored in the storage device 130. Alternatively, the management program PROG may be recorded on a computer-readable recording medium. The management program PROG may be provided via a network.

[0026] The control device 110 transmits and receives data to and from the camera 20 and the user terminal 30 via the communication device 140. Also, the control device 110 functions as a determination processing unit 40 that executes the above-described determination process.

[0027] The storage device 130 stores the registered image IMG-Ti and the target feature amount FEA-Ti. When the above-described temporary feature amount is used in the determination process, the temporary feature amount is stored in the storage device 130.

[0028] Figure 4 is a flowchart showing an example of the processing path of the target management system 1.

[0029] In step S10, the processor 120 acquires the subject image IMG-SJ and calculates the image similarity SML-i between the subject image IMG-SJ and the registered image IMG-Ti.

[0030] In step S11, the processor 120 compares the image similarity SML-i with a threshold value. For example, when the image similarity SML-i is less than the first predetermined value TH1, the processor 120 determines that the subject SJ is not the same as the target Ti. On the other hand, when the image similarity SML-i is greater than the second predetermined value TH2, the processor 120 determines that the subject SJ is the same as the target Ti. Steps S10 and S11 correspond to the determination process. However, when the image similarity SML-i is greater than or equal to the first predetermined value TH1 and less than or equal to the second predetermined value TH2, it is not clear whether the subject SJ is the same as the target Ti, that is, a specific condition is satisfied. In this case, the process proceeds to step S12.

[0031] In step S12, the processor 120 provides the subject image IMG-SJ and the determination request REQ to the user terminal 30. The process proceeds to step S13.

[0032] In step S13, the processor 120 determines the next processing content according to the content of the determination response RES to the determination request REQ transmitted from the user terminal 30. When the content of the determination response RES is "the subject SJ and the specific target Ts are the same" (step S13: YES), the process proceeds to step S14. On the other hand, when the determination response RES indicates that "the subject SJ and the specific target Ts are not the same" (step S13: NO), the process ends.

[0033] In step S14, the processor 120 strengthens the association between the subject SJ and the specific target Ts. Then, the process ends.

[0034] 3. Effects Even when the management device 100 of the target management system 1 in the present disclosure cannot determine whether the subject SJ and the target Ti are the same, by requesting a determination from the user U who can surely distinguish the subject SJ and the target Ti via the determination request REQ, a correct determination becomes possible. Further, since the determination response RES is accumulated, the association between the subject image IMG-SJ and the specific target Ts is strengthened, so that the accuracy of the determination process executed by the management device 100 itself gradually improves. Along with this, the frequency of providing the determination request REQ gradually decreases. In other words, the target management system 1 can efficiently learn the characteristics of the target Ti via the determination request REQ and the determination response RES, and can gradually improve the accuracy of the determination process.

[0035] The above-described temporary feature amounts (such as clothing and hairstyle) are particularly effective immediately after the target management system 1 starts to be used. That is, even at a stage where the feature learning for the target Ti has not progressed, the use of the temporary feature amounts can temporarily improve the determination accuracy.

[0036] 4. Modification Example 4-1. First Modification Example As another example of the specific conditions for which the determination requirement REQ is provided, in the determination process, an image similarity SML-i equal to or greater than a predetermined value is calculated for the target Ti, and then, within a predetermined time, an image similarity SML-i equal to or greater than the predetermined value is calculated again for the same target Ti. Depending on the installation location of the camera 20, it may be unlikely for the same person to appear at the location multiple times within a predetermined time. For example, assume that the camera 20 is installed at the school gate, and during the school arrival time, an image similarity SML-i (equal to or greater than the predetermined value) with the target Ti is calculated for a certain subject. Then, assume that during the same time period, an image similarity SML-i (equal to or greater than the predetermined value) with the same target Ti is calculated for a certain subject again. Since it is usually impossible for the same child (student) to arrive at school twice in a short period, in this case, it is highly likely that one of the determination processes performed by the management device 100 is incorrect. In such a case, by transmitting the determination requirement REQ and using the determination response RES of the user U, the management device 100 can obtain feedback on the determination process result it has performed.

[0037] 4-2. Second Modification Example In this section, the case where there are multiple targets Ti will be described. Here, the target Ti will be distinguished and described as the first target T1 and the second target T2. The registered images of the first target T1 and the second target T2 are referred to as the first registered image IMG-T1 and the second registered image IMG-T2, respectively. FIG. 5 is a flowchart corresponding to the second modification example.

[0038] In step S20, the processor 120 calculates a first image similarity SML-1 between the subject image IMG-SJ and the first registered image IMG-T1. Similarly, the processor 120 also calculates a second image similarity SML-2 between the subject image IMG-SJ and the second registered image IMG-T2.

[0039] In step S21, the processor 120 compares the first image similarity SML-1 with the first predetermined value TH1 and the second predetermined value TH2. Similarly, the processor 120 compares the second image similarity SML-2 with the first predetermined value TH1 and the second predetermined value TH2. When at least one of the first image similarity SML-1 or the second image similarity SML-2 is greater than or equal to the first predetermined value TH1 and less than or equal to the second predetermined value TH2 (step S21: YES), the process proceeds to step S12 in FIG. 4. Otherwise (step S21: NO), the process proceeds to step S22.

[0040] In step S22, the processor 120 compares the first image similarity SML-1 and the second image similarity SML-2 with the second predetermined value TH2. When both the first image similarity SML-1 and the second image similarity SML-2 are greater than the second predetermined value TH2 (step S22: YES), the process proceeds to step S12 in FIG. 4. Otherwise (step S22: NO), the process ends.

[0041] The situation where it is "YES" in step S21 is a situation where "either one of the image similarities SML-i of the target Ti is within a certain range, and it is not clear whether the subject SJ and the target Ti are the same". On the other hand, the situation where it is "YES" in step S22 is a situation where "both of the image similarities SML-i of the target Ti are high, and both of the targets Ti can be determined to be the same as the subject SJ". In such a situation, it is reasonable to proceed to step S12 and provide the subject image IMG-SJ and the determination request REQ to the user terminal 30.

[0042] As described above, the target management system 1 is applicable even when there are multiple targets Ti. In particular, when the appearances of the targets Ti are very similar to each other (for example, when the targets Ti are multiple fetuses such as twins or siblings with very similar appearances), it is considered that it is often impossible to correctly determine only with the management device 100. On the other hand, if the user U (typically the parent of the target Ti) can surely distinguish between the similar targets Ti. Therefore, the feature of the target management system 1 of using the assistance by the user U via the determination request REQ can be particularly utilized.

[0043] 5. Example of determination request display FIG. 6 is a schematic diagram showing an example of a display screen of the user terminal 30 when the subject image IMG-SJ is provided together with the determination request REQ.

[0044] (A) in FIG. 6 is an example of a display screen when only the first target T1 (here, "Satoshi") is registered. As the content of the determination request REQ, three items, "Satoshi", "others", and "unknown", are displayed. As the determination response RES, when "Satoshi" is selected, the feature amount extraction unit 41 strengthens the association between the subject SJ and "Satoshi" (the first target T1). That is, "Satoshi" (the first target T1) is regarded as the specific target Ts. As the determination response RES, when "others" or "unknown" is selected, the feature amount extraction unit 41 does not update the machine learning model. When "others" is selected, the feature amount extraction unit 41 may learn the subject feature amount FEA-SJ included in the subject image IMG-SJ using another machine learning model as a parameter characterizing others (a person different from the first target T1).

[0045] (B) in Fig. 6 is an example of a display screen when, in addition to the first target ("Satoshi"), the second target ("Hiroshi") is registered. As the content of the determination request REQ, four options of "Satoshi", "Hiroshi", "others", and "unknown" are displayed. As the determination response RES, when "Satoshi" or "Hiroshi" is selected, the feature extraction unit 41 strengthens the association between the subject SJ and the selected one (that is, the specific target Ts). When "others" or "unknown" is selected, the subsequent processing is the same as the example of (A) in Fig. 6.

[0046] The following are examples of aspects that can be considered other than the above-described aspects. Note that illustrations are omitted.

[0047] The management device 100 may request the user U to provide an image that satisfies a predetermined condition via the user terminal 30. For example, when the accuracy of the determination process is particularly low for an image taken from a specific angle, the management device 100 requests the user U to provide an image of the target Ti taken from that specific angle. In that case, a message prompting the user U to provide an image of the target Ti taken from a specific angle is displayed on the user terminal 30 together with the determination request REQ. The image provided through this image request improves the factors that have been causing low determination accuracy so far. Therefore, the feature extraction unit 41 can efficiently improve the determination accuracy by learning the features included in this image. Conversely, the user U may request the management device 100 to provide an image that satisfies a predetermined condition. For example, when the determination response RES cannot be obtained for an image of the angle at which the camera 20 has taken a picture, the user U may request the management device 100 to provide an image taken from another angle by a camera other than the camera 20. In this case, a field for the user U to enter a request is provided on the display screen of the determination request REQ.

[0048] Furthermore, when the user U determines that the subject SJ and the specific target Ts are the same, the user U may make a determination response RES after specifying a specific location of the subject image IMG-SJ. For example, the user U specifies the facial part that the user focuses on for determination by tapping or clicking. The feature extraction unit 41 uses the features related to the specified part for the association between the subject SJ and the specific target Ts. In this case, since the information on the features to be focused on is also accumulated together with the content of the determination response RES to the determination request REQ, the association between the subject SJ and the specific target Ts proceeds more efficiently.

Explanation of Signs

[0049] 1…Target management system, 20…Camera, 30…User terminal, 40…Determination processing unit, 41…Feature extraction unit, 42…Similarity calculation unit, 43…Threshold determination unit, 100…Management device, 110…Control device, 120…Processor, 130…Storage device, 140…Communication device, FEA-SJ…Subject feature amount, FEA-Ti…Target feature amount, IMG-SJ…Subject image, IMG-Ti…Registered image, REQ…Determination request, RES…Determination response, SJ…Subject, SML-i…Image similarity, Ti…Target

Claims

1. One or more storage devices that store registration images of each of one or more targets, One or more processors Comprising: The one or more processors Acquire a subject image that is an image of a subject captured by a camera, Based on the registration image and the subject image, execute a determination process for determining whether the subject is identical to any of the one or more targets, When the result of the determination process satisfies a specific condition, provide, to a user terminal, a determination request that requests a determination as to who the subject is, together with the subject image, When a determination response indicating that the subject is identical to a specific target among the one or more targets is received from the user terminal in response to the determination request, strengthen the association between the subject and the specific target in the determination process Configured to Target management system.

2. The target management system according to claim 1, The one or more targets include a first target, In the determination process, the one or more processors calculate a first image similarity between the registration image of the first target and the subject image, The specific condition is that the first image similarity is equal to or greater than a first predetermined value and equal to or less than a second predetermined value Target management system.

3. The target management system according to claim 1, When a determination response indicating that the subject is identical to the specific target is received from the user terminal, the one or more processors Temporarily use, in the determination process, a temporary feature amount of the subject included in the subject image as a temporary feature amount characterizing the specific target, Use, in the determination process, a unique feature amount of the subject included in the subject image as a unique feature amount characterizing the specific target for a longer time than the temporary feature amount Configured to Target management system.

4. The target management system according to claim 1, The one or more targets include a first target and a second target, In the determination process, the one or more processors calculate a first image similarity between the registration image of the first target and the subject image and a second image similarity between the registration image of the second target and the subject image, The specific condition is that "at least one of the first image similarity or the second image similarity is greater than or equal to a first predetermined value and less than or equal to a second predetermined value", or "both the first image similarity and the second image similarity are greater than the second predetermined value". Target management system.

5. The target management system according to claim 4, wherein the first target and the second target are multiple fetuses or siblings. Target management system.

Images