Judgment program, judgment method, and judgment device

Abstract

To determine whether a person on an image is the person himself / herself with high accuracy.SOLUTION: A determination device 1 calculates reference behavior data showing a reference behavior between persons for each of a plurality of kinds of behaviors on the basis of a data group 3 in which data showing behaviors of each person is registered, acquires first behavior data showing behaviors about a person A for each of a plurality of kinds from the data group 3, calculates difference between the first behavior data and the reference behavior data for each of the plurality of kinds, specifies one or more first kinds whose difference is equal to or more than a first threshold from among the plurality of kinds, registers second behavior data showing behaviors of the person A for each of the one or more first kinds in a data group 4, extracts third behavior data showing behaviors of a person from an input image 2, and determines whether the person is the same as the person A on the basis of a comparison result between the third behavior data and the second behavior data.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a determination program, a determination method, and a determination device.

Background Art

[0002] In recent years, the abuse of fake images generated using deep learning techniques, particularly fake images of people, has become a problem. Such fake images are of very high quality and it is difficult to distinguish them as fakes at first glance.

[0003] Therefore, a technique has been considered for determining the identity of these people by comparing the behavior of a person detected from an image taken in the past with the behavior of a person detected from a newly input image. For example, a remote conversation system has been proposed that compares reference behavior information based on the past characteristic behavior of a participant in a remote conversation system with current behavior information based on the characteristic behavior of the participant during the current conversation, and determines the identity of the participant based on the comparison result.

[0004] Also, regarding image recognition, there are the following proposals. For example, a person recognition system has been proposed that detects information indicating the state of each part of the face of a person to be recognized from a plurality of photographed images, arranges them in time series for each part, and determines whether the information can be recognized as the movement of each part in the face of an actual person based on the movement pattern of each part registered in advance.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] Incidentally, as mentioned above, one possible method for determining a person's identity is to compare the characteristic features of a person's past behavior with the characteristic features of their current behavior for a predetermined set of behaviors. However, this method may include not only behaviors unique to the person but also behaviors that differ little from those of others, so there is a possibility of misidentifying someone impersonating the person as the person in question.

[0007] In one aspect, the present invention aims to provide a determination program, determination method, and determination device capable of accurately determining whether a person in an image is the person in question. [Means for solving the problem]

[0008] One proposal provides a determination program that causes a computer to perform the following steps: calculate reference behavior data showing the reference behavior among multiple people for each of multiple types of behavior based on a first data set in which data showing the behavior of each of multiple people is registered; obtain first behavior data showing the behavior of a first person from the first data set for each of multiple types; calculate the difference between the first behavior data and the reference behavior data for each of multiple types; identify one or more first types from among the multiple types whose difference is greater than or equal to a first threshold; register second behavior data showing the behavior of the first person for each of the one or more first types in a second data set; extract third behavior data showing the behavior of the second person from the input image; and determine whether the second person is the same as the first person based on the comparison result between the third behavior data and the second behavior data.

[0009] In another proposal, a determination program is provided that causes a computer to perform a process in which it extracts third behavior data showing the behavior of a second person from an input image, refers to a second data set in which second behavior data showing the behavior of the first person for each of one or more first types of behavior classifications is registered, compares the third behavior data with the second behavior data, and determines whether the second person is the same as the first person based on the result of the comparison. Here, the second data set is generated by calculating reference behavior data showing the reference behavior among multiple people for each of the multiple types of behavior based on the first data set in which data showing the behavior of each of the multiple people including the first person is registered, obtaining first behavior data showing the behavior of the first person from the first data set for each of the multiple types, calculating the difference between the first behavior data and the reference behavior data for each of the multiple types, identifying one or more first types from the multiple types in which the difference is greater than or equal to a predetermined threshold, and registering the data showing the behavior of the first person for each of the one or more first types as second behavior data in the second data set.

[0010] Furthermore, one proposal provides a determination method in which a first computer calculates reference behavior data showing the reference behavior among multiple people for each of multiple types of behavior based on a first data set in which data showing the behavior of each of multiple people is registered, obtains first behavior data showing the behavior of a first person among multiple people from the first data set for each of multiple types, calculates the difference between the first behavior data and the reference behavior data for each of multiple types, identifies one or more first types from among the multiple types whose difference is greater than or equal to a predetermined threshold, registers second behavior data showing the behavior of the first person for each of the one or more first types in a second data set, and either the first or second computer extracts third behavior data showing the behavior of a second person from the input image, and determines whether the second person is the same as the first person based on the comparison result between the third behavior data and the second behavior data registered in the second data set.

[0011] Furthermore, one proposal provides a determination device having a processing unit that, based on a first data set in which data showing the behavior of each of several people is registered, calculates reference behavior data showing the reference behavior among several people for each of several types of behavior, obtains first behavior data showing the behavior of a first person from the first data set for each of several types, calculates the difference between the first behavior data and the reference behavior data for each of several types, identifies one or more first types from among the several types whose difference is greater than or equal to a first threshold, registers second behavior data showing the behavior of the first person for each of the one or more first types in a second data set, extracts third behavior data showing the behavior of the second person from the input image, and determines whether the second person is the same as the first person based on the comparison result between the third behavior data and the second behavior data. [Effects of the Invention]

[0012] In one respect, it can accurately determine whether the person in the image is the actual person. [Brief explanation of the drawing]

[0013] [Figure 1] This figure shows an example of processing by the determination device according to the first embodiment. [Figure 2] This figure shows an example configuration of a video call system according to the second embodiment. [Figure 3] This figure shows an example of the hardware configuration of a control server. [Figure 4] This figure shows an example configuration of the basic processing functions provided by a control server. [Figure 5] This diagram shows a comparative example of methods for detecting impersonation. [Figure 6] This is a diagram intended to explain behaviors that are characteristic of the person in question. [Figure 7] This figure shows an example of the configuration of the processing functions provided by the control server according to Embodiment 2-1. [Figure 8] This is an example flowchart showing the procedure for behavior extraction processing by the behavior extraction unit. [Figure 9] It is a diagram showing an example of the data configuration of time-series feature amounts. [Figure 10] It is an example of a flowchart showing the procedure of behavior determination processing by the behavior determination unit. [Figure 11] It is a diagram showing an example of the data configuration of the behavior DB. [Figure 12] It is an example of a flowchart showing the procedure of reference behavior definition processing by the reference behavior definition unit. [Figure 13] It is a diagram showing an example of the data configuration of the reference behavior DB. [Figure 14] It is an example of a flowchart showing the procedure of behavior difference calculation processing by the behavior difference calculation unit. [Figure 15] It is a diagram showing an example of the data configuration of the behavior difference DB. [Figure 16] It is an example of a flowchart showing the procedure of determination feature amount calculation processing by the determination feature amount calculation unit. [Figure 17] It is a diagram showing an example of the data configuration of the determination feature amount DB. [Figure 18] It is an example of a flowchart (part 1) showing the procedure of spoofing determination processing by the spoofing determination unit. [Figure 19] It is an example of a flowchart (part 2) showing the procedure of spoofing determination processing by the spoofing determination unit. [Figure 20] It is a diagram showing an example of the display screen of the determination result. [Figure 21] It is a diagram showing an example of the configuration of the processing functions provided by the control server according to Embodiment 2-2. [Figure 22] It is a diagram for explaining the spoofing determination method in Embodiment 2-2. <00­00105>It is a diagram for conceptually explaining the determination processing of the behavior that a person always takes. [Figure 24] It is a diagram showing an example of the calculation of the variation range of behavior. [Figure 25] It is a diagram for conceptually explaining the selection process of the determination feature amount. [Figure 26] It is a diagram showing an example of the calculation of the feature amount difference value. [Figure 27] This is an example flowchart showing the procedure for determining individual behavior by the individual behavior determination unit. [Figure 28] This figure shows an example of the data structure in the individual behavior database. [Figure 29] This is an example flowchart showing the procedure for defining the reference behavior in Embodiment 2-2. [Figure 30] This is an example flowchart showing the procedure for calculating behavioral differences in Embodiment 2-2. [Figure 31] This figure shows an example of the configuration of processing functions provided by the control server according to Embodiment 2-3. [Figure 32] This is an example flowchart showing the procedure for the impersonation detection process in Embodiment 2-3. [Modes for carrying out the invention]

[0014] Hereinafter, embodiments of the present invention will be described with reference to the drawings. [First Embodiment] Figure 1 shows an example of processing by a determination device according to the first embodiment. The determination device 1 shown in Figure 1 is a device that determines whether the person in the input image 2 is the same person as a predetermined person (i.e., whether it is the person in the image). The determination device 1 is, for example, a computer equipped with a processor and memory. In this case, at least a part of the processing of the determination device 1 is realized by the processor executing a predetermined program.

[0015] The following explanation illustrates the process for determining whether the person in input image 2 is person A. First, the determination device 1 generates the data to be used for determination using the following procedure. The judgment device 1 calculates reference behavior data (step S1) for each of the multiple types of behavior, based on data set 3 which contains data indicating the behavior of multiple individuals, including person A. In the example in Figure 1, the behaviors are classified into eight types TP1 to TP8, and data set 3 contains behavior data for each of the eight types of behaviors for each individual. The behavior data includes, for example, one or more image features determined for each type of behavior. Note that in Figure 1, the behavior data is shown as integers between 0 and 100 for clarity. The reference behavior data represents the average feature of the same type of behavior among multiple individuals, and is calculated, for example, as the median or average value of the behavior data of multiple individuals.

[0016] Next, the determination device 1 obtains behavioral data for person A from data group 3 for each of types TP1 to TP8 (step S2). Then, the determination device 1 calculates the difference between person A's behavioral data and the reference behavioral data for each of types TP1 to TP8 (step S3). For example, a difference of "10" is calculated between the behavioral data for person A corresponding to type TP1, "80", and the reference behavioral data for type TP1, "70". Also, a difference of "5" is calculated between the behavioral data for person A corresponding to type TP2, "55", and the reference behavioral data for type TP2, "60".

[0017] Next, the determination device 1 identifies the type from types TP1 to TP8 whose difference is greater than or equal to a predetermined threshold, and registers the behavior data of person A corresponding to the identified type as determination behavior data in data group 4 (step S4). In Figure 1, as an example, suppose the threshold is "15", and types TP4, TP7, and TP8 are identified as the corresponding types. In this case, the determination device 1 registers the behavior data of person A corresponding to type TP4 "34", the behavior data of person A corresponding to type TP7 "30", and the behavior data of person A corresponding to type TP8 "50" as determination behavior data in data group 4. These behavior data are registered with the identification number of the type associated with them.

[0018] In this way, the behavioral data for determination corresponding to person A is registered in data group 4, and the determination process is executed by referring to this data group 4. The determination device 1 extracts behavioral data indicating the behavior of a person from the input image 2 (step S5). In this process, behavioral data indicating at least one type of behavior from types TP1 to TP8 is extracted. Next, the determination device 1 compares the extracted behavioral data with the type-specific behavioral data for determination registered in data group 4, and determines whether the person in the input image 2 is the same as person A based on the comparison results (step S6).

[0019] For example, suppose that in step S5, behavioral data representing the behavior of types TP2, TP3, TP4, and TP7 is extracted. However, the judgment behavioral data corresponding to types TP2 and TP3 is not registered in data group 4. Therefore, in step S6, only the comparison for types TP4 and TP7 is performed.

[0020] Let's assume that the behavior data corresponding to types TP4 and TP7 are "40" and "35," respectively. In this case, for example, for type TP4, a difference of "6" is calculated between the behavior data "40" and the judgment behavior data "34," and for type TP7, a difference of "5" is calculated between the behavior data "35" and the judgment behavior data "30." If the threshold is "10," then all of these differences are below the threshold, so for types TP4 and TP7, it is determined that they match the behavior indicated by the judgment behavior data. For example, if two or more types of behavior match to determine that they are the same person, then in the above example, the person in input image 2 is determined to be the same person as person A.

[0021] Here, among the types of behavior, there are some that do not show much difference among multiple people. For example, behaviors such as raising a hand or nodding are similar for any person. Therefore, there is little difference between each person's behavior data and the reference behavior data. Consequently, if behavior data of a type that does not show much difference between people is used in the comparison in step S6, the possibility of misidentifying someone impersonating the person increases. On the other hand, behaviors of a type that show a large difference between a person's past behavior data and the reference behavior data can be said to be behaviors unique to that person (behaviors that are characteristic of the person). Therefore, by using only this type of behavior data in the comparison in step S6, the possibility of misidentification can be suppressed.

[0022] The behavioral data for determination registered in data group 4 represents types of behavior where there is a large difference between person A's past behavioral data and the reference behavioral data (i.e., types of behavior where there is a large difference between person A and other people). Therefore, as described above, by performing the comparison in step S6 using only the types of behavioral data registered in data group 4 from the behavioral data extracted from input image 2, it is possible to determine with high accuracy whether the person in input image 2 is person A or not.

[0023] [Second Embodiment] Next, we will describe a system in which the processing functions of the detection device 1 shown in Figure 1 are used to detect impersonation in a video call system.

[0024] Figure 2 shows an example configuration of a video call system according to the second embodiment. The video call system shown in Figure 2 includes a control server 100 and call terminals 200, 200a, 200b, ...

[0025] The control server 100 is an example of the determination device 1 shown in Figure 1. This control server 100 controls video calls between call terminals. For example, when a video call is made between call terminals 200a and 200b, the control server 100 receives the audio and images captured by call terminal 200a and transmits them to call terminal 200b. At the same time, the control server 100 receives the audio and images captured by call terminal 200b and transmits them to call terminal 200a.

[0026] Furthermore, the control server 100 performs a spoofing detection process to determine whether the person pictured in the image transmitted from the call terminal is the actual person. To perform this spoofing detection process, the control server 100 stores the call images received from the call terminals 200, 200a, 200b, etc., and creates data to be referenced during the spoofing detection process based on the stored images.

[0027] The call terminals 200, 200a, 200b, etc. are terminal devices used by the person making the video call, such as a notebook or desktop personal computer or a smartphone. Each of the call terminals 200, 200a, 200b, etc. is equipped with or connected to a microphone, camera, speaker, display, etc. One call terminal making the video call transmits the audio picked up by the microphone and the image captured by the camera to the control server 100. At the same time, the other call terminal receives the audio picked up by the other call terminal and the image captured by this call terminal from the control server 100, outputs the received audio through the speaker, and displays the received image on the display.

[0028] The video call system may also be a system that allows video calls between three or more call terminals. Figure 3 shows an example of the hardware configuration of a control server. The control server 100 is implemented as a computer, for example, as shown in Figure 3. The control server 100 shown in Figure 3 includes a processor 101, RAM (Random Access Memory) 102, HDD (Hard Disk Drive) 103, GPU (Graphics Processing Unit) 104, input interface (I / F) 105, read device 106, and communication interface (I / F) 107.

[0029] The processor 101 provides comprehensive control over the entire control server 100. The processor 101 may be, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), or a PLD (Programmable Logic Device). Alternatively, the processor 101 may be a combination of two or more elements from among the CPU, MPU, DSP, ASIC, and PLD.

[0030] RAM 102 is used as the main memory of the control server 100. At least a portion of the OS (Operating System) program and application programs to be executed by the processor 101 are temporarily stored in RAM 102. Additionally, various data necessary for processing by the processor 101 are stored in RAM 102.

[0031] HDD103 is used as auxiliary storage for the control server 100. The OS program, application programs, and various data are stored on HDD103. Other types of non-volatile storage devices, such as SSDs (Solid State Drives), can also be used as auxiliary storage.

[0032] A display device 104a is connected to the GPU 104. The GPU 104 displays an image on the display device 104a according to instructions from the processor 101. The display device 104a can be, for example, a liquid crystal display or an organic EL (Electroluminescence) display.

[0033] An input device 105a is connected to the input interface 105. The input interface 105 transmits signals output from the input device 105a to the processor 101. Examples of input devices 105a include keyboards and pointing devices. Examples of pointing devices include mice, touch panels, tablets, touchpads, and trackballs.

[0034] A portable recording medium 106a is attached to and detached from the reading device 106. The reading device 106 reads the data recorded on the portable recording medium 106a and transmits it to the processor 101. The portable recording medium 106a can be an optical disc, a semiconductor memory, or the like.

[0035] The communication interface 107 transmits and receives data with other devices such as call terminals 200, 200a, 200b, etc., via the network 107a. The processing functions of the control server 100 can be realized with the hardware configuration described above. Furthermore, the call terminals 200, 200a, 200b, etc., can also be implemented as computers including processors, main memory, and auxiliary storage devices.

[0036] Figure 4 shows an example of the configuration of the basic processing functions provided by the control server. As shown in Figure 4, the control server 100 includes a storage unit 110, a video call control unit 120, a DB (database) creation unit 130, and a spoofing detection unit 140.

[0037] The memory unit 110 is a memory area reserved in the memory device of the control server 100, such as RAM 102 or HDD 103. The memory unit 110 stores an image DB (database) 111 in which images taken during video calls are stored for each person, and a judgment feature DB (database) 112 in which data referenced when detecting impersonation is registered.

[0038] The processing of the video call control unit 120, the database creation unit 130, and the impersonation detection unit 140 is realized, for example, by the processor 101 executing a predetermined application program.

[0039] The video call control unit 120 controls video calls between call terminals. During a video call, the video call control unit 120 stores the video data transmitted from the call terminals in the image database 111, associating it with a person ID that identifies the person making the call. As a result, the image database 111 accumulates video data that shows the past behavior of each person. Furthermore, during the spoofing detection process by the spoofing detection unit 140, the video call control unit 120 inputs the video data transmitted from the call terminals during the video call, along with the person ID that identifies the person making the call, to the spoofing detection unit 140.

[0040] The DB creation unit 130 creates a feature vector DB 112 for determination by analyzing the past behavior of each person based on the video data stored in the image DB 111. In the feature vector DB 112 for determination, features (determination features) that indicate behavior characteristic of that person (behavior unique to that person) are registered for each person ID. These features include, for example, data indicating the position and movement of hands, face, and head.

[0041] The impersonation detection unit 140 acquires video data transmitted from the call terminal during a video call from the video call control unit 120. The impersonation detection unit 140 compares the characteristic features of the behavior of the person shown in the acquired video with the corresponding characteristic features registered in the characteristic feature database 112 to determine whether the person in the call is the person corresponding to the person ID. If it is determined that the person is not the real person, for example, it will be determined that the person in the image is someone impersonating the real person, or that it is a composite image (fake image) created to resemble the real person.

[0042] The impersonation detection unit 140 may also be provided in the call terminals 200, 200a, 200b, ... In this case, the detection feature quantity DB112 is stored in the storage device of the call terminals 200, 200a, 200b, ... The call terminals 200, 200a, 200b, ... receive video footage captured by the other call terminal, which is the recipient of the video call, via the control server 100 and input it to the impersonation detection unit 140 to determine whether the person on the other end of the call is the real person.

[0043] By the way, as mentioned above, one possible method for determining impersonation by comparing the characteristics of a person's past behavior with the characteristics of their current behavior is the method shown in Figure 5. Figure 5 shows a comparative example of a method for detecting impersonation. In this comparative example, the behavior is pre-classified into multiple behavior patterns. The control server 100 can then determine which behavior pattern the behavior of the person in the input video belongs to. In Figure 5, as an example, the behavior is classified into 20 patterns, and each behavior pattern is assigned a behavior pattern ID for identification.

[0044] The feature vector database 112 for classification registers feature vectors of a person's behavior for each behavioral pattern, taken from video footage of a person stored in the image database 111. In other words, the feature vector database 112 registers feature vectors corresponding to all behavioral patterns for a given person.

[0045] When detecting impersonation, the control server 100 acquires video footage of the person during a video call, detects behaviors corresponding to each of the above behavior patterns from the video footage, and calculates feature quantities for each detected behavior. Then, for each behavior pattern, the control server 100 compares the feature quantities registered in the feature quantity DB 112 for determination with the feature quantities based on the acquired video footage and calculates the difference in feature quantities. The "Difference between Past and Present" shown in Figure 5 represents the difference in feature quantities calculated in this way for each behavior pattern. The control server 100 determines that it is impersonation if the calculated difference is greater than or equal to a predetermined threshold.

[0046] In such comparative examples, the features for each behavior pattern registered in the judgment feature database 112 each represent the characteristics of a person's behavior, but some of these behavior patterns are similar to the movements of other people. Therefore, there is a possibility of failing to detect impersonation or misidentifying the person as an impersonator.

[0047] In the example in Figure 5, when the person in the video call is someone impersonating the person in question, a feature difference of "0.2" is calculated for the behaviors of behavior patterns ID "03" and "04". If the threshold is set to "0.2", the sum of these differences, "0.4", is greater than or equal to the threshold, so it is correctly determined to be an impersonator. On the other hand, when the person in the video call is the person in question, a feature difference of "0.2" is calculated for the behaviors of behavior patterns "02" and "05". The sum of these differences, "0.4", is greater than or equal to the threshold, so it is incorrectly determined to be an impersonator despite being the person in question.

[0048] To suppress the occurrence of such misclassifications, it is considered better to use features that represent "behavior characteristic of the person," which are behaviors unique to the person themselves, rather than features for a predetermined set of behavioral patterns, when registering features in the judgment feature database 112.

[0049] Here, two types of behavioral comparisons can be considered: "comparing the individual's past and present behavior" and "comparing the individual's behavior with that of others." In the former, the behaviors being compared can be classified into those that are likely to coincide with the individual's past and present behavior and those that are unlikely to coincide. In the latter, the behaviors being compared can be classified into those that are likely to coincide with the individual's behavior with that of others and those that are unlikely to coincide.

[0050] In the above comparison example, the fact that the comparison included not only behaviors that were unlikely to match between the individual's past and present, but also behaviors that were likely to match, is considered a factor that makes misjudgment likely. To improve the accuracy of the judgment, it is necessary to compare only behaviors that are unlikely to match between the individual's past and present. Similarly, when comparing the behavior of an individual with that of others, it is necessary to compare only behaviors that are unlikely to match between the individual and that of others.

[0051] Figure 6 is a diagram illustrating behavior that is characteristic of the individual. Figure 6(A) shows a comparison of the individual's past and present behavior, and Figure 6(B) shows a comparison of the individual's behavior with that of others.

[0052] Some behaviors are responses from the limbic system and are unique to the individual, generated by their upbringing and environment. These behaviors are consistent and performed by the individual. Therefore, as shown in Figure 6(A), when analyzing an individual's past behaviors, they can be classified into behaviors that the individual consistently exhibits and behaviors that are temporary. The former behaviors tend to match the individual's behaviors obtained during impersonation detection and can be considered "behaviors characteristic of the individual," while the latter behaviors are less likely to match. Therefore, it is desirable to compare only the former behaviors when detecting impersonation.

[0053] On the other hand, regarding the comparison of the behavior of the person in question with that of others, as shown in Figure 6(B), behaviors can be classified into those with small differences from those of others and those with large differences. The former behaviors are likely to match the behavior of others obtained during impersonation detection, while the latter behaviors are less likely to match the behavior of others and can therefore be called "behaviors that are characteristic of the person in question." For example, the behavior of raising one's hand does not vary much from person to person, so it is likely to match the behavior of others during impersonation detection. On the other hand, the behavior of scratching one's head varies from person to person, with some scratching with their palms and others with their fingers. Therefore, the behavior of scratching one's head is likely to vary greatly from person to person and is less likely to match the behavior of others during impersonation detection. Consequently, it is desirable to compare only the latter behaviors when detecting impersonation.

[0054] Therefore, in Embodiment 2-1 shown below, feature quantities of behaviors that differ significantly from those of others are registered in the Judgment Feature Database 112, and only these feature quantities of behaviors are compared when detecting impersonation. In Embodiment 2-2, feature quantities of behaviors that the person consistently takes that differ significantly from those of others are registered in the Judgment Feature Database 112, and only these feature quantities of behaviors are compared when detecting impersonation.

[0055] <Embodiment 2-1> Figure 7 shows an example of the configuration of processing functions provided by the control server according to Embodiment 2-1. As shown in Figure 7, in the control server 100 according to Embodiment 2-1, the storage unit 110 stores, in addition to the aforementioned image DB 111 and determination feature quantity DB 112, a definition behavior DB (database) 113, a behavior DB (database) 114, a reference behavior DB (database) 115, and a behavior difference DB (database) 116.

[0056] The Defined Behavior DB113 stores a defined feature (defined behavior feature) for each behavior pattern, defining the corresponding behavior. The Defined Behavior DB113 is referenced to determine which behavior pattern corresponds to the behavior of a person in an image.

[0057] The behavior DB114 stores feature quantities for each person, indicating their past behaviors, categorized by behavior pattern. These feature quantities are calculated based on video footage stored in the image DB111.

[0058] The standard behavior DB115 registers standard behavior features for each behavior pattern, representing the average behavior among multiple individuals. These standard behavior features represent the average behavior among the behaviors that multiple individuals have exhibited in the past for the same behavior pattern, and serve as the basis for calculating the judgment features (features that indicate behavior typical of that individual) for each person.

[0059] The Behavior Difference DB116 stores the difference between each person's behavioral feature vector and the baseline behavioral feature vector. The Behavior Difference DB116 is temporarily created when calculating the features for judgment. Furthermore, as shown in Figure 7, the DB creation unit 130 includes a behavior extraction unit 131, a behavior determination unit 132, a reference behavior definition unit 133, a behavior difference calculation unit 134, and a determination feature calculation unit 135.

[0060] The behavior extraction unit 131 extracts features from the video data acquired from the image database 111 using image recognition, and calculates features that indicate the movement of a predetermined body part from the extracted features. The calculated features are calculated continuously over time on the video and are recorded as time-series features in a storage device (such as RAM 102) in association with the person's ID.

[0061] The behavior determination unit 132 compares the recorded time-series features with the defined behavior features defined for each behavior pattern in the defined behavior DB 113 to determine which behavior pattern the behavior indicated by the time-series features corresponds to. If the behavior determination unit 132 determines that the behavior corresponds to one of the behavior patterns, it generates behavior features that indicate the behavior of the corresponding behavior pattern based on the time-series features, and registers the generated behavior features in the behavior DB 114 in association with the person ID and behavior pattern ID. As a result, the behavior DB 114 stores features that indicate the behavior each person has performed in the past, classified and registered according to the behavior pattern.

[0062] The reference behavior definition unit 133 obtains behavioral features for each behavioral pattern from the behavior DB 114, calculates a reference feature (reference behavioral feature) for the corresponding behavioral pattern based on the obtained behavioral features, and registers it in the reference behavior DB 115.

[0063] The behavior difference calculation unit 134 calculates the difference between the past behavior features and the reference behavior features for each person, based on the features registered in the behavior DB 114. Specifically, the following processing is performed for each person. The behavior difference calculation unit 134 obtains features from the behavior DB 114 for each behavior pattern, and obtains the reference behavior features for the corresponding behavior pattern from the reference behavior DB 115, and calculates a feature difference value that shows the difference between these features. The behavior difference calculation unit 134 registers the feature difference value calculated for each behavior pattern in the behavior difference DB 116.

[0064] The feature calculation unit for determination 135 compares the feature difference values ​​registered in the behavior difference DB 116 with a predetermined threshold. If the behavior difference value of a certain behavior pattern for a person is greater than or equal to the threshold, that behavior pattern is considered to differ significantly from that of others and to represent behavior typical of that person. In this case, the feature of this behavior pattern is registered as a feature for determination in the feature determination DB 112. In this way, the feature determination DB 112 registers the behavior features of behavior patterns that represent behavior typical of that person for each individual.

[0065] Furthermore, as shown in Figure 7, the impersonation detection unit 140 includes a behavior extraction unit 141, a behavior determination unit 142, a behavior comparison unit 143, and a determination result output unit 144. The behavior extraction unit 141 acquires video data transmitted from the call terminal during a video call from the video call control unit 120, and extracts time-series features from the acquired video data by image recognition.

[0066] The behavior determination unit 142 compares the extracted time-series features with the defined behavior features defined for each behavior pattern in the defined behavior DB 113 to determine which behavior pattern the behavior shown by the time-series features corresponds to. If the behavior determination unit 142 determines that the behavior corresponds to one of the behavior patterns, it generates behavior features that indicate the behavior of the corresponding behavior pattern based on the time-series features.

[0067] The processing of the behavior extraction unit 141 and the behavior determination unit 142 continues, for example, until a certain period of time has elapsed, and the generated behavioral feature quantities are associated with the behavioral pattern ID and recorded in a storage device (such as RAM 102).

[0068] The behavior comparison unit 143 acquires recorded behavioral features and obtains determination features for behavioral patterns corresponding to those behavioral features from the determination feature DB 112, and compares the difference between these features with a predetermined threshold. If the difference is within the threshold, the behavior comparison unit 143 determines that the behavior indicated by the behavioral features (the behavior of the person on the call) matches the behavior indicated by the determination features (behavior that is characteristic of the person). If the number of behavioral patterns in which the behavior of the person on the call is determined to match the behavior that is characteristic of the person is greater than or equal to a predetermined threshold, the behavior comparison unit 143 determines that the person on the call is the real person. On the other hand, if the number of determined patterns is less than the threshold, the behavior comparison unit 143 determines that it is an impersonation.

[0069] The judgment result output unit 144 outputs the result of the impersonation judgment. For example, the judgment result output unit 144 displays the judgment result on the display device of the call terminal to which the person being judged is on the call.

[0070] The processing of the control server 100 according to Embodiment 2-1 will be described below using a flowchart. Figure 8 is an example flowchart showing the procedure for behavior extraction processing by the behavior extraction unit.

[0071] [Step S11] The behavior extraction unit 131 acquires video data from the image DB 111. The acquired video data has the person ID of the person who was on the phone at the time of recording added to it. [Step S12] The behavior extraction unit 131 extracts features from each frame of the acquired video data by image recognition. For example, the coordinates of a predetermined body part are extracted as features.

[0072] [Step S13] The behavior extraction unit 131 detects head movements based on the extracted feature quantities. [Step S14] The behavior extraction unit 131 detects hand movements based on the extracted feature quantities.

[0073] [Step S15] The behavior extraction unit 131 detects blinking based on the extracted feature quantities. [Step S16] The behavior extraction unit 131 detects eye movement based on the extracted feature quantities.

[0074] The processes in steps S13 to S16 described above may be executed in parallel or sequentially. In the latter case, the order of execution is not particularly limited. [Step S17] The behavior extraction unit 131 records time-series features in a storage device based on the detection results from steps S13 to S16.

[0075] [Step S18] The behavior extraction unit 131 determines whether all video data stored in the image DB 111 has been processed. If there is any unprocessed video data, the process proceeds to step S11, and one of the unprocessed video data is acquired. On the other hand, if all video data has been processed, the behavior extraction process ends.

[0076] Figure 9 shows an example of the data structure for time series features. In step S17 of Figure 8, for example, time series feature 151 as shown in Figure 9 is recorded. The time-series feature vector 151 registers multiple pairs of date and time and feature vectors, associated with the person ID. The date and time indicate the date and time the frame was taken. The feature vectors indicate the features extracted from the frame. For each body part extracted from the frame, the feature vector is registered, consisting of an ID that identifies the body part and the coordinates of that body part on the frame.

[0077] Figure 10 is an example flowchart showing the procedure for behavior determination processing by the behavior determination unit. [Step S21] The behavior determination unit 132 acquires one of the time-series features recorded by the process in Figure 8.

[0078] [Step S22] The behavior determination unit 132 compares the acquired time-series features with the defined behavior features for each behavior pattern registered in the defined behavior DB 113. [Step S23] The behavior determination unit 132 determines whether the time series feature matches the defined behavior feature of any behavior pattern. If it matches, the process proceeds to step S24. If it does not match the defined behavior feature of any behavior pattern, the process proceeds to step S25.

[0079] [Step S24] The behavior determination unit 132 calculates behavioral features corresponding to the matching behavioral patterns based on the time-series features obtained in step S21. The behavior determination unit 132 registers the calculated behavioral features in the behavior DB 114, associating them with at least the person ID and the behavioral pattern ID.

[0080] [Step S25] The behavior determination unit 132 determines whether all time series features recorded by the process in Figure 8 have been processed. If there are any unprocessed time series features, the process proceeds to step S21, and one of the unprocessed time series features is acquired. On the other hand, if all time series features have been processed, the behavior determination process ends.

[0081] Through the above process, feature quantities representing the past behaviors of each individual are registered in the behavior DB114 for each behavior pattern. Figure 11 shows an example of the data structure of the behavior database. As shown in Figure 11, the behavior database 114 contains a table 114a for each person.

[0082] Table 114a associates the person ID with the number of detected behavior patterns. Table 114a also registers records containing the date and time, behavior pattern ID, and behavior feature. The date and time indicates the first date and time among those attached to the time-series feature in which the behavior pattern was detected (i.e., the date and time when the detection of the corresponding behavior began). The behavior pattern ID indicates the behavior pattern of the detected behavior. The behavior feature is the feature calculated in step S24 of Figure 10.

[0083] The data registered as behavioral features is predetermined for each behavioral pattern ID. For example, if behavioral pattern ID "4" represents the behavior of "scratching one's head with one's hand," the behavioral features registered will be the orientation, position, and coordinates of the head and hand, respectively. Similarly, if behavioral pattern ID "8" represents the behavior of "clapping both hands behind the head," the behavioral features registered will be the orientation, position, and coordinates of the head, right hand, and left hand, respectively. Here, for example, the "coordinates" included in the behavioral features represent the coordinates of one or more feature points on the corresponding body part, and the "position" represents the median of the coordinates of these feature points.

[0084] Furthermore, it is desirable that multiple behavioral features be registered in the behavior DB114 for the same behavioral pattern. In other words, it is desirable that the image DB111 accumulates video data such that the same behavioral pattern is captured multiple times for each person. Also, the behavioral features may be time-series features spanning multiple frames.

[0085] Figure 12 is an example flowchart showing the procedure for the reference behavior definition process performed by the reference behavior definition unit. [Step S31] The reference behavior definition unit 133 selects one from all behavior patterns.

[0086] [Step S32] The reference behavior definition unit 133 obtains behavioral features for the selected behavioral pattern from the behavior DB 114. In this process, behavioral features for the corresponding behavioral pattern are obtained regardless of the person ID.

[0087] [Step S33] The reference behavior definition unit 133 determines whether all behavioral features for the corresponding behavior pattern have been obtained from the behavior DB 114. If there are behavioral features that have not been obtained, the process proceeds to step S32, and one of the unobtained behavioral features for the corresponding behavior pattern is obtained. On the other hand, if all behavioral features have been obtained, the process proceeds to step S34.

[0088] [Step S34] The reference behavior definition unit 133 calculates reference behavior features corresponding to the relevant behavior pattern based on the behavior features obtained in step S32. The reference behavior features are calculated, for example, as the median or mean value for each parameter of the behavior features obtained in step S32. If the behavior features are time-series features, the time-series reference behavior features can be calculated, for example, by representing each time-series feature as a vector and calculating the average of those vectors. The reference behavior definition unit 133 registers the calculated reference behavior features in the reference behavior DB 115, associating them with the behavior pattern ID.

[0089] [Step S35] The reference behavior definition unit 133 determines whether all behavior patterns have been processed. If there are any unprocessed behavior patterns, the process proceeds to step S31, and one of the unprocessed behavior patterns is selected. On the other hand, if all behavior patterns have been processed, the reference behavior definition process ends.

[0090] Through the above process, a baseline behavioral feature is calculated for each behavioral pattern, which serves as a reference for calculating characteristic behavioral features of the individual by comparing them with others. Figure 13 shows an example of the data structure of the reference behavior database. As shown in Figure 13, the reference behavior database 115 registers reference behavior features for each behavior pattern ID. The data format of the reference features is the same as that of the behavior features corresponding to each behavior pattern ID. Also, if the behavior features are time-series features, the reference behavior features are also time-series features.

[0091] Figure 14 is an example flowchart showing the procedure for calculating behavioral differences by the behavioral difference calculation unit. [Step S41] The behavior difference calculation unit 134 selects the person to be processed.

[0092] [Step S42] The behavior difference calculation unit 134 refers to the behavior DB 114 and selects one behavior pattern from those associated with the person ID of the selected person. [Step S43] The behavior difference calculation unit 134 obtains behavioral features corresponding to the selected behavioral pattern from the behavior DB 114.

[0093] [Step S44] The behavior difference calculation unit 134 obtains a reference behavior feature corresponding to the behavior pattern selected in step S42 from the reference behavior DB 115, and calculates the difference between this feature and the behavior feature obtained in step S43 as the feature difference value. If multiple behavior features are obtained in step S43, for example, the median or mean of the difference between each behavior feature and the reference behavior feature is calculated as the feature difference value.

[0094] Furthermore, if the behavioral features are time-series features, the feature difference values ​​are calculated, for example, as the difference in vectors (e.g., angle difference) or Euclidean distance for each parameter within the features.

[0095] [Step S45] The behavior difference calculation unit 134 registers the calculated feature difference values ​​in the behavior difference DB 116, associating them with the person ID that indicates the person selected in step S41 and the behavior pattern ID that indicates the behavior pattern selected in step S42.

[0096] [Step S46] The behavior difference calculation unit 134 determines whether all behavior patterns have been processed. If there are any unprocessed behavior patterns, the process proceeds to step S42, and one of the unprocessed behavior patterns is selected. On the other hand, if all behavior patterns have been processed, the process proceeds to step S47.

[0097] [Step S47] The behavior difference calculation unit 134 determines whether all individuals have been processed. If there are any individuals who have not been processed, the process proceeds to step S41, and one of the individuals who has not been processed is selected. On the other hand, if all individuals have been processed, the behavior difference calculation process ends.

[0098] Through the above process, for each individual, a feature difference value is calculated that shows the difference between their past behavior and the behavior of others for each standard pattern. Figure 15 shows an example of the data structure of the behavior difference database. As shown in Figure 15, the behavior difference database 116 has a table 116a for each person. Table 116a associates the person ID with the number of detected behavior patterns. In addition, feature difference values ​​are registered in table 116a for each behavior pattern ID. The feature difference values ​​are the difference values ​​for each parameter included in the corresponding behavior feature.

[0099] For example, suppose behavior pattern ID "01" represents the behavior of "tilting the head to the side," and the behavior features corresponding to this behavior include the orientation, position, and coordinates of the face. In this case, the feature difference values ​​registered would be the difference value for the orientation of the face (orientation difference value) and the difference value for the position of the face (position difference value). Also, suppose behavior pattern ID "04" represents the behavior of "scratching the head with the hand," and the behavior features corresponding to this behavior include the orientation, position, and coordinates of the head and hand, respectively. In this case, the feature difference values ​​registered would be the orientation difference value and the position difference value for the head and hand.

[0100] Figure 16 is an example flowchart showing the procedure for calculating the features used for determination by the feature calculation unit. [Step S51] The feature calculation unit 135 for determination selects the person to be processed.

[0101] [Step S52] The feature calculation unit 135 for determination refers to the behavior difference DB 116 and selects one behavior pattern from those associated with the person ID of the selected person. [Step S53] The feature calculation unit 135 for determination obtains feature difference values ​​corresponding to the selected behavior pattern from the behavior difference DB 116.

[0102] [Step S54] The feature calculation unit 135 for determination determines whether the acquired feature difference value is greater than or equal to a predetermined threshold. If the feature difference value is greater than or equal to the threshold, the process proceeds to step S55; if the feature difference value is less than the threshold, the process proceeds to step S56.

[0103] [Step S55] The judgment feature calculation unit 135 obtains behavioral features from the behavior DB 114 that are associated with the person selected in step S51 and the behavioral pattern selected in step S52. The judgment feature calculation unit 135 associates the obtained behavioral features with the person ID of the person and the behavioral pattern ID of the behavioral pattern and registers them as judgment features in the judgment feature DB 112.

[0104] In practice, a threshold is set for each parameter within a feature, and if, for example, the absolute value of the difference for all parameters is greater than or equal to the corresponding threshold, the process proceeds to step S55. Also, if multiple behavioral features are registered in the behavior DB114, for example, the median or mean of those behavioral features is registered as a judgment feature. Furthermore, if a behavioral feature is a time-series feature, for example, each time-series feature can be represented as a vector, and the time-series judgment feature can be calculated by calculating the average of those vectors.

[0105] [Step S56] The feature calculation unit 135 for determination determines whether all behavior patterns have been processed. If there are any unprocessed behavior patterns, the process proceeds to step S52, and one of the unprocessed behavior patterns is selected. On the other hand, if all behavior patterns have been processed, the process proceeds to step S57.

[0106] [Step S57] The feature calculation unit 135 for determination determines whether all individuals have been processed. If there are individuals who have not been processed, the process proceeds to step S51, and one of the individuals who has not been processed is selected. On the other hand, if all individuals have been processed, the feature calculation process for determination ends.

[0107] Through the above process, behavioral patterns that differ significantly from those of others and are characteristic of that individual are determined, and the feature quantities used for determination of those behavioral patterns are registered in the feature quantity database 112.

[0108] In the process shown in Figure 16, the behavioral features of all behavioral patterns whose feature difference value was greater than or equal to a threshold were registered as features for judgment. However, as another example, the behavioral features of a predetermined number of behavioral patterns whose feature difference value was greater than or equal to a threshold, in descending order of feature difference value, may also be registered as features for judgment.

[0109] Figure 17 shows an example of the data structure of the feature database for determination. As shown in Figure 17, the feature database for determination 112 contains a table 112a for each person. Table 112a associates the person ID with the number of behavior patterns that indicate behavior characteristic of that person. In addition, for each behavior pattern ID of a behavior pattern that indicates behavior characteristic of that person, the feature for determination calculated in step S55 of Figure 16 is registered in table 112a. The data format of the feature for determination is the same as the behavior feature corresponding to each behavior pattern ID. Furthermore, if the behavior feature is a time-series feature, the feature for determination will also be a time-series feature.

[0110] Next, we will explain the impersonation detection process using the feature vector DB112. Figures 18 and 19 are examples of flowcharts showing the procedure for impersonation detection by the impersonation detection unit.

[0111] [Step S61] The behavior extraction unit 141 of the impersonation detection unit 140 starts acquiring video data from the video call control unit 120. This video data is captured on the call terminal during the call and transmitted to the control server 100. The video data also has a person ID attached to it that indicates the real person during the call, and this person ID becomes the number that identifies the person to be judged.

[0112] [Step S62] The behavior extraction unit 141 extracts feature quantities from each frame of the video data by image recognition in the same procedure as steps S12 to S17 in Figure 8, calculates time-series feature quantities based on the feature quantities, and records them in a storage device (such as RAM 102).

[0113] [Step S63] The behavior determination unit 142 of the impersonation detection unit 140 detects behavior that matches any of the behavior patterns by comparing the recorded time-series features with the defined behavior features in the defined behavior DB 113, using the same procedure as in Figure 10. If the time-series features match the defined behavior features of any of the behavior patterns, the behavior determination unit 142 calculates the behavior features corresponding to that behavior pattern based on the time-series features, associates them with the behavior pattern ID, and records them in the storage device.

[0114] [Step S64] The impersonation detection unit 140 determines whether the execution conditions for the behavior comparison process are met. For example, the execution conditions may include that a certain amount of time has elapsed since the start of processing in Figure 18, or that a certain number of behavioral features have been recorded by the processing in step S63. If the execution conditions are not met, the process proceeds to step S62, and the processing in steps S62 and S63 using the acquired video image data continues to be executed. On the other hand, if the execution conditions are met, the process proceeds to step S65, and the behavior comparison process begins.

[0115] [Step S65] The behavior comparison unit 143 of the impersonation detection unit 140 obtains all behavior pattern IDs associated with the person ID of the person to be judged from the detection feature database 112 (i.e., behavior pattern IDs of behavior patterns that show behavior that is characteristic of the person). The behavior comparison unit 143 compares the obtained behavior pattern IDs with the behavior pattern IDs recorded in step S63 (i.e., behavior pattern IDs of behaviors detected for the person to be judged).

[0116] [Step S66] The behavior comparison unit 143 determines whether at least one of the behavior pattern IDs obtained from the judgment feature database 112 in step S65 is included in the behavior pattern IDs recorded in step S63. If at least one of the former behavior pattern IDs is included in the latter behavior pattern, the process proceeds to step S67. On the other hand, if none of the former behavior pattern IDs are included in the latter behavior pattern IDs, the process proceeds to step S74. Possible cases in this case include when behaviors corresponding to the person to be judged are detected, but none of the behaviors that appear to be those of that person are detected, or when the person does not perform any behavior at all (for example, when they are stationary).

[0117] [Step S67] The behavior comparison unit 143 selects one behavior pattern from among the behavior pattern IDs that are included in the behavior pattern IDs recorded in step S63 and that are included in the behavior pattern IDs obtained from the judgment feature DB 112 in step S65.

[0118] [Step S68] The behavior comparison unit 143 retrieves the behavioral features corresponding to the behavioral pattern ID selected in step S67 from the behavioral features recorded in step S63. The behavior comparison unit 143 also retrieves the judgment features corresponding to the behavioral pattern ID selected in step S67 from the judgment feature DB 112. The behavior comparison unit 143 then calculates the difference between these features. If multiple behavioral features corresponding to the relevant behavioral pattern ID are recorded in step S63, for example, the median or mean of those behavioral features is calculated, and the difference between that calculation result and the judgment features is calculated.

[0119] Furthermore, if the behavioral features are time-series features, the feature difference values ​​are calculated, for example, as the difference in vectors (e.g., angle difference) or Euclidean distance for each parameter within the features.

[0120] [Step S69] The behavior comparison unit 143 determines whether the absolute value of the calculated difference is less than or equal to a predetermined threshold. If the absolute value of the difference is less than or equal to the threshold, the process proceeds to step S70; if the absolute value of the difference exceeds the threshold, the process proceeds to step S71. In practice, a threshold is set for each parameter within the feature, and for example, the process proceeds to step S70 if the absolute value of the difference for all parameters is less than or equal to the corresponding threshold.

[0121] [Step S70] The behavior comparison unit 143 records the behavior pattern ID selected in step S67 in the storage device as the behavior pattern ID of the behavior that matches the behavior of the person. [Step S71] The behavior comparison unit 143 determines whether all behavior pattern IDs that match the conditions have been selected in step S67. If there are any unselected behavior pattern IDs, the process proceeds to step S67, and one unselected behavior pattern ID is selected. On the other hand, if all of the relevant behavior patterns have been selected, the process proceeds to step S72.

[0122] [Step S72] The behavior comparison unit 143 determines whether the number of behavior pattern IDs recorded in step S70, that is, the number of behaviors that match the behavior of the person, is equal to or greater than a predetermined threshold. If the number of behaviors is equal to or greater than the threshold, the process proceeds to step S73; if the number of behaviors is less than the threshold, the process proceeds to step S74. Note that different values ​​may be used as the threshold for each behavior pattern.

[0123] [Step S73] The behavior comparison unit 143 determines that the person being judged is indeed the person in question. The judgment result output unit 144 outputs information indicating that judgment result. [Step S74] The behavior comparison unit 143 determines that the person being judged is not the person in question. The judgment result output unit 144 outputs information indicating that judgment result.

[0124] In steps S73 and S74, for example, the judgment result output unit 144 displays the judgment result on the display device of the call terminal to which the person being judged is connected. Also, in step S74, as a judgment result, information indicating that it is an impersonation is displayed.

[0125] In the above process, if multiple behavioral patterns are detected from the video footage during a call, the comparison of the feature quantities for determination is performed only for the behaviors that differ significantly from those of other people. If a predetermined number or more of the detected behaviors match those that differ significantly from those of other people (i.e., behaviors that are characteristic of the person), the person being determined is determined to be the real person. This improves the accuracy of determining whether someone is the real person (whether they are impersonating someone else) compared to the comparative example shown in Figure 5.

[0126] Figure 20 shows an example of a display screen for the judgment result. The display screen 210 shown in Figure 20 is an example of a screen that is displayed on the display device of the call terminal of the recipient in steps S73 and S74. The display screen 210 shows a judgment result display unit 211 that indicates the result of the impersonation detection. Figure 20 illustrates the case when step S74 is executed, and the judgment result display unit 211 shows that it has been determined to be an impersonation.

[0127] Furthermore, the display screen 210 also displays a behavior detection result display unit 212 that shows the results of behavior detection. The behavior detection result display unit 212 displays a record for each behavior that is characteristic of the person in question. Each record displays an ID that identifies the behavior pattern (behavior pattern ID), a description of the behavior, and the difference. The difference displayed is the absolute value of the difference between the features calculated in step S68 of Figure 19.

[0128] In addition to information indicating whether or not the person is the person in question (whether or not it is an impersonation), the information used to show the determination result may also include numerical values ​​indicating the likelihood of the person being the person in question or an impersonation, for example, based on the sum of the absolute difference values ​​calculated in step S68.

[0129] <Embodiment 2-2> Embodiment 2-2 is a modification of part of the processing of the control server 100 in Embodiment 2-1 described above. In Embodiment 2-1, behavioral features that differ significantly from those of others were registered in the judgment feature database 112. In contrast, in Embodiment 2-2, first, behaviors that a person consistently takes are identified based on their past behavior. Then, among the behaviors that a person consistently takes, features that differ significantly from those of others are registered in the judgment feature database 112.

[0130] As shown in Figure 6(A), the behaviors that the person consistently takes tend to match the behaviors of the person obtained during impersonation detection. However, this does not mean that they do not match the behaviors of others. On the other hand, as shown in Figure 6(B), behaviors that differ significantly from those of others tend to not match the behaviors of others during impersonation detection. Therefore, by selecting behaviors that differ significantly from those of others from among the behaviors that the person consistently takes and registering the feature quantities of those behaviors in the detection feature quantity DB112, it becomes possible to use feature quantities of behaviors that tend to match the behavior of the person but do not tend to match the behavior of others during impersonation detection. As a result, the accuracy of determining whether someone is the person they claim to be can be improved.

[0131] Figure 21 is a diagram showing an example of the configuration of processing functions provided by the control server according to Embodiment 2-2. As shown in Figure 21, in the control server 100 according to Embodiment 2-2, the storage unit 110 further stores a personal behavior DB 117. In addition, the DB creation unit 130 further includes a personal behavior determination unit 136.

[0132] The individual behavior determination unit 136 calculates the range of variation (difference from the maximum value to the minimum value) of the behavioral features registered in the behavior DB 114 for each individual, and determines whether the calculated range of variation is less than or equal to the tolerance value set for each behavioral pattern. If the range of variation is less than or equal to the tolerance value, the individual behavior determination unit 136 determines that the behavior corresponding to the behavioral pattern is a behavior that the individual consistently takes, and registers the behavioral features (individual behavioral features) corresponding to that behavioral pattern in the individual behavior DB 117, associating them with the behavioral pattern ID. Therefore, in the individual behavior DB 117, at least the behavioral pattern ID of the behavior that the individual consistently takes and the individual behavioral features that represent that behavior are registered in association with each individual.

[0133] The reference behavior definition unit 133 calculates reference behavior features for each behavior pattern by obtaining behavior features from the individual behavior DB 117 instead of the behavior DB 114, and registers them in the reference behavior DB 115. The behavior difference calculation unit 134 compares the behavior features obtained from the individual behavior DB 117 instead of the behavior DB 114 with the reference behavior features, and registers the feature difference values ​​for each behavior pattern in the behavior difference DB 116.

[0134] Figure 22 is a diagram illustrating the impersonation detection method in Embodiment 2-2. In Figure 22, it is assumed that the behavior is classified into 20 behavior patterns. As described above, the individual behavior determination unit 136 calculates the range of variation of behavioral features for each behavioral pattern from the behavior DB 114 for a given person, and determines whether the range of variation is below an acceptable value. In Figure 22, a range of variation of "small" is shown when the range of variation is below the acceptable value, and a range of variation of "large" is shown when it exceeds the acceptable value. In Figure 22 as an example, the range of variation for behavioral pattern IDs "03" and "04" is "small," and the behaviors corresponding to these have been determined to be behaviors that the person consistently takes.

[0135] Furthermore, for example, suppose that the behavior of behavior patterns "03," "04," and "06" differs significantly from the behavior of others. The judgment feature calculation unit 135 registers the judgment features that represent the behavior of behavior patterns "03" and "04," which are judged to have a small variation range, in the judgment feature DB 112.

[0136] In this case, the impersonation detection unit 140 compares only the feature quantities of behavior patterns "03" and "04" from the behavior detected in the video footage captured by the call terminal with the feature quantities used for detection. For example, suppose a difference of "0.0" is calculated for the feature quantities of both behavior patterns ID "03" and "04" when a video footage of the person in question is input. In this case, the sum of the differences, "0.0", is less than or equal to the threshold "0.2", so the person is correctly determined to be the person in question. On the other hand, suppose a difference of "0.2" is calculated for both behavior patterns ID "03" and "04" when a video footage of someone impersonating the person is input. In this case, the sum of the differences, "0.4", exceeds the threshold "0.2", so the person is correctly determined to be not the person in question (an impersonator).

[0137] Figure 23 is a diagram that conceptually explains the process of determining a person's consistent behavior. In Figure 23, it is assumed that there are two types of behavior feature parameters (feature parameters), with one value plotted on the x-axis and the other on the y-axis. It is also assumed that there are M behavior patterns, and that for a given person, N behaviors are detected for each behavior pattern from the video data in image DB111. Furthermore, it is assumed that tolerance values ​​W1 are set for the x-axis parameters and tolerance values ​​W2 are set for the y-axis parameters.

[0138] In the example in Figure 23, the variation range for behavior pattern 1 exceeds the acceptable limit, but the variation range for behavior pattern M remains within the acceptable limit. In this case, the behavioral features of behavior pattern 1 are not registered in the personal behavior DB117 as personal behavioral features, but the behavioral features of behavior pattern M are registered in the personal behavior DB117 as personal behavioral features. That is, the behavior of behavior pattern M is determined to be a behavior that the person in question consistently exhibits.

[0139] Figure 24 shows an example of calculating the range of behavioral variation. In Figure 24, an example is shown where the behavioral features of a person for a certain behavioral pattern are represented by vectors. The behavioral features of the behavior detected the first time are represented by vector VA1, the behavioral features of the behavior detected the second time are represented by vector VA2, and the behavioral features of the behavior detected the Nth time are represented by vector VAn.

[0140] In this case, the variation range W3 of the behavioral feature is represented, for example, by the difference between the minimum and maximum angles among the vectors VA1, VA2, ..., VAn. If the variation range W3 is less than or equal to an acceptable value, the behavior of the corresponding behavioral pattern is determined to be a behavior that the individual consistently exhibits.

[0141] Figure 25 is a diagram that conceptually explains the selection process for features used for judgment. In Figure 25, the range of variation in behavioral features for the same person (individual variation) is plotted on the x-axis, and the feature difference value for each person (difference value between the behavioral feature and the reference behavioral feature) is plotted on the y-axis. The tolerance value W4 is the tolerance value for the range of variation in behavioral features for the same person. The threshold TH1 is the judgment threshold used for comparison with the feature difference value.

[0142] In the example in Figure 25, for behavior pattern 1, the variation range of behavioral features for all individuals A to C is within the acceptable value W4, but only for individual B, the feature difference value exceeds the threshold TH1. Therefore, for individual B, the behavioral features of behavior pattern 1 are registered as a judgment feature in the judgment feature DB112. Similarly, for behavior pattern 5, the variation range of behavioral features for all individuals A to C is within the acceptable value W4, but only for individual A, the feature difference value exceeds the threshold TH1. Therefore, for individual A, the behavioral features of behavior pattern 5 are registered as a judgment feature in the judgment feature DB112.

[0143] Figure 26 shows an example of calculating feature difference values. Figure 26 illustrates the case where the behavioral features and reference behavioral features for each person are represented by vectors. Vectors VB1, VB2, ..., VBm are vectors representing the reference behavioral features for behavioral patterns 1, 2, ..., M, respectively. Vectors VC1, VC2, ..., VCm are vectors representing the behavioral features for behavioral patterns 1, 2, ..., M, respectively for a given person.

[0144] The difference between a behavioral feature and a reference behavioral feature, i.e., the feature difference value, can be expressed, for example, as the angular difference of a vector. In the example in Figure 26, the feature difference values ​​for behavioral patterns 1, 2, ..., M are represented by angles D1, D2, ..., Dm, respectively.

[0145] Next, we will explain the process in Embodiment 2-2 that differs from that in Embodiment 2-1 using a flowchart. Figure 27 is an example flowchart showing the procedure for the individual behavior determination process performed by the individual behavior determination unit.

[0146] [Step S81] The individual behavior determination unit 136 selects the person to be processed. [Step S82] The individual behavior determination unit 136 refers to the behavior DB 114 and selects one behavior pattern from those associated with the person ID of the selected person.

[0147] [Step S83] The individual behavior determination unit 136 obtains all behavioral features corresponding to the selected behavioral pattern from the behavior DB 114. [Step S84] The individual behavior determination unit 136 calculates the range of variation of the behavioral features obtained in step S83.

[0148] [Step S85] The individual behavior determination unit 136 determines whether the calculated fluctuation range is less than or equal to a predetermined tolerance value. If the fluctuation range is less than or equal to the tolerance value, the process proceeds to step S86; if the fluctuation range exceeds the tolerance value, the process proceeds to step S87.

[0149] [Step S86] The individual behavior determination unit 136 calculates the median or mean of the behavioral features obtained in step S83, associates the calculated value with the person ID of the person selected in step S81 and the behavioral pattern ID of the behavioral pattern selected in step S82, and registers it as an individual behavioral feature in the individual behavior DB 117.

[0150] [Step S87] The individual behavior determination unit 136 determines whether all behavior patterns have been processed. If there are any unprocessed behavior patterns, the process proceeds to step S82, and one of the unprocessed behavior patterns is selected. On the other hand, if all behavior patterns have been processed, the process proceeds to step S88.

[0151] [Step S88] The individual behavior determination unit 136 determines whether all individuals have been processed. If there are individuals who have not been processed, the process proceeds to step S81, and one of the individuals who has not been processed is selected. On the other hand, if all individuals have been processed, the individual behavior determination process ends.

[0152] Through the above process, behavioral features representing the behavioral patterns that each individual consistently exhibits are registered in the Personal Behavior DB117 as personal behavioral features. Figure 28 shows an example of the data structure of the individual behavior database. As shown in Figure 28, the individual behavior database 117 contains a table 117a for each person. Table 117a associates the person ID with the number of behavior patterns corresponding to the behaviors that the person consistently exhibits. In addition, table 117a contains records that include the date and time, behavior pattern ID, and individual behavior features. The contents of the corresponding record in the behavior database 114 are registered directly into this record.

[0153] Figure 29 is an example flowchart showing the procedure for the reference behavior definition process in Embodiment 2-2. In Figure 29, the same step numbers are used for processes similar to those in Figure 12. In the reference behavior definition process shown in Figure 29, steps S32a and S33a are executed instead of steps S32 and S33 in Figure 12, respectively.

[0154] [Step S32a] The reference behavior definition unit 133 obtains individual behavior features for the selected behavior pattern from the individual behavior DB 117. In this process, individual behavior features for the corresponding behavior pattern are obtained regardless of the person ID.

[0155] [Step S33a] The reference behavior definition unit 133 determines whether all individual behavior features for the corresponding behavior pattern have been obtained from the individual behavior DB 117. If there are individual behavior features that have not been obtained, the process proceeds to step S32a, and one of the individual behavior features for the corresponding behavior pattern that has not been obtained is obtained. On the other hand, if all individual behavior features have been obtained, the process proceeds to step S34.

[0156] In step S34, reference behavior features are calculated based on the individual behavior features obtained from the individual behavior DB117 in step S32a. Figure 30 is an example flowchart showing the procedure for calculating behavioral differences in Embodiment 2-2. In Figure 30, the same step numbers are used for processes similar to those in Figure 14. In the behavioral difference calculation process shown in Figure 30, steps S42a and S43a are executed instead of steps S42 and S43 in Figure 14, respectively.

[0157] [Step S42a] The behavior difference calculation unit 134 refers to the individual behavior DB 117 and selects one behavior pattern from those associated with the person ID of the person selected in step S41.

[0158] [Step S43a] The behavior difference calculation unit 134 obtains individual behavior features from the individual behavior DB 117 that correspond to the behavior pattern selected in step S42a. In step S44, the difference between the individual behavior features obtained from the individual behavior DB117 in step S43a and the reference behavior features is calculated as the feature difference value.

[0159] Furthermore, although the processing procedure of the judgment feature calculation unit 135 is the same as in Figure 16, the number of behavior patterns selected in step S52 may be less than in Embodiment 2-1, and as a result, the judgment features registered in the judgment feature DB 112 may be different from those in Embodiment 2-1. In other words, in Embodiment 2-2, only the judgment features that correspond to behaviors that the person consistently takes, among the judgment features registered in Embodiment 2-1, are registered in the judgment feature DB 112.

[0160] Furthermore, in the processing of the judgment feature calculation unit 135, in step S55 of Figure 16, behavioral features may be obtained from the individual behavior DB 117 instead of the behavior DB 114. This reduces the number of records to be searched for in the DB for the relevant features, thereby shortening the processing time.

[0161] As a result of the processing shown in Figure 30, the behavior difference DB116 calculates feature difference values ​​only for behavior patterns that correspond to behaviors that the individual consistently takes. Therefore, the judgment feature DB112 identifies behavior patterns that correspond to behaviors that differ significantly from those of others among the behaviors that the individual consistently takes, using threshold judgment, and registers the behavior features of the identified behavior patterns as judgment features in the judgment feature DB112.

[0162] <Embodiment 2-3> Embodiment 2-3 is a modification of a part of the processing of the control server 100 in Embodiment 2-1 or Embodiment 2-2 described above.

[0163] Figure 31 is a diagram showing an example of the configuration of processing functions provided by the control server according to Embodiment 2-3. As shown in Figure 31, in the control server 100 according to Embodiment 2-3, the impersonation detection unit 140 further comprises a behavior indication unit 145. The behavior indication unit 145 outputs instruction information to the call terminal in which the person to be determined is making a call, instructing the person to perform the behavior of each behavior pattern compared by the behavior comparison unit 143.

[0164] In addition, Figure 31 shows a configuration in which a behavior indication unit 145 is added to the impersonation detection unit 140 in Embodiment 2-2 shown in Figure 21, but it is also possible to add the behavior indication unit 145 to the impersonation detection unit 140 in Embodiment 2-1 shown in Figure 7.

[0165] Figure 32 is an example flowchart showing the procedure for the impersonation detection process in Embodiment 2-3. In the impersonation detection process in Embodiment 2-3, steps S91 to S95 in Figure 32 are executed instead of steps S61 to S64 in Figure 18.

[0166] [Step S91] The behavior presentation unit 145 obtains all behavior pattern IDs associated with the person ID of the person to be judged (i.e., behavior pattern IDs of behavior patterns that show behavior typical of the person) from the judgment feature quantity DB 112. The behavior extraction unit 141 also starts acquiring video image data from the video call control unit 120.

[0167] [Step S92] The behavior presentation unit 145 selects one of the behavior patterns acquired in step S91. The behavior presentation unit 145 sends instruction information to the call terminal that sent the video data, instructing the person to perform the behavior of the selected behavior pattern. Based on this instruction information, the call terminal prompts the person on the call to perform the behavior of the selected behavior pattern, for example, by displaying an image or outputting audio. For example, if prompting the person to "tilt their head to the side," the call terminal outputs the audio "Please tilt your head to the side." The call terminal may also guide the person to perform the corresponding behavior by displaying an image or outputting audio. For example, the call terminal may guide the person to perform the corresponding behavior by outputting a question that prompts the person to perform that behavior.

[0168] [Step S93] The behavior extraction unit 141 extracts features from each frame of the input video data by image recognition, following the same procedure as in steps S12 to S17 in Figure 8, and calculates time-series features based on the features.

[0169] [Step S94] The behavior determination unit 142 detects behavior that matches one of the behavior patterns by comparing the calculated time-series features with the defined behavior features in the defined behavior DB 113. If the time-series features match the defined behavior features of the behavior pattern selected in step S92, the behavior determination unit 142 calculates behavior features corresponding to that behavior pattern based on the time-series features and records them in a storage device (such as RAM 102) in association with the behavior pattern ID.

[0170] [Step S95] The behavior presentation unit 145 determines whether the behavioral features for all behavioral patterns acquired in step S91 have been recorded in the storage device. If there are behavioral patterns for which behavioral features have not been recorded, the process proceeds to step S92, and one of the relevant behavioral patterns is selected. On the other hand, if the behavioral features for all relevant behavioral patterns have already been recorded, the process proceeds to step S65 in Figure 19, and the process is executed using the stored behavioral features.

[0171] In the embodiments 2-3 described above, the behavior comparison unit 143 outputs instruction information that instructs the person to perform the behavior of each behavior pattern being compared, thereby enabling more reliable acquisition of behavioral features of the behavior patterns necessary for judgment. As a result, it becomes possible to improve the accuracy of impersonation detection.

[0172] In the second embodiment described above (Embodiments 2-1 to 2-3), the determination of "whether the person being judged is the person in question" was based on the difference from the person's typical behavior. However, a large difference from the person's typical behavior may occur when the person is exhibiting abnormal behavior that differs from their usual behavior. Such abnormal behavior may occur, for example, when the person is ill, being threatened, or hiding something. Therefore, the above judgment processing procedure can also be used to determine whether the person's behavior is normal or abnormal. Furthermore, the type of abnormal behavior can be determined by using different judgment criteria (thresholds in step S69 of Figure 19) for each behavior pattern.

[0173] Furthermore, in the second embodiment described above, the movement of body parts was detected by image recognition from video data captured by the call terminal, and the detection results were used to determine whether the person was the real person or not. However, for example, speech recognition could be used to detect speech habits and conversational responses from audio data picked up by the call terminal, and the detection results could be combined with the detection results based on the video data described above to determine whether the person was the real person or not.

[0174] Furthermore, the determination result obtained by the process shown in the second embodiment may be combined with the determination result obtained by an existing process that detects fake face images from video data to output a final determination result indicating whether or not the person is the real person.

[0175] Furthermore, in the second embodiment described above, the determination process was performed in real time using video data from a call terminal during a call. However, as another example, the video data to be determined may be stored in a storage device in advance, and the above determination process may be performed on the video data acquired from that storage device.

[0176] Furthermore, the processing functions of the devices shown in each of the above embodiments (for example, the determination device 1 and the control server 100) can be implemented by a computer. In this case, a program describing the processing content of the functions that each device should have is provided, and by executing that program on the computer, the above processing functions are implemented on the computer. The program describing the processing content can be recorded on a recording medium that can be read by a computer. Examples of recording media that can be read by a computer include magnetic storage devices, optical discs, and semiconductor memory. Examples of magnetic storage devices include hard disk drives (HDDs) and magnetic tapes. Examples of optical discs include CDs (Compact Discs), DVDs (Digital Versatile Discs), and Blu-ray Discs (BD, registered trademark).

[0177] When distributing a program, portable recording media such as DVDs and CDs containing the program are sold. Alternatively, the program can be stored in the storage device of a server computer and transferred from the server computer to other computers via a network.

[0178] A computer executing a program stores programs, for example, those recorded on a portable storage medium or transferred from a server computer, in its own memory. The computer then reads the program from its memory and executes the processing according to the program. Alternatively, the computer can directly read the program from the portable storage medium and execute the processing according to that program. Furthermore, the computer can sequentially execute the processing according to the programs received from a server computer connected via a network, each time a program is transferred. [Explanation of Symbols]

[0179] 1 Judgment device 2 Input image 3,4 Datasets S1-S6 Steps TP1~TP8 Types

Claims

1. On the computer, Based on a first data set containing data indicating the behavior of multiple individuals, a baseline behavior data is calculated for each of several types of behavior, indicating the baseline behavior among the aforementioned multiple individuals. From the first data set, first behavior data indicating the behavior of a first person among the multiple persons is obtained for each of the multiple types. The difference between the first behavior data and the reference behavior data is calculated for each of the multiple types. From the aforementioned plurality of types, one or more first types whose difference is greater than or equal to a first threshold are identified, and second behavior data indicating the behavior of the first person for each of the one or more first types is registered in a second data group. A third behavioral data representing the behavior of a second person is extracted from the input image. Based on the comparison result between the third behavior data and the second behavior data, it is determined whether the second person is the same as the first person. A program that determines whether to execute a process.

2. The first behavioral data is acquired multiple times for each of the multiple types, The computer is further instructed to perform a process to identify one or more second types from among the plurality of types in which the variation range of the plurality of first behavior data is less than or equal to a second threshold. The aforementioned one or more first types are selected from the aforementioned one or more second types. The determination program according to claim 1.

3. The standard behavior data for one of the aforementioned multiple types is calculated as the median or average value of the data representing the behavior of that one type for each of the multiple individuals registered in the first data group. The determination program according to claim 1 or 2.

4. In the determination, if none of the one or more first types of behavior are detected from the input image, it is determined that the second person is different from the first person. The determination program according to claim 1 or 2.

5. The computer is further instructed to output instruction information that instructs the second person to take each of the one or more first types of behaviors. The third behavior data is extracted from the input image taken after the output of the instruction information. The determination program according to claim 1 or 2.

6. On the computer, A third behavioral data representing the behavior of a second person is extracted from the input image. The third behavior data and the second behavior data are compared by referring to a second data set in which second behavior data is registered that shows the behavior of a first person for each of one or more first types among several types of behavior classifications. Based on the results of the comparison, it is determined whether the second person is the same as the first person. A decision program that determines whether to execute a process, The second set of data mentioned above is, Based on a first data set in which data indicating the behavior of each of the multiple persons, including the first person, is registered, a reference behavior data indicating the reference behavior among the multiple persons is calculated for each of the multiple types of behavior. From the first data set, first behavioral data indicating the behavior of the first person is obtained for each of the multiple types. The difference between the first behavior data and the reference behavior data is calculated for each of the multiple types. This is generated by identifying one or more first types from among the multiple types whose difference is greater than or equal to a predetermined threshold, and registering data indicating the behavior of the first person for each of the one or more first types as second behavior data in the second data group. Judgment program.

7. The first computer, Based on a first data set containing data indicating the behavior of multiple individuals, a baseline behavior data is calculated for each of several types of behavior, indicating the baseline behavior among the aforementioned multiple individuals. From the first data set, first behavior data indicating the behavior of a first person among the multiple persons is obtained for each of the multiple types. The difference between the first behavior data and the reference behavior data is calculated for each of the multiple types. From among the aforementioned multiple types, one or more first types whose difference is greater than or equal to a predetermined threshold are identified, and second behavior data indicating the behavior of the first person for each of the one or more first types is registered in a second data group. The first computer or the second computer, A third behavioral data representing the behavior of a second person is extracted from the input image. Based on the comparison result between the third behavior data and the second behavior data registered in the second data group, it is determined whether the second person is the same as the first person. Judgment method.

8. Based on a first data set containing data indicating the behavior of multiple individuals, a baseline behavior data is calculated for each of several types of behavior, indicating the baseline behavior among the aforementioned multiple individuals. From the first data set, first behavior data indicating the behavior of a first person among the multiple persons is obtained for each of the multiple types. The difference between the first behavior data and the reference behavior data is calculated for each of the multiple types. From the aforementioned plurality of types, one or more first types whose difference is greater than or equal to a first threshold are identified, and second behavior data indicating the behavior of the first person for each of the one or more first types is registered in a second data group. A third behavioral data representing the behavior of a second person is extracted from the input image. A processing unit determines whether the second person is the same as the first person based on the comparison result between the third behavior data and the second behavior data. A determination device having the following features.

Images