Information processing device, information processing method, and computer program

Abstract

The ratio of incorrect acceptance by others and incorrect rejection by the intended applicant should be appropriately adjusted. [Solution] The information processing device includes: acquisition means for acquiring time-series image data; index calculation means for calculating an integrated feature obtained by integrating a first feature which is a feature of the time-series image data and a second feature which is a feature of pre-registered image data, or a score indicating the similarity between the first feature and the second feature; likelihood ratio calculation means for calculating a likelihood ratio indicating the likelihood of the class to which the time-series image data belongs based on the integrated feature or score; determination means for determining that the time-series image data belongs to a registered class corresponding to the pre-registered image data when the likelihood ratio reaches a first threshold, and determining that the time-series image data belongs to an unregistered class indicating that it has not been pre-registered when the likelihood ratio reaches a second threshold which is a value different from the first threshold.

Description

Technical Field

[0001] This disclosure relates to the technical fields of information processing apparatuses, information processing methods, and computer programs.

Background Art

[0002] As this type of apparatus, there is known one that performs a process of determining the class to which data belongs (so-called class classification). For example, Patent Document 1 discloses a technique for determining the class to which series data to be classified belongs when an individual score or an integrated score calculated based on a likelihood ratio reaches a predetermined threshold value.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] This disclosure aims to provide an information processing apparatus, an information processing method, and a computer program for improving the techniques disclosed in the prior art documents.

Means for Solving the Problems

[0005] One aspect of the information processing device disclosed herein includes: acquisition means for acquiring time-series image data; index calculation means for calculating an integrated feature obtained by integrating a first feature which is a feature of the time-series image data and a second feature which is a feature of pre-registered image data, or a score indicating the similarity between the first feature and the second feature; likelihood ratio calculation means for calculating a likelihood ratio indicating the likelihood of the class to which the time-series image data belongs based on the integrated feature or the score; determination means for determining that the time-series image data belongs to a registered class corresponding to the pre-registered image data when the likelihood ratio reaches a first threshold, and determining that the time-series image data belongs to an unregistered class indicating that it has not been pre-registered when the likelihood ratio reaches a second threshold which is a value different from the first threshold.

[0006] One aspect of the information processing method of this disclosure involves at least one computer acquiring time-series image data, integrating a first feature, which is a feature of the time-series image data, with a second feature, which is a feature of pre-registered image data, to calculate an integrated feature, or a score indicating the similarity between the first feature and the second feature, calculating a likelihood ratio indicating the likelihood of the class to which the time-series image data belongs based on the integrated feature or the score, determining that the time-series image data belongs to a registered class corresponding to the pre-registered image data when the likelihood ratio reaches a first threshold, and determining that the time-series image data belongs to an unregistered class, indicating that it has not been pre-registered, when the likelihood ratio reaches a second threshold that is a different value from the first threshold.

[0007] One aspect of the computer program of this disclosure causes at least one computer to execute an information processing method which involves acquiring time-series image data, integrating a first feature, which is a feature of the time-series image data, with a second feature, which is a feature of pre-registered image data, or calculating a score indicating the similarity between the first feature and the second feature, calculating a likelihood ratio indicating the likelihood of the class to which the time-series image data belongs based on the integrated feature or the score, determining that the time-series image data belongs to a registered class corresponding to the pre-registered image data when the likelihood ratio reaches a first threshold, and determining that the time-series image data belongs to an unregistered class, indicating that it has not been pre-registered, when the likelihood ratio reaches a second threshold that is a different value from the first threshold. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing the hardware configuration of the first information processing device. [Figure 2] This is a block diagram showing the functional configuration of the first information processing device. [Figure 3] This is a flowchart showing the operation flow of the first information processing device. [Figure 4] This graph (Part 1) shows an example of the class threshold, unregistered threshold, and likelihood ratio in the first information processing device. [Figure 5] This is a graph (part 2) showing an example of the class threshold, unregistered threshold, and likelihood ratio in the first information processing device. [Figure 6] This is a graph (part 3) showing an example of the class threshold, unregistered threshold, and likelihood ratio in the first information processing device. [Figure 7] This is graph (number 4) showing an example of the class threshold, unregistered threshold, and likelihood ratio in the first information processing device. [Figure 8] This is a block diagram showing the functional configuration of the second information processing device. [Figure 9] This is a flowchart showing the threshold setting operation flow in the second information processing device. [Figure 10] This is a block diagram showing the functional configuration of the third information processing device. [Figure 11] This is a flowchart showing the operation flow of the third information processing device. [Modes for carrying out the invention]

[0009] The following describes embodiments of the information processing device, information processing method, and computer program with reference to the drawings.

[0010] <First Embodiment> The first information processing device will be described with reference to Figures 1 to 7.

[0011] (Hardware configuration) First, the hardware configuration of the first information processing device will be described with reference to Figure 1. Figure 1 is a block diagram showing the hardware configuration of the first information processing device.

[0012] As shown in Figure 1, the first information processing device 1 includes a processor 11, RAM (Random Access Memory) 12, ROM (Read Only Memory) 13, a storage device 14, an input device 15, and an output device 16. The processor 11, RAM 12, ROM 13, storage device 14, input device 15, and output device 16 are all connected via a data bus 17. Note that the data bus 17 may be an interface other than a data bus (for example, LAN or USB).

[0013] The processor 11 reads a computer program. For example, the processor 11 is configured to read a computer program stored in at least one of the RAM 12, ROM 13, and storage device 14. Alternatively, the processor 11 may read a computer program stored in a computer-readable storage medium using a storage medium reading device (not shown). The processor 11 may also obtain (i.e., read) a computer program from a device (not shown) located outside the first information processing device 1 via a network interface. The processor 11 performs various processes by executing the read computer program. When the processor 11 executes the read computer program, a functional block related to the processing performed by the first information processing device 1 is realized within the processor 11. That is, the processor 11 may function as a controller that performs various processes and controls in the first information processing device 1.

[0014] The processor 11 may be configured as, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an FPGA (field-programmable gate array), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), or a quantum processor. The processor 11 may consist of one of these, or it may be configured to use multiple of them in parallel.

[0015] RAM12 temporarily stores computer programs executed by processor 11. RAM12 also temporarily stores data that processor 11 uses temporarily while executing computer programs. RAM12 may be, for example, D-RAM (Dynamic Random Access Memory) or SRAM (Static Random Access Memory). Alternatively, other types of volatile memory may be used instead of RAM12.

[0016] The ROM 13 stores a computer program executed by the processor 11. The ROM 13 may also store other fixed data. The ROM 13 may be, for example, a P-ROM (Programmable Read Only Memory) or an EPROM (Erasable Read Only Memory). Also, instead of the ROM 13, other types of non-volatile memories may be used.

[0017] The storage device 14 stores data that the first information processing device 1 stores in the long term. The storage device 14 may operate as a temporary storage device of the processor 11. The storage device 14 may store a computer program executed by the processor 11. The storage device 14 may include, for example, at least one of a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and a disk array device.

[0018] The input device 15 is a device that receives an input instruction from a user of the first information processing device 1. The input device 15 may include, for example, at least one of a keyboard, a mouse, a touch panel, and a touch pen. The input device 15 may be a device capable of voice input including, for example, a microphone.

[0019] The output device 16 is a device that outputs information regarding the first information processing device 1 to the outside. For example, the output device 16 may be a display device (e.g., a display, a monitor, etc.) capable of displaying information regarding the first information processing device 1. Also, the output device 16 may be a speaker or the like capable of voice outputting information regarding the information processing device 1.

[0020] The first information processing device may be configured to include some of the components described in Figure 1. For example, the first information processing device 1 may be configured to include only the processor 11, RAM 12, and ROM 13 from the components described above. In this case, the storage device 14, input device 15, and output device 16 may each be provided as external devices to the first information processing device 1. Furthermore, some of the arithmetic functions of the first information processing device 1 may be implemented by an external server or cloud service.

[0021] (Functional configuration) Next, the functional configuration of the first information processing device 1 will be described with reference to Figure 2. Figure 2 is a block diagram showing the functional configuration of the first information processing device.

[0022] In Figure 2, the first information processing device 1 is configured as a device for classifying incoming time-series data. More specifically, the first information processing device 1 is configured to perform matching processing (in other words, authentication processing) of objects contained in image data by classifying incoming time-series image data into classes corresponding to registered image data. The first information processing device 1 is configured to include an image acquisition unit 110, an index calculation unit 120, a registered image storage unit 125, a likelihood ratio calculation unit 130, and a class determination unit 140 as components for realizing its function. Note that each of the image acquisition unit 110, the index calculation unit 120, the likelihood ratio calculation unit 130, and the class determination unit 140 may be processing blocks realized by the processor 11 (see Figure 1) described above.

[0023] The image acquisition unit 110 is configured to acquire time-series image data. The image acquisition unit 110 may acquire image data of multiple frames sequentially. For example, the image acquisition unit 110 may be configured to acquire images of each frame captured from a camera that shoots video. The time-series image data acquired by the image acquisition unit 110 is acquired in order to perform a matching process with pre-registered registered image data. For example, the time-series image data may be face image data including the face of the target. In this case, the acquired face image data may be used to perform face recognition using the face of the target. Note that face image data is merely an example, and the time-series image data may include image data of parts other than the face of the target (for example, fingerprints or irises). Also, the time-series image data may include subjects other than people. The time-series image data acquired by the image acquisition unit 110 is configured to be output to the index calculation unit 120.

[0024] The index calculation unit 120 is configured to calculate an integrated feature or score from time-series image data acquired by the image acquisition unit 110. The integrated feature is a feature obtained by integrating a first feature, which is a feature of the time-series image data, and a second feature, which is a feature of pre-registered image data. The score is a score that indicates the similarity (in other words, the degree of agreement) between the first feature and the second feature described above. The index calculation unit 120 may also have a function to extract features from image data in order to calculate the integrated feature or score. For example, the index calculation unit 120 may have a function to extract features of the target face from face image data that includes the target face. The index calculation unit 120 acquires time-series image data from the image acquisition unit 110 while simultaneously acquiring registered image data from the registered image storage unit 125 to calculate the integrated feature or score. The integrated feature or score calculated by the index calculation unit 120 is output to the likelihood ratio calculation unit 130.

[0025] The registered image storage unit 125 is configured to store registered image data used by the index calculation unit 120 when calculating integrated features or scores. The registered image storage unit 125 may be configured to store features extracted from the registered image data (i.e., second features) rather than the registered image data itself. The registered image storage unit 125 may be implemented using the storage device 14 (see Figure 1) described above. Alternatively, the registered image storage unit 125 may be implemented by a database or the like provided outside the first information processing device 1. The registered image storage unit 125 is configured to store multiple registered image data. The index calculation unit 120 may calculate N integrated features or N scores using first features extracted from time-series image data and N second features extracted from each of the N registered image data. The registered image data stored in the registered image storage unit 125 may be, for example, face image data of a registered user used for face recognition. In this case, the integrated feature quantities and scores calculated by the index unit 120 serve as indicators showing the degree of match between the target face included in the time-series image data and the face of the registered user.

[0026] The likelihood ratio calculation unit 130 is configured to calculate the likelihood ratio based on the integrated feature or score calculated by the index calculation unit 120. The likelihood ratio calculated by the likelihood ratio calculation unit 130 is a value that indicates the likelihood of the time-series image data belonging to a particular class. For example, the likelihood ratio calculation unit 130 calculates the likelihood that the time-series image data belongs to each of the multiple registration classes set for each registered image data. In this case, the likelihood ratio calculation unit 130 may calculate multiple likelihood ratios corresponding to each of the multiple registration classes (i.e., the same number of likelihood ratios as the number of registration classes). For example, if N registered image data is registered in the registered image storage unit 125, N registration classes are set to correspond to them, and the likelihood ratio calculation unit 130 may calculate N likelihood ratios corresponding to each of the N registration classes. The likelihood ratio calculation unit 130 also calculates the likelihood that the time-series image data belongs to an unregistered class, which is the case when the image data does not belong to any of the registered classes (i.e., is unregistered). The likelihood ratio calculation unit 130 is not particularly limited in its method for calculating the likelihood ratio. The likelihood ratio calculation unit 130 may calculate the likelihood ratio using various existing methods. For example, the likelihood ratio calculation unit 130 may calculate the likelihood ratio using a pre-trained estimation model (specifically, a neural network trained by deep learning, etc.). The likelihood ratio calculation unit 130 may calculate the likelihood ratio based on two or more consecutive image data from a time series of image data. For example, the likelihood ratio calculation unit 130 may calculate the likelihood ratio using integrated features or scores calculated from previously acquired image data in addition to integrated features or scores calculated from the most recently acquired image data. The likelihood ratio calculated by the likelihood ratio calculation unit 130 is output to the class determination unit 140.

[0027] The class determination unit 140 is configured to determine the class to which time-series image data belongs based on the likelihood ratio calculated by the likelihood ratio calculation unit 130. In other words, the class determination unit 140 is configured to perform class classification processing based on likelihood ratios. Specifically, the class determination unit 140 determines the class to which time-series image data belongs by comparing the likelihood ratio corresponding to each class calculated by the likelihood ratio calculation unit 130 with a pre-set threshold. The class determination unit 140 uses two different thresholds depending on the type of likelihood ratio. Specifically, the class determination unit 140 uses a class threshold for the likelihood ratio corresponding to a registered class (hereinafter referred to as the "first likelihood ratio" as appropriate). The class determination unit 140 also uses an unregistered threshold for the likelihood ratio corresponding to an unregistered class (hereinafter referred to as the "second likelihood ratio" as appropriate). The class threshold and the unregistered threshold are set to different values ​​from each other. The class determination unit 140 determines that time-series image data belongs to a registered class corresponding to registered image data when the first likelihood ratio reaches the class threshold. That is, the class determination unit 140 determines that time-series image data matches one of the registered image data when the first likelihood ratio reaches the class threshold. On the other hand, the class determination unit 140 determines that time-series image data belongs to an unregistered class (in other words, does not belong to any of the registered classes) when the second likelihood ratio reaches the unregistered threshold. That is, the class determination unit 140 determines that time-series image data does not match any of the registered image data (in other words, is unregistered) when the second likelihood ratio reaches the unregistered threshold.

[0028] (Flow of operations) Next, the operation flow of the first information processing device 1 will be explained with reference to Figure 3. Figure 3 is a flowchart showing the operation flow of the first information processing device.

[0029] As shown in Figure 3, when the operation of the first information processing device 1 begins, the image acquisition unit 110 first acquires image data (step S101). The image data acquired here may be the image data of the most recently captured single frame from the time-series image data.

[0030] Next, the index calculation unit 120 extracts a first feature from the image data acquired by the image acquisition unit 110 and a second feature from the registered image data read from the registered image storage unit 125 (step S102). Then, the index calculation unit 120 calculates an integrated feature or score based on the extracted first and second features (step S103).

[0031] Next, the likelihood ratio calculation unit 130 calculates a likelihood ratio indicating the likelihood of the time-series image data belonging to a particular class, based on the integrated features or scores calculated by the index calculation unit 120 (step S104). Specifically, the likelihood ratio calculation unit 130 calculates a first likelihood ratio corresponding to the registered class and a second likelihood ratio corresponding to the unregistered class.

[0032] Next, the class determination unit 140 determines whether the second likelihood ratio calculated by the likelihood ratio calculation unit 130 has reached the unregistered threshold (step S105). If the second likelihood ratio has reached the unregistered threshold (step S105: YES), the class determination unit 140 determines that the time-series image data does not belong to any registered class (i.e., it is unregistered) (step S106). In this case, the time-series image data is determined not to match the registered image data (in other words, the target is an unregistered user), and the matching process ends.

[0033] On the other hand, if the second likelihood ratio has not reached the unregistered threshold (step S105: NO), the class determination unit 140 determines whether the first likelihood ratio has reached the class threshold (step S107). If the first likelihood ratio has reached the class threshold (step S107: YES), the class determination unit 140 determines that the time-series image data belongs to a registered class (step S108). Specifically, the class determination unit 140 determines that the time-series image data belongs to a registered class corresponding to the first likelihood ratio that has exceeded the class threshold. In this case, the time-series image data is determined to match the registered image data (in other words, the target is a registered user), and the matching process ends.

[0034] On the other hand, if the first likelihood ratio does not reach the class threshold (step S107: NO), processing starts again from step S101. That is, the image data acquisition unit 110 acquires new image data (for example, image data of the next frame), and the series of processes described above are executed again. As the processing is repeated in this way, the likelihood ratio calculated by the likelihood ratio calculation unit 130 gradually changes. The determination process (in other words, the matching process) continues until the likelihood ratio exceeds the class threshold or the unregistered threshold.

[0035] The first information processing device 1 may be configured to execute various processes related to the target based on the determination result of the class determination unit 140. For example, the first information processing device 1 may be configured to execute a process to permit or prohibit passage of the target to a predetermined area based on the determination result of the class determination unit 140. More specifically, if it is determined that the time-series image data matches the registered image data, the first information processing device 1 may control the gate installed in the predetermined area to open, thereby permitting passage of the target. If it is determined that the time-series image data does not match the registered image data (i.e., unregistered), the first information processing device 1 may control the gate installed in the predetermined area to close, thereby prohibiting passage of the target.

[0036] (Example of operation) Next, with reference to Figures 4 to 7, specific examples of the operation of the first information processing device 1 (in particular, examples of the operation of determining the class to which time-series image data belongs based on the likelihood ratio) will be described. Figure 4 is a graph (1) showing an example of the class threshold, unregistered threshold, and likelihood ratio in the first information processing device. Figure 5 is a graph (2) showing an example of the class threshold, unregistered threshold, and likelihood ratio in the first information processing device. Figure 6 is a graph (3) showing an example of the class threshold, unregistered threshold, and likelihood ratio in the first information processing device. Figure 7 is a graph (4) showing an example of the class threshold, unregistered threshold, and likelihood ratio in the first information processing device.

[0037] In the examples shown in Figures 4 and 5, registered image data A, registered image data B, and registered image data C are registered. Classes A, B, and C are set as the corresponding registration classes for each of the registered image data A, B, and C. In addition to the registered classes, an unregistered class is also set.

[0038] The likelihood ratio calculation unit 130 calculates first likelihood ratios based on the image data acquired by the image acquisition unit 110 and the registered image data A, B, and C. Specifically, the likelihood ratio calculation unit 130 calculates first likelihood ratios corresponding to class A, class B, and class C. The likelihood ratio calculation unit 130 also calculates second likelihood ratios corresponding to unregistered classes. These likelihood ratios gradually change over time (i.e., as image data is acquired sequentially).

[0039] In the examples shown in Figures 4 and 5, the class threshold used to determine whether or not a class belongs to a registered class is set to a higher value than the unregistered threshold used to determine whether or not a class belongs to an unregistered class.

[0040] In the example shown in Figure 4, the first likelihood ratio corresponding to class A has reached the class threshold. On the other hand, the first likelihood ratios corresponding to classes B and C have not reached the class threshold. Furthermore, the second likelihood ratio corresponding to the unregistered class has not reached the unregistered threshold. In such a case, the time-series image data acquired by the image acquisition unit 110 is determined to belong to class A. That is, the time-series image data is determined to match registered image data A, but not to registered image data B and C. As a result, a matching result is obtained in which the subject included in the time-series image data is the user corresponding to registered image data A.

[0041] In the example shown in Figure 5, none of the first likelihood ratios corresponding to classes A, B, and C have reached the class threshold. On the other hand, the second likelihood ratio corresponding to the unregistered class has reached the unregistered threshold. In such a case, the time-series image data acquired by the image acquisition unit 110 is determined to belong to the unregistered class. In other words, the time-series image data is determined to be unregistered image data that does not belong to any of classes A, B, and C. Therefore, the time-series image data is determined not to match any of the registered image data A, B, and C. As a result, the matching result is obtained that the subject included in the time-series image data is an unregistered user.

[0042] As shown in Figures 4 and 5, setting the class threshold to a value higher than the unregistered threshold makes it less likely for time-series image data to be determined to belong to a registered class, while making it more likely to be determined to belong to an unregistered class. As a result, when the correct class to which time-series image data belongs is "unregistered class," the accuracy of the unregistered determination increases. Conversely, when the correct class to which time-series image data belongs is "registered class," the accuracy of the unregistered determination decreases. Therefore, the number of false acceptances by others (determining that a match exists in the registered image data when there is no match) and false rejections by the correct person (determining that a match does not exist in the registered image data when there is a match) in the matching process decreases.

[0043] In the examples shown in Figures 6 and 7, most of the conditions are the same as in the examples shown in Figures 4 and 5, except that the class threshold for determining whether or not a class belongs to a registered class is set to a lower value than the unregistered threshold for determining whether or not a class belongs to an unregistered class.

[0044] In the example shown in Figure 6, the second likelihood ratio corresponding to the unregistered class is higher than the other first likelihood ratios, but it does not reach the unregistered threshold. On the other hand, the first likelihood ratio corresponding to class A is lower than the second likelihood ratio, but it does reach the class threshold. In such a case, the time-series image data acquired by the image acquisition unit 110 is determined to belong to class A. That is, the time-series image data is determined to match registered image data A, but not to registered image data B and C. As a result, the matching result is obtained that the subject included in the time-series image data is the user corresponding to registered image data A.

[0045] In the example shown in Figure 7, none of the first likelihood ratios corresponding to classes A, B, and C have reached the class threshold. On the other hand, the second likelihood ratio corresponding to the unregistered class has reached the unregistered threshold. In such a case, the time-series image data acquired by the image acquisition unit 110 is determined to belong to the unregistered class. In other words, the time-series image data is determined to be unregistered image data that does not belong to any of classes A, B, and C. Therefore, the time-series image data is determined not to match any of the registered image data A, B, and C. As a result, the matching result is obtained that the subject included in the time-series image data is an unregistered user.

[0046] As shown in Figures 6 and 7, setting the class threshold to a value lower than the unregistered threshold makes it easier to determine that time-series image data belongs to a registered class, while making it harder to determine that it belongs to an unregistered class. As a result, when the correct class to which the time-series image data belongs is "unregistered class," the accuracy of the unregistered determination decreases. Conversely, when the correct class to which the time-series image data belongs is "registered class," the accuracy of the unregistered determination increases. Therefore, the number of false acceptances by others and the number of false rejections by others in the matching process increases, while the number of false rejections by others decreases.

[0047] (Technical effects) Next, we will explain the technical effects obtained by the first information processing device 1.

[0048] As explained in Figures 1 to 7, in the first information processing device 1, a class threshold for determining whether or not time-series image data belongs to a registered class and an unregistered threshold for determining whether or not it belongs to an unregistered class are set to different values. In this way, the rate of misacceptance by others and misrejection by the intended recipient can be adjusted by changing the values ​​of the class threshold and the unregistered threshold, respectively.

[0049] <Second Embodiment> The second information processing device 1 will be described with reference to Figures 8 and 9. Note that the second information processing device 1 differs from the first information processing device 1 described above in some configurations and operations, while other parts may be the same as those of the first information processing device 1. Therefore, the parts that differ from the first embodiment will be described in detail below, while other overlapping parts will be omitted as appropriate.

[0050] (Functional configuration) First, the functional configuration of the second information processing device 1 will be explained with reference to Figure 8. Figure 8 is a block diagram showing the functional configuration of the second information processing device. Note that in Figure 8, the same reference numerals are used for elements similar to those described in Figure 2.

[0051] In Figure 8, the second information processing device 1 is configured to include an image acquisition unit 110, an index calculation unit 120, a registered image storage unit 125, a likelihood ratio calculation unit 130, a class determination unit 140, and a threshold setting unit 150 as components for realizing its functions. That is, the second information processing device 1 further includes a threshold setting unit 150 in addition to the configuration described in the first embodiment (see Figure 2). The threshold setting unit 150 may be a processing block realized by the processor 11 (see Figure 1) described above.

[0052] The threshold setting unit 150 is configured to set an unregistered threshold used by the class determination unit 140. Specifically, the threshold setting unit 150 sets the unregistered threshold based on the class threshold. As a result, the class threshold and the unregistered threshold become correlated values. For example, the threshold setting unit 150 may calculate the unregistered threshold using the class threshold and an adjustment parameter. The adjustment parameter is a parameter for trading off between incorrect acceptance by others and incorrect rejection by the correct person, and may be determined based on a desired ratio between incorrect acceptance by others and incorrect rejection by the correct person.

[0053] If the class threshold is λ1, the unregistered threshold is λ2, and the adjustment parameter is δ, the threshold setting unit 150 may calculate and set the unregistered threshold λ2 as λ2 = λ1 + δ. That is, the unregistered threshold λ2 may be set as the class threshold λ1 plus the adjustment parameter δ. Although an example of adding the adjustment parameter is given here, the unregistered threshold may also be calculated by multiplying the class threshold λ1 by the adjustment parameter. Alternatively, the unregistered threshold may be calculated using a more complex function.

[0054] The class threshold λ1 can be pre-set based on the required speed and accuracy for the determination. For example, lowering the class threshold shortens the time it takes to determine if an item is a registered class, thus increasing the determination speed. On the other hand, raising the class threshold lengthens the time it takes to determine if an item is a registered class (i.e., allows more time for determination), thus increasing the determination accuracy. In this way, the class threshold functions as a parameter to adjust the speed and accuracy of the determination. Therefore, by using the class threshold and the unregistered threshold set based on the adjustment parameter, it is possible to adjust the rate of incorrect acceptance by others and incorrect rejection by others, as well as the speed and accuracy of the determination.

[0055] (Threshold setting operation) Next, with reference to Figure 9, the flow of the threshold setting operation in the second information processing device 1 (i.e., the operation when the threshold setting unit 150 sets an unregistered threshold) will be explained. Figure 9 is a flowchart showing the flow of the threshold setting operation in the second information processing device.

[0056] As shown in Figure 9, when the operation of the second information processing device 1 begins, the threshold setting unit 150 first acquires the adjustment parameters (step S201). The adjustment parameters may be input by, for example, the user of the device. In this case, the user may input the value of the adjustment parameter itself, or they may input the desired percentage of misacceptance by others and misrejection by others.

[0057] Next, the threshold setting unit 150 calculates an unregistered threshold using the pre-set class threshold and the adjustment parameters obtained in step S201 (step S202). Then, the threshold setting unit 150 outputs the calculated unregistered threshold value to the class determination unit 140 and sets it as the threshold used for determining the likelihood ratio (step S203).

[0058] The threshold setting operation described above typically only needs to be performed once before starting operation of the second information processing device 1. However, if a situation arises where the rates of incorrect acceptance by others and incorrect rejection by others need to be changed, the threshold setting operation may be performed again. In this case, the values ​​of the adjustment parameters should be changed and the series of operations described above should be performed.

[0059] (Technical effects) Next, we will explain the technical effects obtained by the second information processing device 1.

[0060] As explained in Figures 6 and 7, in the second information processing device 1, the class threshold and the unregistered threshold are set as correlated values. In this way, it is possible to set an appropriate unregistered threshold according to the value of the class threshold. In this case, by using adjustment parameters to trade off false acceptance by others and false rejection by others, false acceptance by others and false rejection by others in the judgment process can be appropriately adjusted. For example, if the unregistered threshold λ2 is calculated as λ2 = λ1 + δ, the unregistered threshold can be easily and accurately calculated from the class threshold.

[0061] <Third Embodiment> The third information processing device 1 will be described with reference to Figures 10 and 11. Note that the third information processing device 1 differs in some configurations and operations from the first and second information processing devices 1 described above, while other parts may be the same as those of the first and second information processing devices 1. Therefore, the following will provide a detailed explanation of the parts that differ from the embodiments already described, while explanations of other overlapping parts will be omitted as appropriate.

[0062] (Functional configuration) First, the functional configuration of the third information processing device 1 will be explained with reference to Figure 10. Figure 10 is a block diagram showing the functional configuration of the third information processing device. Note that in Figure 10, the same reference numerals are used for elements similar to those described in Figure 2.

[0063] In Figure 10, the third information processing device 1 is configured to include an image acquisition unit 110, an index calculation unit 120, a registered image storage unit 125, a likelihood ratio calculation unit 130, a class determination unit 140, and a threshold change unit 160 as components for realizing its functions. That is, the third information processing device 1 further includes a threshold change unit 160 in addition to the configuration described in the first embodiment (see Figure 2). The threshold change unit 160 may be a processing block realized by the processor 11 (see Figure 1) described above.

[0064] The threshold change unit 160 is configured to change the unregistered threshold used by the class determination unit 140. Specifically, the threshold change unit 160 is configured to dynamically change the unregistered threshold according to the time-series image data acquired by the image acquisition unit 110. As already explained, the unregistered threshold has the function of adjusting the ratio of false acceptances and false rejections of correct entries in the matching process. Therefore, by dynamically changing the unregistered threshold, the ratio of false acceptances and false rejections of correct entries can be changed during the determination process. However, it is not common to change the ratio of false acceptances and false rejections of correct entries midway through the matching process. On the other hand, if the circumstances under which the matching process is performed change significantly, the ratio of false acceptances and false rejections of correct entries may change unintentionally. In such cases, the threshold change unit 160 may change the unregistered threshold so that the ratio of false acceptances and false rejections of correct entries does not change in response to the change in circumstances.

[0065] For example, the threshold changing unit 160 may estimate the direction of movement of the object included in the time-series image data (i.e., the flow of people at the location where the time-series image data is taken) and change the unregistered threshold based on the estimated direction of movement of the object. When the direction of movement of the object changes, the orientation of the object included in the time-series image data changes. For example, if the time-series image data is face image data including the face of the object, when the direction of movement of the object changes, the orientation of the face of the object included in the face image data changes. In such a case, the matching rate may change due to the change in face orientation, and as a result, the rate of false acceptance of other persons and false rejection of persons may change. Therefore, the threshold changing unit 160 may change the unregistered threshold based on the direction of movement of the object in order to minimize the change in the rate of false acceptance of other persons and false rejection of persons.

[0066] Alternatively, the threshold changing unit 160 may estimate the orientation of the target's face included in the time-series image data and change the unregistered threshold based on the estimated orientation of the target's face. For example, the threshold changing unit 160 may change the unregistered threshold depending on whether the target's face included in the time-series image data is facing right, left, or forward. As described above, if the orientation of the target's face changes, the matching rate may change, and as a result, the rate of false acceptances and false rejections of the target may change. Therefore, the threshold changing unit 160 may change the unregistered threshold based on the orientation of the target's face in order to minimize the change in the rate of false acceptances and false rejections of the target.

[0067] Furthermore, the threshold modification unit 160 may also modify the class threshold when the unregistered threshold is modified. For example, the threshold modification unit 160 may modify both the class threshold and the unregistered threshold so as not to disrupt the correlation between the class threshold and the unregistered threshold.

[0068] (Flow of operations) Next, the operation flow of the third information processing device 1 will be explained with reference to Figure 11. Figure 11 is a flowchart showing the operation flow of the third information processing device. Note that in Figure 11, the same reference numerals are used for the same processes as shown in Figure 3.

[0069] As shown in Figure 11, when the operation of the third information processing device 1 is started, the image acquisition unit 110 first acquires image data (step S101).

[0070] Next, the index calculation unit 120 extracts a first feature from the image data acquired by the image acquisition unit 110 and a second feature from the registered image data read from the registered image storage unit 125 (step S102). Then, the index calculation unit 120 calculates an integrated feature or score based on the extracted first and second features (step S103).

[0071] Next, the likelihood ratio calculation unit 130 calculates a likelihood ratio indicating the likelihood of the time-series image data belonging to a particular class, based on the integrated features or scores calculated by the index calculation unit 120 (step S104). Specifically, the likelihood ratio calculation unit 130 calculates a first likelihood ratio corresponding to the registered class and a second likelihood ratio corresponding to the unregistered class.

[0072] Next, the threshold changing unit 160 estimates the current situation (for example, the direction of movement of the object and the orientation of its face as described above) from the image data acquired by the image acquisition unit 110 (step S301). Then, the threshold changing unit 160 determines whether the estimated situation has changed from the previous situation (step S302). For example, if the situation in which the object has been moving from right to left as viewed from the camera continues, and then the object starts moving from left to right, the threshold changing unit 160 may determine that the situation has changed. Alternatively, if the situation in which the object's face has been photographed from the front continues, and then the object's face starts being photographed from the right side, the threshold changing unit 160 may determine that the situation has changed.

[0073] If it is determined that the situation has changed (step S302: YES), the threshold change unit 160 changes the unregistered threshold based on the changed situation (step S303). For example, the threshold change unit 160 may change the unregistered threshold so that the change in the rate of incorrect acceptance by others and incorrect rejection by the person themselves due to the change in the situation becomes smaller. The threshold change unit 160 may change the unregistered threshold by changing the adjustment parameter δ, for example. If it is determined that the situation has not changed (step S302: NO), the process in step S303 described above may be omitted. In other words, the unregistered threshold does not need to be changed.

[0074] Next, the class determination unit 140 determines whether the second likelihood ratio calculated by the likelihood ratio calculation unit 130 has reached the unregistered threshold (step S105). If the unregistered threshold has been changed in step S303, the class determination unit 140 uses the changed unregistered threshold for the determination. If the second likelihood ratio has reached the unregistered threshold (step S105: YES), the class determination unit 140 determines that the time-series image data does not belong to any registered class (i.e., it is unregistered) (step S106). In this case, the time-series image data is determined not to match the registered image data (in other words, the target is an unregistered user), and the matching process ends.

[0075] On the other hand, if the second likelihood ratio has not reached the unregistered threshold (step S105: NO), the class determination unit 140 determines whether the first likelihood ratio has reached the class threshold (step S107). If the first likelihood ratio has reached the class threshold (step S107: YES), the class determination unit 140 determines that the time-series image data belongs to a registered class (step S108). Specifically, the class determination unit 140 determines that the time-series image data belongs to a registered class corresponding to the first likelihood ratio that has exceeded the class threshold. In this case, the time-series image data is determined to match the registered image data (in other words, the target is a registered user), and the matching process ends.

[0076] On the other hand, if the first likelihood ratio does not reach the class threshold (step S107: NO), processing starts again from step S101. That is, the image data acquisition unit 110 acquires new image data (for example, image data of the next frame), and the series of processes described above are executed again.

[0077] (Technical effects) Next, we will explain the technical effects obtained by the third information processing device 1.

[0078] As explained in Figures 10 and 11, the unregistered threshold is changed in the third information processing device 1 according to the time-series image data. In this way, the unregistered threshold can be changed to an appropriate value depending on the circumstances under which the time-series image data has been acquired. Therefore, the judgment process using the unregistered threshold can be executed more appropriately.

[0079] The processing method of recording a program that operates the configuration of each embodiment in order to realize the functions of each embodiment described above on a recording medium, reading the program recorded on the recording medium as code, and executing it on a computer is also included in the scope of each embodiment. In other words, a computer-readable recording medium is also included in the scope of each embodiment. Furthermore, not only the recording medium on which the above-mentioned program is recorded, but also the program itself is included in each embodiment.

[0080] As recording media, for example, floppy disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, magnetic tapes, non-volatile memory cards, and ROMs can be used. Furthermore, the scope of each embodiment is not limited to programs that perform processing on the recording media alone, but also includes programs that operate on the OS and perform processing in cooperation with other software and the functions of expansion boards. In addition, the program itself may be stored on a server, and part or all of the program may be made available for download from the server to the user terminal. The program may be provided to the user in, for example, SaaS (Software as a Service) format.

[0081] <Note> The embodiments described above may also be described in the following appendix, but are not limited to these.

[0082] (Note 1) The information processing device described in Appendix 1 is an information processing device comprising: acquisition means for acquiring time-series image data; index calculation means for calculating an integrated feature obtained by integrating a first feature which is a feature of the time-series image data and a second feature which is a feature of pre-registered image data, or a score indicating the similarity between the first feature and the second feature; likelihood ratio calculation means for calculating a likelihood ratio indicating the likelihood of the class to which the time-series image data belongs based on the integrated feature or the score; determination means for determining that the time-series image data belongs to a registered class corresponding to the pre-registered image data when the likelihood ratio reaches a first threshold, and determining that the time-series image data belongs to an unregistered class indicating that it has not been pre-registered when the likelihood ratio reaches a second threshold which is a value different from the first threshold.

[0083] (Note 2) The information processing device described in Appendix 2 is the information processing device described in Appendix 1, wherein the first threshold and the second threshold are correlated values.

[0084] (Note 3) The information processing device described in Appendix 3 is the information processing device described in Appendix 2, wherein the second threshold is a value calculated using an adjustment parameter that trades off the first threshold with the possibility of incorrect acceptance by others and incorrect rejection by the intended person.

[0085] (Note 4) The information processing device described in Appendix 4 is the information processing device described in Appendix 2, wherein, when the first threshold is λ1, the second threshold is λ2, and the adjustment parameter is δ, the second threshold λ2 is a value calculated as λ2 = λ1 + δ.

[0086] (Note 5) The information processing device described in Appendix 5 is the information processing device described in any one of Appendix 1 to 4, further comprising a threshold changing means for dynamically changing the second threshold according to the acquired time-series image data.

[0087] (Note 6) The information processing device described in Appendix 6 is the information processing device described in Appendix 5, wherein the time-series image data includes a moving object, and the threshold changing means changes the second threshold based on the direction of movement of the object at the location where the time-series image data is acquired.

[0088] (Note 7) The information processing device described in Appendix 7 includes the subject's face in the time-series image data. The threshold changing means is an information processing device as described in Appendix 5, which changes the second threshold based on the orientation of the face included in the time-series image data.

[0089] (Note 8) The information processing method described in Appendix 8 is an information processing method in which at least one computer acquires time-series image data, calculates an integrated feature by integrating a first feature which is a feature of the time-series image data and a second feature which is a feature of pre-registered image data, or calculates a score indicating the similarity between the first feature and the second feature, calculates a likelihood ratio indicating the likelihood of the class to which the time-series image data belongs based on the integrated feature or the score, determines that the time-series image data belongs to a registered class corresponding to the pre-registered image data when the likelihood ratio reaches a first threshold, and determines that the time-series image data belongs to an unregistered class indicating that it has not been pre-registered when the likelihood ratio reaches a second threshold which is a different value from the first threshold.

[0090] (Note 9) The computer program described in Appendix 9 is a computer program that causes at least one computer to execute an information processing method which includes acquiring time-series image data, integrating a first feature, which is a feature of the time-series image data, with a second feature, which is a feature of pre-registered image data, or calculating a score indicating the similarity between the first feature and the second feature, calculating a likelihood ratio indicating the likelihood of the class to which the time-series image data belongs based on the integrated feature or the score, determining that the time-series image data belongs to a registered class corresponding to the pre-registered image data when the likelihood ratio reaches a first threshold, and determining that the time-series image data belongs to an unregistered class, indicating that it has not been pre-registered, when the likelihood ratio reaches a second threshold that is a different value from the first threshold.

[0091] (Note 10) The recording medium described in Appendix 10 is a recording medium on which a computer program is recorded that causes at least one computer to execute an information processing method, which involves acquiring time-series image data, integrating a first feature, which is a feature of the time-series image data, with a second feature, which is a feature of pre-registered image data, or calculating a score indicating the similarity between the first feature and the second feature, calculating a likelihood ratio indicating the likelihood of the class to which the time-series image data belongs based on the integrated feature or the score, determining that the time-series image data belongs to a registered class corresponding to the pre-registered image data when the likelihood ratio reaches a first threshold, and determining that the time-series image data belongs to an unregistered class, indicating that it has not been pre-registered, when the likelihood ratio reaches a second threshold that is a different value from the first threshold.

[0092] This disclosure may be modified as appropriate, insofar as it does not contradict the gist or idea of ​​the invention as can be inferred from the claims and the specification as a whole, and information processing devices, information processing methods, and computer programs that include such modifications are also included in the technical idea of ​​this disclosure. [Explanation of Symbols]

[0093] 1. Information Processing Device 11 processors 12 RAM 13 ROM 14 Storage device 15 Input device 16 Output device 17 Data bus 110 Image acquisition unit 120 Indicator calculation section 125 Registered Image Storage Unit 130 Likelihood Ratio Calculation Unit 140 Class determination unit 150 Threshold setting section 160 Threshold change section

Claims

1. A means of acquiring time-series image data, An index calculation means for calculating an integrated feature obtained by integrating a first feature, which is a feature of the time-series image data, and a second feature, which is a feature of pre-registered registered image data, or a score indicating the similarity between the first feature and the second feature. A likelihood ratio calculation means that calculates a likelihood ratio indicating the likelihood of the class to which the time-series image data belongs, based on the integrated features or the score, A determination means that determines that the time-series image data belongs to a registration class corresponding to the registered image data when the likelihood ratio reaches a first threshold, and determines that the time-series image data belongs to an unregistered class indicating that it has not been previously registered when the likelihood ratio reaches a second threshold that is a different value from the first threshold, An information processing device equipped with the following features.

2. The first threshold and the second threshold are correlated values. The information processing apparatus according to claim 1.

3. The second threshold is a value calculated using the first threshold and an adjustment parameter that trades off incorrect acceptance by others and incorrect rejection by the individual. The information processing apparatus according to claim 2.

4. When the first threshold is λ1, the second threshold is λ2, and the adjustment parameter is δ, The second threshold value λ2 is a value calculated as λ2 = λ1 + δ. The information processing apparatus according to claim 2.

5. The system further includes threshold changing means for dynamically changing the second threshold according to the acquired time-series image data. The information processing apparatus according to any one of claims 1 to 4.

6. The aforementioned time-series image data includes moving objects. The threshold changing means changes the second threshold based on the direction of movement of the target at the location where the time-series image data is acquired. The information processing apparatus according to claim 5.

7. The aforementioned time-series image data includes the subject's face. The threshold changing means changes the second threshold based on the orientation of the face included in the time-series image data. The information processing apparatus according to claim 5.

8. At least one computer, Acquire time-series image data, A combined feature is calculated by integrating the first feature, which is a feature of the time-series image data, and the second feature, which is a feature of pre-registered image data, or a score indicating the similarity between the first feature and the second feature is calculated. Based on the aforementioned integrated features or the aforementioned score, a likelihood ratio is calculated that indicates the likelihood of the class to which the time-series image data belongs. When the likelihood ratio reaches a first threshold, it is determined that the time-series image data belongs to the registration class corresponding to the registered image data. When the likelihood ratio reaches a second threshold that is different from the first threshold, it is determined that the time-series image data belongs to the unregistered class, indicating that it has not been previously registered. Information processing methods.

9. On at least one computer, Acquire time-series image data, A combined feature is calculated by integrating the first feature, which is a feature of the time-series image data, and the second feature, which is a feature of pre-registered image data, or a score indicating the similarity between the first feature and the second feature is calculated. Based on the aforementioned integrated features or the aforementioned score, a likelihood ratio is calculated that indicates the likelihood of the class to which the time-series image data belongs. When the likelihood ratio reaches a first threshold, it is determined that the time-series image data belongs to the registration class corresponding to the registered image data. When the likelihood ratio reaches a second threshold that is different from the first threshold, it is determined that the time-series image data belongs to the unregistered class, indicating that it has not been previously registered. A computer program that executes information processing methods.

Images