Information processing device, information processing method, and computer program

Abstract

Determine appropriately whether the data is not registered or not. [Solution] The information processing device includes: acquisition means for acquiring time-series image data; index calculation means for calculating an integrated feature or score by integrating a first feature, which is a feature of the time-series image data, and a second feature, which is a feature of the registered image data; first likelihood ratio calculation means for calculating N first likelihood ratios indicating the likelihood of belonging to each of the registered classes based on the integrated feature or score; second likelihood ratio calculation means for calculating a second likelihood ratio indicating the likelihood of belonging to an unregistered class based on the N first likelihood ratios; and determination means for determining that the time-series image data belongs to a registered class when the first likelihood ratio reaches a predetermined threshold, and determining that the time-series image data does not belong to any of the registered classes when the second likelihood ratio reaches a predetermined threshold.

Description

Technical Field

[0001] This disclosure relates to the technical fields of information processing apparatuses, information processing methods, and computer programs.

Background Art

[0002] As this type of apparatus, there is known one that performs a process of determining the class to which data belongs (so-called class classification). For example, Patent Document 1 discloses a technique for determining the class to which series data to be classified belongs when an individual score or an integrated score calculated based on a likelihood ratio reaches a predetermined threshold value.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] An object of this disclosure is to provide an information processing apparatus, an information processing method, and a computer program for improving the techniques disclosed in the prior art documents.

Means for Solving the Problems

[0005] One aspect of the information processing device of this disclosure includes: acquisition means for acquiring time-series image data; index calculation means for calculating an integrated feature obtained by integrating a first feature which is a feature of the time-series image data and a second feature which is a feature of pre-registered image data, or a score indicating the similarity between the first feature and the second feature; first likelihood ratio calculation means for calculating N first likelihood ratios indicating the likelihood that the time-series image data belongs to each of N (N is a natural number) registered classes corresponding to the registered image data, based on the integrated feature or the score; second likelihood ratio calculation means for calculating a second likelihood ratio indicating the likelihood that the time-series image data belongs to an unregistered class, which would be the case if the time-series image data were not pre-registered, based on the N first likelihood ratios; and determination means for determining that the time-series image data belongs to a registered class when the first likelihood ratio reaches a predetermined threshold, and determining that the time-series image data does not belong to any of the registered classes when the second likelihood ratio reaches a predetermined threshold.

[0006] One aspect of the information processing method of this disclosure involves at least one computer acquiring time-series image data, integrating a first feature, which is a feature of the time-series image data, with a second feature, which is a feature of pre-registered image data, to calculate an integrated feature, or a score indicating the similarity between the first feature and the second feature, calculating N first likelihood ratios based on the integrated feature or the score, indicating the likelihood that the time-series image data belongs to each of N (N is a natural number) registered classes corresponding to the registered image data, calculating a second likelihood ratio based on the N first likelihood ratios, indicating the likelihood that the time-series image data belongs to an unregistered class, which would be the case if the time-series image data was not pre-registered, determining that the time-series image data belongs to a registered class when the first likelihood ratio reaches a predetermined threshold, and determining that the time-series image data does not belong to any of the registered classes when the second likelihood ratio reaches a predetermined threshold.

[0007] One aspect of the computer program of this disclosure causes at least one computer to execute an information processing method which involves integrating a first feature, which is a feature of the time-series image data, and a second feature, which is a feature of pre-registered image data, or a score indicating the similarity between the first feature and the second feature; calculating N first likelihood ratios based on the integrated feature or the score, which indicate the likelihood that the time-series image data belongs to each of N (N is a natural number) registered classes corresponding to the registered image data; calculating a second likelihood ratio based on the N first likelihood ratios, which indicates the likelihood that the time-series image data belongs to an unregistered class, which would be the case if the time-series image data was not pre-registered; determining that the time-series image data belongs to a registered class when the first likelihood ratio reaches a predetermined threshold; and determining that the time-series image data does not belong to any of the registered classes when the second likelihood ratio reaches a predetermined threshold. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing the hardware configuration of the first information processing device. [Figure 2] This is a block diagram showing the functional configuration of the first information processing device. [Figure 3] This is a flowchart showing the operation flow of the first information processing device. [Figure 4] This graph (part 1) shows an example of a likelihood ratio calculated by the first information processing device. [Figure 5] This is a graph (part 2) showing an example of a likelihood ratio calculated in the first information processing device. [Figure 6] This is a block diagram showing the configuration of the likelihood ratio calculation unit in the second information processing device. [Figure 7] This is a block diagram showing a modified configuration of the likelihood ratio calculation unit in the second information processing device. [Figure 8] This is a block diagram showing the functional configuration of the third information processing device. [Figure 9]This is a flowchart showing the operation flow of the third information processing device. [Modes for carrying out the invention]

[0009] The following describes embodiments of the information processing device, information processing method, and computer program with reference to the drawings.

[0010] <First Embodiment> The first information processing device will be described with reference to Figures 1 to 5.

[0011] (Hardware configuration) First, the hardware configuration of the first information processing device will be described with reference to Figure 1. Figure 1 is a block diagram showing the hardware configuration of the first information processing device.

[0012] As shown in Figure 1, the first information processing device 1 includes a processor 11, RAM (Random Access Memory) 12, ROM (Read Only Memory) 13, a storage device 14, an input device 15, and an output device 16. The processor 11, RAM 12, ROM 13, storage device 14, input device 15, and output device 16 are all connected via a data bus 17. Note that the data bus 17 may be an interface other than a data bus (for example, LAN or USB).

[0013] The processor 11 reads a computer program. For example, the processor 11 is configured to read a computer program stored in at least one of the RAM 12, ROM 13, and storage device 14. Alternatively, the processor 11 may read a computer program stored in a computer-readable storage medium using a storage medium reading device (not shown). The processor 11 may also obtain (i.e., read) a computer program from a device (not shown) located outside the first information processing device 1 via a network interface. The processor 11 performs various processes by executing the read computer program. When the processor 11 executes the read computer program, a functional block related to the processing performed by the first information processing device 1 is realized within the processor 11. That is, the processor 11 may function as a controller that performs various processes and controls in the first information processing device 1.

[0014] The processor 11 may be configured as, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an FPGA (field-programmable gate array), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), or a quantum processor. The processor 11 may consist of one of these, or it may be configured to use multiple of them in parallel.

[0015] RAM12 temporarily stores computer programs executed by processor 11. RAM12 also temporarily stores data that processor 11 uses temporarily while executing computer programs. RAM12 may be, for example, D-RAM (Dynamic Random Access Memory) or SRAM (Static Random Access Memory). Alternatively, other types of volatile memory may be used instead of RAM12.

[0016] The ROM 13 stores the computer programs executed by the processor 11. The ROM 13 may store other fixed data. The ROM 13 may be, for example, a P-ROM (Programmable Read Only Memory) or an EPROM (Erasable Read Only Memory). Also, instead of the ROM 13, other types of non-volatile memories may be used.

[0017] The storage device 14 stores the data that the first information processing device 1 stores in the long term. The storage device 14 may operate as a temporary storage device of the processor 11. The storage device 14 may store the computer programs executed by the processor 11. The storage device 14 may include, for example, at least one of a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and a disk array device.

[0018] The input device 15 is a device that receives an input instruction from the user of the first information processing device 1. The input device 15 may include, for example, at least one of a keyboard, a mouse, a touch panel, and a touch pen. The input device 15 may be a device capable of voice input including, for example, a microphone.

[0019] The output device 16 is a device that outputs information regarding the first information processing device 1 to the outside. For example, the output device 16 may be a display device (e.g., a display, a monitor, etc.) capable of displaying information regarding the first information processing device 1. Also, the output device 16 may be a speaker or the like capable of voice outputting information regarding the information processing device 1.

[0020] The first information processing device may be configured to include some of the components described in Figure 1. For example, the first information processing device 1 may be configured to include only the processor 11, RAM 12, and ROM 13 from the components described above. In this case, the storage device 14, input device 15, and output device 16 may each be provided as external devices to the first information processing device 1. Furthermore, some of the arithmetic functions of the first information processing device 1 may be implemented by an external server or cloud service.

[0021] (Functional configuration) Next, the functional configuration of the first information processing device 1 will be described with reference to Figure 2. Figure 2 is a block diagram showing the functional configuration of the first information processing device.

[0022] In Figure 2, the first information processing device 1 is configured as a device for classifying incoming time-series data. More specifically, the first information processing device 1 is configured to perform matching processing (in other words, authentication processing) of objects contained in image data by classifying incoming time-series image data into classes corresponding to registered image data. The first information processing device 1 is configured to include an image acquisition unit 110, an index calculation unit 120, a registered image storage unit 125, a likelihood ratio calculation unit 130, and a class determination unit 140 as components for realizing its function. Note that each of the image acquisition unit 110, the index calculation unit 120, the likelihood ratio calculation unit 130, and the class determination unit 140 may be processing blocks realized by the processor 11 (see Figure 1) described above.

[0023] The image acquisition unit 110 is configured to acquire time-series image data. The image acquisition unit 110 may acquire image data of multiple frames sequentially. For example, the image acquisition unit 110 may be configured to acquire images of each frame captured from a camera that shoots video. The time-series image data acquired by the image acquisition unit 110 is acquired in order to perform a matching process with pre-registered registered image data. For example, the time-series image data may be face image data including the face of the target. In this case, the acquired face image data may be used to perform face recognition using the face of the target. Note that face image data is merely an example, and the time-series image data may include image data of parts other than the face of the target (for example, fingerprints or irises). Also, the time-series image data may include subjects other than people. The time-series image data acquired by the image acquisition unit 110 is configured to be output to the index calculation unit 120.

[0024] The index calculation unit 120 is configured to calculate an integrated feature or score from time-series image data acquired by the image acquisition unit 110. The integrated feature is a feature obtained by integrating a first feature, which is a feature of the time-series image data, and a second feature, which is a feature of pre-registered image data. The score is a score that indicates the similarity (in other words, the degree of agreement) between the first feature and the second feature described above. The index calculation unit 120 may also have a function to extract features from image data in order to calculate the integrated feature or score. For example, the index calculation unit 120 may have a function to extract features of the target face from face image data that includes the target face. The index calculation unit 120 acquires time-series image data from the image acquisition unit 110 while simultaneously acquiring registered image data from the registered image storage unit 125 to calculate the integrated feature or score. The integrated feature or score calculated by the index calculation unit 120 is output to the likelihood ratio calculation unit 130.

[0025] The registered image storage unit 125 is configured to store registered image data used by the index calculation unit 120 when calculating integrated features or scores. The registered image storage unit 125 may be configured to store features extracted from the registered image data (i.e., second features) rather than the registered image data itself. The registered image storage unit 125 may be implemented using the storage device 14 (see Figure 1) described above. Alternatively, the registered image storage unit 125 may be implemented by a database or the like provided outside the first information processing device 1. The registered image storage unit 125 is configured to store multiple registered image data. The index calculation unit 120 may calculate N integrated features or N scores using first features extracted from time-series image data and N second features extracted from each of the N registered image data. The registered image data stored in the registered image storage unit 125 may be, for example, face image data of a registered user used for face recognition. In this case, the integrated feature quantities and scores calculated by the index unit 120 serve as indicators showing the degree of match between the target face included in the time-series image data and the face of the registered user.

[0026] The likelihood ratio calculation unit 130 comprises a first calculation unit 1301 and a second calculation unit 1302. The likelihood ratio calculation unit 130 is configured to output the first likelihood ratio calculated by the first calculation unit 1301 and the second likelihood ratio calculated by the second calculation unit 1302 to the class determination unit 140. The first calculation unit 1301 and the second calculation unit 1302 will be described in detail below.

[0027] The first calculation unit 1301 is configured to calculate the first likelihood ratio. The first likelihood ratio is a value that indicates the likelihood of a time-series image data belonging to a particular class. Specifically, the first likelihood ratio is a value that indicates the likelihood of a time-series image data belonging to one of several registered classes set for each registered image data. The first calculation unit 1301 calculates multiple likelihood ratios corresponding to each of the multiple registered classes (i.e., the same number of likelihood ratios as the number of registered classes). For example, if N registered image data is registered in the registered image storage unit 125, N registered classes are set to correspond to them. In this case, the first calculation unit 1301 calculates N likelihood ratios corresponding to each of the N registered classes. The specific method used by the first calculation unit 1301 to calculate the first likelihood ratio is not particularly limited. The first calculation unit 1301 may use various existing methods to calculate the first likelihood ratio. For example, the first calculation unit 1301 may calculate the first likelihood ratio using a pre-trained estimation model (specifically, a neural network trained by deep learning, etc.). The first calculation unit 1301 may also calculate the first likelihood ratio based on two or more consecutive image data from a time-series of image data. For example, the first calculation unit 1301 may calculate the first likelihood ratio using integrated features or scores calculated from previously acquired image data in addition to integrated features or scores calculated from the most recently acquired image data.

[0028] The second calculation unit 1302 is configured to calculate the second likelihood ratio. The second likelihood ratio is a value that indicates the likelihood that the time-series image data belongs to an unregistered class. Here, the unregistered class is the class to which the time-series image data belongs if it does not belong to any registered class (i.e., it is unregistered). The second calculation unit 1302 calculates the second likelihood ratio based on the N first likelihood ratios calculated by the first calculation unit 1301. Even if there are N registered classes, the second calculation unit 1302 only needs to calculate one second likelihood ratio. In this case, the likelihood ratio calculation unit 130 as a whole will calculate N+1 likelihood ratios. The specific method of calculating the second likelihood ratio by the second calculation unit 1302 will be explained in detail in other embodiments described later.

[0029] The class determination unit 140 is configured to determine the class to which time-series image data belongs based on the likelihood ratio calculated by the likelihood ratio calculation unit 130. In other words, the class determination unit 140 is configured to perform class classification processing based on the likelihood ratio. Specifically, the class determination unit 140 determines that time-series image data belongs to a registered class corresponding to registered image data when the first likelihood ratio calculated by the first calculation unit 1301 reaches a preset threshold. In other words, the class determination unit 140 determines that time-series image data matches one of the registered image data when the first likelihood ratio reaches a threshold. On the other hand, the class determination unit 140 determines that time-series image data belongs to a registered class (in other words, does not belong to any of the registered classes) when the second likelihood ratio calculated by the second calculation unit 1302 reaches a threshold. In other words, the class determination unit 140 determines that time-series image data does not match any of the registered image data (in other words, is unregistered) when the second likelihood ratio reaches a threshold. The threshold used by the class determination unit is a threshold used to determine which class a time-series image data belongs to, and may be a pre-set value.

[0030] (Flow of operations) Next, the operation flow of the first information processing device 1 will be explained with reference to Figure 3. Figure 3 is a flowchart showing the operation flow of the first information processing device.

[0031] As shown in Figure 3, when the operation of the first information processing device 1 begins, the image acquisition unit 110 first acquires image data (step S101). The image data acquired here may be the image data of the most recently captured single frame from the time-series image data.

[0032] Next, the index calculation unit 120 extracts a first feature from the image data acquired by the image acquisition unit 110 and a second feature from the registered image data read from the registered image storage unit 125 (step S102). Then, the index calculation unit 120 calculates an integrated feature or score based on the extracted first and second features (step S103).

[0033] Next, the first calculation unit 1301 calculates N first likelihood ratios that indicate the likelihood of the registration class to which the time-series image data belongs, based on the integrated features or scores calculated by the index calculation unit 120 (step S104). Then, the second calculation unit 1302 calculates a second likelihood ratio based on the N first likelihood ratios calculated by the first calculation unit 1301 (step S105).

[0034] Next, the class determination unit 140 determines whether the first likelihood ratio calculated by the first calculation unit 1301 or the second likelihood ratio calculated by the second calculation unit 1302 has reached a threshold (step S106). If the first likelihood ratio or the second likelihood ratio has reached a threshold (step S106: YES), the class determination unit 140 determines which class the time-series image data belongs to (step S107).

[0035] Specifically, if the first likelihood ratio reaches a threshold, the class determination unit 140 determines that the time-series image data belongs to a registered class (i.e., the registered class corresponding to the first likelihood ratio that exceeds the threshold). In this case, the time-series image data is determined to match registered image data (in other words, the target is a registered user), and the matching process ends. On the other hand, if the second likelihood ratio reaches a threshold, the class determination unit 140 determines that the time-series image data belongs to an unregistered class. In this case, the time-series image data is determined not to match registered image data (in other words, the target is an unregistered user), and the matching process ends.

[0036] On the other hand, if neither the first likelihood ratio nor the second likelihood ratio reaches the threshold (step S106: NO), the process restarts from step S101. That is, the image data acquisition unit 110 acquires new image data (for example, image data of the next frame), and the series of processes described above are executed again. As the process is repeated in this way, the first likelihood ratio and the second likelihood ratio calculated by the likelihood ratio calculation unit 130 gradually change. The determination process (in other words, the matching process) continues until either the first likelihood ratio or the second likelihood ratio reaches the threshold.

[0037] The first information processing device 1 may be configured to execute various processes related to the target based on the determination result of the class determination unit 140. For example, the first information processing device 1 may be configured to execute a process to permit or prohibit passage of the target to a predetermined area based on the determination result of the class determination unit 140. More specifically, if it is determined that the time-series image data matches the registered image data, the first information processing device 1 may control the gate installed in the predetermined area to open, thereby permitting passage of the target. If it is determined that the time-series image data does not match the registered image data (i.e., unregistered), the first information processing device 1 may control the gate installed in the predetermined area to close, thereby prohibiting passage of the target.

[0038] (Example of operation) Next, with reference to Figures 4 and 5, we will explain specific examples of the operation of the first information processing device 1 (in particular, an example of the operation of determining the class to which time-series image data belongs based on the likelihood ratio). Figure 4 is a graph (1) showing an example of the likelihood ratio calculated by the first information processing device. Figure 5 is a graph (2) showing an example of the likelihood ratio calculated by the first information processing device.

[0039] In the examples shown in Figures 4 and 5, registered image data A, registered image data B, and registered image data C are registered. Classes A, B, and C are set as the corresponding registration classes for each of the registered image data A, B, and C. In addition to the registered classes, an unregistered class is also set.

[0040] The first calculation unit 1301 calculates first likelihood ratios based on the image data acquired by the image acquisition unit 110 and the registered image data A, B, and C. Specifically, the first calculation unit 1301 calculates the first likelihood ratio corresponding to class A, the first likelihood ratio corresponding to class B, and the first likelihood ratio corresponding to class C. The second calculation unit 1302 then calculates the second likelihood ratio corresponding to the unregistered class based on the three first likelihood ratios calculated by the first calculation unit 1301. These likelihood ratios gradually change over time (i.e., as image data is acquired sequentially).

[0041] In the example shown in Figure 4, the first likelihood ratio corresponding to class A has reached the threshold. On the other hand, the first likelihood ratios corresponding to classes B and C have not reached the threshold. Furthermore, the second likelihood ratio corresponding to the unregistered class has also not reached the threshold. In such a case, the time-series image data acquired by the image acquisition unit 110 is determined to belong to class A. That is, the time-series image data is determined to match registered image data A, but not to registered image data B and C. As a result, a matching result is obtained in which the subject included in the time-series image data is the user corresponding to registered image data A.

[0042] In the example shown in Figure 5, none of the first likelihood ratios corresponding to classes A, B, and C have reached the threshold. On the other hand, the second likelihood ratio corresponding to the unregistered class has reached the threshold. In such a case, the time-series image data acquired by the image acquisition unit 110 is determined to belong to the unregistered class. In other words, the time-series image data is determined to be unregistered image data that does not belong to any of classes A, B, and C. Therefore, the time-series image data is determined not to match any of the registered image data A, B, and C. As a result, the matching result is obtained that the subject included in the time-series image data is an unregistered user.

[0043] (Technical effects) Next, we will explain the technical effects obtained by the first information processing device 1.

[0044] As explained in Figures 1 to 5, the first information processing device 1 determines whether time-series image data is registered based on the first likelihood ratio, while determining whether time-series image data is unregistered based on the second likelihood ratio. This makes it possible to shorten the time required for matching with registered image data when performing a matching process with class classification. For example, if class classification is performed using only the first likelihood ratio, a matching result cannot be obtained if the first likelihood ratio continues to transition without reaching the threshold. However, in this embodiment, since the second likelihood ratio corresponding to the unregistered class is also used, even if the first likelihood ratio does not reach the threshold, a matching result indicating that the image data is unregistered can be obtained when the second likelihood ratio reaches the threshold.

[0045] <Second Embodiment> The second information processing device 1 will be described with reference to Figures 6 and 7. Note that the second information processing device 1 differs from the first information processing device 1 described above in some configurations and operations, while other parts may be the same as those of the first information processing device 1. Therefore, the parts that differ from the first embodiment will be described in detail below, while other overlapping parts will be omitted as appropriate.

[0046] (Likelihood ratio calculation unit) First, the configuration and operation of the likelihood ratio calculation unit 130 in the second information processing device 1 will be explained with reference to Figure 6. Figure 6 is a block diagram showing the configuration of the likelihood ratio calculation unit in the second information processing device. In Figure 6, the same reference numerals are used for elements as those described in Figure 2.

[0047] In Figure 6, in the likelihood ratio calculation unit 130 of the second information processing device 1, the first calculation unit 1301 calculates N first likelihood ratios λ1 to λN. In particular, the second calculation unit 1302 calculates one second likelihood ratio λ{N+1} from the N first likelihood ratios λ1 to λN using a nonlinear function F. The nonlinear function F may use a parameter p that imposes a condition in addition to the first likelihood ratios λ1 to λN. Specific examples of the nonlinear function F include, for example, equations (1) and (2) below.

[0048]

number

[0049]

number

[0050] Note that z in equations (1) and (2) above i This is a feature of the registered image data (i.e., the second feature), and z q is the feature vector (i.e., the first feature vector) of the acquired time-series image data. α and β are parameters whose range is real numbers.

[0051] (modified version) Next, with reference to Figure 7, the configuration and operation of a modified example of the likelihood ratio calculation unit 130 in the second information processing device 1 will be described. Figure 7 is a block diagram showing the configuration of a modified example of the likelihood ratio calculation unit in the second information processing device. Note that in Figure 7, the same reference numerals are used for elements similar to those described in Figure 6.

[0052] In Figure 7, in a modified example of the likelihood ratio calculation unit 130 in the second information processing device 1, the first calculation unit 1301 calculates N first likelihood ratios λ1 to λN. In particular, the second calculation unit 1302 uses a neural network to calculate one second likelihood ratio λ{N+1} from the N first likelihood ratios λ1 to λN. The neural network may be an estimation model learned by deep learning. This estimation model may be a model that takes the first likelihood ratios λ1 to λN and a conditioning parameter p as input and outputs a second likelihood ratio λ{N+1}.

[0053] (Technical effects) Next, we will explain the technical effects obtained by the second information processing device 1.

[0054] As explained in Figures 6 and 7, the second information processing device 1 calculates the second likelihood ratio by a nonlinear process using N first likelihood ratios. In this way, it is possible to appropriately calculate the second likelihood ratio corresponding to the unregistered class. If the second likelihood ratio were to be calculated using a linear process (for example, a process to calculate the average value), the process would be relatively simple, and there is a risk that each of the N first likelihood ratios would not be sufficiently considered. However, according to this embodiment, since the second likelihood ratio is calculated by a nonlinear process, it is possible to calculate the second likelihood ratio while fully considering each of the N first likelihood ratios.

[0055] As explained in Figure 6, the second likelihood ratio can be easily and appropriately calculated using a nonlinear function. Furthermore, as explained in Figure 7, the second likelihood ratio can be appropriately calculated using a neural network based on a pre-trained estimation model.

[0056] <Third Embodiment> The third information processing device 1 will be described with reference to Figures 8 and 9. Note that the third information processing device 1 differs in some configurations and operations from the first and second information processing devices 1 described above, while other parts may be the same as those of the first and second information processing devices 1. Therefore, the following will provide a detailed explanation of the parts that differ from the embodiments already described, while explanations of other overlapping parts will be omitted as appropriate.

[0057] (Functional configuration) First, the functional configuration of the third information processing device 1 will be explained with reference to Figure 8. Figure 8 is a block diagram showing the functional configuration of the third information processing device. Note that in Figure 8, the same reference numerals are used for elements similar to those described in Figure 2.

[0058] In Figure 8, the third information processing device 1 is configured to include an image acquisition unit 110, an index calculation unit 120, a registered image storage unit 125, a likelihood ratio calculation unit 130, a likelihood ratio storage unit 135, and a class determination unit 140 as components for realizing its functions. In other words, the third information processing device 1 further includes a likelihood ratio storage unit 135 in addition to the configuration described in the first embodiment (see Figure 2).

[0059] The likelihood ratio storage unit 135 is configured to store the first likelihood ratio calculated by the first calculation unit 1301. The likelihood ratio storage unit 135 may be implemented using the storage device 14 (see Figure 1) described above. Alternatively, the likelihood ratio storage unit 135 may be implemented by a database or the like provided outside the third information processing device 1. The likelihood ratio storage unit 135 may be configured to store the calculated first likelihood ratio each time the first calculation unit 1301 calculates the first likelihood ratio. That is, the likelihood ratio storage unit 135 may be configured to sequentially accumulate the first likelihood ratios calculated by the first calculation unit 1301. The first likelihood ratio stored in the likelihood ratio storage unit 135 is readable by the second calculation unit 1302.

[0060] The second calculation unit 1302 in the third information processing device 1 is configured to calculate the second likelihood ratio using the first likelihood ratio most recently calculated by the first calculation unit 1301, as well as the first likelihood ratio previously calculated by the first calculation unit 1301. Specifically, the second calculation unit 1302 calculates the second likelihood ratio using the N first likelihood ratios calculated by the first calculation unit 1301, as well as the N first likelihood ratios read from the likelihood ratio storage unit 135.

[0061] (Flow of operations) Next, the operation flow of the third information processing device 1 will be explained with reference to Figure 9. Figure 9 is a flowchart showing the operation flow of the third information processing device. Note that in Figure 9, the same reference numerals are used for the same processes as shown in Figure 3.

[0062] As shown in Figure 9, when the operation of the third information processing device 1 begins, the image acquisition unit 110 first acquires image data (step S101).

[0063] Next, the index calculation unit 120 extracts a first feature from the image data acquired by the image acquisition unit 110 and a second feature from the registered image data read from the registered image storage unit 125 (step S102). Then, the index calculation unit 120 calculates an integrated feature or score based on the extracted first and second features (step S103).

[0064] Next, the first calculation unit 1301 calculates N first likelihood ratios that indicate the likelihood of the registration class to which the time-series image data belongs, based on the integrated features or scores calculated by the index calculation unit 120 (step S104).

[0065] Next, the second calculation unit 1302 reads out N previously calculated first likelihood ratios from the likelihood ratio storage unit 135 (step S301). Then, the second calculation unit 1302 calculates the second likelihood ratio based on the N first likelihood ratios calculated by the first calculation unit 1301 and the N previously read second likelihood ratios from the likelihood ratio storage unit 135 (step S302).

[0066] Next, the class determination unit 140 determines whether the first likelihood ratio calculated by the first calculation unit 1301 or the second likelihood ratio calculated by the second calculation unit 1302 has reached a threshold (step S106). If the first likelihood ratio or the second likelihood ratio has reached a threshold (step S106: YES), the class determination unit 140 determines which class the time-series image data belongs to (step S107).

[0067] (Technical effects) Next, we will explain the technical effects obtained by the third information processing device 1.

[0068] As explained in Figures 8 and 9, the third information processing device 1 calculates the second likelihood ratio using the most recently calculated first likelihood ratio as well as the first likelihood ratio calculated in the past. In this way, the second likelihood ratio is calculated taking into account past first likelihood ratios, making it possible to calculate the second likelihood ratio more appropriately compared to using only the most recently calculated first likelihood ratio. For example, information such as how high or low past first likelihood ratios were, or whether the slope of past first likelihood ratios was steep or gentle, can be taken into consideration. Therefore, based on the second likelihood ratio, it becomes possible to more appropriately determine whether or not time-series image data belongs to the unregistered class.

[0069] In the example described above, the case where N first likelihood ratios corresponding to the image data of one past frame are used was explained, but it is also possible to use first likelihood ratios corresponding to the image data of multiple past frames. For example, the second calculation unit 1302 may calculate one second likelihood ratio using, in addition to the N first likelihood ratios corresponding to the image data of the most recently calculated first frame, N first likelihood ratios corresponding to the image data of the past second frame, N first likelihood ratios corresponding to the image data of the past third frame, and N first likelihood ratios corresponding to the image data of the past third frame.

[0070] The processing method of recording a program that operates the configuration of each embodiment in order to realize the functions of each embodiment described above on a recording medium, reading the program recorded on the recording medium as code, and executing it on a computer is also included in the scope of each embodiment. In other words, a computer-readable recording medium is also included in the scope of each embodiment. Furthermore, not only the recording medium on which the above-mentioned program is recorded, but also the program itself is included in each embodiment.

[0071] As recording media, for example, floppy disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, magnetic tapes, non-volatile memory cards, and ROMs can be used. Furthermore, the scope of each embodiment is not limited to programs that perform processing on the recording media alone, but also includes programs that operate on the OS and perform processing in cooperation with other software and the functions of expansion boards. In addition, the program itself may be stored on a server, and part or all of the program may be made available for download from the server to the user terminal. The program may be provided to the user in, for example, SaaS (Software as a Service) format.

[0072] <Note> The embodiments described above may also be described in the following appendix, but are not limited to these.

[0073] (Note 1) The information processing device described in Appendix 1 is an information processing device comprising: acquisition means for acquiring time-series image data; index calculation means for calculating an integrated feature obtained by integrating a first feature which is a feature of the time-series image data and a second feature which is a feature of pre-registered image data, or a score indicating the similarity between the first feature and the second feature; first likelihood ratio calculation means for calculating N first likelihood ratios indicating the likelihood that the time-series image data belongs to each of N (N is a natural number) registered classes corresponding to the registered image data, based on the integrated feature or the score; second likelihood ratio calculation means for calculating a second likelihood ratio indicating the likelihood that the time-series image data belongs to an unregistered class, which would be the case if the time-series image data was not pre-registered, based on the N first likelihood ratios; and determination means for determining that the time-series image data belongs to a registered class when the first likelihood ratio reaches a predetermined threshold, and determining that the time-series image data does not belong to any of the registered classes when the second likelihood ratio reaches a predetermined threshold.

[0074] (Note 2) The information processing device described in Appendix 2 is the information processing device described in Appendix 1, wherein the second likelihood ratio calculation means calculates the second likelihood ratio by a nonlinear process using N first likelihood ratios.

[0075] (Note 3) The information processing device described in Appendix 3 is the same as the information processing device described in Appendix 2, wherein the nonlinear processing is processing using a nonlinear function.

[0076] (Note 4) The information processing device described in Appendix 4 is the same as the information processing device described in Appendix 2, wherein the nonlinear processing is processing using a neural network.

[0077] (Note 5) The information processing device described in Appendix 5 is the information processing device described in any one of Appendix 1 to 4, wherein the second likelihood ratio calculation means calculates the second likelihood ratio based on N first likelihood ratios calculated from the first frame of the time-series image data and N first likelihood ratios calculated from the second frame acquired before the first frame.

[0078] (Note 6) The information processing method described in Appendix 6 is an information processing method in which at least one computer acquires time-series image data, calculates an integrated feature by integrating a first feature which is a feature of the time-series image data and a second feature which is a feature of pre-registered image data, or calculates a score indicating the similarity between the first feature and the second feature, calculates N first likelihood ratios indicating the likelihood that the time-series image data belongs to each of N (N is a natural number) registered classes corresponding to the registered image data based on the integrated feature or the score, calculates a second likelihood ratio indicating the likelihood that the time-series image data belongs to an unregistered class which would be the case if the time-series image data was not pre-registered based on the N first likelihood ratios, determines that the time-series image data belongs to the registered class when the first likelihood ratio reaches a predetermined threshold, and determines that the time-series image data does not belong to any of the registered classes when the second likelihood ratio reaches the predetermined threshold.

[0079] (Note 7) The computer program described in Appendix 7 is a computer program that causes at least one computer to execute an information processing method which involves integrating a first feature, which is a feature of the time-series image data, and a second feature, which is a feature of pre-registered image data, to calculate an integrated feature, or a score indicating the similarity between the first feature and the second feature; calculating N first likelihood ratios, which indicate the likelihood that the time-series image data belongs to each of N (N is a natural number) registered classes corresponding to the registered image data, based on the integrated feature or the score; calculating a second likelihood ratio, which indicates the likelihood that the time-series image data belongs to an unregistered class, based on the N first likelihood ratios, to determine that the time-series image data belongs to an unregistered class, which would be the case if the time-series image data was not pre-registered; determining that the time-series image data belongs to a registered class when the first likelihood ratio reaches a predetermined threshold; and determining that the time-series image data does not belong to any of the registered classes when the second likelihood ratio reaches a predetermined threshold.

[0080] (Note 8) The recording medium described in Appendix 8 is a recording medium on which a computer program is recorded that causes at least one computer to execute an information processing method, which involves integrating a first feature, which is a feature of the time-series image data, and a second feature, which is a feature of pre-registered image data, or a score indicating the similarity between the first feature and the second feature; calculating N first likelihood ratios, which indicate the likelihood that the time-series image data belongs to each of N (N is a natural number) registered classes corresponding to the registered image data, based on the integrated feature or the score; calculating a second likelihood ratio, which indicates the likelihood that the time-series image data belongs to an unregistered class, based on the N first likelihood ratios; determining that the time-series image data belongs to a registered class when the first likelihood ratio reaches a predetermined threshold; and determining that the time-series image data does not belong to any of the registered classes when the second likelihood ratio reaches a predetermined threshold.

[0081] This disclosure may be modified as appropriate, insofar as it does not contradict the gist or idea of ​​the invention as can be inferred from the claims and the specification as a whole, and information processing devices, information processing methods, and computer programs that include such modifications are also included in the technical idea of ​​this disclosure. [Explanation of Symbols]

[0082] 1. Information Processing Device 11 processors 12 RAM 13 ROM 14 Storage device 15 Input device 16 Output device 17 Data bus 110 Image acquisition unit 120 Indicator calculation section 125 Registered Image Storage Unit 130 Likelihood Ratio Calculation Unit 1301 First Calculation Unit 1302 Second Calculation Unit 135 Likelihood Ratio Memory Unit 140 Class determination unit

Claims

1. A means of acquiring time-series image data, An index calculation means for calculating an integrated feature obtained by integrating a first feature, which is a feature of the time-series image data, and a second feature, which is a feature of pre-registered registered image data, or a score indicating the similarity between the first feature and the second feature. A first likelihood ratio calculation means that calculates N first likelihood ratios indicating the likelihood that the time-series image data belongs to each of the N (N is a natural number) registration classes corresponding to the registered image data, based on the integrated features or the score, A second likelihood ratio calculation means calculates a second likelihood ratio indicating the likelihood that the time-series image data belongs to the unregistered class, which is the case when the time-series image data is not pre-registered, based on the N first likelihood ratios, A determination means that determines that the time-series image data belongs to the registration class when the first likelihood ratio reaches a predetermined threshold, and determines that the time-series image data does not belong to any of the registration classes when the second likelihood ratio reaches a predetermined threshold, An information processing device equipped with the following features.

2. The second likelihood ratio calculation means calculates the second likelihood ratio by a nonlinear process using N of the first likelihood ratios. The information processing apparatus according to claim 1.

3. The aforementioned nonlinear processing is a process that uses a nonlinear function. The information processing apparatus according to claim 2.

4. The aforementioned nonlinear processing is a process using a neural network. The information processing apparatus according to claim 2.

5. The second likelihood ratio calculation means calculates the second likelihood ratio based on N first likelihood ratios calculated from the first frame of the time-series image data and N first likelihood ratios calculated from the second frame acquired before the first frame. The information processing apparatus according to any one of claims 1 to 4.

6. At least one computer, Acquire time-series image data, A combined feature is calculated by integrating the first feature, which is a feature of the time-series image data, and the second feature, which is a feature of pre-registered image data, or a score indicating the similarity between the first feature and the second feature is calculated. Based on the integrated features or the score, N first likelihood ratios are calculated that indicate the likelihood that the time-series image data belongs to each of the N (N is a natural number) registration classes corresponding to the registered image data. Based on the N first likelihood ratios, a second likelihood ratio is calculated that indicates the likelihood of the time-series image data belonging to the unregistered class, which corresponds to the case where the image data is not pre-registered. When the first likelihood ratio reaches a predetermined threshold, it is determined that the time-series image data belongs to the registration class, and when the second likelihood ratio reaches a predetermined threshold, it is determined that the time-series image data does not belong to any of the registration classes. Information processing methods.

7. On at least one computer, A combined feature is calculated by integrating the first feature, which is a feature of the time-series image data, and the second feature, which is a feature of pre-registered image data, or a score indicating the similarity between the first feature and the second feature is calculated. Based on the integrated features or the score, N first likelihood ratios are calculated that indicate the likelihood that the time-series image data belongs to each of the N (N is a natural number) registration classes corresponding to the registered image data. Based on the N first likelihood ratios, a second likelihood ratio is calculated that indicates the likelihood of the time-series image data belonging to the unregistered class, which corresponds to the case where the image data is not pre-registered. When the first likelihood ratio reaches a predetermined threshold, it is determined that the time-series image data belongs to the registration class, and when the second likelihood ratio reaches a predetermined threshold, it is determined that the time-series image data does not belong to any of the registration classes. A computer program that executes information processing methods.

Images