Information processing device, information processing method, and recording medium

Abstract

The present invention comprises an acquisition unit 111 that acquires a determination-subject image SD in which a plurality of faces are captured, a determination unit 114 that determines if two faces among the plurality of faces captured in the determination-subject image SD are similar to at least a prescribed degree, and a storage control unit 115 that, if the two faces are similar to at least the prescribed degree, associates and stores: two face-images respectively containing the two faces; and a different-person label which indicates that two people corresponding to said two face images are not the same person.

Description

[Technical Field]

[0001] The present disclosure relates to the technical fields of an information processing device, an information processing method, and a recording medium. [Background technology]

[0002] Patent Document 1 discloses a technology that maintains a certain matching performance and security level even when similar face patterns exist in a dictionary for face matching by grouping similar face patterns into a similarity group when a plurality of similar face patterns are registered in a dictionary and determining whether or not face patterns belonging to the similarity group can be matched by a special process different from the normal matching process. Also, Patent Document 2 discloses a technology that performs stable authentication even when extremely similar people, such as twins, are registered in personal authentication based on the face image of the person to be authenticated by using not only the similarity to the person's own dictionary but also the similarity to other people's dictionaries as a criterion for authentication. Patent Document 3 also discloses a technology for authenticating a person with high accuracy even when persons with similar faces are registered, by storing the correspondence between registered face information of a registered person and complementary persons, comparing input face information representing facial features included in a face area extracted from an input image with each of the registered face information of a plurality of registered people, identifying a registered person among the plurality of registered people whose registered face information is similar to the input face information as a candidate person for authentication, and determining that a face appearing in a plurality of input images is the face of the registered person based on the number of times that either the registered person or the complementary person associated with the registered person is identified as a candidate person for authentication in a plurality of input images taken at different times. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2008-071366 [Patent Document 2] Japanese Patent Application Laid-Open No. 2004-078686 [Patent Document 3] Japanese Patent Application Laid-Open No. 2014-071684 Summary of the Invention [Problem to be solved by the invention]

[0004] An object of this disclosure is to provide an information processing device, an information processing method, and a recording medium that aim to improve upon the techniques described in prior art documents. [Means for solving the problem]

[0005] A first aspect of the information processing device includes an acquisition means for acquiring an image to be judged that includes multiple faces; a judgment means for judging whether two of the multiple faces that appear in the image to be judged are similar to each other to a predetermined degree or more; and a storage means for, if the two faces are similar to each other to a predetermined degree or more, storing two face images each containing the two faces and a "other person" label indicating that the two people corresponding to the two face images are not the same person.

[0006] A second aspect of the information processing device includes an acquisition means for acquiring a dataset including a plurality of facial images each including the faces of a plurality of persons whose faces are similar to each other to a predetermined degree or more, and label information regarding a correct class to which the plurality of persons belong in common among a plurality of classes; an extraction means for extracting facial features of each of the plurality of persons based on the plurality of facial images; a class identification means for generating class identification information regarding an estimated class to which the plurality of persons belong in common among the plurality of classes based on the features; and a learning means for performing machine learning to set operating characteristics of the extraction means based on the label information and the class identification information.

[0007] A first aspect of the information processing method acquires an image to be judged that includes multiple faces, determines whether two of the multiple faces that appear in the image to be judged are similar to each other to a predetermined degree or more, and if the two faces are similar to each other to a predetermined degree or more, stores two face images that respectively include the two faces and a "other person" label that indicates that the two people corresponding to the two face images are not the same person. A second aspect of the information processing method acquires a dataset including a plurality of facial images each containing the faces of a plurality of people whose faces are similar to each other to a predetermined degree or more, and label information relating to a correct class among a plurality of classes to which the plurality of people belong in common; extracts facial features of each of the plurality of people based on the plurality of facial images; generates class identification information relating to an estimated class among the plurality of classes to which the plurality of people belong in common based on the features; and performs machine learning to set the operating characteristics of the extraction means based on the label information and the class identification information.

[0008] A first aspect of the recording medium has recorded thereon a computer program for causing a computer to execute an information processing method, which involves acquiring a judgment target image containing a plurality of faces, determining whether two of the plurality of faces contained in the judgment target image are similar to each other to a predetermined degree or more, and, if the two faces are similar to each other to a predetermined degree or more, storing two facial images each containing the two faces and a "other person" label indicating that the two people corresponding to the two facial images are not the same person. A second aspect of the recording medium has recorded thereon a computer program for causing a computer to execute an information processing method for performing machine learning, which includes acquiring a dataset including a plurality of facial images each including the faces of a plurality of persons whose faces are similar to each other to a predetermined degree or more, and label information regarding a correct class among a plurality of classes to which the plurality of persons belong in common, extracting facial features of each of the plurality of persons based on the plurality of facial images, generating class identification information regarding an estimated class among the plurality of classes to which the plurality of persons belong in common based on the features, and setting operating characteristics of the extraction means based on the label information and the class identification information. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing the configuration of an information processing device according to the first embodiment. [Figure 2] FIG. 2 is a schematic diagram of the annotation operation in the first embodiment. [Figure 3]FIG. 3 is a flowchart showing the flow of the annotation operation performed by the information processing device in the first embodiment. [Figure 4] FIG. 4 is a block diagram showing the configuration of an information processing device according to the second embodiment. [Figure 5] FIG. 5 is a conceptual diagram of the learning data used in the second embodiment. [Figure 6] FIG. 6 is a conceptual diagram of a loss function in the second embodiment. [Figure 7] FIG. 7 is a flowchart showing the flow of the learning operation performed by the information processing device in the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of an information processing device, an information processing method, and a recording medium will be described with reference to the drawings. [1: First embodiment]

[0011] First, a first embodiment of an information processing device, an information processing method, and a recording medium will be described. Hereinafter, the first embodiment of the information processing device, the information processing method, and the recording medium will be described using an information processing device 1 to which the first embodiment of the information processing device, the information processing method, and the recording medium is applied. [1-1: Configuration of information processing device 1]

[0012] First, the configuration of an information processing device 1 in the first embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the information processing device 1 in the first embodiment.

[0013] 1, the information processing device 1 includes a calculation device 11 and a storage device 12. The information processing device 1 may further include a communication device 13, an input device 14, and an output device 15. However, the information processing device 1 does not necessarily include at least one of the communication device 13, the input device 14, and the output device 15. The calculation device 11, the storage device 12, the communication device 13, the input device 14, and the output device 15 may be connected via a data bus 16.

[0014] The arithmetic device 11 includes, for example, at least one of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and an FPGA (Field Programmable Gate Array). The arithmetic device 11 reads a computer program. For example, the arithmetic device 11 may read a computer program stored in the storage device 12. For example, the arithmetic device 11 may read a computer program stored in a computer-readable, non-transitory recording medium using a recording medium reading device (e.g., an input device 14 described later) not shown in the drawings that is included in the information processing device 1. The arithmetic device 11 may acquire (i.e., download or read) the computer program from a device (not shown) located outside the biometric authentication device 2 via the communication device 13 (or another communication device). The arithmetic device 11 executes the read computer program. As a result, logical functional blocks for executing operations to be performed by the information processing device 1 are realized within the arithmetic device 11. That is, the arithmetic device 11 can function as a controller for realizing logical functional blocks for executing the operations (in other words, processing) that the information processing device 1 should perform.

[0015] 1 shows an example of logical functional blocks realized within the arithmetic device 11 to execute information processing operations. As shown in Fig. 1, an acquisition unit 111 which is a specific example of "acquisition means", a face extraction unit 112, a feature extraction unit 113, a determination unit 114 which is a specific example of "determination means", and a memory control unit 115 which is a specific example of "storage means" are realized within the arithmetic device 11. Note that at least one of the face extraction unit 112 and the feature extraction unit 113 does not necessarily have to be realized within the arithmetic device 11.

[0016] The operations of the acquisition unit 111, face extraction unit 112, feature extraction unit 113, and determination unit 114 will be described in detail later with reference to Figures 2 and 3. However, the calculation device 11 does not necessarily have to include the face extraction unit 112.

[0017] The storage device 12 can store desired data. For example, the storage device 12 may temporarily store a computer program executed by the arithmetic device 11. The storage device 12 may temporarily store data that the arithmetic device 11 temporarily uses when the arithmetic device 11 is executing a computer program. The storage device 12 may store data that the information processing device 1 stores long-term. The storage device 12 may include at least one of a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and a disk array device. In other words, the storage device 12 may include a non-temporary recording medium.

[0018] The storage device 12 may store sample data SD used by the information processing device 1 for information processing operations. However, the storage device 12 does not have to store the sample data SD. If the storage device 12 does not store the sample data SD, the communication device 13 may acquire the sample data SD from a device external to the information processing device 1, or the input device 14 may accept input of the sample data SD from a device external to the information processing device 1. The storage device 12 may also store the facial image pairs IP generated by the information processing operation of the information processing device 1.

[0019] Here, the information processing device 1 in the first embodiment may use an image containing a face as sample data SD. The information processing device 1 may use the sample data SD to generate a dataset to be used for machine learning of the face recognition engine. Since the dataset to be used for machine learning preferably includes a large amount of data, for example, 10,000 or more, it is preferable that a large amount of sample data SD can be collected.

[0020] The communication device 13 is capable of communicating with devices external to the information processing device 1 via a communication network (not shown).

[0021] The input device 14 is a device that accepts information input to the information processing device 1 from outside the information processing device 1. For example, the input device 14 may include an operation device (for example, at least one of a keyboard, a mouse, and a touch panel) that can be operated by an operator of the information processing device 1. For example, the input device 14 may include a reading device that can read information recorded as data on a recording medium that can be externally attached to the information processing device 1.

[0022] The output device 15 is a device that outputs information to the outside of the information processing device 1. For example, the output device 15 may output information as an image. That is, the output device 15 may include a display device (a so-called display) that can display an image showing the information to be output. For example, the output device 15 may output information as sound. That is, the output device 15 may include an audio device (a so-called speaker) that can output sound. For example, the output device 15 may output information on paper. That is, the output device 15 may include a printing device (a so-called printer) that can print desired information on paper. [1-2: Information processing operation performed by information processing device 1]

[0023] Next, an information processing operation performed by the information processing device 1 in the first embodiment will be described with reference to FIGS. 2 and 3. The information processing operation performed by the information processing device 1 in the first embodiment may be an annotation operation of labeling a face image. More specifically, the information processing operation performed by the information processing device 1 in the first embodiment may be an annotation operation of associating two face images, each including two faces that are similar to each other to a predetermined degree, with a "other" label indicating that the two people corresponding to the two face images are not the same person. The two face images, each including two faces that are similar to each other to a predetermined degree, may include a first face image including a first face and a second face image including a second face. [1-3: Overview of annotation operation]

[0024] First, an overview of the annotation operation in the first embodiment will be described with reference to Fig. 2. Fig. 2 is a diagram showing an overview of the annotation operation in the first embodiment. As shown in Fig. 2(a), when multiple people (three people, person A, person B, and person C in Fig. 2(a)) appear in one image, these multiple people can be considered to be different people. For example, when three people, person A, person B, and person C, look very similar, three similar faces will be lined up, but it can be determined that each of these three faces belongs to a different person.

[0025] Therefore, in the first embodiment, even if two people have similar faces, if they appear in the same image, the information processing device 1 determines that they are not the same person. Then, the information processing device 1 assigns a "different person" label to each person, indicating that they are not the same person with similar faces. For example, it is difficult to distinguish between similar faces, such as twins. This is also difficult for machines such as neural networks, and there is a demand for machines that can distinguish between similar faces. The annotation operation in the first embodiment can generate learning data that can be used for machine learning to identify similar faces.

[0026] The three people, person A, person B, and person C, shown in FIG. 2(a) are, for example, triplets, and will be described as an example where they all have similar faces. For a person who is not very close to person A, person B, and person C, it is often difficult to distinguish between person A, person B, and person C. In this way, the information processing device 1 may generate face image pairs IP in order to generate learning data useful for machine learning that can accurately identify faces even when it is difficult to distinguish between them.

[0027] In the annotation operation in the first embodiment, the information processing device 1 may perform processing for each pair of all people appearing in the image. When three people appear in the image as shown in Fig. 2(a), the information processing device 1 may perform three processes: processing for the pair of person A and person B shown in Fig. 2(b-1), processing for the pair of person B and person C shown in Fig. 2(b-2), and processing for the pair of person C and person A shown in Fig. 2(b-3).

[0028] In the process shown in FIG. 2(b-1), the information processing device 1 assigns a "different person label 1" to each of the images of person A and person B, indicating that person A and person B are not the same person. Also, in the process shown in FIG. 2(b-2), the information processing device 1 assigns a "different person label 2" to each of the images of person B and person C, indicating that person B and person C are not the same person. Also, in the process shown in FIG. 2(b-3), the information processing device 1 assigns a "different person label 3" to each of the images of person C and person A, indicating that person B and person C are not the same person.

[0029] The flow of the annotation operation performed by the information processing device 1 in the first embodiment will be described with reference to Fig. 3. Fig. 3 is a flowchart showing the flow of the annotation operation performed by the information processing device 1 in the first embodiment.

[0030] As shown in FIG. 3, the acquisition unit 111 acquires one sample data SD as a determination target image (step S11). The acquisition unit 111 determines whether the sample data SD is a composite image in which a plurality of images are combined (step S12). If the sample data SD is not a composite image in which multiple images are combined (step S12: No), the face extraction unit 112 extracts a face region from the sample data SD (step S13). The face extraction unit 112 determines whether two or more faces appear in the sample data SD (step S14). The face extraction unit 112 may determine whether the face regions extracted in step S13 are two or more. If two or more faces appear in the sample data SD (step S14: Yes), the face extraction unit 112 selects a pair of two faces from the two or more faces (step S15).

[0031] The feature extraction unit 113 extracts the feature of each face included in the selected pair (step S16). The determination unit 114 calculates the similarity of each feature amount of the faces included in the selected pair (step S17). Based on the calculated similarity, the determination unit 114 determines whether the two faces included in the selected pair are similar to each other to a predetermined degree or more (step S18). The determination unit 114 may calculate, for example, cosine similarity as the similarity. In this case, the determination unit 114 may determine that the two faces are similar to each other to a predetermined degree or more when the similarity is equal to or greater than a predetermined threshold. If the two faces included in the selected pair are similar to each other at a predetermined level (step S18: Yes), the storage control unit 115 associates two face images each including the two faces with a different person label indicating that the two people corresponding to the two face images are not the same person, and stores the associated images in the storage device 12 (step S19). The storage control unit 115 may also associate a first face image including the first face and a second face image including the second face with a different person label, and store the associated images in the storage device 12.

[0032] In the example shown in FIG. 2, it is assumed that three people, person A, person B, and person C, are similar to each other to a predetermined degree or more. In this case, (1) for the pair of person A and person B, the storage control unit 115 may store in the storage device 12 a face image pair IP0 with an other label, in which an image of person A and an image of person B (face image pair 1) are associated with an "other label 1" indicating that they are not the same person, as shown in FIG. 2(b-1). (2) For the pair of person B and person C, the storage control unit 115 may store in the storage device 12 a face image pair IP0 with an other label, in which an image of person B and an image of person C (face image pair 2) are associated with an "other label 2" indicating that they are not the same person, as shown in FIG. 2(b-2). (3) For the pair of person A and person B, the storage control unit 115 may store in the storage device 12 a face image pair IP0 with an other-label, which associates an image of person C and an image of person A (face image pair 3) with an “other-label 3” indicating that they are not the same person, as shown in FIG. 2(b-3).

[0033] If the two faces included in the selected pair are not similar to each other to a predetermined degree (step S18: No), the process proceeds to step S20. The face extraction unit 112 determines whether or not there is a pair of two faces that has not yet been selected. If there is a pair that has not yet been selected (step S20: Yes), the process proceeds to step S15. If there is no pair that has not yet been selected (step S20: No), the operation for one sample data SD ends. If the sample data SD is a composite image in which a plurality of images are combined (step S12: Yes), the operation for one sample data SD is completed. The arithmetic unit 11 performs the processes of steps S15 to S19 on one piece of sample data SD. The arithmetic unit 11 may perform the processes of steps S11 to S19 on each of a plurality of pieces of sample data SD. [1-4: Application example of face image pair IP generated by information processing device 1]

[0034] The face image pair IP0 with a different label generated by the information processing device 1 in the first embodiment may be used to construct a feature extraction model EM1. The feature extraction model EM1 may be a model for identifying faces of different persons as faces of different persons and identifying faces of the same person as faces of the same person. In this case, a face image pair IP1 with a true person label may also be prepared, in which two different images of the same person (face image pair) are associated with a "true person label" indicating that they are the same person. Then, a face image pair IP including both the face image pair IP0 and the face image pair IP1 may be prepared, and the face image pair IP may be used as training data TD for constructing the feature extraction model EM1. Specifically, the feature extraction model EM1 may be a model that determines that a face image pair associated with an other-person label is a face image of a different person, and determines that a face image pair associated with an identical-person label is a face image of the same person. More specifically, when a face image pair IP is input, the feature extraction model EM1 may extract each feature using a shared weight network and determine whether the face image pair is a face image of a different person or a face image of the same person based on the distance or similarity of each feature. In this case, the feature extraction model EM1 may be trained to minimize the distance or maximize the similarity when the face image pair IP1 is input, and to maximize the distance or minimize the similarity when the face image pair IP0 is input. The feature extraction model EM1 may also be a model that compares the distance or similarity between sample data SD. In the first embodiment, the term "other person's label / personal label" is used, but the same technical content may be expressed using other terms, such as "negative example / positive example," "negative class / positive class," or "0 / 1." [1-5: Technical Effects of Information Processing Device 1]

[0035] The information processing device 1 in the first embodiment can assign a "different person label" indicating that the faces are different to a pair of face images of similar faces that are known to be faces of different people. By generating data with such labels, it is possible to generate learning data that is useful for machine learning to identify people with similar faces. This can improve the authentication accuracy for face images that are difficult to authenticate, such as twins. Furthermore, the determination unit 114 determines that the two faces are similar to or more than a predetermined value when the matching score between the feature amounts of the two faces is equal to or greater than a predetermined value. In this way, the determination unit 114 can determine that two faces that are so similar that they would be erroneously determined to be the same person during matching are similar to or more than a predetermined value. [2: Second embodiment]

[0036] Next, a second embodiment of the information processing device, the information processing method, and the recording medium will be described. Hereinafter, the second embodiment of the information processing device, the information processing method, and the recording medium will be described using an information processing device 2 to which the second embodiment of the information processing device, the information processing method, and the recording medium is applied.

[0037] It is often difficult for humans to accurately identify the owners of faces of multiple people whose faces are similar to or above a certain level, such as twins, triplets, or other multiple births, through human observation. Therefore, it is often difficult to prepare training data in which accurate correct labels are assigned to face images of multiple people whose faces are similar to or above a certain level. Furthermore, even in face recognition using machines such as neural networks, it is often difficult to accurately identify the faces of multiple people whose faces are similar to or above a certain level. On the other hand, it is relatively easy to group multiple people whose faces are similar to or above a certain level and assign them to the same group. Furthermore, it is relatively easy to identify face images of people belonging to the same group as belonging to the same group. Therefore, it is relatively easy to assign the same label to face images of people belonging to the same group. In other words, it is relatively easy to prepare sample data SD in which the same label is assigned to face images of people belonging to the same group.

[0038] Therefore, the information processing device 2 in the second embodiment prepares sample data SD in which the same label is assigned to face images of people who belong to the same group, and uses this sample data SD to construct a feature extraction model EM2 that extracts features so as to accurately determine which group a face image belongs to.

[0039] In the first embodiment, a "different label" is assigned to a pair of face images that are known to belong to different people, even though it is difficult to accurately identify who the face belongs to. In contrast, in the second embodiment, a "twin ID label" is assigned to a face image that is difficult to accurately identify who the face belongs to and it is not known whether the faces belong to the same person or different people. Here, the "twin ID label" may be the name of a label assigned to a face image of a person who belongs to a group of multiple people whose faces are similar to each other by a predetermined amount or more. In other words, the "twin ID label" may be a label shared by multiple people whose faces are similar to each other by a predetermined amount or more. In the second embodiment, information such as "someone belonging to a group (not an individual)" may be used instead of "someone (an individual)."

[0040] The information processing operation performed by the information processing device 2 in the second embodiment may be a learning operation for identifying the faces of multiple people whose faces are similar to each other by a predetermined amount or more, such as multiple births, as belonging to the same class. More specifically, the information processing operation performed by the information processing device 2 in the second embodiment may be a learning operation for setting facial feature extraction operation characteristics so that the faces of multiple people whose faces are similar to each other by a predetermined amount or more, such as multiple births, belong to the same class. Furthermore, the information processing device 2 in the second embodiment may construct a feature extraction model EM2 for performing face recognition of multiple non-identical people whose faces are difficult for others to distinguish, such as multiple births such as twins. [2-1: Configuration of information processing device 2]

[0041] The configuration of the information processing device 2 in the second embodiment will be described with reference to Fig. 4. Fig. 4 is a block diagram showing the configuration of the information processing device 2 in the second embodiment.

[0042] 4, the information processing device 2 includes a calculation device 21 and a storage device 22. The information processing device 2 may further include a communication device 23, an input device 24, and an output device 25. However, the information processing device 2 does not necessarily include at least one of the communication device 23, the input device 24, and the output device 25. The calculation device 21, the storage device 22, the communication device 23, the input device 24, and the output device 25 may be connected via a data bus 26.

[0043] Just as the arithmetic device 11 can function as a controller for realizing logical functional blocks for executing operations that the information processing device 1 should perform, the arithmetic device 21 can function as a controller for realizing logical functional blocks for executing operations that the information processing device 2 should perform.

[0044] Fig. 4 shows an example of logical functional blocks realized in the arithmetic device 21 to execute information processing operations. As shown in Fig. 4, an acquisition unit 211 which is a specific example of "acquisition means", a feature extraction unit 212 which is a specific example of "extraction means", a class identification unit 213 which is a specific example of "class identification means", and a learning unit 214 which is a specific example of "learning means" are realized in the arithmetic device 21. Details of the operations of the acquisition unit 211, feature extraction unit 212, class identification unit 213, and learning unit 214 will be described with reference to Figs. 5 to 7.

[0045] The storage device 22 can store desired data, like the storage device 12. The storage device 22 may store training data TD. However, the storage device 22 does not have to store the training data TD. If the storage device 22 does not store the training data TD, the communication device 23 may acquire the training data TD from a device external to the information processing device 2, or the input device 24 may accept input of the training data TD from a device external to the information processing device 2. Details of the training data TD will be described with reference to FIG. 5.

[0046] Next, the information processing operation performed by the information processing device 2 in the second embodiment will be described with reference to FIGS. [2-2: Concept of twin ID class to be processed by information processing device 2]

[0047] First, the concept of the twin ID class that is the target of processing by the information processing device 2 in the second embodiment will be described. In the second embodiment, face images of multiple non-identical people whose faces are similar to or more than a predetermined level and difficult for others to distinguish are grouped into the same twin ID class. The training data TD used in the second embodiment includes data in which the same "twin ID label" is attached to face images belonging to the same twin ID class. The face images included in the same twin ID class may be, for example, face images of siblings such as twins, triplets, or quadruplets, or may be face images of other people who look very similar. The number of people whose faces are included in the same twin ID class may be known.

[0048] FIG. 5 is a conceptual diagram of training data TD used in the second embodiment. In the example shown in FIG. 5, the training data TD includes data belonging to four twin ID classes: twin ID class CA, twin ID class CB, twin ID class CC, and twin ID class CD. Each twin ID class includes face images of multiple people. In the example shown in FIG. 5, the twin ID class CA includes L face images 1a, 2a, 3a, 4a, . . . , La, and each face image included in the twin ID class CA is assigned "LA" as a "twin ID label." Information is also added indicating that the number of people owned by the faces of the face images belonging to the twin ID class CA is K A, e.g., two. It is unknown which of the K A people corresponds to each of the L face images. The twin ID class CB includes M face images 1b, 2b, 3b, 4b, . . . , Mb, and each face image included in the twin ID class CB is assigned "LB" as a "twin ID label." Furthermore, information is also attached that the number of people owned by the faces of face images belonging to twin ID class CB is KB, e.g., 3. Furthermore, it is unknown which of the KB people corresponds to each of the M face images. Twin ID class CC includes N face images 1c, 2c, 3c, 4c, . . . , Nc, and each face image included in twin ID class CC is labeled with "LC" as a "twin ID label." Further, information is attached that the number of people owned by the faces of face images belonging to twin ID class CC is KC, e.g., 2. Furthermore, it is unknown which of the KC people corresponds to each of the N face images. Twin ID class CD includes O face images 1d, 2d, 3d, 4d, . . . , Od, and each face image included in twin ID class CD is labeled with "LD" as a "twin ID label." Also, information is added that the number of people who own the faces of the face images belonging to the twin ID class CD is KD people, for example, 4. Furthermore, it is unknown which of the KD people corresponds to each of the O face images. [2-3: Overview of information processing operations performed by information processing device 2]

[0049] The information processing device 2 in the second embodiment performs machine learning to set operating characteristics of an operation for extracting facial features based on the acquired label information and the generated class identification information. The label information is information about a correct class, among multiple classes, to which multiple people whose faces are similar to each other at a predetermined level belong in common. The label information may indicate the correct class using a correct value of the probability that multiple people belong to each of the multiple classes in common. Furthermore, the class identification information is information about an estimated class, among multiple classes, to which multiple people whose faces are similar to each other at a predetermined level belong in common. The class identification information may indicate the estimated class using the probability that multiple people belong to each of the multiple classes in common. The feature extraction model EM2 may extract common facial features of people belonging to the same class, and the information processing device 2 may accurately identify that they belong to the same class. [2-3-1: Introduction of cross-entropy error]

[0050] The information processing device 2 in the second embodiment may perform machine learning based on a cross-entropy error calculated based on label information and class identification information to construct a feature extraction model EM2. The information processing device 2 in the second embodiment may perform machine learning based on a cross-entropy error calculated using a cross-entropy loss function shown in the following [Equation 1], for example. [Formula 1] TIFF0007718506000001.tif12150 The function shown in the above [Equation 1] is a loss function based on label information and class identification information. yi indicates the correct class, which in the case shown in Figure 5 corresponds to the twin ID label (LA, LB, LC, or LD). exp(scos(θ i,yi +m)) is a function related to the correct class, and Σ j exp(scos(θ i,j)) is a function related to multiple classes other than the correct class. Furthermore, the cross-entropy loss function shown in [Formula 1] above has a margin m added to the correct class compared to general cross-entropy loss functions. That is, in the second embodiment, the cross-entropy loss function shown in [Formula 1] above, which adds a margin m to the correct class, may be adopted to reduce intra-class variance. The feature extraction model EM2 constructed through machine learning in this way can extract features such that the features of face images belonging to the same twin ID class are close to each other. The feature extraction model EM2 may be a model that compares the distance or similarity between the sample data SD and a center. The center may be a feature that represents a class. [2-3-2-1: Introducing subcenters into cross-entropy loss functions]

[0051] Incidentally, as a countermeasure for a noisy data set, i.e., a data set including sample data that is difficult to classify, there is a method (e.g., SubcenterArcFace) of defining multiple subclasses for each of multiple classes. That is, this method includes multiple subcenters (multiple center positions) in each of the multiple classes. With this method, it is possible to extract the feature of the sample data SD so that the feature of the sample data SD resembles one of the multiple subcenters.

[0052] A face image belonging to the twin ID class belongs to a single class because it is difficult to distinguish which person's face it belongs to. However, in reality, a face image belonging to the twin ID class is a face image of one of multiple people, so for a machine that identifies classes, it can be considered a noisy data set, that is, a data set containing sample data that is difficult to identify. Therefore, in the information processing device 2 in the second embodiment, each of multiple classes may include multiple subclasses.

[0053] Furthermore, in the second embodiment, since the twin ID class includes facial images of a known number of people, it is expected that there will be as many centers of probability distribution as the number of people. Therefore, in the second embodiment, the number of subclasses included in each of the multiple classes may be the same as the number of people belonging to the class. In other words, there is no need to prepare many subcenters. Furthermore, since the number of subcenters can be reduced, the amount of calculation can be reduced. The feature extraction model EM2 may be a model that compares the distance or similarity between the sample data SD and a sub-center. A sub-center may be a feature that represents a sub-class. [2-3-2-2: Concept of the learning process when sub-centers are introduced]

[0054] FIG. 6 is a conceptual diagram of a learning process performed by the information processing device 2 in the second embodiment. As described above, in the second embodiment, the same number of sub-centers as the number of owners of face images belonging to twin ID class j may be prepared. In the example shown in FIG. 6, the number of owners of face images belonging to twin ID class j is two, and two sub-centers W j1 ,W j2 are available.

[0055] For example, when training feature extraction model EM2 using the dataset shown in Figure 5, K A sub-centers corresponding to twin ID class CA, K B sub-centers corresponding to twin ID class CB, K C sub-centers corresponding to twin ID class CC, and K D sub-centers corresponding to twin ID class CD may be prepared.

[0056] The information processing device 2 in the second embodiment may construct a feature extraction model EM2 so that each feature is close to one of the sub-centers. Face images belonging to the twin ID class belong to a single class because they are difficult to distinguish, but since they actually represent the faces of two people, two sub-classes may be prepared. When a twin ID class includes twins, each feature is extracted from the face of one of the two people, so it can be expected that they will be distributed around two distribution centers. Therefore, if two sub-centers are prepared, the feature extraction operation can be trained so that the facial features of one person will be close to one sub-center and the facial features of the other person will be close to the other sub-center.

[0057] To achieve this, θ in the above [Equation 1] i,j θ expressed by the following [Equation 2] i,j may be applied. [Formula 2] TIFF0007718506000002.tif22122The parentheses in the arkcos function in the above [Equation 2] indicate a max process for selecting one of the multiple sub-centers prepared. That is, the learning unit 214 selects multiple sub-centers W jk Among them, the sub-center W with the largest inner product with the extracted feature jk In other words, the learning unit 214 may use cosθ i,j W jk and θ i,j W jk may be used to calculate the cross entropy error of the above [Equation 2]. Furthermore, the feature extraction model EM2 may assign the subclass of the subcenter selected by the max process to the class of the face image from which features are extracted. In other words, the feature extraction model EM2 can also assign classes during training.

[0058] In other words, the class identification information may indicate an estimated class by using the probability that each of the multiple people belongs to one of multiple subclasses included in one class as the probability that multiple people commonly belong to one of multiple classes.Also, the class identification information may indicate an estimated class by using the probability that each of the multiple people belongs to one of multiple subclasses included in one class that corresponds to a subclass feature that is most similar to the feature extracted by the feature extraction model EM2.

[0059] For example, if twins belong to a certain twin ID class, two sub-centers W1 and W2 may be prepared. Then, the feature extraction model EM2 may be machine-trained to extract features of a face image of one of the twins so as to be closer to that of the sub-center W1, and extract features of a face image of the other twin so as to be closer to that of the sub-center W2. In other words, in the second embodiment, the multiple sub-centers can capture the respective features of multiple people belonging to the twin ID class. Although the above description has been given of an example in which a sub-center is selected by max processing, a sub-center may be selected using other methods such as an attention mechanism. [2-4: Learning operation by information processing device 2]

[0060] FIG. 7 is a flowchart showing the flow of the learning operation performed by the information processing device 2 in the second embodiment. 7, the acquiring unit 211 acquires a dataset including a plurality of face images each including faces of a plurality of people whose faces are similar to each other by a predetermined amount or more, and label information related to a correct class to which the plurality of people belong in common among a plurality of classes (step S21). In the case shown in FIG. 5, the acquiring unit 211 may acquire a dataset including L face images 1a, 2a, 3a, 4a, . . ., La and label information "LA" related to the correct class "CA", a dataset including M face images 1b, 2b, 3b, 4b, . . ., Mb and label information "LB" related to the correct class "CB", a dataset including N face images 1c, 2c, 3c, 4c, . . ., Nc and label information "LC" related to the correct class "CC", and a dataset including O face images 1d, 2d, 3d, 4d, . . ., Od and label information "LD" related to the correct class "CD". The label information may indicate the correct class using a correct value of the probability that multiple people commonly belong to each of multiple classes.

[0061] The feature extraction unit 212 extracts facial features of each of the plurality of people based on the plurality of face images (step S22). The feature extraction unit 212 may extract facial features of each of the plurality of people using a feature extraction model EM2. The class identification unit 213 generates class identification information for an estimated class to which a plurality of people commonly belong among the plurality of classes, based on the feature amount (step S23). The class identification information may indicate the estimated class using the probability that a plurality of people commonly belong to each of the plurality of classes.

[0062] The learning unit 214 performs machine learning to set operating characteristics of the feature extraction unit 212 based on the label information and class identification information (step S24). The learning unit 214 may perform machine learning based on a cross-entropy error calculated based on the label information and class identification information. The learning unit 214 may perform machine learning based on a cross-entropy error calculated using a cross-entropy loss function that uses the label information and class identification information. The learning unit 214 causes the feature extraction unit 212 to learn a method for extracting features from a face image. Specifically, the learning unit 214 may cause the feature extraction model EM2 used by the feature extraction unit 212 to learn a method for extracting features from a face image, and construct the feature extraction model EM2.

[0063] The learning unit 214 may calculate the gradient of the learning parameters included in the feature extraction model EM2 based on the cross-entropy error, and update the values of the learning parameters included in the feature extraction model EM2 using the gradient of the learning parameters. Updating the values of the learning parameters corresponds to learning the feature extraction model EM2. For example, the learning unit 214 may optimize the values of the learning parameters so that the value of the cross-entropy error is minimized.

[0064] At least the operation of step S24 may be executed for each batch of sample data SD. There is no particular limitation on the value of the batch size, and any value can be used.

[0065] The acquisition unit 211 determines whether or not there is unprocessed training data TD (step S25). If there is no unprocessed training data TD (step S25: No), the calculation device 21 stores the feature extraction model EM2 in the storage device 22 (step S26). The process proceeds to step S22. If there is unprocessed training data TD (step S25: Yes), the process proceeds to step S22.

[0066] The learning unit 214 may store the optimized feature extraction model EM2 including the optimally updated learning parameters in the storage device 22.

[0067] Since learning is performed to make the faces resemble one of the two, the extraction operation may be learned so that the feature amount extracted from one person's face is closer to one sub-center, and the feature amount extracted from the other person's face is closer to the other sub-center. If two faces are closer to different sub-classes, the two faces may be determined to be of different people.

[0068] It should be noted that the face image pair IP generated in the first embodiment may be used to learn the extraction operation so that the feature amounts extracted from the two faces tend to fall into different subclasses. In the second embodiment, for each pair of twin ID classes classified into the same class, it may be determined whether the pair is an unrelated pair or a true-identical pair using the feature extraction model EM2 generated in the second embodiment. [2-5: Technical Effects of Information Processing Device 2]

[0069] The feature extraction model EM2 can extract features such that the features of facial images belonging to the same twin ID class are similar to each other, thereby enabling accurate identification of facial images belonging to the same twin ID class. The learning unit 214 performs machine learning based on the cross-entropy error calculated based on the label information and class identification information, thereby enabling machine learning to be advanced so that feature values extracted from the faces of multiple people whose faces are similar to each other to a predetermined degree are similar to each other. Furthermore, if the number of sub-centers is the same as the number of people belonging to the class, there is no need to prepare many sub-centers. Furthermore, the number of sub-centers can be reduced, thereby reducing the amount of calculation. Furthermore, the feature extraction model EM2 is constructed so that each feature value is similar to one of the sub-centers, enabling identification of which person's facial image it belongs to. [3: Note]

[0070] The following additional notes are provided regarding the above-described embodiment. [Appendix 1] an acquisition means for acquiring a determination target image including a plurality of faces; a determination means for determining whether two faces among the plurality of faces captured in the determination target image are similar to each other to a predetermined degree; a storage means for storing, when the two faces are similar to each other to a predetermined degree, two face images each including the two faces and a "other person" label indicating that the two people corresponding to the two face images are not the same person, in association with each other; An information processing device comprising: [Appendix 2] The determining means determines that the two faces are similar to each other to a predetermined degree or more when a matching score between the feature amounts of the two faces is equal to or greater than a predetermined value. 10. The information processing device according to claim 1. [Appendix 3] an acquisition means for acquiring a data set including a plurality of face images each including faces of a plurality of people whose faces are similar to each other by a predetermined degree or more, and label information regarding a correct class to which the plurality of people belong in common among a plurality of classes; extraction means for extracting facial features of each of the plurality of people based on the plurality of facial images; a class identification means for generating class identification information regarding an estimated class to which the plurality of people commonly belong, among the plurality of classes, based on the feature amount; learning means for performing machine learning to set operating characteristics of the extraction means based on the label information and the class identification information; An information processing device comprising: [Appendix 4] the label information indicates the correct class using a correct value of a probability that the plurality of people belong to each of the plurality of classes in common; the class identification information indicates the estimated class using a probability that the plurality of people commonly belong to each of the plurality of classes; The learning means performs the machine learning based on a cross-entropy error calculated based on the label information and the class identification information. 4. The information processing device according to claim 3. [Appendix 5] Each of the plurality of classes includes a plurality of subclasses, the class identification information indicates the estimated class using a probability that each of the plurality of people belongs to one of the plurality of subclasses included in the one class as a probability that each of the plurality of people belongs to one of the plurality of classes in common; The number of the plurality of subclasses included in each class is the same as the number of the plurality of people. 5. The information processing device according to claim 4. [Appendix 6] The class identification information indicates the estimated class using a probability that each of the plurality of people belongs to one of the subclasses included in the one class, the subclass corresponding to the subclass feature most similar to the feature extracted by the extraction means. 6. The information processing device according to claim 5. [Appendix 7] Acquire an image to be judged that contains multiple faces, determining whether two faces among the plurality of faces captured in the determination target image are similar to each other to a predetermined degree or more; If the two faces are similar to each other to a predetermined degree, two face images each containing the two faces are stored in association with a "other person" label indicating that the two people corresponding to the two face images are not the same person. Information processing methods. [Appendix 8] Acquire a dataset including a plurality of face images each including faces of a plurality of people whose faces are similar to each other by a predetermined degree or more, and label information regarding a correct class to which the plurality of people belong in common among a plurality of classes; extracting facial features of each of the plurality of people based on the plurality of facial images; generating class identification information relating to an estimated class to which the plurality of people commonly belong among the plurality of classes based on the feature amount; Performing machine learning to set operating characteristics of the extraction means based on the label information and the class identification information. Information processing methods. [Appendix 9] On the computer, Acquire an image to be judged that contains multiple faces, determining whether two faces among the plurality of faces captured in the determination target image are similar to each other to a predetermined degree or more; If the two faces are similar to each other to a predetermined degree, two face images each containing the two faces are stored in association with a "other person" label indicating that the two people corresponding to the two face images are not the same person. A recording medium on which a computer program for executing an information processing method is recorded. [Appendix 10] On the computer, Acquire a dataset including a plurality of face images each including faces of a plurality of people whose faces are similar to each other by a predetermined degree or more, and label information regarding a correct class to which the plurality of people belong in common among a plurality of classes; extracting facial features of each of the plurality of people based on the plurality of facial images; generating class identification information relating to an estimated class to which the plurality of people commonly belong among the plurality of classes based on the feature amount; Performing machine learning to set operating characteristics of the extraction means based on the label information and the class identification information. A recording medium on which a computer program for executing an information processing method is recorded.

[0071] At least some of the constituent elements of each of the above-described embodiments can be appropriately combined with at least some of the other constituent elements of each of the above-described embodiments. Some of the constituent elements of each of the above-described embodiments may not be used. Furthermore, to the extent permitted by law, the disclosures of all documents (e.g., published patent applications) cited in this disclosure are incorporated by reference as part of the description of this disclosure.

[0072] This disclosure may be modified as appropriate within the scope of the claims and the technical idea that can be read from the entire specification. Information processing devices, information processing methods, and recording media that involve such modifications are also included in the technical idea of this disclosure. [Explanation of symbols]

[0073] 1,2 Information processing device 11,12 arithmetic unit 12,22 Storage device 111,211 Acquisition Department 112 Face Extraction Unit 113,212 Feature Extraction Unit 114 Judgment section 115 Memory control unit 213 Class Identification Unit 214 Learning Department IP Face Image Pair SD sample data TD training data

Claims

1. an acquisition means for acquiring a determination target image including a plurality of faces; a determination means for determining whether the two faces are similar to each other to a predetermined degree when a matching score between feature amounts of two faces among the plurality of faces captured in the determination target image is equal to or greater than a predetermined value; a storage means for storing, when the two faces are similar to each other to a predetermined degree, two face images each including the two faces and a "other person" label indicating that the two people corresponding to the two face images are not the same person, in association with each other; An information processing device comprising:

2. an acquisition means for acquiring a data set including a plurality of face images each including faces of a plurality of people whose faces are similar to each other by a predetermined degree or more, and label information regarding a correct class to which the plurality of people belong in common among a plurality of classes; extraction means for extracting facial features of each of the plurality of people based on the plurality of facial images; a class identification means for generating class identification information regarding an estimated class to which the plurality of people commonly belong, among the plurality of classes, based on the feature amount; a learning means for performing machine learning to set operating characteristics of the extraction means based on the label information and the class identification information; An information processing device comprising:

3. the label information indicates the correct class using a correct value of a probability that the plurality of people belong to each of the plurality of classes in common; the class identification information indicates the estimated class using a probability that the plurality of people commonly belong to each of the plurality of classes; The learning means performs the machine learning based on a cross-entropy error calculated based on the label information and the class identification information. The information processing device according to claim 2 .

4. Each of the plurality of classes includes a plurality of subclasses, the class identification information indicates the estimated class using a probability that each of the plurality of people belongs to one of the plurality of subclasses included in the one class as a probability that each of the plurality of people belongs to one of the plurality of classes in common; The number of the plurality of subclasses included in each class is the same as the number of the plurality of people. The information processing device according to claim 3 .

5. The class identification information indicates the estimated class using a probability that each of the plurality of people belongs to one of the subclasses included in the one class, the subclass corresponding to the subclass feature most similar to the feature extracted by the extraction means. The information processing device according to claim 4 .

6. 1. A computer-implemented information processing method, comprising: Acquire an image to be judged that contains multiple faces, determining whether the two faces are similar to each other to a predetermined degree or more when a matching score between respective feature amounts of two faces among the plurality of faces captured in the determination target image is equal to or greater than a predetermined value; If the two faces are similar to each other by a predetermined amount or more, two face images each containing the two faces are stored in association with a "other person" label indicating that the two people corresponding to the two face images are not the same person. Information processing methods.

7. 1. A computer-implemented information processing method, comprising: Acquire a dataset including a plurality of face images each including faces of a plurality of people whose faces are similar to each other by a predetermined degree or more, and label information regarding a correct class to which the plurality of people belong in common among a plurality of classes; extracting facial features of each of the plurality of people based on the plurality of facial images; generating class identification information relating to an estimated class to which the plurality of people commonly belong among the plurality of classes based on the feature amount; Performing machine learning to set operating characteristics of the extraction based on the label information and the class identification information. Information processing methods.

8. On the computer, Acquire an image to be judged that contains multiple faces, determining whether the two faces are similar to each other to a predetermined degree or more when a matching score between respective feature amounts of two faces among the plurality of faces captured in the determination target image is equal to or greater than a predetermined value; If the two faces are similar to each other by a predetermined amount or more, two face images each containing the two faces are stored in association with a "other person" label indicating that the two people corresponding to the two face images are not the same person. A computer program for executing an information processing method.

9. On the computer, Acquire a dataset including a plurality of face images each including faces of a plurality of people whose faces are similar to each other by a predetermined degree or more, and label information regarding a correct class to which the plurality of people belong in common among a plurality of classes; extracting facial features of each of the plurality of people based on the plurality of facial images; generating class identification information relating to an estimated class to which the plurality of people commonly belong among the plurality of classes based on the feature amount; Performing machine learning to set operating characteristics of the extraction based on the label information and the class identification information. A computer program for executing an information processing method.

Images