Apparatus for generating fingerprint data conversion models, method for generating fingerprint data conversion models, and system for generating fingerprint data conversion models

Abstract

This process generates a trained model that converts biometric data obtained using a different method than the one used during registration into biometric data that can be used for biometric authentication. [Solution] The biodata conversion model generation device acquires one or more first biodata acquired by the first acquisition method and second biodata acquired by the second acquisition method, converts the second biodata to biodata equivalent to the first biodata using a learning model, and if it is determined that the degree of agreement between the features of the first biodata and the features of the converted second biodata does not satisfy predetermined conditions, it performs learning on the learning model with the degree of agreement as the loss, and if it is determined that the predetermined conditions are satisfied, it outputs the learning model as a trained model that converts the second biodata.

Description

Technical Field

[0001] The present disclosure relates to an apparatus for generating a fingerprint data conversion model, a method for generating a fingerprint data conversion model, and a system for generating a fingerprint data conversion model.

Background Art

[0002] Patent Document 1 discloses a biometric authentication apparatus including: a teacher data generation unit that generates a plurality of pairs each of which is a combination of a first biometric image and a second biometric image from a plurality of first biometric images and a plurality of second biometric images; a learning data generation unit that extracts feature amounts from the first biometric image and the second biometric image using a plurality of different temporary parameters; a calculation unit that calculates the degree of coincidence of the feature amounts extracted from the paired first biometric image and second biometric image with respect to the plurality of pairs generated by the teacher data generation unit; and an optimal solution determination unit that determines the temporary parameters based on the calculation result of the calculation unit.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In view of the above-described conventional circumstances, the present disclosure provides an apparatus for generating a fingerprint data conversion model, a method for generating a fingerprint data conversion model, and a system for generating a fingerprint data conversion model, which generate a learned model for converting fingerprint data acquired by a method different from that at the time of registration into fingerprint data capable of fingerprint authentication.

Means for Solving the Problems

[0005] This disclosure provides a fingerprint data conversion model generation device comprising: an acquisition unit that acquires image captures of first fingerprint data of multiple fingertips of authenticated persons acquired by a first acquisition method and image captures of second fingerprint data of the same fingertips acquired by a second acquisition method different from the first acquisition method; a data conversion unit that uses a learning model to convert the image captures of the second fingerprint data into image data corresponding to the image captures of the first fingerprint data acquired by the first acquisition method; a calculation unit that calculates the degree of agreement between the fingertip features included in the image captures of the first fingerprint data and the fingertip features included in the converted image captures of the second fingerprint data; and a control unit that, if it is determined that the calculated degree of agreement does not satisfy predetermined conditions, performs learning on the learning model with the degree of agreement as the loss, and if it is determined that the degree of agreement satisfies predetermined conditions, outputs the learning model as a trained model that converts the image captures of the second fingerprint data into image data acquired by the first acquisition method.

[0006] Furthermore, this disclosure provides a method for generating a fingerprint data conversion model performed by one or more processors, comprising: acquiring captured images of one or more first fingerprint data of multiple fingertips of authenticated persons acquired by a first acquisition method, and captured images of second fingerprint data of the same fingertips acquired by a second acquisition method different from the first acquisition method; using a learning model, converting the captured images of the second fingerprint data into image data corresponding to the captured images of the first fingerprint data acquired by the first acquisition method; calculating the degree of agreement between the fingertip features included in the captured images of the first fingerprint data and the fingertip features included in the captured images of the converted second fingerprint data; if it is determined that the calculated degree of agreement does not satisfy a predetermined condition, performing learning on the learning model with the degree of agreement as the loss; and if it is determined that the calculated degree of agreement satisfies a predetermined condition, outputting the learning model as a trained model that converts the captured images of the second fingerprint data into image data acquired by the first acquisition method.

[0007] Furthermore, this disclosure relates to a fingerprint data conversion model generation system comprising: a first acquisition device that acquires captured images of first fingerprint data of multiple authenticated persons' fingertips using a first acquisition method; a second acquisition device that acquires captured images of second fingerprint data of the fingertips acquired using a second acquisition method different from the first acquisition method; and a conversion model generation device that can communicate between the first acquisition device and the second acquisition device, wherein the first acquisition device acquires captured images of one or more of the first fingerprint data and transmits them to the conversion model generation device; the second acquisition device acquires captured images of the second fingerprint data and transmits them to the conversion model generation device; and the conversion model generation device uses a learning model to perform the conversion model generation. The present invention provides a fingerprint data conversion model generation system that converts second fingerprint data into image data corresponding to the captured image of the first fingerprint data acquired by the first acquisition method, calculates the degree of agreement between the fingertip features included in the captured image of the first fingerprint data and the fingertip features included in the captured image of the converted second fingerprint data, and if it is determined that the calculated degree of agreement does not satisfy a predetermined condition, performs learning on the learning model with the degree of agreement as the loss, and if it is determined that the calculated degree of agreement satisfies the predetermined condition, outputs the learning model as a trained model that converts the captured image of the second fingerprint data into image data acquired by the first acquisition method. [Effects of the Invention]

[0008] According to this disclosure, it is possible to generate a trained model that converts fingerprint data obtained using a different acquisition method than the one used during registration into fingerprint data that can be authenticated. [Brief explanation of the drawing]

[0009] [Figure 1] Block diagram showing a first use case example of the biometric data conversion model generation system according to the embodiment. [Figure 2] A flowchart illustrating the procedure for generating training data for an information processing device. [Figure 3] A flowchart illustrating the procedure for generating the first trained model of an information processing device. [Figure 4] This figure illustrates an example of generating the first pre-trained model. [Figure 5] A flowchart illustrating the procedure for generating the second trained model of an information processing device. [Figure 6] A diagram illustrating an example of generating a second pre-trained model. [Figure 7] A flowchart illustrating the procedure for generating the third trained model of an information processing device. [Figure 8] This figure illustrates an example of generating a third pre-trained model. [Figure 9] Block diagram showing a second use case example of the biometric data conversion model generation system according to the embodiment. [Figure 10] Flowchart showing the procedure for inferring biometric data for authentication using the first to third trained models. [Modes for carrying out the invention]

[0010] (Background leading to this disclosure) In methods for acquiring biometric data used in biometric authentication (e.g., fingerprints, veins, or palm prints), there are two approaches: acquiring biometric data from a user's body parts in a non-contact state (e.g., fingers or palms) and acquiring biometric data from a user's body parts in a contact state. However, biometric data acquired in a contact state may differ in shape from biometric data acquired in a non-contact state because pressure is applied to the body part when it is placed on the surface (contact). Therefore, in biometric authentication using biometric data acquired by different methods, it is necessary to estimate the registration biometric data acquired by the different method using the authentication biometric data. However, because the characteristics indicating the user's individuality contained in the biometric data differ due to the different acquisition methods, there was a possibility that the authentication accuracy would decrease.

[0011] Furthermore, in recent years, there is a technology that estimates biometric data acquired in a contact state from biometric data acquired in a non-contact state by performing a simulation in which a 3D model of a biological part (e.g., finger) of a real or fictional person is pressed against a surface. However, biological parts have different 3D shapes for each user and for each part (e.g., index finger, middle finger, or ring finger). Therefore, biometric authentication using biometric data estimated using a 3D model of a real or fictional person can only improve authentication accuracy if the person has the same 3D shape as the 3D model, making it difficult to improve the authentication accuracy of biometric authentication that is widely used by various users.

[0012] Reference 1 discloses a biometric authentication device that extracts features from biometric images acquired using various operational styles, such as fingerprints taken in advance for criminal investigations or fingerprints left at crime scenes. This biometric authentication device performs learning based on the degree of separation between the degree of similarity of features obtained from the same type of finger and the degree of similarity of features obtained from different types of fingers. It can be said that the aim is to more clearly distinguish whether fingerprints taken by different methods belong to the same finger or different fingers, and to optimize the parameters for extracting fingerprint features regardless of the acquisition method.

[0013] Therefore, in order to improve authentication accuracy in biometric authentication using biometric data acquired by different methods, such as contact and non-contact states, a technology is required that uses authentication biometric data to estimate registration biometric data with higher accuracy.

[0014] Hereinafter, while appropriately referring to the drawings, each embodiment specifically disclosed for the configuration and operation of the apparatus for generating a fingerprint data conversion model, the method for generating a fingerprint data conversion model, and the generation system of the fingerprint data conversion model according to the present disclosure will be described in detail. However, detailed descriptions that are more than necessary may be omitted. For example, detailed descriptions of well-known matters and redundant descriptions of substantially the same configurations may be omitted. This is to avoid making the following description unnecessarily redundant and to facilitate understanding by those skilled in the art. Note that the accompanying drawings and the following description are provided for those skilled in the art to fully understand the present disclosure, and it is not intended to limit the subject matter described in the claims thereby.

[0015] Note that the term "biometric data" in the present disclosure is data including the characteristics of a biological part used for biometric authentication. For example, it is a term including image data obtained by imaging a biological part, data of one or more feature points indicating individuality in a biological part, or data of a feature amount indicating individuality obtained from a biological part. Further, the term "feature" in the present disclosure indicates an element used to identify a fingerprint in the authentication process of biometric authentication, and is a term including elements such as a pattern or shape of a biological part, a feature amount of a biological part, or a feature point of a biological part.

[0016] <First Use Case Example> Referring to FIG. 1, a first use case example of the biometric data conversion model generation system 100 according to Embodiment 1 will be described. FIG. 1 is a block diagram showing a first use case example of the biometric data conversion model generation system 100 according to the embodiment.

[0017] In the first use case example, the biological data conversion model generation system 100 generates a learned model for converting biological data used for authentication so that in the authentication of the same biological part of the same person, the features indicating the individuality of the person included in the registered biological data match the features indicating the individuality of the person included in the biological data obtained in a different way from the registration time and used for authentication. The biological data conversion model generation system 100 includes a first data acquisition device P1, a second data acquisition device P2, and an information processing device S1. Note that the first data acquisition device P1 or the second data acquisition device P2 shown in FIG. 1 may be integrally configured with the information processing device S1. Also, when the information processing device S1 has already stored the biological data obtained in the same way as at the time of registration in the learning data storage unit 13 in advance, or when it is possible to read the biological data obtained and stored in the same way as at the time of registration from another device or storage medium, the first data acquisition device P1 may be omitted as it is not essential.

[0018] In the present disclosure, as an example of biometric authentication, a method for generating a conversion model of biological data in the case of performing fingerprint authentication will be described. However, the biological data used for biometric authentication is not limited to fingerprints. Biometric authentication may be, for example, vein authentication or palmprint authentication, or may be authentication using any two or more of fingerprint, vein, and palmprint biological data.

[0019] The information processing device S1 is communicably connected to each of the first data acquisition device P1 and the second data acquisition device P2. The information processing device S1 is realized by a Personal Computer (hereinafter referred to as "PC"), a notebook PC, a tablet terminal, a cloud server, or an on-premises server, etc. The information processing device S1 generates a learned model for converting the biological data acquired by the second data acquisition device P2 into biological data having features that can be authenticated (matched) with the previously registered biological data acquired by the first data acquisition device P1. The information processing device S1 includes a communication unit 10, a processor 11, a memory 12, a learning data storage unit 13, and a learned model storage unit 14.

[0020] The communication unit 10 is connected to the first data acquisition device P1 and the second data acquisition device P2 via wireless or wired communication, respectively. Wireless communication here refers to short-range wireless communication such as Bluetooth® or NFC®, or communication via a wireless Local Area Network (LAN) such as Wi-Fi®.

[0021] The communication unit 10 outputs to the processor 11 either the biometric data acquired by the first data acquisition device P1 (i.e., an image of a fingerprint taken when the fingertip is in contact with an arbitrary object (hereinafter referred to as the "contact image")) or the biometric data acquired by the second data acquisition device P2 (i.e., an image of a fingerprint taken when the fingertip is not in contact with an arbitrary object (hereinafter referred to as the "non-contact image")).

[0022] The processor 11 is configured using, for example, a Central Processing Unit (hereinafter referred to as "CPU"), a Field Programmable Gate Array (hereinafter referred to as "FPGA"), or a Graphics Processing Unit (hereinafter referred to as "GPU"), and works in cooperation with the memory 12 to perform various processing and control. Specifically, the processor 11 refers to the programs and data held in the memory 12 and executes those programs to realize functions such as the learning unit 111 and the feature point data generation unit 112, and generates a trained model. Note that the feature point data generation unit 112 is not a required configuration and may be omitted.

[0023] The learning unit 111 uses a learning model that transforms the biometric data acquired by the second data acquisition device P2 to enable authentication of the features contained in the biometric data acquired by the first data acquisition device P1. This learning unit uses multiple pairs of learning data stored in the learning data storage unit 13 (described later), or multiple pairs of learning data contained in at least one set of learning data. The learning unit 111 repeats the learning process, using the degree of agreement as the loss, until the degree of agreement between the features contained in the biometric data acquired by the second data acquisition device P2 and the features contained in the biometric data acquired by the first data acquisition device P1 satisfies a predetermined condition. When the learning unit 111 determines that the degree of agreement between the features of the transformed biometric data transformed by the learning model and the features of the biometric data acquired by the first data acquisition device P1 satisfies a predetermined condition, it terminates the learning of the learning model and outputs this learning model as a trained model to the trained model storage unit 14 for storage.

[0024] The feature point data generation unit 112 generates feature points of biological parts from biological data. When learning using feature points of biological parts is performed, the feature point data generation unit 112 generates feature points of biological parts and stores (stores) the generated feature points of biological parts as learning data in the learning data storage unit 13.

[0025] Memory 12 includes, for example, Random Access Memory (hereinafter referred to as "RAM"), which serves as work memory used when executing various processes of the processor 11, and Read Only Memory (hereinafter referred to as "ROM"), which stores programs and data that define the operation of the processor 11. Data or information generated or acquired by the processor 11 is temporarily stored in RAM. Programs that define the operation of the processor 11 are written to ROM.

[0026] The learning data storage unit 13 stores the learning data used for learning. The learning data storage unit 13 stores (saves) information relating to the biometric data of multiple different individuals acquired by the second data acquisition device P2 (for example, the biometric data itself or the file name of the biometric data) and information relating to the biometric data of multiple different individuals acquired by the first data acquisition device P1 (for example, the biometric data itself or the file name of the biometric data). The learning data storage unit 13 may be configured separately from the information processing device S1 and implemented by an external storage medium capable of data communication with the information processing device S1.

[0027] Here, we will explain the biometric data (training data) stored in the training data storage unit 13 and used to generate the learning model. The training data storage unit 13 associates one or more biometric data acquired by the first data acquisition device P1 with one biometric data acquired by the second data acquisition device P2, and stores these biometric data as pairs of training data used in one training process. For example, in the example shown in Figure 4, the training data storage unit 13 stores the biometric data with the file name "001-LEFT-INDEX-CL.png" and the biometric data with the file name "001-LEFT-INDEX-C.png" as the first pair, and the biometric data with the file name "002-RIGHT-RING-CL.png" and the biometric data with the file name "002-RIGHT-RING-C.png" as the second pair.

[0028] Furthermore, the training data storage unit 13 may store multiple sets of training data as a single set of training data containing multiple pairs of training data used to generate (train) a trained model. For example, in the example shown in Figure 4, the training data storage unit 13 stores the first pair and the second pair mentioned above as a single set of training data. In addition, multiple sets of training data may be set; for example, a first set of training data containing three pairs of training data and a second set of training data containing five pairs of training data may be set. Note that the setting of training data pairs and training data sets may be performed by any person or may be performed automatically.

[0029] The trained model storage unit 14 stores (saves) the trained model generated by the processor 11.

[0030] The first data acquisition device P1 is connected to the information processing device S1 so as to be able to communicate data. The first data acquisition device P1 images or reads a biological body part when the biological body part is in contact with an arbitrary object (for example, a glass surface). The first data acquisition device P1 includes a communication unit 20, a processor 21, and a memory 22.

[0031] The communication unit 20 is connected to the information processing device S1 via wireless or wired communication. The communication unit 20 transmits the biometric data (contact data) output from the processor 21 to the information processing device S1.

[0032] The processor 21 is configured using, for example, a CPU, FPGA, or GPU, and works in cooperation with the memory 22 to perform various processes and controls. Specifically, the processor 21 refers to the programs and data held in the memory 22 and executes those programs to realize the function of acquiring biological data of a biological part in contact with an arbitrary object. The processor 21 outputs the biological data (contact data) output from the contact data acquisition unit 23 to the communication unit 20 and has it transmitted to the information processing device S1.

[0033] Memory 22 includes, for example, RAM as work memory used when executing each process of processor 21, and ROM which stores programs and data that define the operation of processor 21. Data or information generated or acquired by processor 21 is temporarily stored in RAM. Programs that define the operation of processor 21 are written to ROM.

[0034] The contact data acquisition unit 23 is implemented, for example, by a sensor or an optical system including a lens and an image sensor, and acquires biological data of a biological part in contact with an arbitrary object. The contact data acquisition unit 23 outputs the acquired biological data (contact data) to the processor 21. In this disclosure, the contact data acquisition unit 23 is mainly composed of an optical system and an example of imaging a biological part is shown, but this does not exclude a configuration using a sensor.

[0035] The second data acquisition device P2 is connected to the information processing device S1 so as to be able to communicate data. The second data acquisition device P2 is implemented by, for example, a visible light camera device, or any device equipped with a camera (for example, a smartphone or tablet terminal). The second data acquisition device P2 images the biological body part in a non-contact state, in which the biological body part is not in contact with any object. The second data acquisition device P2 includes a communication unit 30, a processor 31, and a memory 32.

[0036] The communication unit 30 is connected to the information processing device S1 via wireless or wired communication. The communication unit 30 transmits biometric data (non-contact data) output from the processor 31 to the information processing device S1.

[0037] The processor 31 is configured using, for example, a CPU, FPGA, or GPU, and works in cooperation with the memory 32 to perform various processing and control. Specifically, the processor 31 refers to the programs and data held in the memory 32 and executes those programs to realize the function of acquiring biological data of living parts that are in a non-contact state with any object. The processor 31 outputs the biological data (non-contact data) output from the non-contact data acquisition unit 33 to the communication unit 30 and has it transmitted to the information processing device S1.

[0038] Memory 32 includes, for example, RAM as work memory used when executing each process of the processor 31, and ROM which stores programs and data that define the operation of the processor 31. Data or information generated or acquired by the processor 31 is temporarily stored in RAM. Programs that define the operation of the processor 31 are written in ROM.

[0039] The non-contact data acquisition unit 33 is implemented, for example, by an optical system including a lens and an image sensor, and captures biological data of a living body part in a non-contact state with any object. The non-contact data acquisition unit 33 outputs the acquired biological data (non-contact data) to the processor 31.

[0040] In the following explanation, when biometric data acquired using different methods during registration and authentication is used, we will describe how to generate each pre-trained model (specifically, the first pre-trained model, the second pre-trained model, and the third pre-trained model) that transforms the contactless data so that the state of features contained in the contactless data (biometric data) acquired using the same method as during authentication corresponds to the state of the registered data or the contact data (biometric data) acquired using the same method as during registration. Furthermore, in order to make the explanation easier to understand, the following explanation will assume fingerprint authentication and specifically describe an example where the biometric part is the fingertip (fingerprint).

[0041] Here, the first pre-trained model is a pre-trained model for transforming a fingerprint image (hereinafter referred to as "non-contact image") taken when the fingertip, a biological part, is not in contact with any object, and is a pre-trained model for transforming a non-contact image into a fingerprint image (hereinafter referred to as "contact image") taken when the fingertip is in contact with any object.

[0042] Furthermore, the second pre-trained model is a pre-trained model for transforming non-contact images, specifically a pre-trained model for transforming non-contact images into fingerprint feature points (hereinafter referred to as "contact feature points") in a state of contact with an object.

[0043] Furthermore, the third pre-trained model is a pre-trained model for converting fingerprint feature points (hereinafter referred to as "non-contact feature points") where the fingertip, a biological part, is in a non-contact state with any object, and is a pre-trained model for converting non-contact feature points into contact feature points.

[0044] In the following explanation, we will describe how to generate each pre-trained model, in the order of the first pre-trained model, the second pre-trained model, and the third pre-trained model.

[0045] <Method for generating the first pre-trained model> First, let's explain how to generate the first pre-trained model. The information processing device S1 uses a non-contact image and a contact image to generate a first pre-trained model for converting an input non-contact image into a non-contact image corresponding to the state of the contact image.

[0046] First, with reference to Figure 2, we will explain how to generate the first trained model, that is, how to generate the training data Dt10 (see Figure 4) used to train the trained model MD. Figure 2 is a flowchart showing the procedure for generating training data Dt10 to Dt30 by the information processing device S1. Note that the processing in steps St11 to St12 may be repeated based on the number of training data pairs contained in training data Dt10.

[0047] Note that the generation procedures for each of the training data shown in Figure 2 include the generation procedures for the training data Dt10 to Dt30 used to generate the first to third trained models, respectively. However, here we will only explain the method for generating training data Dt10, which is used to generate the first trained model.

[0048] The processor 11 of the information processing device S1 acquires at least one contact image Dt131 to Dt13N acquired by the first data acquisition device P1 and a non-contact image Dt11 acquired by the second data acquisition device P2 (St11). The timing of image acquisition by each data acquisition device may be at any arbitrary timing.

[0049] When the processor 11 generates a first trained model, it associates the non-contact image Dt11 and contact images Dt131~Dt13N, which are images of the same person's body parts, and stores them in the training data storage unit 13 (St12).

[0050] Here, the processor 11 stores in the training data storage unit 13 one training data pair, in which one or more contact images Dt131 to Dt13N are associated with one non-contact image Dt11. In other words, if there are multiple non-contact images, the processor 11 generates multiple training data pairs, in which one or more contact images Dt131 to Dt13N are associated with each non-contact image, and stores them in the training data storage unit 13. The processor 11 may also store two or more training data pairs from those stored in the training data storage unit 13 as a single training data set used to generate the first trained model.

[0051] Based on the above, the information processing device S1 generates learning data Dt10 containing information on multiple pairs or sets of learning data. In Figure 4, the learning data Dt10 stored in the learning data storage unit 13 is managed as a single pair of learning data, with the following linked information: a person ID that can identify the person who owns the biological part shown in the non-contact image and the contact image, information about the biological part shown in the non-contact image and the contact image, the file name of the non-contact image, and the file name of the contact image. However, it is not limited to this example. It is sufficient that the learning data Dt10 is linked to the file name of the non-contact image.

[0052] Next, referring to Figures 3 and 4, we will explain the method for generating the first trained model, that is, the training method using the trained model MD. Figure 3 is a flowchart showing the procedure for generating the first trained model of the information processing device S1. Figure 4 is a diagram illustrating an example of generating the first trained model.

[0053] The learning unit 111 of the information processing device S1 performs learning (St20) using all pairs of learning data or all sets of learning data that are stored in the learning data storage unit 13 and have been set in advance as learning data Dt10 to be used for learning. Specifically, if the learning data storage unit 13 has N (an integer greater than or equal to 2) pairs of learning data or M (an integer greater than or equal to 1) sets of learning data, the learning unit 111 performs the processing in steps St21 to St23 for each of the N pairs of learning data or M sets of learning data. In other words, the learning unit 111 repeatedly performs the processing in steps St21 to St23 a number of times corresponding to the number of pairs or sets of learning data. Here, we will describe an example of a learning process using learning data Dt10 containing N pairs of learning data.

[0054] The learning unit 111 acquires a non-contact image Dt11 of a biological body part and at least one contact image Dt131 to Dt13N associated with this non-contact image Dt11 from the learning data Dt10 stored in the learning data storage unit 13 (St21).

[0055] The learning unit 111 uses the learning model MD to convert the non-contact image Dt11 into a non-contact image Dt12 (St22) such that the non-contact image Dt11 becomes a non-contact image corresponding to the state of the biological parts reflected in the contact images Dt131~Dt13N (for example, the way the biological parts are crushed, the way feature points or features change, etc.).

[0056] The learning unit 111 calculates the degree of agreement between the converted non-contact image Dt12 and each of the contact images Dt131 to Dt13N for each combination of non-contact feature points and contact feature points. The learning unit 111 updates the learning model MD by performing learning with the calculated degree of agreement as the loss (St23). Note that the combinations referred to here include all three possible combinations of non-contact and contact images, for example, in the case of a pair in which one non-contact image is associated with three contact images.

[0057] For example, in the example shown in Figure 4, the training data Dt10 contains information for N pairs of training data. The training unit 111 uses a non-contact image with the filename "001-LEFT-INDEX-CL.png" associated with the biological parts "LEFT (hand)" and "INDEX (finger)" of person ID "001," and a contact image with the filename "001-LEFT-INDEX-C.png" as the first pair of training data to perform the first training process (processing in steps St21 to St23). Then, using a non-contact image with the filename "002-RIGHT-RING-CL.png" associated with the biological parts "RIGHT (hand)" and "RING (finger)" of person ID "002," and a contact image with the filename "002-RIGHT-RING-C.png" as the second pair of training data to perform the second training process (processing in steps St21 to St23). The learning unit 111 repeats this process and executes the Nth learning process (the processes from step St21 to step St23).

[0058] The learning unit 111 determines, based on all the matching scores calculated in step St20 (i.e., N learning processes), whether the matching score satisfies predetermined conditions and whether the learning model MD is capable of generating a non-contact image Dt12 (St24).

[0059] The predetermined condition here is that the average value of the degree of agreement is greater than or equal to a predetermined value. The predetermined value here can be any value, as long as it is judged to be sufficiently high. As a result, the learning unit 111 can generate a first pre-trained model that enables transformation (inference) with higher consistency and higher stability for various contact images in the transformation (inference) process of non-contact images using the first pre-trained model.

[0060] If the learning unit 111 determines in step St24 that the degree of agreement satisfies a predetermined condition (St24, YES), it saves the updated learning model MD as the first learned model in the learned model storage unit 14 (St25).

[0061] On the other hand, if the learning unit 111 determines in step St24 that the degree of agreement does not meet the predetermined conditions (St24, NO), it returns to the process of step St20 and re-executes the processes of steps St21 to St23 using all pairs or sets of learning data included in the learning data Dt10, thereby re-updating the learning model MD.

[0062] As described above, the information processing device S1 can generate and acquire a first trained model that can generate a non-contact image (transformed non-contact image) in which the state of the biological part shown in the non-contact image acquired in a non-contact state corresponds to the state of the biological part shown in the contact image acquired in a contact state (for example, the way the biological part is crushed, the way feature points or features change, etc.). Furthermore, since the first trained model of this disclosure is trained using biometric data from multiple people, it can correspond to the 3D shape of various users' biological parts (for example, fingers) and can infer biometric data corresponding to various contact states (for example, pressing force, direction of pressure, and location of pressure).

[0063] In other words, even if the three-dimensional shape of a biological part differs depending on the type of person or biological part, and deformation of the biological part occurs due to arbitrary pressure applied to the biological part during registration, the information processing device S1 can use the first trained model to convert the biometric data (non-contact image) acquired during authentication into data (non-contact image) that has biometric data corresponding to the state of the biometric data (contact image) registered during registration (for example, the way the biological part is deformed, the way feature points or features change, etc.). Therefore, the information processing device S1 can improve the authentication accuracy of biometric authentication using biometric data acquired by different acquisition methods.

[0064] <Method for generating the second pre-trained model> Next, the method for generating the second pre-trained model will be explained. The information processing device S1 generates fingerprint contact feature points from each contact image. The information processing device S1 generates a second pre-trained model for converting the input non-contact image into non-contact feature points corresponding to the contact feature points generated from the contact image.

[0065] First, referring to Figure 2, we will explain how to generate the second trained model, that is, how to generate the training data Dt20 (see Figure 6) used to train the trained model MD. Note that steps St11 to St12 may be repeated based on the number of training data pairs contained in the training data Dt20.

[0066] The processor 11 of the information processing device S1 acquires at least one contact image Dt231 to Dt23N acquired by the first data acquisition device P1 and a non-contact image Dt21 acquired by the second data acquisition device P2 (St11). The timing of image acquisition by each data acquisition device may be at any arbitrary timing.

[0067] The processor 11 associates the non-contact image Dt21 and the contact images Dt231~Dt23N, which are of the same person and the same biological part (for example, the middle finger of the right hand), and stores them in the learning data storage unit 13 (St12).

[0068] Here, the processor 11 stores in the training data storage unit 13 one training data pair, where one or more contact images Dt231 to Dt23N are associated with one non-contact image Dt21. In other words, if there are multiple non-contact images, the processor 11 associates one or more contact images Dt231 to Dt23N with each non-contact image and stores them in the training data storage unit 13. The processor 11 may also store two or more training data pairs from those stored in the training data storage unit 13 as a single training data set used to generate a second trained model.

[0069] Based on the above, the information processing device S1 generates learning data Dt20 containing information on multiple pairs or sets of learning data. In Figure 6, the learning data Dt20 stored in the learning data storage unit 13 is managed as a single pair of learning data, with the following linked elements: a person ID that can identify the owner of the biological parts shown in the non-contact image and the contact image, information about the biological parts shown in the non-contact image and the contact image, the file name of the non-contact image, and the file name of the contact image. However, this is not an example.

[0070] The processor 11 may perform image analysis on each of the contact images Dt231 to Dt23N as shown in Figure 6, and generate information on fingerprint contact feature points from each of the contact images Dt231 to Dt23N. In this case, the processor 11 generates training data (for example, training data Dt20 shown in Figure 6) for each non-contact image, which is further associated with the file names of the contact feature points of the fingerprint (biological part) that are in contact, generated from at least one contact image, or training data in which the file names of the contact feature points are associated with the file names of the non-contact images instead of the file names of the contact images, and stores these in the training data storage unit 13. The training data Dt20 only needs to have the file names of the non-contact images associated with the file names of the contact images or the file names of the contact feature point information.

[0071] Next, referring to Figures 5 and 6, we will explain the method for generating the second trained model, that is, the training method using the trained model MD. Figure 5 is a flowchart showing the procedure for generating the second trained model of the information processing device S1. Figure 6 is a diagram illustrating an example of generating the second trained model.

[0072] The learning unit 111 of the information processing device S1 performs learning (St30) using all pairs of learning data or all sets of learning data that are stored in the learning data storage unit 13 and have been set in advance as learning data Dt20 to be used for learning. Specifically, if the learning data storage unit 13 has N pairs of learning data or M sets of learning data, the learning unit 111 performs the processing in steps St31 to St34 for each of the N pairs of learning data or M sets of learning data. In other words, the learning unit 111 repeatedly performs the processing in steps St31 to St34 a number of times corresponding to the number of pairs or sets of learning data. Here, we will explain an example of a learning process using learning data Dt20 containing N pairs of learning data.

[0073] The learning unit 111 acquires a non-contact image Dt21 of a biological body part and at least one contact image Dt231 to Dt23N associated with this non-contact image Dt21 from the learning data Dt20 stored in the learning data storage unit 13 (St31).

[0074] Furthermore, if the learning unit 111 has information on contact feature points generated from each of the contact images Dt231 to Dt23N stored in the learning data storage unit 13 (Dt20), the learning unit 111 may acquire the contact feature point information. In such a case, the processing in step St33 may be omitted.

[0075] The learning unit 111 uses the learning model MD to convert the non-contact image Dt21 into non-contact feature points Dt22 (St32) such that the non-contact image Dt21 becomes a non-contact feature point corresponding to the state of the contact feature points Dt241~Dt24N (for example, a change in the feature point or a change in the positional relationship between the feature points).

[0076] Furthermore, in the processing of step St30, if the processing of step St32 has not yet been executed and contact feature points Dt241~Dt24N have not been generated, the learning unit 111 estimates the state of the contact feature points and, based on the non-contact image and the estimated state of the contact feature points, converts the non-contact image into non-contact feature points that have changes in the feature points or changes in the positional relationship between feature points corresponding to the estimated state of the contact feature points. In addition, the learning unit 111 may execute the processing of step St32 after executing the processing of step St33.

[0077] The feature point data generation unit 112 generates fingerprint contact feature points Dt241 to Dt24N from each contact image Dt231 to Dt23N (St33).

[0078] The learning unit 111 calculates the degree of agreement between the transformed non-contact feature point Dt22 and each of the contact feature points Dt241 to Dt24N for each combination of non-contact and contact feature points. The learning unit 111 updates the learning model MD (St34) by performing learning with all the degree of agreement calculated in step St30 (i.e., N learning processes) as the loss.

[0079] For example, in the example shown in Figure 6, the training data Dt20 contains information for N pairs of training data. The training unit 111 uses a non-contact image with the file name "001-LEFT-INDEX-CL.png" associated with the biological parts "LEFT (hand)" and "INDEX (finger)" of person ID "001", a contact image with the file name "001-LEFT-INDEX-C.png", or contact feature point information with the file name "001-LEFT-INDEX-C.feat" as the first pair of training data to perform the first training process (processing in steps St31 to St34). Furthermore, the learning unit 111 uses a non-contact image with the filename "002-RIGHT-RING-CL.png" associated with the biological parts "RIGHT (hand)" and "RING (finger)" of person ID "002", a contact image with the filename "002-RIGHT-RING-C.png", or contact feature point information with the filename "002-RIGHT-RING-C.feat", as the second pair of learning data to execute the second learning process (processing in steps St31 to St34). The learning unit 111 repeats this process to execute the Nth learning process (processing in steps St31 to St34).

[0080] The learning unit 111 determines, based on all the matching scores calculated in step St30 (i.e., N learning processes), whether the matching score satisfies predetermined conditions and whether the learning model MD is capable of generating non-contact feature points Dt22 (St35). Note that the predetermined conditions here are the same as those described in the generation of the first trained model, so the explanation is omitted.

[0081] If the learning unit 111 determines in step St35 that the degree of agreement satisfies a predetermined condition (St35, YES), it saves the updated learning model MD as the second learned model in the learned model storage unit 14 (St36).

[0082] On the other hand, if the learning unit 111 determines in step St35 that the degree of agreement does not meet the predetermined conditions (St35, NO), it returns to the process of step St30 and re-executes the processes of steps St31 to St34 using all pairs or sets of learning data included in the learning data Dt20, thereby updating the learning model MD again.

[0083] As described above, the information processing device S1 can generate and acquire a second trained model that can generate non-contact feature points (transformed non-contact feature points) by changing the state of the biological parts shown in the non-contact image acquired in a non-contact state to a state corresponding to the state of contact feature points acquired and generated in a contact state (for example, changes in feature points or changes in the positional relationship between feature points). Furthermore, since the second trained model of this disclosure is trained using biometric data from multiple people, it can correspond to the 3D shapes of various users' biological parts (for example, fingers) and can infer biometric data corresponding to various contact states (for example, pressure, direction of pressure, and location of pressure).

[0084] In other words, even if the three-dimensional shape of a biological part differs depending on the type of person or biological part, and deformation of the biological part occurs due to arbitrary pressure applied to the biological part during registration, the information processing device S1 can use a second trained model to convert the biometric data (non-contact image) acquired during authentication into biometric data (non-contact feature points) that corresponds to the state of the biometric data (contact feature points) registered during registration. Therefore, the information processing device S1 can improve the authentication accuracy of biometric authentication using biometric data acquired by different acquisition methods.

[0085] <Method for generating a pre-trained model (3)> Next, the method for generating the third pre-trained model will be explained. The information processing device S1 generates non-contact feature points from each non-contact image and contact feature points from each contact image. The information processing device S1 generates a third pre-trained model for transforming the state of the generated non-contact feature points so that they correspond to the state of the generated contact feature points.

[0086] First, referring to Figure 2, we will explain how to generate the third pre-trained model, that is, how to generate the training data Dt30 (see Figure 8) used to train the trained model MD. Note that steps St11 to St12 may be repeated based on the number of training data pairs contained in the training data Dt30.

[0087] The processor 11 acquires at least one contact image Dt331 to Dt33N acquired by the first data acquisition device P1 and a non-contact image (not shown) acquired by the second data acquisition device P2 (St11). The timing of image acquisition by each data acquisition device may be at any arbitrary timing.

[0088] When the processor 11 generates a third pre-trained model, it performs image analysis on the non-contact image (not shown) and the contact images Dt331 to Dt33N, respectively, and generates information on fingerprint feature points (non-contact feature points and contact feature points, respectively) from the non-contact image (not shown) and the contact images Dt331 to Dt33N (St13).

[0089] The processor 11 stores the non-contact feature point Dt31 and the contact feature points Dt341~Dt34N, which are feature points of the same person, as a pair of training data in the training data storage unit 13 (St14).

[0090] Based on the above, the information processing device S1 generates learning data Dt30 containing information on multiple pairs or sets of learning data. The learning data Dt30 stored in the learning data storage unit 13 is managed as a single pair of learning data, with each pair linked to a person ID that can identify the owner of the biological part shown in the non-contact image and the contact image, information about the biological part shown in the non-contact image and the contact image, the file name of the non-contact image, and the file name of the contact image. This is an example, but is not limited to this.

[0091] The training data used to generate the third trained model may be generated using the same procedure as for the first trained model; for example, it is sufficient if the file names of the non-contact images and the file names of the contact images are linked. In such cases, the learning unit 111 executes the generation process of non-contact feature points and contact feature points in the flowchart shown in Figure 7.

[0092] Furthermore, the training data used to generate the third trained model may be linked as shown in Figure 8, with the file names of non-contact images, non-contact feature point information, contact images, and contact feature point information being associated with each other.

[0093] Next, with reference to Figures 7 and 8, we will explain the method for generating the third pre-trained model, that is, the learning method using the trained model MD. Figure 7 is a flowchart showing the procedure for generating the third pre-trained model of the information processing device S1. Figure 8 is a diagram illustrating an example of generating the third pre-trained model.

[0094] The learning unit 111 of the information processing device S1 performs learning (St40) using all pairs of learning data or all sets of learning data that are stored in the learning data storage unit 13 and have been set in advance as learning data Dt30 to be used for learning. Specifically, if the learning data storage unit 13 has N pairs of learning data or M sets of learning data, the learning unit 111 performs the processing in steps St41 to St44 for each of the N pairs of learning data or M sets of learning data. In other words, the learning unit 111 repeatedly performs the processing in steps St41 to St44 a number of times corresponding to the number of pairs or sets of learning data. Here, we will explain an example of learning processing using learning data Dt20 which contains N pairs of learning data.

[0095] The learning unit 111 acquires a non-contact image (not shown) of a biological part and at least one contact image Dt331 to Dt33N associated with this non-contact image (not shown) from the learning data Dt30 stored in the learning data storage unit 13 (St41).

[0096] The feature point data generation unit 112 generates non-contact feature points Dt32 of the fingerprint from the non-contact image (not shown) and contact feature points Dt341 to Dt34N of the fingerprint from the contact images Dt331 to Dt33N, respectively (St42). Note that the processing in step St42 may be executed only if the training data Dt30 does not contain non-contact feature point information or contact point information.

[0097] The learning unit 111 uses the learning model MD to convert the non-contact feature points Dt31 into non-contact feature points Dt32 that correspond to the state of the contact feature points Dt341~Dt34N (for example, the 3D shape of the biological part estimated from the contact feature points, the way it is deformed, or the way the feature points change, etc.) (St43).

[0098] The learning unit 111 calculates the degree of agreement between the transformed non-contact feature point Dt32 and each of the contact feature points Dt341 to Dt34N for each combination of non-contact and contact feature points. The learning unit 111 updates the learning model MD by performing learning with the calculated degree of agreement as the loss (St44).

[0099] For example, in the example shown in Figure 8, the training data Dt30 contains information for N pairs of training data. The training unit 111 uses a non-contact image with the filename "001-LEFT-INDEX-CL.png" associated with the biological parts "LEFT (hand)" and "INDEX (finger)" of person ID "001," and a contact image with the filename "001-LEFT-INDEX-C.png" as the first pair of training data to perform the first training process (processing in steps St41 to St43). Then, using a non-contact image with the filename "002-RIGHT-RING-CL.png" associated with the biological parts "RIGHT (hand)" and "RING (finger)" of person ID "002," and a contact image with the filename "002-RIGHT-RING-C.png" as the second pair of training data to perform the second training process (processing in steps St41 to St43). The learning unit 111 repeats this process and executes the Nth learning process (the processes from step St41 to step St43).

[0100] The learning unit 111 determines, based on all the matching scores calculated in step St40 (i.e., N learning processes), whether the matching score satisfies predetermined conditions and whether the learning model MD is capable of generating non-contact feature points Dt32 (St45). Note that the predetermined conditions here are the same as those described in the generation of the first trained model, so the explanation is omitted.

[0101] If the learning unit 111 determines in step St45 that the degree of agreement satisfies a predetermined condition (St45, YES), it saves the updated learning model MD as the third learned model in the learned model storage unit 14 (St46).

[0102] On the other hand, if the learning unit 111 determines in step St45 that the degree of agreement does not meet the predetermined conditions (St45, NO), it returns to the process of step St40 and re-executes the process of steps St41 to St44 using all pairs or sets of learning data included in the learning data Dt30, thereby updating the learning model MD again.

[0103] As described above, the information processing device S1 can generate and acquire a third pre-trained model that can generate non-contact feature points (transformed non-contact feature points) that are obtained by changing the state of the biological part indicated by non-contact feature points obtained in a non-contact state to a state corresponding to the state of the biological part indicated by contact feature points obtained in a contact state (for example, the way the biological part is crushed, the way the feature points or features change, etc.). Furthermore, since the third pre-trained model of this disclosure is trained using biometric data from multiple people, it can correspond to the 3D shapes of various users' biological parts (for example, fingers) and can infer biometric data corresponding to various contact states (for example, pressing force, direction of pressure, and location of pressure).

[0104] In other words, even if the three-dimensional shape of a biological part differs depending on the type of person or biological part, and deformation of the biological part occurs due to arbitrary pressure applied to the biological part during registration, the information processing device S1 can use a third pre-trained model to convert the biometric data (non-contact feature points) acquired during authentication into biometric data (non-contact feature points) that corresponds to the state of the biometric data (contact feature points) registered during registration. Therefore, the information processing device S1 can improve the authentication accuracy of biometric authentication using biometric data acquired by different acquisition methods.

[0105] <Example of a second use case> The above explanation described the generation methods for the first to third trained models. Therefore, the following explanation will refer to Figures 9 and 10 to describe the overall configuration of the biometric data conversion model generation system 100A and the method for inferring (generating) contactless images or contactless feature points used during authentication, using the generated first to third trained models.

[0106] Figure 9 is a block diagram showing a second use case example of the biodata conversion model generation system 100A according to the embodiment. Note that the overall configuration of the biodata conversion model generation system 100A shown in Figure 9 is just one example and is not limited thereto.

[0107] For example, the bio-data conversion model generation system 100A may be realized by replacing the processor 11 of the information processing device S1 shown in Figure 1 with the processor 11A of the information processing device S1A shown in Figure 9, and configuring the information processing device S1 shown in Figure 1 to be able to perform the functions of the processor 11A. In the description of the bio-data conversion model generation system 100A shown in Figure 9, the same configuration and functions as the bio-data conversion model generation system 100 shown in Figure 1 are assigned the same reference numerals, and the explanation is omitted.

[0108] The biometric data conversion model generation system 100A includes a second data acquisition device P2 capable of acquiring biometric data used for authentication, and an information processing device S1A that infers (generates) biometric data used for authentication, which has features of a biological part similar to or matching the biometric data at the time of registration, using a pre-generated trained model (at least one trained model from the first to third trained models). The second data acquisition device P2 may be configured integrally with the information processing device S1A.

[0109] The information processing device S1A is connected to the second data acquisition device P2 in a data communication manner. The information processing device S1A can be implemented as a PC, notebook PC, tablet terminal, cloud server, or on-premise server. The information processing device S1A includes a communication unit 10, a processor 11A, a memory 12, a trained model storage unit 14, a contactless data acquisition unit 15, and a registered data storage unit 16. Note that the contactless data acquisition unit 15 and the registered data storage unit 16 are not mandatory and may be omitted.

[0110] The processor 11A is configured using, for example, a CPU, FPGA, or GPU, and works in cooperation with the memory 12 to perform various processing and control. Specifically, the processor 11A refers to the programs and data held in the memory 12 and executes those programs to realize functions such as the feature point data generation unit 112, the inference unit 113, the registration unit 114, or the authentication unit 115, and performs biometric data inference. Note that the feature point data generation unit 112, the registration unit 114, and the authentication unit 115 are not mandatory configurations and may be omitted or realized by external devices that are communicatively connected to the information processing device S1A.

[0111] The inference unit 113 uses the first trained model, second trained model, or third trained model, which have been generated and stored (stored) in the trained model storage unit 14, to infer biometric data corresponding to the method of acquiring biometric data at the time of registration from the biometric data acquired by the second data acquisition device P2 or the non-contact data acquisition unit 15.

[0112] The registration unit 114 functions when registering biometric data (contactless data) acquired by the second data acquisition device P2 or the contactless data acquisition unit 15 as registration biometric data to be compared with the biometric data acquired at the time of authentication. The registration unit 114 links the biometric data generated (inferred) by the inference unit 113 as registration biometric data with the information of the person and biological part corresponding to this biometric data and stores it in the registration data storage unit 16.

[0113] The authentication unit 115 functions when performing biometric authentication using biometric data (contactless data) acquired by the second data acquisition device P2 or the contactless data acquisition unit 15. The authentication unit 115 outputs the biometric data generated (inferred) by the inference unit 113 as biometric data for authentication to an application that performs authentication processing, and performs biometric authentication of the person corresponding to the biometric data for authentication by comparing it with the biometric data stored in the registration data storage unit 16.

[0114] The contactless data acquisition unit 15 is provided when the biometric data used for authentication is acquired by the information processing device S1A itself, rather than by the second data acquisition device P2. The contactless data acquisition unit 15 is implemented, for example, by a visible light camera. The contactless data acquisition unit 15 outputs the captured contactless image (biometric data) to the processor 11A.

[0115] The registration data storage unit 16 is used for biometric authentication and stores (saves) biometric data registered before authentication is performed, linked to information about the person and body part that owns the biometric data. The registration data storage unit 16 is configured separately from the information processing device S1A and may be implemented by an external storage medium capable of data communication with the information processing device S1A.

[0116] <Inference method using the first pre-trained model> Referring to Figure 10, the method for inferring biological data using the first trained model will be explained. In the flowchart shown in Figure 10, as an example, the biological data acquired by the second data acquisition device P2 or the non-contact data acquisition unit 15 is a non-contact image, and the first trained model converts it to generate (infer) a non-contact image corresponding to the state of the biological part shown in the contact image.

[0117] The inference unit 113 acquires the non-contact image (biometric data) acquired by the second data acquisition device P2 (St51).

[0118] The inference unit 113 uses the first trained model to transform the acquired contactless image (St52) and outputs the transformed contactless image (biometric data) to the registration unit 114 or the authentication unit 115 (St53). Specifically, if the inference unit 113 wants to store the transformed contactless image in the registration data storage unit 16, it outputs the transformed contactless image to the registration unit 114, and if it wants to perform biometric authentication using the transformed contactless image, it outputs the transformed contactless image to the authentication unit 115.

[0119] The registration unit 114 adds or overwrites the converted contactless image as a contact image to be registered in advance for biometric authentication (St54). The authentication unit 115 then inputs the converted contactless image into an application or the like that performs the authentication process and performs biometric authentication (St54).

[0120] As described above, the information processing device S1A can use the first trained model to transform non-contact images, thereby inferring (generating) a non-contact image equivalent to a contact image of a fingerprint taken while in contact with any object, based on a non-contact image of a fingerprint taken while in contact with any object. This allows the information processing device S1A to use the first trained model to transform non-contact images acquired in a different way from pre-registered biometric data (contact images), and add or overwrite the transformed non-contact image as a new contact image for registration. Furthermore, the information processing device S1A can also perform biometric authentication using the non-contact image acquired for authentication and the newly registered biometric data (i.e., the non-contact image transformed using the first trained model).

[0121] <Inference method using the second pre-trained model> Referring to Figure 10, the method for inferring biometric data using the second trained model will be explained. In the flowchart shown in Figure 10, as an example, the biometric data acquired by the second data acquisition device P2 or the non-contact data acquisition unit 15 is a non-contact image, and the second trained model generates (infers) non-contact feature points corresponding to the state of contact feature points through transformation.

[0122] The inference unit 113 acquires non-contact images (biometric data) obtained by the second data acquisition device P2 (St61).

[0123] The inference unit 113 uses a second trained model to convert the acquired non-contact image into non-contact feature points corresponding to contact feature points (St63), and outputs the converted non-contact feature points (biometric data) to the registration unit 114 or the authentication unit 115 (St64). Specifically, if the inference unit 113 stores the converted non-contact feature points in the registration data storage unit 16, it outputs the converted non-contact feature points to the registration unit 114, and if it performs biometric authentication using the converted non-contact feature points, it outputs the converted non-contact feature points to the authentication unit 115.

[0124] The registration unit 114 adds or overwrites the converted contactless feature points as contact feature points that are pre-registered in biometric authentication (St65). The authentication unit 115 then inputs the converted contactless feature points into an application that performs authentication processing and performs biometric authentication (St65).

[0125] As described above, the information processing device S1A can infer (generate) non-contact feature points (biometric data) that correspond to the state of contact feature points generated from a fingerprint in contact with any object, based on a non-contact image in which a fingerprint has been captured without contact with any object. This allows the information processing device S1A to use the second trained model to transform non-contact feature points, thereby transforming a non-contact image acquired in a different way than the previously registered biometric data (contact feature points), and add or overwrite the transformed non-contact feature points as new contact feature points for registration. Furthermore, the information processing device S1A can also perform biometric authentication using the non-contact image acquired for authentication and the newly registered biometric data (i.e., non-contact feature points transformed using the second trained model).

[0126] <Inference method using a pre-trained model (3)> Referring to Figure 10, the method for inferring biological data using the third pre-trained model will be explained. In the flowchart shown in Figure 10, as an example, the biological data acquired by the second data acquisition device P2 or the non-contact data acquisition unit 15 is a non-contact image, and the third pre-trained model generates (infers) non-contact feature points corresponding to the state of feature points of biological parts included in the contact image through transformation.

[0127] The inference unit 113 acquires non-contact images (biometric data) obtained by the second data acquisition device P2 (St61).

[0128] The feature point data generation unit 112 generates non-contact feature points of fingerprints from the acquired non-contact image using the second trained model (St62).

[0129] The inference unit 113 uses a second trained model to convert the acquired non-contact feature points into non-contact feature points corresponding to the contact feature points (St63), and outputs the converted non-contact feature points (biometric data) to the registration unit 114 or the authentication unit 115 (St64). Specifically, if the inference unit 113 stores the converted non-contact feature points in the registration data storage unit 16, it outputs the converted non-contact feature points to the registration unit 114, and if it performs biometric authentication using the converted non-contact feature points, it outputs the converted non-contact feature points to the authentication unit 115.

[0130] The registration unit 114 adds or overwrites the converted contactless feature points as contact feature points that are pre-registered in biometric authentication (St65). The authentication unit 115 then inputs the converted contactless feature points into an application that performs authentication processing and performs biometric authentication (St65).

[0131] As described above, the information processing device S1A can infer (generate) non-contact feature points (biometric data) that correspond to the contact feature points of a fingerprint in a contact state when it is in contact with an arbitrary object, based on the non-contact feature points of a fingerprint in a contact state when it is in contact with an arbitrary object, by transforming the non-contact feature points using the third pre-trained model. In this way, the information processing device S1A can use the third pre-trained model to transform non-contact feature points acquired in a different way from the previously registered biometric data (contact feature points), and add or overwrite the transformed non-contact image as new contact feature points for registration. Furthermore, the information processing device S1A can also perform biometric authentication using the non-contact feature points acquired for authentication and the newly registered biometric data (i.e., non-contact feature points transformed using the third pre-trained model).

[0132] (Note) Based on the descriptions of the embodiments described above, the following technologies are disclosed.

[0133] (Technology 1) An acquisition unit (communication unit 10) acquires one or more first biometric data (contact data, contact image, or contact feature points) of multiple biological parts of a person to be authenticated acquired by a first acquisition method (contact state), and second biometric data (non-contact data, non-contact image, or non-contact feature points) of the same biological parts acquired by a second acquisition method (non-contact state) different from the first acquisition method (contact state), A data conversion unit (learning unit 111) uses a learning model MD to convert the second biological data into biological data corresponding to the first biological data acquired by the first acquisition method (contact state), A calculation unit (learning unit 111) calculates the degree of agreement between the characteristics of the biological site included in the first biological data and the characteristics of the biological site included in the converted second biological data, The system includes a control unit (learning unit 111) that, if it is determined that the calculated degree of agreement does not meet predetermined conditions, performs learning on the learning model with the degree of agreement as the loss, and if it is determined that the degree of agreement meets predetermined conditions, outputs the learning model MD as a trained model (first trained model, second trained model, or third trained model) that converts the second biological data into biological data acquired by the first acquisition method (contact state), A device for generating biometric data conversion models (information processing device S1). As described above, the information processing device S1 can generate and acquire a trained model that can generate second biometric data in which the state of the biological part indicated by the second biometric data acquired by the second acquisition method corresponds to the state of the biological part indicated by the first biometric data acquired by the first acquisition method (for example, the way the biological part is deformed, the way feature points or features change, the change in the positional relationship between feature points, etc.). Therefore, even if the three-dimensional shape of the biological part differs for each type of person or biological part, and deformation of the biological part occurs due to arbitrary pressure applied to the biological part during registration, the information processing device S1 can improve the authentication accuracy of biometric authentication using biometric data acquired by different acquisition methods by using the trained model.

[0134] (Technology 2) The first acquisition method (contact state) is a method in which biometric data of the body part of the person to be authenticated, which is registered in advance, is acquired in biometric authentication. The second acquisition method (contactless state) is a method in which biometric data of the biometric body part of the person being authenticated is acquired in the biometric authentication. A device for generating the biometric data conversion model described in (Technology 1) (information processing device S1). As described above, the information processing device S1 can generate a trained model that can convert biometric data acquired by the authentication acquisition method (second acquisition method) into data equivalent to biometric data acquired (imaged) by the registration acquisition method (first acquisition method).

[0135] (Technology 3) The first acquisition method is a method in which biological data of a biological part is acquired in a contact state in which the biological part is in contact with an arbitrary object, The second acquisition method is a method in which biological data of the biological part is acquired in a non-contact state in which the biological part does not come into contact with the object. A device for generating a bio-data conversion model (information processing device S1) as described in (Technology 1) or (Technology 2). As described above, the information processing device S1 can generate a trained model that can convert biological data acquired when the biological part is not in contact (second acquisition method) into data equivalent to biological data acquired (imaged) when the biological part is in contact (first acquisition method).

[0136] (Technology 4) The first biological data and the second biological data are each biological data from which the biological body part has been imaged. A device for generating a bio-data conversion model (information processing device S1) described in any one of (Technology 1) to (Technology 3). As described above, the information processing device S1 can acquire a trained model (first trained model) that converts a non-contact image acquired (imaged) by the second acquisition method into image data equivalent to a contact image acquired (imaged) by the first acquisition method, and that can convert a biological part visible in the image data acquired (imaged) by the second acquisition method into image data equivalent to a contact image acquired (imaged) by the first acquisition method.

[0137] (Technology 5) The first biological data and the second biological data are each characteristic point information of the biological site, A device for generating a bio-data conversion model (information processing device S1) described in any one of (Technology 1) to (Technology 3). As described above, the information processing device S1 can acquire a trained model (third trained model) that converts non-contact feature points acquired by the second acquisition method into data corresponding to contact feature points acquired by the first acquisition method, and a trained model (third trained model) capable of converting non-contact feature points acquired by the second acquisition method into feature points corresponding to contact feature points acquired (imaged) by the first acquisition method.

[0138] (Technology 6) The system further includes a feature point data generation unit 112 that generates first feature point information of the biological part based on the first biological data, The data conversion unit (learning unit 111) uses the learning model MD2 to convert the second biological data into second feature point information corresponding to the first feature point information based on the first biological data acquired by the first acquisition method (contact state), The calculation unit (processor 11, 11A) calculates the degree of agreement between the first feature point information and the converted second feature point information. A device for generating a biodata conversion model (information processing device S1) described in any one of (Technology 1) to (Technology 3). As described above, the information processing device S1 can acquire a trained model (second trained model) that converts a non-contact image acquired (imaged) by the second acquisition method into data corresponding to contact feature points generated from a contact image acquired by the first acquisition method, and a trained model (second trained model) that can convert a non-contact image acquired by the second acquisition method into feature point information corresponding to contact feature points acquired (imaged) by the first acquisition method.

[0139] (Technology 7) The control unit (learning unit 111) determines, when there is only one first biological data, whether the degree of agreement between the first biological data and the second biological data is greater than or equal to a predetermined value. A device for generating a biodata conversion model (information processing device S1) as described in any one of (Technology 1) to (Technology 6). As described above, when the information processing device S1 uses one biometric data (non-contact data) and one biometric data (contact data) as training data, by setting a higher value for the degree of agreement, it is possible to generate a trained model that can achieve high reproducibility and higher conversion accuracy in the conversion (inference) process that converts the input biometric data (non-contact data) into biometric data (non-contact data) that has features corresponding to the biometric data (contact data).

[0140] (Technology 8) The control unit (learning unit 111), when there are multiple first biological data, calculates the degree of agreement between each of the first biological data and the second biological data, and determines whether the average value of the multiple calculated degrees of agreement is greater than or equal to a predetermined value. A device for generating a biodata conversion model (information processing device S1) as described in any one of (Technology 1) to (Technology 6). As described above, when the information processing device S1 uses one biometric data (non-contact data) and multiple biometric data (contact data) as training data, it can generate a trained model that enables transformation (inference) with higher consistency and higher stability for various biometric data (non-contact data) in the transformation (inference) process of the biometric data (non-contact data) described above.

[0141] (Technology 9) A method for generating a biometric data conversion model performed by one or more processors (processors 11, 11A), The system acquires one or more first biometric data (contact data, contact images, or contact feature points) of multiple biological parts of a person to be authenticated, acquired by a first acquisition method (contact state), and second biometric data (non-contact data, non-contact images, or non-contact feature points) of the same biological parts, acquired by a second acquisition method (non-contact state) different from the first acquisition method (contact state). Using the learning model MD, the second biometric data is converted into biometric data corresponding to the first biometric data acquired by the first acquisition method (contact state). The degree of agreement between the characteristics of the biological region included in the first biological data and the characteristics of the biological region included in the converted second biological data is calculated. If it is determined that the calculated degree of agreement does not meet the predetermined conditions, the learning model is trained using the degree of agreement as the loss. If it is determined that the calculated degree of agreement satisfies predetermined conditions, the learning model MD is output as a trained model (first trained model, second trained model, or third trained model) that converts the second biological data into biological data acquired by the first acquisition method (contact state). Method for generating biometric data conversion models. As described above, the information processing device S1 can generate and acquire a trained model that can generate second biometric data in which the state of the biological part indicated by the second biometric data acquired by the second acquisition method corresponds to the state of the biological part indicated by the first biometric data acquired by the first acquisition method (for example, the way the biological part is deformed, the way feature points or features change, the change in the positional relationship between feature points, etc.). Therefore, even if the three-dimensional shape of the biological part differs for each type of person or biological part, and deformation of the biological part occurs due to arbitrary pressure applied to the biological part during registration, the information processing device S1 can improve the authentication accuracy of biometric authentication using biometric data acquired by different acquisition methods by using the trained model.

[0142] (Technology 10) A first acquisition device (first data acquisition device P1) acquires first biometric data (contact data, contact image, or contact feature points) of multiple biometric body parts of multiple authenticated individuals using a first acquisition method (contact state), A second acquisition device (second data acquisition device P2) acquires second biological data (non-contact data, non-contact image, or non-contact feature points) of the biological part acquired by a second acquisition method (non-contact state) that is different from the first acquisition method (contact state), A bio-data conversion model generation system (biological data conversion model generation system 100) comprising a conversion model generation device (information processing device S1) capable of communicating with the first acquisition device (first data acquisition device P1) and the second acquisition device (second data acquisition device P2), The first acquisition device (first data acquisition device P1) is, One or more of the first biological data are acquired and transmitted to the conversion model generation device (information processing device S1), The second acquisition device (second data acquisition device P2) is, The second biological data is acquired and transmitted to the conversion model generation device (information processing device S1), The aforementioned conversion model generation device (information processing device S1) Using the learning model MD, the second biometric data is converted into biometric data corresponding to the first biometric data acquired by the first acquisition method (contact state). The degree of agreement between the characteristics of the biological region included in the first biological data and the characteristics of the biological region included in the converted second biological data is calculated. If it is determined that the calculated degree of agreement does not meet the predetermined conditions, the learning model is trained using the degree of agreement as the loss. If it is determined that the calculated degree of agreement satisfies predetermined conditions, the learning model MD is output as a trained model (first trained model, second trained model, or third trained model) that converts the second biological data into biological data acquired by the first acquisition method (contact state). A system for generating biometric data conversion models (biometric data conversion model generation system 100). As described above, the biometric data conversion model generation system 100 can generate and acquire a trained model capable of generating second biometric data in which the state of the biological part indicated by the second biometric data acquired by the second acquisition method corresponds to the state of the biological part indicated by the first biometric data acquired by the first acquisition method (for example, the degree of deformation of the biological part, the way of feature points or changes in features, changes in the positional relationship between feature points, etc.). Therefore, even if the three-dimensional shape of the biological part differs for each type of person or biological part, and deformation of the biological part occurs due to arbitrary pressure applied to the biological part during registration, the information processing device S1 can improve the authentication accuracy of biometric authentication using biometric data acquired by different acquisition methods by using the trained model.

[0143] Although various embodiments have been described above with reference to the attached drawings, this disclosure is not limited to such examples. It will be clear to those skilled in the art that various modifications, alterations, substitutions, additions, deletions, and equivalents can be conceived within the scope of the claims, and these will also be understood to fall within the technical scope of this disclosure. Furthermore, the components of the various embodiments described above can be combined arbitrarily without departing from the spirit of the invention. [Industrial applicability]

[0144] This disclosure is useful as a presentation of a fingerprint data conversion model generation apparatus, a fingerprint data conversion model generation method, and a fingerprint data conversion model generation system, which generate a trained model that converts fingerprint data obtained by a different acquisition method than the one used at the time of registration into fingerprint data that can be authenticated. [Explanation of symbols]

[0145] 10, 20, 30 Communications Department 11,11A,21,31 processors 12,22,32 memory 13. Learning Data Storage Unit 14 Model Memory Unit 15. Contactless data acquisition unit 16. Registered Data Storage Unit 23 Contact data acquisition unit 33 Contactless data acquisition unit 100,100A Bio-data Conversion Model Generation System 111 Learning Department 112 Feature Point Data Generation Unit Dt10, Dt20, Dt30 Training Data Dt11, Dt12, Dt21 Non-contact images Contact images of Dt131, Dt13N, Dt231, Dt23N, Dt331, Dt33N Dt241, Dt24N, Dt341, Dt34N Contact feature points Dt22, Dt31, Dt32 Non-contact feature points MD Learning Model P1 First data acquisition device P2 Second data acquisition device S1, S1A Information Processing Device

Claims

1. An acquisition unit that acquires an image of first fingerprint data of multiple authenticated persons' fingertips acquired by a first acquisition method, and an image of second fingerprint data of the same fingertips acquired by a second acquisition method different from the first acquisition method, A data conversion unit that uses a learning model to convert the captured image of the second fingerprint data into image data corresponding to the captured image of the first fingerprint data acquired by the first acquisition method, A calculation unit that calculates the degree of agreement between the fingertip features included in the captured image of the first fingerprint data and the fingertip features included in the captured image of the second fingerprint data after conversion, The system includes a control unit that, if it is determined that the calculated degree of agreement does not meet predetermined conditions, performs training on the learning model with the degree of agreement as the loss, and if it is determined that the degree of agreement meets predetermined conditions, outputs the learning model as a trained model that converts the captured image of the second fingerprint data into the image data acquired by the first acquisition method. A device for generating fingerprint data conversion models.

2. The first acquisition method is a method in which, in fingerprint authentication, the fingerprint data of the fingertip of the person to be authenticated, which is registered in advance, is acquired. The second acquisition method is a method in which, in the fingerprint authentication, fingerprint data of the fingertip of the person to be authenticated is acquired. A device for generating a fingerprint data conversion model according to claim 1.

3. The first acquisition method is a method in which fingerprint data of the fingertip is acquired in a contact state when the fingertip is in contact with an arbitrary object. The second acquisition method is a method in which fingerprint data of the fingertip is acquired in a non-contact state in which the fingertip does not come into contact with the object. A device for generating a fingerprint data conversion model according to claim 1.

4. The captured image of the first fingerprint data and the captured image of the second fingerprint data include the characteristic point information of the fingertip. A device for generating a fingerprint data conversion model according to claim 1.

5. The system further comprises a feature point data generation unit that generates first feature point information of the fingertip based on the captured image of the first fingerprint data, The data conversion unit uses the learning model to convert the captured image of the second fingerprint data into second feature point information corresponding to the first feature point information based on the captured image of the first fingerprint data acquired by the first acquisition method, The calculation unit calculates the degree of agreement between the first feature point information and the converted second feature point information. A device for generating a fingerprint data conversion model according to claim 1.

6. The control unit determines, if there is only one captured image of the first fingerprint data, whether the degree of agreement between the captured image of the first fingerprint data and the captured image of the second fingerprint data is greater than or equal to a predetermined value. A device for generating a fingerprint data conversion model according to claim 1.

7. If there are multiple captured images of the first fingerprint data, the control unit calculates the degree of agreement between each of the captured images of the first fingerprint data and the captured image of the second fingerprint data, and determines whether the average value of the multiple calculated degrees of agreement is greater than or equal to a predetermined value. A device for generating a fingerprint data conversion model according to claim 1.

8. A method for generating a fingerprint data conversion model performed by one or more processors, The system acquires an image of one or more first fingerprint data of the fingertips of multiple authenticated persons acquired by the first acquisition method, and an image of the second fingerprint data of the same fingertips acquired by a second acquisition method different from the first acquisition method. Using a learning model, the captured image of the second fingerprint data is converted into image data corresponding to the captured image of the first fingerprint data acquired by the first acquisition method. The degree of agreement between the fingertip features included in the captured image of the first fingerprint data and the fingertip features included in the captured image of the converted second fingerprint data is calculated. If it is determined that the calculated degree of agreement does not meet the predetermined conditions, the learning model is trained using the degree of agreement as the loss. If it is determined that the calculated degree of agreement satisfies predetermined conditions, the learning model is output as a trained model that converts the captured image of the second fingerprint data into the image data obtained by the first acquisition method. Method for generating a fingerprint data conversion model.

9. A first acquisition device that acquires captured images of first fingerprint data from the fingertips of multiple authenticated persons using a first acquisition method, A second acquisition device for acquiring an image of the second fingerprint data of the fingertip acquired by a second acquisition method different from the first acquisition method, A fingerprint data conversion model generation system comprising a conversion model generation device capable of communicating between the first acquisition device and the second acquisition device, The first acquisition device is The captured images of one or more of the first fingerprint data are acquired and transmitted to the conversion model generation device. The second acquisition device is The captured image of the second fingerprint data is acquired and transmitted to the conversion model generation device. The aforementioned conversion model generation device is Using a learning model, the second fingerprint data is converted into image data corresponding to the captured image of the first fingerprint data obtained by the first acquisition method. The degree of agreement between the fingertip features included in the captured image of the first fingerprint data and the fingertip features included in the captured image of the converted second fingerprint data is calculated. If it is determined that the calculated degree of agreement does not meet the predetermined conditions, the learning model is trained using the degree of agreement as the loss. If it is determined that the calculated degree of agreement satisfies predetermined conditions, the learning model is output as a trained model that converts the captured image of the second fingerprint data into the image data obtained by the first acquisition method. A system for generating fingerprint data conversion models.

Images