Training device, biometric-information estimation device, training method, and recording medium

Abstract

In a training device according to the present invention, a first setting means sets, as training content for pre-training of a model, first training content for carrying out training using first training data that includes a plurality of images representing differences and / or changes in at least a part of a human body. A second setting means sets, as training content for transfer training of a model that has been subjected to pre-training, second training content for carrying out training using second training data that includes images in which at least a part of a human body of an actual person has been photographed and labels attached to the images. The training device can contribute to optimization of training of a model.

Description

Learning device, biological information estimation device, learning method, and recording medium

[0001] The present disclosure relates to techniques that can be used to train models.

[0002] A method for extracting biological features using a model trained by machine learning or the like is known.

[0003] Specifically, Patent Document 1 discloses a method for extracting multiple features of a person, such as their complexion, by applying multiple trained models to still or moving images obtained by photographing the person and information about the medical department the person visits.

[0004] Japanese Patent Application Laid-Open No. 2021-196998

[0005] However, the method disclosed in Patent Document 1 has a problem in that, for example, in order to extract multiple features of a person, a learning device (model) must be trained using a large amount of training data corresponding to each of the multiple features.

[0006] One object of the present disclosure is to provide a learning device that can efficiently and effectively construct a model used to estimate human body information.

[0007] In one aspect of the present disclosure, a learning device includes a first setting means for setting a first learning content for learning using first learning data including a plurality of images representing differences and / or changes in at least a portion of a human body as learning content for pre-learning of a model, and a second setting means for setting a second learning content for learning using second learning data including images of at least a portion of a human body of a real person and labels assigned to the images as learning content for transfer learning of the model for which the pre-learning has been performed.

[0008] In another aspect of the present disclosure, a biometric information estimation device includes a first setting means for setting, as learning content for pre-learning of a model, a first learning content for performing learning using first learning data including a plurality of images representing differences and / or changes in at least a portion of a human body; a second setting means for setting, as learning content for transfer learning of the model for which the pre-learning has been performed, a second learning content for performing learning using second learning data including images of at least a portion of a human body of a real person and labels assigned to the images; and an estimation means for performing processing to estimate human body information using the model for which the pre-learning and transfer learning including machine learning have been performed.

[0009] In yet another aspect of the present disclosure, a computer-implemented learning method sets, as learning content for pre-learning of a model, a first learning content for learning using first learning data including a plurality of images showing differences and / or changes in at least a portion of a human body, and sets, as learning content for transfer learning of the model for which the pre-learning has been performed, a second learning content for learning using second learning data including images of at least a portion of a human body of a real person and labels assigned to the images.

[0010] In yet another aspect of the present disclosure, a recording medium records a program that causes a computer to execute a process of setting, as learning content for pre-learning of a model, a first learning content for learning using first learning data including a plurality of images representing differences and / or changes in at least a portion of a human body, and setting, as learning content for transfer learning of the model for which the pre-learning has been performed, a second learning content for learning using second learning data including images of at least a portion of a human body of a real person and labels assigned to the images.

[0011] According to the present disclosure, a model used for estimating human body information can be constructed efficiently and effectively.

[0012] 1 is a block diagram showing an example of the hardware configuration of a learning device according to the present disclosure; 2 is a block diagram showing an example of the functional configuration of a learning device according to the present disclosure; 3 is a diagram showing an example of the configuration of a deep learning model learned by a learning device according to the present disclosure; 4 is a diagram for explaining an example of learning data acquired by a learning device according to the present disclosure; 5 is a diagram for explaining an example of processing performed by a learning device according to the present disclosure; 6 is a flowchart showing an example of processing performed in a learning device according to the present disclosure; 7 is a block diagram showing another example of the functional configuration of a learning device according to the present disclosure; 8 is a flowchart showing another example of processing performed in a learning device according to the present disclosure.

[0013] Hereinafter, preferred embodiments of the present disclosure will be described with reference to the drawings.

[0014] First Embodiment [Hardware Configuration] Fig. 1 is a block diagram showing an example of the hardware configuration of a learning device according to the present disclosure. As shown in Fig. 1, the learning device 100 includes an interface (IF) 111, a processor 112, a memory 113, a recording medium 114, and a database (DB) 115. The learning device 100 can contribute to optimizing model learning.

[0015] The IF 111 inputs and outputs data to and from an external device. For example, learning data used for learning a deep learning model is input to the learning device 100 via the IF 111.

[0016] The processor 112 is a computer such as a CPU (Central Processing Unit), and executes a program prepared in advance to control the entire learning device 100. Specifically, the processor 112 performs processes related to learning of a deep learning model.

[0017] The memory 113 is configured by a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The memory 113 is also used as a working memory while the processor 112 is executing various processes.

[0018] Recording medium 114 is a non-volatile, non-transitory recording medium such as a disk-shaped recording medium or semiconductor memory, and is configured to be detachable from learning device 100. Recording medium 114 records various programs executed by processor 112. When learning device 100 executes various processes, the programs recorded on recording medium 114 are loaded into memory 113 and executed by processor 112.

[0019] DB115 stores, for example, learning data input through IF111 and a learned deep learning model obtained by processing by processor 112.

[0020] [Functional Configuration] Fig. 2 is a block diagram showing an example of the functional configuration of a learning device 100 according to the present disclosure. As shown in Fig. 2, the learning device 100 includes a data acquisition unit 11, a learning processing unit 12, and a learning control unit 13.

[0021] The data acquisition unit 11 functions as a data acquisition means. The data acquisition unit 11 acquires learning data GD in response to a control signal CA output from the learning control unit 13 and outputs the acquired learning data GD to the learning processing unit 12. In response to the control signal CA, the data acquisition unit 11 acquires, for example, learning data GDP from pre-learning or learning data GDT from transfer learning as the learning data GD. When pre-learning is being performed, the data acquisition unit 11 acquires an attribute value ZV corresponding to the learning data GDP and outputs the acquired attribute value ZV to the learning control unit 13. When transfer learning is being performed, the data acquisition unit 11 outputs a label LB assigned to an image IT included in the learning data GDT to the learning control unit 13. Details of the attribute value ZV and the label LB will be described later.

[0022] The learning processing unit 12 has a deep learning model DLM as a model to be learned using the learning data GD. The learning processing unit 12 performs processing related to learning of the deep learning model DLM in response to a control signal CB output from the learning control unit 13. In response to the control signal CB, the learning processing unit 12, for example, inputs the learning data GD to the deep learning model DLM to obtain a feature vector FV corresponding to the extraction result of the features of the learning data GD. The learning processing unit 12 also performs a calculation using the feature vector FV to calculate an estimated variation value EV or an estimated value SV corresponding to the estimation result of the deep learning model DLM. The learning processing unit 12 also outputs the feature vector FV to the learning control unit 13 as necessary. The learning processing unit 12 also outputs the estimated variation value EV or the estimated value SV to the learning control unit 13. The learning processing unit 12 also applies parameters set by the learning control unit 13 to the deep learning model DLM.

[0023] The learning control unit 13 performs processing according to the current learning status of the deep learning model DLM. The learning control unit 13 also sets a learning content LC that includes information related to learning of the deep learning model DLM. The learning control unit 13 sets, for example, a learning content LCP for pre-learning and a learning content LCT for transfer learning as the learning content LC. The learning content LCP includes information indicating, for example, the batch size, the number of iterations, the number of epochs, and the learning strategy. The learning content LCT includes information indicating, for example, the batch size, the number of iterations, the number of epochs, and the learning strategy. A batch corresponds to a subset of multiple data sets. The batch size included in the learning content LCP and the LCT represents the number of data sets in one batch. The number of iterations included in the learning content LCP and the LCT represents the number of batches that make up one epoch. In one epoch of learning, all subsets obtained by dividing multiple data sets by batch size are used for learning. For example, if m subsets divided by batch size are used as training data, m training sessions are performed per epoch. The number of epochs included in the training content LCP represents the number of epochs in pre-training. The number of epochs included in the training content LCT represents the number of epochs in transfer learning. Specific examples of the training policy in the training content LCP will be described later.

[0024] The learning control unit 13 functions as a parameter update unit. The learning control unit 13 generates a control signal CA for acquiring learning data GD according to the learning content LC and outputs the generated control signal CA to the data acquisition unit 11. The learning control unit 13 generates a control signal CB for performing learning according to the learning content LC and outputs the generated control signal CB to the learning processing unit 12. When pre-learning is being performed, the learning control unit 13 calculates a loss LPA using a feature vector FV obtained from the learning processing unit 12. The loss LPA is calculated as a value corresponding to the learning status of facial differences between different people, such as loss LPA1 described below. When pre-learning is being performed, the learning control unit 13 calculates a loss LPB using an attribute value ZV obtained from the data acquisition unit 11 and an estimated variation value EV obtained from the learning processing unit 12. The loss LPB is calculated as a value corresponding to the learning status of facial changes within the same person, such as LPB1 to LPB4 described below. Furthermore, when pre-learning is being performed, the learning control unit 13 acquires parameters PMP corresponding to the losses LPA and LPB and sets the acquired parameters PMP as new parameters of the deep learning model DLM. Furthermore, when transfer learning is being performed, the learning control unit 13 calculates the loss LT using the label LB obtained from the data acquisition unit 11 and the estimated variation value EV or the estimated value SV obtained from the learning processing unit 12. Furthermore, when transfer learning is being performed, the learning control unit 13 acquires parameters PMT corresponding to the loss LT and sets the acquired parameters PMT as new parameters of the deep learning model DLM. Furthermore, the learning control unit 13 includes a learning content setting unit 13A that performs processing related to setting the learning content LC and a parameter storage unit 13B that stores the acquired parameters PMP and PMT. Furthermore, the learning content setting unit 13A functions as a setting unit. Specific examples of processing performed by the learning content setting unit 13A will be described later.

[0025] 3 is a diagram illustrating an example of the configuration of a deep learning model trained by a learning device according to the present disclosure. As shown in FIG. 3, the deep learning model DLM includes a data extension unit 12A, a backbone unit 12B, an output data storage unit 12C, and an estimation processing unit 12D.

[0026] The data extension unit 12A functions as a data extension means, and acquires training data ED by applying data extension to the training data GD, and outputs the acquired training data ED to the backbone unit 12B.

[0027] The backbone unit 12B has a model configured by a neural network such as a CNN (Convolutional Neural Network). According to the present disclosure, the backbone unit 12B preferably has a model usable for image recognition, such as a ResNet (Residual Network) or a Transformer. The backbone unit 12B also receives training data ED according to the training content LC indicated by the control signal CB, acquires a feature vector FV corresponding to the extraction result of the features of the training data ED, and outputs the acquired feature vector FV to the output data storage unit 12C. When pre-training is being performed, the backbone unit 12B sets the parameter PMP obtained by the processing of the learning control unit 13 as a new parameter. When transfer learning is being performed, the backbone unit 12B also sets the parameter PMT obtained by the processing of the learning control unit 13 as a new parameter.

[0028] The output data storage unit 12C stores the feature vector FV output from the backbone unit 12B, and outputs the feature vector FV to the learning control unit 13 as necessary.

[0029] The estimation processing unit 12D is configured with a computing unit such as a multilayer perceptron. Furthermore, the estimation processing unit 12D calculates an estimated variation value EV corresponding to a scalar value indicating the degree of difference between the multiple feature vectors FV read from the output data storage unit 12C in accordance with the learning content LC indicated by the control signal CB, and outputs the calculated estimated variation value EV to the learning control unit 13. Furthermore, the estimation processing unit 12D calculates an estimated value SV by performing processing such as converting the feature vector FV read from the output data storage unit 12C into a scalar value in accordance with the learning content LC indicated by the control signal CB, and outputs the calculated estimated value SV to the learning control unit 13. That is, in the present disclosure, the estimated variation value EV or the estimated value SV calculated by the estimation processing unit 12D can be treated as an estimation result of the deep learning model DLM.

[0030] [Specific Example] Next, a specific example according to the present disclosure will be described. Note that, hereinafter, unless otherwise specified, an example in which learning is performed using images including human faces as training data will be described. Furthermore, in this specific example, at least one of pre-learning and transfer learning may be performed as learning including machine learning.

[0031] (Specific Example of Pre-Learning Process) First, a specific example of pre-learning process will be described.

[0032] The learning content setting unit 13A sets a learning content LCP such that, for example, four images IMA1 to IMA4 containing faces of different people are included in one batch of learning data GDPA. The learning content setting unit 13A also sets a learning content LCP such that, for example, four images IMB1 to IMB4, which are the same person as the person included in each of images IMA1 to IMA4 but with different facial states, are included in one batch of learning data GDPB. The learning content setting unit 13A also generates, as the learning content LCP, information indicating the batch size, number of iterations, number of epochs, and learning policy for pre-learning. The learning content setting unit 13A also generates a control signal CA for acquiring learning data GDPA and GDPB according to the learning content LCP and outputs the generated control signal CA to the data acquisition unit 11.

[0033] According to the above-described process, the learning content setting unit 13A can cause the data acquisition unit 11 to acquire, as learning data GDPA, images IMA1-IMA4 showing facial differences between different people (see FIG. 4). According to the above-described process, the learning content setting unit 13A can cause the data acquisition unit 11 to acquire, as learning data GDPB, images IMB1-IMB4 showing facial differences between different people (see FIG. 4). According to the above-described process, the learning content setting unit 13A can cause the data acquisition unit 11 to acquire, as learning data GDPA and GDPB, images IMA1-IMA4 and IMB1-IMB4 showing facial changes within the same person (see FIG. 4). FIG. 4 is a diagram illustrating an example of learning data acquired by a learning device according to the present disclosure.

[0034] Images IMA1 to IMA4 and images IMB1 to IMB4 can be generated using a generative model such as "StyleGAN." Furthermore, images IMA1 and IMB1 can be generated as images in which, for example, a parameter corresponding to the face transformation parameter "α" in "InterFaceGAN," a face transformation method using "StyleGAN," is set to different values. Furthermore, images IMA2 and IMB2 can be generated as images in which, for example, a parameter corresponding to "α" in "InterFaceGAN" is set to different values. Furthermore, images IMA3 and IMB3 can be generated as images in which, for example, a parameter corresponding to "α" in "InterFaceGAN" is set to different values. Furthermore, images IMA4 and IMB4 can be generated as images in which, for example, a parameter corresponding to "α" in "InterFaceGAN" is set to different values.

[0035] In "InterFaceGAN," for example, by setting a parameter corresponding to "α" to 0, an image can be generated in which a predetermined attribute related to the face of a fictional character is in a standard state. Furthermore, in "InterFaceGAN," for example, by setting a parameter corresponding to "α" to a positive value, an image can be generated in which a predetermined attribute related to the face of a fictional character is enhanced relative to the standard state. Furthermore, in "InterFaceGAN," for example, by setting a parameter corresponding to "α" to a negative value, an image can be generated in which a predetermined attribute related to the face of a fictional character is attenuated relative to the standard state. According to such "InterFaceGAN" processing, for example, an image can be generated in which the facial state changes depending on changes in the weight of the same person. The attributes related to a person's face that change depending on changes in the parameter corresponding to "α" in "InterFaceGAN" can include, for example, at least one of weight, gender, information indicating the degree of edema, and age. The "α" value written near each image in FIG. 4 indicates an example of the setting value when the image is generated using "InterFaceGAN."

[0036] The data acquisition unit 11 acquires learning data GDPA and GDPB according to the learning content LCP indicated by a control signal CA output from the learning control unit 13. The data acquisition unit 11 also outputs the learning data GDPA and GDPB to the learning processing unit 12. According to this processing, the data acquisition unit 11 can acquire a group of multiple images representing differences and changes in faces as learning data for pre-learning.

[0037] The data acquisition unit 11 acquires attribute values ​​ZVA1 to ZVA4 corresponding to each of the images IMA1 to IMA4 included in the learning data GDPA, and outputs the acquired attribute values ​​ZVA1 to ZVA4 to the learning control unit 13. Furthermore, the data acquisition unit 11 acquires attribute values ​​ZVB1 to ZVB4 corresponding to each of the images IMB1 to IMB4 included in the learning data GDPB, and outputs the acquired attribute values ​​ZVB1 to ZVB4 to the learning control unit 13. In this specific example, the data acquisition unit 11 acquires a parameter corresponding to "α" in "InterFaceGAN" as the attribute values ​​ZVA1 to ZVA4 and ZVB1 to ZVB4. Specifically, the data acquisition unit 11 acquires, for example, "0.10" as the attribute value ZVA1 corresponding to image IMA1, "-4.70" as the attribute value ZVA2 corresponding to image IMA2, "-1.30" as the attribute value ZVA3 corresponding to image IMA3, and "-4.20" as the attribute value ZVA4 corresponding to image IMA4 (see FIG. 4). Also, the data acquisition unit 11 acquires, for example, "0.50" as the attribute value ZVB1 corresponding to image IMB1, "-2.00" as the attribute value ZVB2 corresponding to image IMB2, "3.70" as the attribute value ZVB3 corresponding to image IMB3, and "-2.40" as the attribute value ZVB4 corresponding to image IMB4 (see FIG. 4).

[0038] The data expansion unit 12A applies the data expansion DEA to each of the images IMA1 to IMA4 included in the training data GDPA to obtain images EMA1 to EMA4 corresponding to the data-augmented image group (see FIG. 5). The data expansion unit 12A also applies the data expansion DEB to each of the images IMA1 to IMA4 included in the training data GDPA to obtain images EMA5 to EMA8 corresponding to the data-augmented image group (see FIG. 5). The data expansion DEA and DEB include at least one process selected from a plurality of image processing processes, such as cropping, horizontal flipping, color conversion, and grayscale conversion. The data expansion DEA and DEB also include different image processing processes. FIG. 5 is a diagram for explaining an example of processing performed by a learning device according to the present disclosure.

[0039] The data extension unit 12A outputs training data EDA1 including images EMA1 to EMA4 to the backbone unit 12B. The data extension unit 12A also outputs training data EDA2 including images EMA5 to EMA8 to the backbone unit 12B.

[0040] According to the above-described processing, the data extension unit 12A can obtain two mutually different learning data EDA1 and EDA2 by applying the data extensions DEA and DEB to one batch of learning data GDPA.

[0041] The backbone unit 12B acquires feature vectors FVA1 to FVA4 corresponding to the extraction results of features for each of the images EMA1 to EMA4 included in the training data EDA1 in response to the learning content LCP indicated by the control signal CB, and outputs the acquired feature vectors FVA1 to FVA4 to the output data storage unit 12C. The backbone unit 12B also acquires feature vectors FVA5 to FVA8 corresponding to the extraction results of features for each of the images EMA5 to EMA8 included in the training data EDA2 in response to the learning content LCP indicated by the control signal CB, and outputs the acquired feature vectors FVA5 to FVA8 to the output data storage unit 12C. Through this processing, the backbone unit 12B can acquire feature vectors FVA1 to FVA4 and feature vectors FVA5 to FVA8 as two groups of feature vectors corresponding to one batch of training data GDPA.

[0042] The learning control unit 13 calculates the loss LPA1 related to the contrastive learning using the feature vectors FVA1 to FVA4 and feature vectors FVA5 to FVA8 read from the output data storage unit 12C. Specifically, the learning control unit 13 calculates the loss LPA1 having a relatively small value when, for example, the feature vectors between images containing the same person's face are close and the feature vectors between images containing different people's faces are far apart. Furthermore, the learning control unit 13 calculates the loss LPA1 having a relatively large value when, for example, the feature vectors between images containing the same person's face are far apart and the feature vectors between images containing different people's faces are close. In this specific example, the pair of feature vectors FVA1 and FV5, the pair of feature vectors FVA2 and FV6, the pair of feature vectors FVA3 and FV7, and the pair of feature vectors FVA4 and FV8 can be treated as feature vectors between images containing the same person's face. In this specific example, each pair other than the four pairs described above can be treated as a feature vector between images containing faces of different people.

[0043] According to the above-described process, the learning control unit 13 performs processing related to updating the parameters of the backbone unit 12B in pre-learning using "SimCLR," which is one of the contrastive learning methods. Note that, according to this specific example, the learning control unit 13 may update the parameters of the backbone unit 12B using a method other than "SimCLR," as long as the learning control unit 13 performs processing related to contrastive learning in pre-learning.

[0044] The data expansion unit 12A applies the data expansion DEC to each of the images IMB1 to IMB4 included in the training data GDPB to obtain images EMB1 to EMB4 corresponding to the data-expanded image group (see FIG. 5). The data expansion DEC includes at least one process selected from a plurality of processes related to image processing, such as cropping, horizontal flipping, color conversion, and grayscale conversion. Furthermore, it is desirable that the data expansion DEC include a process related to image processing that is different from both the data expansion DEA and the data expansion DEB.

[0045] The data extension unit 12A outputs the training data EDB1 including the images EMB1 to EMB4 to the backbone unit 12B.

[0046] The backbone unit 12B acquires feature vectors FVB1 to FVB4 corresponding to the extraction results of the features of each of the images EMB1 to EMB4 included in the training data EDB1 in accordance with the learning content LCP indicated by the control signal CB, and outputs the acquired feature vectors FVB1 to FVB4 to the output data storage unit 12C. Through this processing, the backbone unit 12B can acquire the feature vectors FVB1 to FVB4 as one feature vector group corresponding to one batch of training data GDPB.

[0047] The estimation processing unit 12D acquires either the feature vectors FVA1 to FVA4 or the feature vectors FVA5 to FVA8, and the feature vectors FVB1 to FVB4, from the output data storage unit 12C in accordance with the learning content LCP indicated by the control signal CB. For ease of explanation, the following description will be given assuming that the estimation processing unit 12D acquires the feature vectors FVA1 to FVA4 and the feature vectors FVB1 to FVB4 from the output data storage unit 12C.

[0048] The estimation processing unit 12D calculates a value indicating the degree of difference between feature vectors FVA1 and FVB1 as an estimated variation value EVA1. The estimation processing unit 12D also calculates a value indicating the degree of difference between feature vectors FVA2 and FVB2 as an estimated variation value EVA2. The estimation processing unit 12D also calculates a value indicating the degree of difference between feature vectors FVA3 and FVB3 as an estimated variation value EVA3. The estimation processing unit 12D also calculates a value indicating the degree of difference between feature vectors FVA4 and FVB4 as an estimated variation value EVA4. The estimation processing unit 12D then outputs the estimated variation values ​​EVA1 to EVA4 to the learning control unit 13.

[0049] In this specific example, the estimation processing unit 12D can calculate the estimated variation value EVA1 using, for example, the difference between feature vectors FVA1 and FVB1. Specifically, the estimation processing unit 12D can calculate the estimated variation value EVA1 by performing arithmetic processing using a multilayer perceptron or the like using the difference between feature vectors FVA1 and FVB1 as input. Also, in this specific example, the estimation processing unit 12D can calculate the estimated variation value EVA2 by performing arithmetic processing using a multilayer perceptron or the like using the difference between feature vectors FVA2 and FVB2. Also, in this specific example, the estimation processing unit 12D can calculate the estimated variation value EVA3 by performing arithmetic processing using a multilayer perceptron or the like using the difference between feature vectors FVA3 and FVB3. Also, in this specific example, the estimation processing unit 12D can calculate the estimated variation value EVA4 by performing arithmetic processing using a multilayer perceptron or the like using the difference between feature vectors FVA4 and FVB4. When performing the process for calculating the estimated variation value, the estimation processing unit 12D may use, for example, the sum of the feature vectors or the concatenation of the feature vectors instead of the difference between the feature vectors.

[0050] The learning control unit 13 calculates the difference between attribute values ​​ZVA1 and ZVB1 as attribute variation value ZCA1. The learning control unit 13 also calculates the difference between attribute values ​​ZVA2 and ZVB2 as attribute variation value ZCA2. The learning control unit 13 calculates the difference between attribute values ​​ZVA3 and ZVB3 as attribute variation value ZCA3. The learning control unit 13 calculates the difference between attribute values ​​ZVA4 and ZVB4 as attribute variation value ZCA4. The attribute variation values ​​ZCA1 to ZCA4 can be treated as correct values ​​in learning of the deep learning model DLM. Furthermore, according to this specific example, the learning control unit 13 can obtain "-0.40" as the calculation result of attribute variation value ZCA1, "-2.70" as the calculation result of attribute variation value ZCA2, "-5.00" as the calculation result of attribute variation value ZCA3, and "-1.80" as the calculation result of attribute variation value ZCA4.

[0051] The learning control unit 13 calculates the mean square error between the absolute value |ZCA1| of the attribute variation value ZCA1 and the absolute value |EVA1| of the estimated variation value EVA1 as the loss LPB1. The learning control unit 13 also calculates the mean square error between the absolute value |ZCA2| of the attribute variation value ZCA2 and the absolute value |EVA2| of the estimated variation value EVA2 as the loss LPB2. The learning control unit 13 also calculates the mean square error between the absolute value |ZCA3| of the attribute variation value ZCA3 and the absolute value |EVA3| of the estimated variation value EVA3 as the loss LPB3. The learning control unit 13 also calculates the mean square error between the absolute value |ZCA4| of the attribute variation value ZCA4, the absolute value |EVA4| of the estimated variation value EVA4, and the absolute value |ZCA4| of the attribute variation value ZCA4 as the loss LPB4.

[0052] The learning control unit 13 optimizes the currently set parameters of the backbone unit 12B according to the losses LPA1 and LPB1 to LPB4, thereby acquiring parameters PMP that minimize the losses LPA1 and LPB1 to LPB4. Specifically, the learning control unit 13 acquires, as the parameters PMP, parameters that minimize the calculation result of "β × LPA1 + (1-β) × (LPB1 + LPB2 + LPB3 + LPB4)," a loss function including a weight β. The learning control unit 13 then sets the parameters PMP as new parameters for the backbone unit 12B. The learning control unit 13 also repeatedly updates the parameters of the backbone unit 12B by repeating the above-described process until the total number of learning cycles corresponding to the number of epochs included in the learning content LCP is reached. The learning control unit 13 also completes pre-learning of the deep learning model DLM when the total number of learning cycles corresponding to the number of epochs included in the learning content LCP is reached.

[0053] According to this specific example, the learning control unit 13 may acquire the parameters PMP without referring to the currently set parameters of the backbone unit 12 B. In such a case, the learning control unit 13 may set, as the parameters of the backbone unit 12 B at the completion of pre-learning, the parameters PMPX that will cause the smallest loss among the parameters PMP stored in the parameter storage unit 13 B until the total number of learning cycles corresponding to the number of epochs included in the learning content LCP is reached.

[0054] According to this specific example, in the pre-learning, the learning control unit 13 may calculate the loss using another calculation method instead of calculating the mean square error between the absolute value of the attribute variation value and the absolute value of the estimated variation value as the loss. For example, in the pre-learning, the learning control unit 13 may calculate the mean square error between the attribute variation value ZCA1 and the estimated variation value EVA1 as the loss LPB1. Furthermore, the learning control unit 13 may calculate the mean square error between the attribute variation value ZCA2 and the estimated variation value EVA2 as the loss LPB2. Furthermore, the learning control unit 13 may calculate the mean square error between the attribute variation value ZCA3 and the estimated variation value EVA3 as the loss LPB3. The learning control unit 13 may also calculate the mean square error between the attribute variation value ZCA4 and the estimated variation value EVA4 as the loss LPB4.

[0055] According to this specific example, the learning control unit 13 may convert feature vectors into embedding vectors in a latent space by performing processing using a multilayer perceptron or the like in the pre-learning process, and calculate the loss using the embedding vectors. Furthermore, the learning control unit 13 may perform contrastive learning using the embedding vectors in the latent space when calculating the loss LPA. Through such processing, the learning control unit 13 can acquire parameters PMP that can improve the expression acquisition capability of the backbone unit 12B.

[0056] According to this specific example, in the pre-learning, a plurality of feature vector groups FVPA corresponding to one batch or multiple batches of learning data GDPA and a plurality of feature vector groups FVPB corresponding to multiple batches of learning data GDPB may be stored in the output data storage unit 12C. In such a case, the estimation processing unit 12D can calculate values ​​similar to the estimated variation values ​​EVA1 to EVA4 using one feature vector group selected from the plurality of feature vector groups FVPA and one feature vector group selected from the plurality of feature vector groups FVPB.

[0057] According to this specific example, the learning control unit 13 may repeatedly perform either a process of acquiring a parameter PMP corresponding to the loss LPA1 or a process of acquiring a parameter PMP corresponding to the losses LPB1 to LPB4 as a process related to updating the parameters of the backbone unit 12B. That is, the learning control unit 13 acquires the parameter PMP by performing a process using the loss LPA1 and / or the losses LPB1 to LPB4, and can update the parameters of the backbone unit 12B using the acquired parameter PMP.

[0058] In the process of acquiring the parameter PMP, the learning control unit 13 can use losses calculated by other calculation methods in addition to the losses LPA1 and LPB1 to LPB4. Specifically, in the process of acquiring the parameter PMP, the learning control unit 13 can use, for example, a cross-entropy loss calculated based on the result of classifying the estimated variation values ​​EVA1 to EVA4 by attribute. Furthermore, in the process of acquiring the parameter PMP, the learning control unit 13 can use, for example, a loss corresponding to the result of estimating or classifying information about a randomly masked region of an image based on information about an unmasked region of the image. The learning control unit 13 can estimate or classify information about the masked regions by, for example, performing processing using the feature vectors FVA1 to FVA8 and FVB1 to FVB4.

[0059] In the process of acquiring the parameter PMP, the learning control unit 13 can use, for example, the loss LPX between the image IPX included in the learning data and the text TPX input to the generative model when generating the image IPX. The loss LPX can be calculated, for example, using a feature vector calculated from the image IPX and a feature vector calculated from the text TPX. By using the loss LPX calculated as described above, the learning control unit 13 can perform learning such as, for example, bringing the feature vectors of a pair of matching image IPX and text TPX closer together and moving the feature vectors of a pair of mismatching image IPX and text TPX farther apart. Note that the text TPX is not limited to the text input to the generative model, and may be, for example, text included in the data linked to the image IPX.

[0060] The pre-training described in this specific example is not limited to being performed on an untrained deep learning model, but may also be performed on a deep learning model trained to suit a specific application. Specifically, the pre-training described in this specific example may be performed on a deep learning model trained to suit, for example, image recognition or face recognition.

[0061] According to this specific example, the learning content setting unit 13A sets, as the learning content for pre-learning of the deep learning model DLM, a learning content LCP for performing learning using learning data including a plurality of images showing differences and / or changes in at least a portion of the human body. In such a case, the data acquisition unit 11 can acquire a plurality of images showing differences and / or changes in the face as learning data according to the learning content LCP indicated by the control signal CA.

[0062] According to this specific example, the data acquisition unit 11 preferably acquires, as training data for pre-learning, a plurality of image groups in which facial differences between different people are represented by a plurality of images included in one data set, and in which facial changes within the same person are represented by a plurality of data sets in addition to the people included in each image of the one data set. Note that, in this specific example, the aforementioned "one data set" can be rephrased as "one batch of training data." Also, in this specific example, the aforementioned "multiple data sets" can be rephrased as "multiple batches of training data." Furthermore, according to this specific example, the data acquisition unit 11 can acquire, as images corresponding to the plurality of image sets, images of a fictional person generated by a generative model and / or images of a real person.

[0063] According to this specific example, the data acquisition unit 11 may acquire, as the learning data for the pre-learning, images that include at least a part of a face, such as upper body images and full-body images. Furthermore, according to this specific example, the data acquisition unit 11 may acquire, as the learning data for the pre-learning, images that include parts of a human body other than a face, but do not include a face. Furthermore, according to this specific example, the data acquisition unit 11 may acquire, as multiple images showing changes in the face or parts other than the face of the same person, images in which one attribute has changed, or images in which multiple attributes have changed.

[0064] (Learning Strategy for Pre-Learning) Next, the learning strategy for pre-learning will be described.

[0065] The learning content setting unit 13A generates information indicating the acquisition order and / or attribute ratio of the images included in the learning data GDP as a learning policy of the learning content LCP.

[0066] The learning content setting unit 13A generates information JA as a learning policy for the learning content LCP, for example, to set the acquisition order of images included in the learning data GDP to an order that minimizes facial changes within the same person according to the current learning count CLK. In other words, the learning content setting unit 13A can set the acquisition order of images included in the learning data GDP in the learning content LCP to an order that minimizes facial changes within the same person according to the current learning count CLK. The learning content setting unit 13A can count, for example, the number of times learning using one epoch of the learning data GDP is completed as the current learning count CLK. According to the learning content LCP including the information JA, the data acquisition unit 11 can acquire learning data GDPA including images IMA1 to IMA4 whose attribute variation values ​​ZCA1 to ZCA4 are large when the current learning count CLK is low, for example. Furthermore, by using a learning content LCP including information JA, the data acquisition unit 11 can acquire learning data GDPA including images IMA1 to IMA4 such that the attribute variation values ​​ZCA1 to ZCA4 become smaller when the current learning count CLK is high. Furthermore, by using a learning content LCP including information JA, the learning difficulty can be gradually increased depending on the learning situation in pre-learning. Therefore, when information JA is included in the learning content LCP, the learning control unit 13 can efficiently and effectively perform pre-learning of the deep learning model DLM.

[0067] The learning content setting unit 13A generates, as a learning policy for the learning content LCP, an information JB for changing the attribute ratio of images included in the learning data GDP according to the current learning situation. In other words, the learning content setting unit 13A can set the attribute ratio of images included in the learning data GDP in the learning content LCP to a ratio according to the current learning situation. The learning content setting unit 13A can grasp the current learning situation by, for example, acquiring verification data KDP instead of the learning data GDP each time learning using one epoch of the learning data GDP is completed and calculating the loss LPK corresponding to the verification data KDP. Note that the verification data KDP may include, for example, the same number of images as one batch of learning data GDP, but may include images different from the images already acquired as the learning data GDP. According to the learning content LCP including the information JB, for example, when the loss LPK corresponding to one attribute of each image included in the verification data KDP is relatively large and the loss LPK corresponding to another attribute of each image is relatively small, the data acquisition unit 11 can acquire learning data GDP that includes more images related to the one attribute than images related to the other attributes. Furthermore, according to the learning content LCP including the information JB, for example, when the loss LPK corresponding to one attribute of each image included in the verification data KDP is relatively small and the loss LPK corresponding to the other attribute of each image is relatively large, the data acquisition unit 11 can acquire learning data GDP that includes more images related to the other attributes than images related to the one attribute. In other words, when the learning content LCP includes the information JB, the learning control unit 13 can adjust the number of data items included in the learning data GDP acquired by the data acquisition unit 11 according to the learning status in the pre-learning. By making such adjustments, the data acquisition unit 11 can acquire, as the learning data GDP, data that includes, for example, a relatively small number of images relating to attributes for which learning is going well and a relatively large number of images relating to attributes for which learning is not going well.

[0068] (First Specific Example of Processing Related to Transfer Learning) Next, a first specific example of processing related to transfer learning will be described. Note that, hereinafter, for convenience of explanation, the pre-trained deep learning model DLM will also be referred to as the pre-trained model PLM. Furthermore, in this specific example, an example will be described in which the pre-trained model PLM is applied as a model for edema estimation. Furthermore, in this specific example, a patient or the like who has symptoms related to edema will be referred to as a real person PS.

[0069] The learning content setting unit 13A sets the learning content LCT so that, for example, an image ITA including the face of a real person PS before edema and a label LBA assigned to the image ITA are included in one batch of learning data GDTA. The learning content setting unit 13A also sets the learning content LCT so that, for example, an image ITB including the face of a real person PS after edema has occurred and a label LBB assigned to the image ITB are included in one batch of learning data GDTB. The learning content setting unit 13A also generates, as the learning content LCT, information indicating the batch size, number of iterations, number of epochs, and learning policy for transfer learning. The learning content setting unit 13A also generates a control signal CA for acquiring the learning data GDTA and GDTB according to the learning content LCT and outputs the generated control signal CA to the data acquisition unit 11.

[0070] The labels LBA and LBB may each include the same type of measurement value related to the edema state of the real person PS. Specifically, the labels LBA and LBB may each include at least one value of the following as a measurement value related to the edema state of the real person PS: a value indicating body weight, a value indicating the amount of water removal, a CTR (cardio thoracic ratio) value, and a BNP (brain nutrient peptide) value. In this specific example, a case will be described in which the label LBA includes a measurement value MVA and the label LBB includes a measurement value MVB of the same type as the measurement value MVA.

[0071] According to the above-described process, the learning content setting unit 13A can cause the data acquisition unit 11 to acquire the learning data GDTA and GDTB that indicate changes in the face of the same person contained in the images ITA and ITB.

[0072] The data acquisition unit 11 acquires learning data GDTA and GDTB according to the learning content LCT indicated by a control signal CA output from the learning control unit 13. Through this processing, the data acquisition unit 11 can acquire, as learning data for transfer learning, image ITA and image ITB, a label LBA assigned to image ITA, and a label LBB assigned to image ITB. The data acquisition unit 11 also outputs the learning data GDTA and GDTB to the learning processing unit 12. The data acquisition unit 11 also outputs, to the learning control unit 13, the label LBA assigned to image ITA included in the learning data GDTA and the label LBB assigned to image ITB included in the learning data GDTB.

[0073] The data expansion unit 12A applies the data-expanded DEM to the image ITA included in the training data GDTA to obtain an image ETA corresponding to the data-expanded image. The data expansion unit 12A also applies the data-expanded DEN to the image ITB included in the training data GDTB to obtain an image ETB corresponding to the data-expanded image. The data-expanded DEM and DEN include at least one process selected from a plurality of processes related to image processing, such as cropping, horizontal flipping, color conversion, and grayscale conversion. The data-expanded DEM and DEN also include different processes related to image processing.

[0074] The data extension unit 12A outputs training data EDTA including an image ETA and a label LBA to the backbone unit 12B. The data extension unit 12A also outputs training data EDTB including an image ETB and a label LBB to the backbone unit 12B.

[0075] The backbone unit 12B acquires a feature vector FVTA corresponding to the extraction result of the features of the image ETA included in the training data EDTA in accordance with the learning content LCT indicated by the control signal CB, and outputs the acquired feature vector FVTA to the output data storage unit 12C. Also, the backbone unit 12B acquires a feature vector FVTB corresponding to the extraction result of the features of the image ETB included in the training data EDTB in accordance with the learning content LCPT indicated by the control signal CB, and outputs the acquired feature vector FVTB to the output data storage unit 12C.

[0076] The estimation processing unit 12D acquires the feature vectors FVTA and FVTB from the output data storage unit 12C in accordance with the learning content LCT indicated by the control signal CB. The estimation processing unit 12D also calculates an estimated variation value EVT, which indicates the degree of difference between the feature vectors FVTA and FVTB. In this specific example, the estimated variation value EVT is calculated as the same type of value as the measurement values ​​MVA and MVB.

[0077] The learning control unit 13 calculates the difference between the measurement value MVA included in the label LBA and the measurement value MVB included in the label LBB as the measured variation value MCV. The learning control unit 13 also calculates the difference between the measured variation value MCV and the estimated variation value EVT as the loss LTA. The learning control unit 13 also optimizes the currently set parameters of the backbone unit 12B in accordance with the loss LTA, thereby obtaining parameters PMTA that minimize the loss LTA. The learning control unit 13 also sets the parameters PMTA as new parameters of the backbone unit 12B.

[0078] The learning control unit 13 continuously updates the parameters of the backbone unit 12B by repeating the above-described process until the total number of learning times corresponding to the number of epochs included in the learning content LCT is reached. Furthermore, the learning control unit 13 completes transfer learning of the pre-training model PLM when the total number of learning times corresponding to the number of epochs included in the learning content LCT is reached.

[0079] For example, the learning processing unit 12 can input two images similar to images ITA and ITB into a pre-learning model PLM that has undergone transfer learning, thereby estimating the state of edema of the person depicted in the two images. That is, according to this specific example, the learning processing unit 12 can acquire the function of an estimation unit that performs processing to estimate human body information, including information related to the state of edema of the person, using a model that has undergone pre-learning and transfer learning, including machine learning. Furthermore, according to this specific example, the learning device 100 can be treated as a biological information estimation device having the aforementioned estimation unit.

[0080] According to this specific example, even if there is little training data available for transfer learning of the pre-training model PLM, it is possible to construct a model that can estimate the edema state of various people with high accuracy. Furthermore, according to this specific example, by inputting two images of a specific person into the pre-training model PLM that has undergone transfer learning, it is possible to obtain a value indicating the fluctuation state of at least one of the body weight, the amount of water removed, the CTR value, and the BNP value as an estimation result indicating the edema state of the specific person.

[0081] According to this specific example, the learning content setting unit 13A sets, as the learning content for transfer learning of the pre-learning model PLM, learning content LCT for learning using learning data including an image of at least a part of the human body of a real person and a label assigned to the image.

[0082] According to this specific example, the learning control unit 13 may calculate the difference between the absolute value of the measured variation value and the absolute value of the estimated variation value as the loss in transfer learning. Specifically, the learning control unit 13 may calculate the difference between the absolute value |MCV| of the measured variation value MCV and the absolute value |EVT| of the estimated variation value EVT as the loss LTA.

[0083] According to this specific example, in transfer learning, a plurality of feature vectors FVTA corresponding to a plurality of batches of learning data GDTA and a plurality of feature vectors FVTB corresponding to a plurality of batches of learning data GDTB may be stored in the output data storage unit 12C. In such a case, the estimation processing unit 12D can calculate a value similar to the estimated variation value EVT using one feature vector selected from the plurality of feature vectors FVTA and one feature vector selected from the plurality of feature vectors FVTB.

[0084] According to this specific example, transfer learning may be performed on the entire pre-training model PLM, or on a portion of the pre-training model PLM. Specifically, the learning control unit 13 may perform transfer learning on the entire pre-training model PLM, including, for example, the backbone unit 12B and the estimation processing unit 12D. Alternatively, the learning control unit 13 may perform transfer learning on, for example, the backbone unit 12B of the pre-training model PLM, while not performing transfer learning on other portions of the pre-training model PLM. Alternatively, the learning control unit 13 may perform transfer learning on, for example, the estimation processing unit 12D of the pre-training model PLM, while not performing transfer learning on other portions of the pre-training model PLM.

[0085] (Second Specific Example of Processing Related to Transfer Learning) Next, a second specific example of processing related to transfer learning will be described. In this specific example, an example will be described in which the pre-training model PLM is applied as a model for estimating person information.

[0086] The learning content setting unit 13A sets a learning content LCT such that, for example, an image ITR containing the face of a real person PR and a label LBR assigned to the image ITR are included in one batch of learning data GDTR. The learning content setting unit 13A also generates, as the learning content LCT, information indicating the batch size, number of iterations, number of epochs, and learning policy for transfer learning. The learning content setting unit 13A also generates a control signal CA for acquiring learning data GDTR according to the learning content LCT, and outputs the generated control signal CA to the data acquisition unit 11.

[0087] The label LBR may include a state value related to the facial state of the real person PR. Specifically, the label LBR may include, as the state value related to the facial state of the real person PR, at least one value selected from the group consisting of a BMI (Body Mass Index) value, a value indicating age, a value indicating the type or intensity of facial expression, and a value indicating the degree of edema. In this specific example, a case will be described in which the label LBR includes the state value JVR.

[0088] The data acquisition unit 11 acquires training data GDTR according to the training content LCT indicated by a control signal CA output from the training control unit 13. Through this processing, the data acquisition unit 11 can acquire an image ITR and a label LBR assigned to the image ITR as training data for transfer learning. The data acquisition unit 11 also outputs the training data GDTR to the training processing unit 12. The data acquisition unit 11 also outputs the label LBR assigned to the image ITR included in the training data GDTR to the training control unit 13.

[0089] The data extension unit 12A applies the data extension DER to the image ITR included in the training data GDTR to obtain an image ETR corresponding to the data-extended image. The data extension DER includes at least one process selected from a plurality of processes related to image processing, such as cropping, horizontal flipping, color conversion, and grayscale conversion.

[0090] The data extension unit 12A outputs the training data EDTR including the image ETR and the label LBR to the backbone unit 12B.

[0091] The backbone unit 12B acquires a feature vector FVTR corresponding to the extraction result of the features of the image ETR contained in the training data EDTR in accordance with the learning content LCT indicated by the control signal CB, and outputs the acquired feature vector FVTR to the output data storage unit 12C.

[0092] The estimation processing unit 12D acquires the feature vector FVTR from the output data storage unit 12C in response to the learning content LCT indicated by the control signal CB. The estimation processing unit 12D also calculates the estimated value SVR by converting the feature vector FVTR into a scalar value using a multilayer perceptron or the like. In this specific example, the estimated value SVR is calculated as a value of the same type as the state value JVR.

[0093] The learning control unit 13 calculates the difference between the state value JVR and the estimated value SVR as the loss LTR. The learning control unit 13 also optimizes the currently set parameters of the backbone unit 12B in accordance with the loss LTR to obtain parameters PMTR that minimize the loss LTR. The learning control unit 13 also sets the parameters PMTR as new parameters of the backbone unit 12B.

[0094] The learning control unit 13 continuously updates the parameters of the backbone unit 12B by repeating the above-described process until the total number of learning times corresponding to the number of epochs included in the learning content LCT is reached. Furthermore, the learning control unit 13 completes transfer learning of the pre-training model PLM when the total number of learning times corresponding to the number of epochs included in the learning content LCT is reached.

[0095] The learning processing unit 12 can estimate personal information about a person included in an image ITR by inputting the image ITR into a pre-learning model PLM that has undergone transfer learning. In other words, according to this specific example, the learning processing unit 12 can acquire the function of an estimation unit that performs processing to estimate human body information, including personal information, using a model that has undergone pre-learning and transfer learning, including machine learning. Furthermore, according to this specific example, the learning device 100 can be treated as a biometric information estimation device having the above-mentioned estimation unit.

[0096] According to this specific example, even if there is little training data available for transfer learning of the pre-training model PLM, it is possible to construct a model that can estimate personal information of various people with high accuracy. Furthermore, according to this specific example, by inputting an image of a specific person's face into the pre-training model PLM that has undergone transfer learning, it is possible to obtain at least one piece of information from among BMI value, age, type or intensity of facial expression, and a value indicating the degree of edema as an estimation result of the specific person's personal information.

[0097] According to this specific example, in transfer learning, a plurality of feature vectors FVTR corresponding to a plurality of batches of training data GDTR may be stored in the output data storage unit 12C. In such a case, the estimation processing unit 12D can calculate a value similar to the estimated value SVR using one feature vector selected from the plurality of feature vectors FVTR.

[0098] According to this specific example, transfer learning may be performed on the entire pre-training model PLM, or on a portion of the pre-training model PLM. Specifically, the learning control unit 13 may perform transfer learning on the entire pre-training model PLM, including, for example, the backbone unit 12B and the estimation processing unit 12D. Alternatively, the learning control unit 13 may perform transfer learning on, for example, the backbone unit 12B of the pre-training model PLM, while not performing transfer learning on other portions of the pre-training model PLM. Alternatively, the learning control unit 13 may perform transfer learning on, for example, the estimation processing unit 12D of the pre-training model PLM, while not performing transfer learning on other portions of the pre-training model PLM.

[0099] [Processing Flow] Next, a description will be given of the flow of processing performed in the learning device 100. Fig. 6 is a flowchart showing an example of processing performed in the learning device according to the present disclosure.

[0100] First, the learning device 100 sets the learning content LCP of the pre-learning to be performed on the deep learning model DLM (step S11).

[0101] Next, the learning device 100 acquires learning data GDP of pre-learning according to the learning content LCP set in step S11 (step S12).

[0102] Next, the learning device 100 performs pre-learning of the deep learning model DLM using the learning data GDP acquired in step S12 according to the learning content LCP set in step S11 (step S13).

[0103] Next, the learning device 100 determines whether or not the pre-learning of the deep learning model DLM has been completed (step S14). Specifically, the learning device 100 determines whether or not the number of times the deep learning model DLM has been trained has reached the total number of times corresponding to the number of epochs included in the learning content LCP set in step S11.

[0104] If the pre-learning of the deep learning model DLM is completed (step S14: YES), the learning device 100 performs the process of step S15 described below. If the pre-learning of the deep learning model DLM is not completed (step S14: NO), the learning device 100 returns to step S13 and continues the pre-learning of the deep learning model DLM.

[0105] The learning device 100 sets the learning content LCT of the transfer learning to be performed on the pre-learning model PLM obtained through the processing of step S14 (step S15).

[0106] Next, the learning device 100 acquires learning data GDT for transfer learning according to the learning content LCT set in step S15 (step S16).

[0107] Next, the learning device 100 performs transfer learning of the pre-learning model PLM using the learning data GDT acquired in step S16 according to the learning content LCT set in step S15 (step S17).

[0108] Next, the learning device 100 determines whether or not the transfer learning of the pre-training model PLM has been completed (step S18). Specifically, the learning device 100 determines whether or not the number of times the pre-training model PLM has been trained has reached the total number of times of training corresponding to the number of epochs included in the learning content LCT set in step S15.

[0109] If the transfer learning of the pre-training model PLM is completed (step S18: YES), the learning device 100 ends the series of processes in Fig. 6. If the transfer learning of the pre-training model PLM is not completed (step S18: NO), the learning device 100 returns to step S17 and continues the transfer learning of the pre-training model PLM.

[0110] As described above, according to this embodiment, pre-training of a deep learning model is performed using training data including multiple images representing differences and / or changes in at least a portion of the human body. Furthermore, according to this embodiment, transfer learning of the pre-trained deep learning model is performed using training data including labeled images. Furthermore, according to this embodiment, even if the amount of training data for transfer learning is small, for example, by performing transfer learning of the pre-trained deep learning model using the training data, it is possible to construct a model capable of estimating various pieces of human body information corresponding to the images and labels included in the training data. Therefore, according to this embodiment, a model used for estimating human body information can be constructed efficiently and effectively.

[0111] Second Embodiment FIG. 7 is a block diagram showing another example of the functional configuration of a learning device according to the present disclosure.

[0112] The learning device 500 has the same hardware configuration as the learning device 100. The learning device 500 also has a first setting unit 511 and a second setting unit 512.

[0113] The first setting means 511 can be realized, for example, by using a function possessed by the learning content setting unit 13A. The second setting means 512 can be realized, for example, by using a function possessed by the learning content setting unit 13A.

[0114] FIG. 8 is a flowchart illustrating another example of processing performed in the learning device according to the present disclosure.

[0115] The first setting means 511 sets a first learning content as the learning content for pre-learning of the model, for learning using first learning data including a plurality of images showing differences and / or changes in at least a part of the human body (step S51).

[0116] The second setting means 512 sets second learning content for learning using second learning data including an image of at least a part of the human body of a real person and a label assigned to the image as learning content for transfer learning of the pre-trained model (step S52).

[0117] According to this embodiment, a model used for estimating human body information can be constructed efficiently and effectively.

[0118] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0119] (Supplementary Note 1) A learning device comprising: a first setting means for setting a first learning content for learning using first learning data including a plurality of images showing differences and / or changes in at least a part of a human body as learning content for pre-learning of a model; and a second setting means for setting a second learning content for learning using second learning data including images of at least a part of a human body of a real person and labels assigned to the images as learning content for transfer learning of the model for which the pre-learning has been performed.

[0120] (Supplementary Note 2) The learning device of Supplementary Note 1 further comprises a data acquisition means for acquiring the first learning data according to the first learning content and the second learning data according to the second learning content, wherein the data acquisition means acquires a plurality of images representing differences and / or changes in faces as the first learning data.

[0121] (Supplementary Note 3) The data acquisition means is a learning device according to Supplementary Note 2, which acquires, as the first learning data, a group of multiple images in which facial differences between different people are represented by multiple images included in one data set, and in which facial changes within the same person as the people included in each image of the one data set are represented by multiple data sets.

[0122] (Supplementary Note 4) The learning device according to Supplementary Note 3, wherein the data acquisition means acquires images of fictional characters generated by a generative model and / or images of real people as images corresponding to the plurality of image groups.

[0123] (Supplementary Note 5) The learning device of Supplementary Note 3 further comprises: a first data augmentation means for acquiring a plurality of data-augmented image groups by performing data augmentation on each of the plurality of image groups; and a first parameter update means for calculating losses corresponding to a plurality of feature vector groups obtained by inputting the plurality of data-augmented image groups into the model, and updating parameters of the model in accordance with the calculated losses.

[0124] (Appendix 6) The learning device of Appendix 2, wherein the first setting means sets the order of acquisition of images included in the first learning data in the first learning content to an order that minimizes facial changes within the same person according to the current number of learning rounds.

[0125] (Supplementary Note 7) The learning device according to Supplementary Note 1, wherein the first setting means sets, in the first learning content, a proportion of attributes of images included in the first learning data to a proportion according to a current learning situation.

[0126] (Supplementary Note 8) The learning device of Supplementary Note 2, wherein the data acquisition means acquires, as the second learning data, a first image including a face of a person before edema occurs and a first label assigned to the first image, a second image including a face of the person after edema occurs and a second label assigned to the second image.

[0127] (Supplementary Note 9) The learning device according to Supplementary Note 8, wherein the first label and the second label contain the same type of measurement value related to the edema state of the person.

[0128] (Supplementary Note 10) The learning device of Supplementary Note 9, wherein the measurement values ​​include at least one value of a value indicating body weight, a value indicating the amount of water removal, a CTR (Cardio Thoracic Ratio) value, and a BNP (Brain Natriuretic Peptide) value.

[0129] (Supplementary Note 11) The learning device of Supplementary Note 8, further comprising: second data augmentation means for performing first data augmentation on the first image to obtain a first data-augmented image, and performing second data augmentation on the second image to obtain a second data-augmented image; and second parameter update means for calculating a loss using a first feature vector obtained by inputting the first data-augmented image to the pre-trained model and a second feature vector obtained by inputting the second data-augmented image to the pre-trained model, and updating parameters of the pre-trained model in accordance with the calculated loss.

[0130] (Supplementary Note 12) A biological information estimation device including an estimation unit that performs processing to estimate human body information using the model that has been subjected to the pre-learning and transfer learning including machine learning by the learning device of Supplementary Note 1.

[0131] (Supplementary Note 13) A learning method executed by a computer, the learning method comprising: setting, as learning content for pre-learning of a model, a first learning content for learning using first learning data including a plurality of images showing differences and / or changes in at least a part of a human body; and setting, as learning content for transfer learning of the model for which the pre-learning has been performed, a second learning content for learning using second learning data including images of at least a part of the human body of a real person and labels assigned to the images.

[0132] (Supplementary Note 14) The learning method according to Supplementary Note 13, wherein a plurality of images showing differences and / or changes in faces are acquired as the first learning data according to the first learning content.

[0133] (Supplementary Note 15) The learning method of Supplementary Note 14, wherein the first learning data is a group of multiple images in which facial differences between different people are represented by multiple images included in one data set, and facial changes within the same person are represented by multiple data sets.

[0134] (Supplementary Note 16) The learning method according to Supplementary Note 15, wherein images of a fictional character generated by a generative model and / or images of a real person are acquired as images corresponding to the plurality of image groups.

[0135] (Supplementary Note 17) A recording medium having recorded thereon a program that causes a computer to execute the following processes: setting a first learning content for learning using first learning data including a plurality of images showing differences and / or changes in at least a part of a human body as learning content for pre-learning of a model; and setting a second learning content for learning using second learning data including images of at least a part of a human body of a real person and labels assigned to the images as learning content for transfer learning of the model for which the pre-learning has been performed.

[0136] (Appendix 18) The recording medium of Appendix 17, having recorded thereon a program for causing a computer to execute a process of acquiring a plurality of images showing differences and / or changes in a face as the first learning data according to the first learning content.

[0137] (Appendix 19) A recording medium according to Appendix 18, having recorded thereon a program for causing a computer to execute a process of acquiring, as the first learning data, a group of multiple images in which facial differences between different persons are represented by multiple images included in one data, and in which facial changes within the same person as the person included in each image of the one data are represented by multiple data.

[0138] (Appendix 20) A recording medium according to Appendix 19, having recorded thereon a program for causing a computer to execute a process of acquiring images of a fictional character generated by a generative model and / or images of a real person as images corresponding to the plurality of image groups.

[0139] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments and examples. Various modifications that would be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Each embodiment can be combined with other embodiments as appropriate. Furthermore, some or all of the configurations described in Supplements 2 to 12, which are dependent on Supplement 1, may also be dependent on Supplements 13 and 17 in the same manner as Supplements 2 to 12. Furthermore, without departing from the scope of each of the above-described embodiments, not limited to Supplement 1, Supplement 13, and Supplement 17, some or all of the configurations described as Supplements may also be dependent on various hardware, software, various recording means for recording software, or systems.

[0140] 11 Data acquisition unit 12 Learning processing unit 13 Learning control unit 100 Learning device

Claims

1. A learning device comprising: a first setting means for setting a first learning content for learning using first learning data including a plurality of images showing differences and / or changes in at least a part of a human body as learning content for pre-learning of a model; and a second setting means for setting a second learning content for learning using second learning data including images of at least a part of a human body of a real person and labels assigned to the images as learning content for transfer learning of the model for which the pre-learning has been performed.

2. The learning device of claim 1, further comprising a data acquisition means for acquiring the first learning data according to the first learning content and the second learning data according to the second learning content, wherein the data acquisition means acquires a plurality of images representing differences and / or changes in faces as the first learning data.

3. The learning device according to claim 2, wherein the data acquisition means acquires, as the first learning data, a group of multiple images in which facial differences between different people are represented by multiple images contained in one data set, and in which facial changes within the same person as the people contained in each image of the one data set are represented by multiple data sets.

4. A learning device as described in claim 3, wherein the data acquisition means acquires images of fictional characters generated by a generative model and / or images of real people as images corresponding to the plurality of image groups.

5. The learning device according to claim 3, further comprising: a first data augmentation means for acquiring a plurality of data-augmented image groups by performing data augmentation on each of the plurality of image groups; and a first parameter update means for calculating losses corresponding to a plurality of feature vector groups obtained by inputting the plurality of data-augmented image groups into the model, and updating the parameters of the model in accordance with the calculated losses.

6. The learning device described in claim 2, wherein the first setting means sets the order in which images included in the first learning data are acquired in the first learning content to an order that minimizes facial changes within the same person according to the current number of learning rounds.

7. A learning device as described in claim 1, wherein the first setting means sets the attribute ratio of images contained in the first learning data in the first learning content to a ratio corresponding to the current learning situation.

8. The learning device described in claim 2, wherein the data acquisition means acquires, as the second learning data, a first image including a face of a person before edema occurs and a first label assigned to the first image, and a second image including a face of the person after edema occurs and a second label assigned to the second image.

9. The learning device of claim 8, wherein the first label and the second label include the same type of measurement related to the edema state of the person.

10. A learning device as described in claim 9, wherein the measurement values ​​include at least one of a value indicating body weight, a value indicating the amount of water removal, a CTR (cardio thoracic ratio) value, and a BNP (brain nutrient peptide) value.

11. The learning device according to claim 8, further comprising: second data augmentation means for performing first data augmentation on the first image to obtain a first data-augmented image, and performing second data augmentation on the second image to obtain a second data-augmented image; and second parameter update means for calculating a loss using a first feature vector obtained by inputting the first data-augmented image to the pre-trained model, and a second feature vector obtained by inputting the second data-augmented image to the pre-trained model, and updating parameters of the pre-trained model in accordance with the calculated loss.

12. A biometric information estimation device having estimation means for performing processing to estimate human body information using the model that has undergone the pre-learning and transfer learning including machine learning by the learning device of claim 1.

13. A computer-implemented learning method, comprising: setting, as learning content for pre-learning of a model, a first learning content for learning using first learning data including a plurality of images showing differences and / or changes in at least a part of a human body; and setting, as learning content for transfer learning of the pre-learned model, a second learning content for learning using second learning data including images of at least a part of the human body of a real person and labels assigned to the images.

14. The learning method according to claim 13, wherein a plurality of images showing differences and / or changes in faces are acquired as the first learning data according to the first learning content.

15. A learning method as described in claim 14, wherein the first learning data is a group of multiple images in which facial differences between different people are represented by multiple images contained in a single data set, and facial changes within the same person as the people contained in each image of the single data set are represented by multiple data sets.

16. A learning method according to claim 15, wherein images corresponding to the plurality of image groups are images of fictional characters generated by a generative model and / or images of real people.

17. A recording medium having recorded thereon a program that causes a computer to execute the following processes: set a first learning content for learning using first learning data including a plurality of images showing differences and / or changes in at least a part of the human body as learning content for pre-learning of a model; and set a second learning content for learning using second learning data including images of at least a part of the human body of a real person and labels assigned to the images as learning content for transfer learning of the model that has undergone the pre-learning.

18. A recording medium according to claim 17, which stores a program for causing a computer to execute a process of acquiring a plurality of images showing differences and / or changes in a face as the first learning data according to the first learning content.

19. A recording medium as described in claim 18, which stores a program for causing a computer to execute a process of acquiring, as the first learning data, a group of multiple images in which facial differences between different people are represented by multiple images contained in one data set, and in which facial changes within the same person as the people contained in each image of the one data set are represented by multiple data sets.

20. A recording medium as described in claim 19, which stores a program that causes a computer to execute a process of acquiring images of fictional characters generated by a generative model and / or images of real people as images corresponding to the plurality of image groups.

Images