Learning device, estimation device, learning method, and learning program

Abstract

A learning device (10) that has a data processing unit (11) that generates a plurality of pieces of data that represent different ways of seeing or different ways of hearing from facial image data or voice data for the same person, a feature quantity conversion unit (121) that uses a first NN (1211) to convert each of the plurality of pieces of data to a feature quantity vector, an age estimation unit (122) that uses a second NN (1221) to estimate the age of a target person from a plurality of feature quantity vectors converted by the feature quantity conversion unit (121), and an updating unit (13) that updates parameters for the first NN (1211) and the second NN (1221) such that the plurality of feature quantity vectors converted by the feature quantity conversion unit (121) for the same person approach each other in a feature space and the error between the actual age of a person and the age of the person estimated by the age estimation unit (122) from each of the feature quantity vectors for the same person is reduced.

Description

[Technical field]

[0001] The present invention relates to a learning device, an estimation device, a learning method, and a learning program. [Background technology]

[0002] There is a demand for age estimation technology that estimates a person's age from face image data or voice data in the fields of call centers and marketing. In response to this demand, in recent years, in the fields of voice processing and image processing, a person age estimation method using a neural network (NN) has been known (e.g., Non-Patent Documents 1 and 2).

[0003] Non-Patent Document 1 describes how age can be estimated with high accuracy by linking a NN that converts voice data into a feature vector with a NN that estimates the posterior probability of an age label from the feature vector, and simultaneously training these NNs to maximize the posterior probability for the correct age value.

[0004] Similarly, Non-Patent Document 2 describes that age can be estimated with high accuracy by linking a NN that converts face image data into a feature vector with a NN that estimates the posterior probability of an age label from the feature vector, and training these NNs simultaneously to maximize the posterior probability for the correct age value. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] P.Ghahremani, et al. “End-to-End Deep Neural Network Age Estimation”, Proc. Interspeech, pp. 277-281, 2018., [online], [Retrieved June 19, 2021], Internet<URL:https: / / www.isca-speech.org / archive / Interspeech_2018 / pdfs / 2015.pdf> [Non-Patent Document 2] R. Rothe, et al. “Deep Expectation of Real and Apparent Age from a Single Image Without Facial Landmarks”, in International Journal of Computer Vision, vol. 126, no. 2-4, pp. 144-157, Springer, 2018., [online], [Retrieved June 19, 2021], Internet <URL: https: / / www.research-collection.ethz.ch / bitstream / 20.500.11850 / 204027 / 1 / eth-50296-01.pdf%20%3E> Summary of the Invention [Problem to be solved by the invention]

[0006] Here, the age estimation technologies described in Non-Patent Documents 1 and 2 have a problem in that the age estimation results using NN can change even for the same person's facial image data and voice data due to changes in how the person looks and sounds due to differences in facial direction and speech content, etc.

[0007] To solve this problem, for example, Non-Patent Document 1 describes a method for improving the robustness of a model by adding noise and convolving multiple reverberation characteristics to training speech data to increase the apparent amount of data.

[0008] Furthermore, for example, Non-Patent Document 2 describes a method for improving the robustness of a model by performing geometric processing such as shrinking, enlarging, and rotating face image data for training to increase the apparent amount of data.

[0009] In this way, by using the techniques described in Non-Patent Documents 1 and 2 to increase the variation of the training data and train the model, the robustness of the model is improved.

[0010] However, the methods described in Non-Patent Documents 1 and 2 do not impose constraints on the learning of the NN so that it produces the same output for data of the same person. Therefore, the methods described in Non-Patent Documents 1 and 2 have a problem in that it is not guaranteed that the NN will behave in the same way when data of the same person is input. In other words, even if data of the same person is input, the NN may return significantly different results for each piece of data.

[0011] The present invention has been made in consideration of the above, and aims to provide a learning device, an estimation device, a learning method, and a learning program that can obtain an estimator that outputs similar age estimation results with high accuracy when different data of the same person is input. [Means for solving the problem]

[0012] In order to solve the above-mentioned problems and achieve the object, the learning device of the present invention is characterized by having a data processing unit that generates multiple pieces of data that look or sound different from facial image data or voice data of the same person; a conversion unit that uses a first neural network to convert each of the multiple pieces of data into a feature vector; an estimation unit that uses a second neural network to estimate the age of a target person from the multiple feature vectors converted by the conversion unit; and an update unit that updates each parameter of the first neural network and the second neural network so that the multiple feature vectors of the same person converted by the conversion unit are close to each other in feature space, while reducing an error between the age of the person estimated by the estimation unit from each feature vector for the same person and the person's true age.

[0013] In addition, the estimation device of the present invention has a conversion unit that converts facial image data or voice data into a feature vector using a first neural network, and an estimation unit that estimates the age of a target person from the feature vector converted by the conversion unit using a second neural network, and is characterized in that the first neural network and the second neural network are trained so that multiple feature vectors based on multiple data on how the same person looks or sounds differently, converted by the first neural network, are close to each other in feature space, while the second neural network is trained to reduce an error between the age of the person estimated from each feature vector for the same person and the person's true age. Effect of the Invention

[0014] According to the present invention, when different data of the same person are input, it is possible to obtain an estimator that outputs similar age estimation results with high accuracy. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of a learning device according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating a process flow in the learning device shown in FIG. [Diagram 3] FIG. 3 is a diagram illustrating an example of the configuration of the first NN. [Figure 4] FIG. 4 is a diagram illustrating an example of the configuration of the first NN. [Diagram 5] FIG. 5 is a diagram illustrating an example of the configuration of the second NN. [Figure 6] FIG. 6 is a flowchart illustrating a processing procedure of the learning process according to the embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of a configuration of an estimation device according to an embodiment. [Figure 8] FIG. 8 is a flowchart showing the procedure of the estimation process executed by the estimation device shown in FIG. [Figure 9]FIG. 9 is a diagram illustrating an example of a computer that realizes the learning device and the estimation device by executing a program. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to this embodiment. In addition, in the description of the drawings, the same parts are denoted by the same reference numerals. Note that, in the following, when "~A" is written for a vector A, it is equivalent to "a symbol with "~" written immediately above "A".

[0017] [Embodiment Mode] In this embodiment, learning of an estimation model that estimates a person's age from input face image data or voice data using a neural network (NN) is described. In this embodiment, when learning the estimator, a learning criterion is used such that the feature amount of input data obtained by the NN in the front stage of the estimator shows a similar value as the data corresponds to the same person, thereby realizing the acquisition of an estimator that outputs a close age estimation result with high accuracy when data showing different appearances and sounds of the same person is input.

[0018] [Learning device] Next, a learning device according to an embodiment will be described. Fig. 1 is a diagram showing a schematic configuration of a learning device according to an embodiment. Fig. 2 is a diagram showing a process flow in the learning device shown in Fig. 1.

[0019] The learning device 10 according to the embodiment is realized by, for example, loading a predetermined program into a computer or the like including a ROM (Read Only Memory), a RAM (Random Access Memory), a CPU (Central Processing Unit), etc., and having the CPU execute the predetermined program. The learning device 10 also has a communication interface for transmitting and receiving various information to and from other devices connected via a wired connection or a network, etc.

[0020] 1 and 2, a learning device 10 includes a data processing unit 11, an estimation unit 12, an update unit 13, and a control processing unit 14. The learning data is face image data or voice data to which the true age of a target person is assigned as the correct age.

[0021] The data processing unit 11 applies changes to the appearance or sound of face image data or voice data, which are learning data. The data processing unit 11 artificially processes the face image data or voice data, which are learning data, to create multiple data with different appearances or sounds from the face image data or voice data of the same person.

[0022] For example, when processing voice data, the data processing unit 11 adds multiple pre-recorded noises, convolves impulse responses recorded in different reverberation environments, deletes parts of the voice signal, and / or extracts only parts of the voice signal to create multiple pieces of data that sound different from the voice data of the same person.

[0023] For example, when the data processing unit 11 is processing facial image data, it creates multiple data that look different from the facial image data of the same person by adding noise to the facial image data, performing geometric transformations such as enlarging / reducing or rotating, and / or hiding part of the image data.

[0024] The estimation unit 12 estimates the age of a target person based on a plurality of pieces of face image data or a plurality of pieces of voice data based on face image data or voice data of the same person created by the data processing unit 11. The estimation unit 12 has a feature conversion unit 121 (conversion unit) and an age estimation unit 122 (estimation unit).

[0025] The feature conversion unit 121 converts each of a plurality of pieces of face image data or voice data into a feature vector of fixed dimension using the first NN 1211. The first NN 1211 is an NN that converts a plurality of pieces of face image data or a plurality of pieces of voice data of a person generated by the data processing unit 11 into a feature vector.

[0026] When facial image data is the target, the first NN 1211 is realized by an NN that converts the facial image data into a feature vector, for example, by using the technology described in Non-Patent Document 2. FIG. 3 is a diagram for explaining an example of the configuration of the first NN 1211. In this case, the first NN 1211 is realized by an NN having a structure as shown in FIG. 3, for example. As an example, the first NN 1211 is realized by a convolutional NN consisting of a plurality of residual blocks that employ Squeeze-and-Excitation.

[0027] When speech data is the target, the first NN 1211 is realized by an NN that converts speech data into a feature vector, for example, by using the technology described in Non-Patent Document 1. FIG. 4 is a diagram for explaining an example of the configuration of the first NN 1211. In this case, the first NN 1211 is realized by an NN having a structure as shown in FIG. 4, for example. As an example, the first NN 1211 is realized by a convolutional NN consisting of a plurality of time-delay layers and a statistical pooling layer.

[0028] The age estimation unit 122 uses the second NN 1221 to estimate the age of the target person from the multiple feature vectors converted by the feature conversion unit 121. The second NN 1221 is an NN that estimates the age of each target person from a series of feature vectors converted by the first NN 1211.

[0029] The second NN 1221 is realized by an NN that estimates the age value of a target person from a feature vector, for example, using the technology described in Non-Patent Document 1. FIG. 5 is a diagram for explaining an example of the configuration of the second NN 1221. This second NN 1221 is realized by an NN having a structure as shown in FIG. 5, for example. As an example, the second NN 1221 is realized by a fully connected NN consisting of multiple 512-dimensional fully connected layers and a one-dimensional fully connected layer.

[0030] The update unit 13 updates each parameter of the first NN 1211 and the second NN 1221 based on each feature vector based on multiple facial image data or voice data converted by the feature conversion unit 121, the age estimation result of the target person estimated by the age estimation unit 122, and the correct age assigned to the learning data.

[0031] The update unit 13 updates each parameter of the first NN 1211 and the second NN 1221 so that multiple feature vectors of the same person converted by the feature conversion unit 121 are close to each other in feature space, while reducing the error between the person's age estimated by the age estimation unit 122 from each feature vector for the same person and the person's true age.

[0032] For example, the update unit 13 applies the first NN 1211 to the N pieces of processed data obtained from the data processing unit 11, and converts the obtained feature vectors into x1, x2, . . . , x n Then, the second NN1221 is applied to each feature vector, and the age estimation results obtained are ~y1, ~y2, . . . , ~y n The true age of the person is denoted by y, the loss by L, and the parameter to be updated by θ, and the parameters (parameters of the first NN 1211 and the second NN 1221) are updated by the formula (1).

[0033]

number

[0034] The first term of formula (1) is the distance between feature vectors (see FIG. 2), and can be obtained from the output of the first NN 1211. The second term of formula (1) is the estimation error (see FIG. 2) between the estimated age and the true age (correct age) of the target person, and can be obtained from the correct age assigned to the learning data and the output of the second NN 1221. Note that λ and μ in formula (1) are preset learning weights, and both are positive constants.

[0035] The control processing unit 14 repeatedly executes the processes by the data processing unit 11, the feature conversion unit 121, the age estimation unit 122, and the update unit 13 until a predetermined condition is satisfied. The control processing unit 14 repeatedly executes the update of the parameters of the first NN 1211 and the second NN 1221 by the update unit 125 until a predetermined condition is satisfied. The predetermined condition is, for example, a condition that the learning of the first NN 1211 and the second NN 1221 is sufficiently performed, such as reaching a predetermined number of repetitions or the update amount of the parameters of the first NN 1211 and the second NN 1221 being less than a predetermined threshold.

[0036] [Learning process procedure] Next, a description will be given of the learning process executed by the learning device 10. Fig. 6 is a flowchart showing the processing procedure of the learning process according to the embodiment.

[0037] As shown in FIG. 6, in the learning device 10, the data processing unit 11 generates multiple pieces of data that look or sound different from the face image data or voice data of the same person among the learning data by performing data processing such as adding noise or extracting random areas (step S1).

[0038] Then, the feature conversion unit 121 converts each of the multiple data generated in step S1 into a feature vector using the first NN 1211 (step S2). The age estimation unit 122 estimates the age of the target person from the multiple feature vectors converted by the feature conversion unit 121 using the second NN 1221 (step S3).

[0039] The update unit 13 updates each parameter of the first NN 1211 and the second NN 1221 so that multiple feature vectors of the same person converted by the feature conversion unit 121 are close to each other in feature space, while reducing the error between the person's age estimated by the age estimation unit 122 from each feature vector for the same person and the person's true age (step S4).

[0040] The control processing unit 14 determines whether or not a predetermined condition is satisfied (step S5). If the predetermined condition is not satisfied (step S5: No), the learning device 10 returns to step S1 and performs each process of data processing, feature conversion, age estimation, and parameter update. On the other hand, if the predetermined condition is satisfied (step S5: Yes), the learning device 10 ends the learning process.

[0041] [Estimation device] Next, an estimation device according to an embodiment will be described. Fig. 7 is a diagram illustrating an example of the configuration of the estimation device according to an embodiment. Fig. 8 is a flowchart illustrating an estimation process procedure executed by the estimation device illustrated in Fig. 7.

[0042] 7 includes a feature conversion unit 221 (conversion unit) having a first NN 1211, and an age estimation unit 222 (estimation unit) having a second NN 1221. The first NN 1211 and the second NN 1221 are NNs that have been trained by the learning device 10.

[0043] When the feature conversion unit 221 receives input of face image data or voice data (step S11 in FIG. 8), it converts the face image data or voice data into features using the first NN 1211 (step S12 in FIG. 8).

[0044] The age estimation unit 222 uses the second NN 1221 to estimate the age of the target person from the feature vector converted by the feature conversion unit 221 (step S13 in FIG. 8), and outputs the estimated age (step S14 in FIG. 8).

[0045] [Evaluation experiment] Next, an evaluation experiment was performed on the first NN1211 and the second NN1221 trained based on formula (1) by the learning device 10. Here, the first NN1211 and the second NN1221 were trained using 3750 utterances from 375 speakers recorded by a microphone mounted on a smartphone as training data. After that, the estimation device 20 estimated the ages of the speakers using the first NN1211 and the second NN1221 for the same 1300 utterances from 120 speakers.

[0046] As a result, the mean absolute error between the correct age value and the speaker's age estimation results using the first NN1211 and the second NN1221 was 5.87 years. In addition, the correlation coefficient between the correct age value and the speaker's age estimation results was 0.92.

[0047] For reference, when the first and second NNs were trained using only the second term of equation (1), the absolute error between the correct age value and the estimated speaker age was 6.72 years, and the correlation coefficient was 0.88.

[0048] [Effects of the embodiment] Therefore, as in this embodiment, by training the first NN1211 and the second NN1221 by imposing constraints so that the feature vector approaches positive for data of the same person who looks and sounds different, it was found that when different data of the same person was input, close age estimation results were output with high accuracy, and it was confirmed that this functions effectively.

[0049] In this way, according to this embodiment, it is possible to obtain a feature extractor (first NN1211) that is invariant to input data that looks or sounds different, and an age estimator (second NN1221) that is robust to the influence of these variations. In other words, in this embodiment, when training an NN that estimates age from face image data or voice data, when "different data of the same person" is given, it is possible to realize an NN that can output similar age estimation results regardless of changes in the content of speech or the angle of the face by imposing a constraint that the intermediate feature amounts inside the NN have similar values.

[0050] The learning device 10 is similar to the techniques described in Non-Patent Documents 1 and 2 in that input data that has been processed in some way is used as learning data, but differs in the following points.

[0051] In Non-Patent Document 1, speech is processed in the same manner as the data processing unit in this embodiment, and in Non-Patent Document 2, images are processed in the same manner as the data processing unit in this embodiment, and the processed data are used as learning data. However, in both of the techniques in Non-Patent Documents 1 and 2, the processed data is used only as normal learning data. In other words, in the techniques described in Non-Patent Documents 1 and 2, only the second term of the loss L in Equation (1) is used.

[0052] In contrast, in this embodiment, by applying a constraint to multiple pieces of data of the same person that have been subjected to different processing so that the first term of the loss L in formula (1) is close to positive in the feature vector space, it is possible to estimate the age more robustly against differences in how the person sees or hears. In other words, while the techniques described in Non-Patent Documents 1 and 2 simply increase the variation of the learning data, in this embodiment, in addition to this, an explicit constraint is applied so that the behavior of the NN is invariant to each variation, thereby realizing a NN that is more robust against these variations.

[0053] This embodiment is applicable regardless of the difference in input, such as image data, audio data, etc. Specifically, the first NN 1211 may be changed to one suitable for each type of input data.

[0054] [System configuration of the embodiment] Each component of the learning device 10 and the estimation device 20 is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of the functions of the learning device 10 and the estimation device 20 is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed or integrated in any unit depending on various loads, usage conditions, etc.

[0055] Furthermore, all or any part of the processes performed in the learning device 10 and the estimation device 20 may be realized by a CPU, a GPU (Graphics Processing Unit), and a program analyzed and executed by the CPU and the GPU. Furthermore, each process performed in the learning device 10 and the estimation device 20 may be realized as hardware using wired logic.

[0056] Furthermore, among the processes described in the embodiments, all or part of the processes described as being performed automatically can be performed manually. Alternatively, all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the process procedures, control procedures, specific names, and information including various data and parameters described above and shown in the drawings can be changed as appropriate unless otherwise specified.

[0057] [program] 9 is a diagram showing an example of a computer in which a program is executed to realize the learning device 10 and the estimation device 20. The computer 1000 has, for example, a memory 1010 and a CPU 1020. The computer 1000 also has a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0058] The memory 1010 includes a ROM 1011 and a RAM 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

[0059] The hard disk drive 1090 stores, for example, an OS (Operating System) 1091, an application program 1092, a program module 1093, and program data 1094. That is, a program that defines each process of the learning device 10 and the estimation device 20 is implemented as a program module 1093 in which code executable by the computer 1000 is written. The program module 1093 is stored, for example, in the hard disk drive 1090. For example, a program module 1093 for executing processes similar to the functional configurations of the learning device 10 and the estimation device 20 is stored in the hard disk drive 1090. The hard disk drive 1090 may be replaced with an SSD (Solid State Drive).

[0060] Furthermore, setting data used in the processing of the above-described embodiment is stored as program data 1094, for example, in memory 1010 or hard disk drive 1090. Then, CPU 1020 reads out program module 1093 or program data 1094 stored in memory 1010 or hard disk drive 1090 into RAM 1012 as necessary, and executes the program.

[0061] Note that the program module 1093 and the program data 1094 are not limited to being stored in the hard disk drive 1090, but may be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and the program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or wide area network (WAN)). The program module 1093 and the program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070.

[0062] Although the embodiment of the invention made by the present inventor has been described above, the present invention is not limited by the description and drawings that form a part of the disclosure of the present invention according to the present embodiment. In other words, other embodiments, examples, operation techniques, etc. made by those skilled in the art based on the present embodiment are all included in the scope of the present invention. [Explanation of symbols]

[0063] 10 Learning Device 11 Data Processing Department 12 Estimation part 13 Update section 14 Control processing section 121,221 Feature conversion unit 122,222 Age Estimation Department 1211 1st NN 1221 Second NN

Claims

1. A data processing unit that generates multiple pieces of data that look or sound different from face image data or voice data of the same person; a conversion unit that converts each of the plurality of data into a feature vector using a first neural network; an estimation unit that estimates an age of a target person from the plurality of feature vectors converted by the conversion unit using a second neural network; an update unit that updates each parameter of the first neural network and the second neural network so that a plurality of feature vectors of the same person converted by the conversion unit are close to each other in a feature vector space, while reducing an error between an age of the person estimated by the estimation unit from each feature vector of the same person and a true age of the person; having the update unit updates each parameter of the first neural network and the second neural network using a loss in equation (1) in which a first term is a distance between multiple feature vectors of the same person converted by the conversion unit, and a second term is an error between the age of the person estimated from each feature vector of the same person by the estimation unit and the true age of the person, in a direction that reduces the loss. [0010]

2. 2. The learning device according to claim 1, further comprising a control processing unit that repeatedly executes the processes by the data processing unit, the conversion unit, the estimation unit and the update unit until a predetermined condition is satisfied.

3. The learning device according to claim 1 or 2, characterized in that the data processing unit generates multiple pieces of data that look or sound different from the facial image data or voice data of the same person by adding noise or extracting random areas to the facial image data or voice data of the same person.

4. a conversion unit that converts face image data or voice data into a feature vector using a first neural network; an estimation unit that estimates an age of a target person from the feature vector converted by the conversion unit using a second neural network; having the first neural network and the second neural network are trained so that a plurality of feature vectors based on a plurality of data on different ways of seeing or hearing the same person, which are converted by the first neural network, are close to each other in a feature space, while an error between an age of the person estimated by the second neural network from each feature vector for the same person and a true age of the person is reduced; wherein the parameters of the first neural network and the second neural network are updated in a direction that reduces a loss expressed by equation (2), in which a first term is a distance between a plurality of feature vectors of the same person transformed by the first neural network, and a second term is an error between the age of the person estimated from each feature vector of the same person by the second neural network and the true age of the person. [0025]

5. A learning method executed by a learning device, comprising: A step of generating a plurality of data items each having different appearances or sounds from face image data or voice data of the same person; converting each of the plurality of data into a feature vector using a first neural network; using a second neural network to estimate the age of the target person from the plurality of feature vectors converted by the converting step; updating each parameter of the first neural network and the second neural network so that a plurality of feature vectors of the same person converted in the converting step are close to each other in a feature space, while reducing an error between the age of the person estimated from each feature vector for the same person in the estimating step and a true age of the person; Including, the updating step updates each parameter of the first neural network and the second neural network in a direction that reduces a loss given by equation (3), in which a distance between a plurality of feature vectors of the same person converted in the converting step is used as a first term, and an error between the age of the person estimated from each feature vector for the same person in the estimating step and the true age of the person is used as a second term. [0030]

6. A learning program for causing a computer to function as the learning device according to any one of claims 1 to 3.

Images