Training device, inference device, training method, and program

Abstract

This training device that executes training of a model for performing face detection and attribute estimation comprises a training unit that trains the model so as to continuously output, from an input image, a result of the face detection and a result of the attribute estimation in a text format.

Description

Learning device, inference device, learning method, and program

[0001] The present invention relates to a technique for detecting human faces from an image containing multiple people and estimating their attributes.

[0002] The technology to detect human faces from images containing multiple people and estimate their attributes can be applied to camera-based surveillance systems, attribute analysis systems for store visitors, systems for measuring the effectiveness of digital signage advertisements, etc. The elemental tasks that realize these systems include face detection and attribute estimation such as age estimation, gender estimation, and emotion estimation.

[0003] In recent years, a method has been proposed to solve various image tasks with a single model, using a conditional text generation model that estimates text (hereafter referred to as output text) from an image (hereafter referred to as input image) and text (hereafter referred to as input text). A conditional text generation model is a model that has the ability to generate text when certain conditions are given.

[0004] For example, Non-Patent Document 1 discloses a technique for solving various image tasks with a single model using a conditional text generation model consisting of an encoder and a decoder.

[0005] In this case, the input text is a text that indicates the task to be solved for the input image, and the output text is a text that shows the result.

[0006] For example, if the task is image recognition, which classifies an input image into classes, the input text can be text such as "What's in the image?" or a special token that indicates image recognition. For example, if the input image shows a sea snake, the output text can be " <s>Sea snake <e>" is a text indicating the class of the recognition result.

[0007] For example, if the task is object detection, which involves detecting object regions from an input image and assigning classes, the input text may be text such as "What kind of object is in the image?" or a special token that indicates object detection. For example, if a chair appears in the rectangular region with top left and top right coordinates (100, 745) and (495, 991) in the input image, respectively, and a bed appears in the rectangular region with coordinates (293, 100) and (753, 763), the output text may be " <s><100> <745> <495> <991> chair <293> <100> <753> <763> Bed <e>" is a sequence of text indicating a token representing a rectangular area and a class.

[0008] In the above example, character-based tokens are used, and tokens are separated by a half-width space. <s>、 <e>are the tokens that represent the start of the text (hereafter referred to as the start token) and the tokens that represent the end of the text (hereafter referred to as the end token), respectively. <n>is a special token (hereinafter referred to as coordinate token) that represents coordinate n.

[0009] J. Lu, C. Clark, R. Zellers, R. Mottaghi, and A. Kembhavi, "Unified-IO: A unified model for vision, language, and multimodal tasks," in Proceedings of the International Conference on Learning Representations (ICLR), 2022.

[0010] If it were possible to realize a process for detecting human faces from images containing multiple people and estimating their attributes using a single model, it would be possible to prevent errors in face detection from propagating to subsequent processing, and to improve accuracy by using face region information in attribute estimation, compared to a case where face detection and attribute estimation are performed in cascade. However, conventional technology has not been able to realize a process for detecting human faces from images containing multiple people and estimating their attributes using a single model.

[0011] The present invention has been made in consideration of the above points, and aims to provide a technology for implementing a process of detecting human faces from an image containing multiple people and estimating their attributes using a single model.

[0012] According to the disclosed technology, there is provided a learning device that executes learning of a model that performs face detection and attribute estimation, the learning device including a learning unit that trains the model so as to continuously output the results of the face detection and the results of the attribute estimation in text format from an input image.

[0013] The disclosed technology provides a technology for implementing a process of detecting human faces from an image containing multiple people and estimating their attributes using a single model.

[0014] 1 is a diagram showing an example of a conditional text generation model. FIG. 2 is a diagram showing an example of the configuration of the learning device 100. FIG. 3 is a diagram for explaining an example of the operation of the teacher data generation unit 110 in the first embodiment. FIG. 4 is a diagram showing the processing flow of the learning device 100. FIG. 5 is a diagram showing an example of the configuration of the inference device 200. FIG. 6 is a diagram showing the processing flow of the inference device 200. FIG. 7 is a diagram for explaining an example of the operation of the teacher data generation unit 110 in the second embodiment. FIG. 8 is a diagram showing an example of the configuration of the learning device 100 in the third embodiment. FIG. 9 is a diagram for explaining an example of the operation of the teacher data generation unit 110 in the third embodiment. FIG. 10 is a diagram for explaining an example of the operation of the teacher data generation unit 110 in the fourth embodiment. FIG. 11 is a diagram showing an example of the hardware configuration of the device. FIG. 12 is a diagram showing experimental results.

[0015] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment.

[0016] As described above, conventional techniques have not realized a process for detecting human faces from an image containing multiple people and estimating their attributes using a single model. However, it is expected that such a process can be realized by using a single conditional text generation model.

[0017] However, an effective modeling method for such a model has not yet been established. Even if a modeling method were established, training such a model would require a large amount of training data in which face regions and desired attributes are all annotated, and collecting such training data would be expensive.

[0018] Therefore, in this embodiment, the output text of the conditional text generation model is defined as a sequence of tokens indicating face areas and tokens indicating their attributes, thereby realizing a process in which a single model can detect human faces from an image containing multiple people and estimate their attributes.

[0019] Note that "text" is a string of characters, symbols, or numbers. One character (one symbol, one number) is also an example of "text." A "token" is a unit used in machine learning, and is a division of a sentence (text) into certain units. "Tokens" are included in "text." Also, a sequence is data in which characters, symbols, or numbers are arranged consecutively.

[0020] Outputting a sequence of tokens indicating face regions and tokens indicating their attributes can also be rephrased as "continuously outputting the results of face detection and attribute estimation in text format."

[0021] In addition, in this embodiment, a switching token is introduced into the input text, making it possible to explicitly specify, for example, face detection or only some attribute estimation as the target task during training. This makes it possible to explicitly provide the model with information about which tasks are being treated as the population. As a result, efficient training is possible from training data that is annotated with only some tasks.

[0022] Hereinafter, a description will be given of an example in which the technology according to this embodiment is applied to a conditional text generation model having an encoder-decoder type autoregressive decoder that obtains output text from an input image and input text. Such a conditional text generation model is the model used in Non-Patent Document 1. Such a conditional text generation model is also used in the reference "T. Chen, S. Saxena, L. Li, T.-Y. Lin, DJ Fleet, and G.E. Hinton, "A unified sequence interface for vision tasks," in Advances in Neural Information Processing Systems (NeurIPS), 2022, vol. 35, pp. 31333-31346."

[0023] An example of the conditional text generation model is shown in Figure 1. As shown in Figure 1, the conditional text generation model includes an encoder 1 and a decoder 2. The encoder 1 includes an image encoder 3, a text encoder 4, and a cross-modal encoder 5.

[0024] An input image and an input text are input to an encoder 3 and an encoder 4, respectively, and features obtained from the input sequence (input image and input text) are input from a cross-modal encoder 5 to a decoder 2. The decoder 2 uses the features to generate an output. The decoder 2 generates output text by sequentially inputting its own output.

[0025] Note that the model to which the technology according to this embodiment can be applied is not limited to a conditional text generation model of encoder-decoder type having an autoregressive decoder.

[0026] In other words, the technology according to this embodiment can be applied to any technology that obtains output text from an image and input text using any end-to-end sequence-to-sequence model. For example, a causal language model structure consisting only of a decoder may be used as a conditional text generation model.

[0027] Furthermore, as will be described later, in this embodiment, an example is taken in which the area obtained as a result of face detection is represented by a total of four tokens, which are the x-coordinates and y-coordinates of the upper left and lower right, respectively; however, the method of representing the results of face detection in this embodiment is not limited to this.

[0028] For example, all coordinates in an image may be assigned consecutive numbers, and the area obtained as a result of face detection may be represented by two tokens, the coordinate numbers of the top left and bottom right, or by tokens corresponding to any number of points surrounding the area.

[0029] Furthermore, in this embodiment, age estimation and gender estimation are described as examples of attribute estimation, but attribute estimation in this embodiment is not limited to age estimation and gender estimation. In other words, the technology according to this embodiment can be applied to general tasks of estimating any attribute obtained from a face, and can be applied to estimating any number of attributes. For example, attribute estimation may include emotion estimation, key point estimation, face direction estimation, estimation of emotional valence or arousal level, gaze estimation, and estimation of belongings such as glasses or a mask.

[0030] The following describes the configuration and operation of an apparatus for detecting human faces from an image containing multiple people using a single model and estimating their attributes, using first to fourth embodiments. Note that the input image is not limited to an image containing multiple people, and may be an image containing one person.

[0031] In the following, the learning device 100 that learns a model and the inference device 200 that performs inference using the trained model will be described as separate devices. However, the learning device 100 and the inference device 200 may be the same device. For example, the learning device 100 itself may perform inference using a model that it has learned. The learning device 100 that performs inference may also be called the "inference device."

[0032] (First embodiment) <Configuration and operation of the learning device 100> Fig. 2 shows an example configuration of the learning device 100 in the first embodiment. The learning device 100 is a device that trains a conditional text generation model that performs face detection and attribute estimation. In the following description of the embodiment, this conditional text generation model will be referred to as a face detection / attribute detection model.

[0033] 2, the learning device 100 includes a teacher data generation unit 110, a face detection and attribute estimation unit 120, and a parameter update unit 130. The face detection and attribute estimation unit 120 includes a face detection and attribute detection model 121. The face detection and attribute estimation unit 120 may also include a tokenizer or the like. The "face detection and attribute estimation unit 120 and the parameter update unit 130" may be referred to as a learning unit.

[0034] Furthermore, a teacher data DB 150 is provided outside the learning device 100. The teacher data DB 150 stores teacher data for all tasks.

[0035] "All tasks" refers to all tasks assumed in the embodiment. For example, if it is assumed that a face detection task, a gender estimation task, and an age estimation task are executed as all tasks, "all tasks" refers to the face detection task, the gender estimation task, and the age estimation task.

[0036] If the "all tasks" are a face detection task, a gender estimation task, and an age estimation task, the training data for all tasks is, for example, a collection of "images of people's faces, with the position of the face, the gender of the person, and the age of the person who has that face annotated for each face." The operation of each part is as follows.

[0037] The teacher data generation unit 110 receives as input the teacher data for all tasks read from the teacher data DB 150, and generates and outputs teacher data that can be used for training the face detection / attribute detection model 121. In the first embodiment, the teacher data that can be used for training the face detection / attribute detection model 121 is a set of "input image, input text, and correct output text."

[0038] An example of the operation of the teacher data generating unit 110 in the first embodiment will be described with reference to FIG.

[0039] As shown in Fig. 3, the teacher data generation unit 110 receives teacher data for all tasks as input and generates the sets of "input image, input text, correct output text" shown in Fig. 3. The teacher data generation unit 110 generates multiple sets of "input image, input text, correct output text", but Fig. 3 shows one set of "input image, input text, correct output text" as an example.

[0040] The input text constituting the training data available for training the face detection and attribute detection model 121 is, for example, text such as "Please perform face detection, age estimation, and gender estimation" as shown in FIG. 3. The input text is always fixed. "Always fixed" means that the input text is the same in multiple pairs of "input image, input text, and correct output text" generated by the training data generation unit 110.

[0041] In the first embodiment (and the second embodiment), a model that omits the text encoder may be used as the face detection / attribute detection model 121 (conditional text generation model). When a model that omits the text encoder is used as the face detection / attribute detection model 121, the training data generation unit 110 omits generating input text. In other words, in this case, the training data generation unit 110 generates a set of "input image, correct output text."

[0042] The correct output text is a sequence of coordinate tokens representing the rectangular area of ​​the face and text indicating the age and gender for each face. In the example shown in Figure 3, when a 13-year-old male appears in the rectangular area with the top left and top right coordinates (48, 97) and (80, 119) in the input image, an 8-year-old female appears in the rectangular area with the coordinates (155, 126) and (183, 150), and a 26-year-old male appears in the rectangular area with the coordinates (239, 49) and (272, 75), the output text is " <s><48> <97> <80> <119> 13-year-old male <155> <126> <183> <150> 8-year-old girl <239> <49> <272> <75> 26-year-old male <e>" It has been shown that

[0043] The updated parameters are set in the face detection and attribute estimation model 121. The parameter updates are repeated during the learning process.

[0044] The face detection and attribute estimation unit 120 executes a task using a face detection and attribute estimation model 121. Specifically, the face detection and attribute estimation unit 120 inputs an input image and input text to the face detection and attribute estimation model 121. The face detection and attribute estimation model 121 solves a task for the input image based on the instructions expressed in the input text from the input image and input text, and outputs the result as estimated output text. The face detection and attribute estimation unit 120 acquires and outputs the estimated output text.

[0045] As described above, in the first embodiment (and the second embodiment), the input text is always fixed text. Alternatively, in the first embodiment (and the second embodiment), a conditional text generation model that omits the text encoder may be used, in which case input of the input text to the face detection and attribute estimation unit 120 can be omitted.

[0046] In addition to tokens for natural language processing, start tokens, end tokens, and coordinate tokens are registered in the dictionary used in the face detection and attribute estimation model 121. Note that the dictionary may be separate for input text and for output text.

[0047] The parameter update unit 130 receives the estimated output text and the correct output text as input, derives updated parameters for the face detection / attribute estimation model 121 (neural network) so as to reduce a predefined error function, outputs the updated parameters, and updates the parameters of the face detection / attribute estimation model 121 to the updated parameters.

[0048] The error function may be any function whose value decreases as the difference between the estimated output text and the correct output text decreases, and for example, the cross entropy between the estimated output text and the correct output text may be used.

[0049] <Processing Flow of Learning Device 100> Next, a processing flow of the learning device 100 in the first embodiment will be described with reference to FIG.

[0050] In S101 (step 101), the teacher data generation unit 110 performs a teacher data generation process. Specifically, teacher data for all tasks is input to the teacher data generation unit 110, and the teacher data generation unit 110 generates and outputs a set of "input image, input text, and correct output text."

[0051] In S102, the face detection / attribute estimation unit 120 performs face detection / attribute estimation processing. Specifically, an input image and input text are input to the face detection / attribute estimation unit 120, and the face detection / attribute estimation unit 120 obtains estimated output text using a face detection / attribute estimation model 121.

[0052] In S103, the parameter update unit 130 performs a parameter update process. Specifically, the estimated output text and the correct output text are input to the parameter update unit 130, the parameter update unit 130 outputs updated parameters, and the parameters of the face detection / attribute estimation model 121 are updated with the updated parameters.

[0053] The learning device 100 repeats the training data generation process, face detection and attribute estimation process, and parameter update process until a learning termination condition is met (S104). The learning termination condition may be, for example, until a fixed number of epochs is reached. Alternatively, part of the training data may be used as development data, and the error function for the development data may stop decreasing a certain number of times.

[0054] Furthermore, the training data generation process, face detection and attribute estimation process, and parameter update process may be performed by dividing the training data into mini-batches and repeating the process for each mini-batch.

[0055] 5 shows an example of the configuration of the inference device 200 according to the first embodiment. The inference device 200 is a device that performs inference using the face detection and attribute detection model 121 trained by the learning device 100.

[0056] 5, the inference device 200 has a face detection and attribute estimation unit 120 that includes a face detection and attribute detection model 121. The face detection and attribute detection model 121 has parameters that have been updated (optimized) by the learning device 100.

[0057] An input image and input text are input to the face detection and attribute estimation unit 120. The face detection and attribute estimation unit 120 inputs the input image and input text to a face detection and attribute detection model 121. The face detection and attribute detection model 121 estimates text corresponding to the input image and input text through face detection and attribute estimation, and outputs the text as estimated output text. The face detection and attribute estimation unit 120 acquires and outputs the estimated output text.

[0058] <Processing Flow of Inference Device 200> Next, a processing flow of the inference device 200 in the first embodiment will be described with reference to FIG.

[0059] In S111, the face detection and attribute estimation unit 120 performs face detection and attribute estimation processing. Specifically, an input image and input text are input to the face detection and attribute estimation unit 120. The face detection and attribute estimation unit 120 inputs the input image and input text to a face detection and estimation model 121, thereby obtaining estimated output text from the face detection and estimation model 121. The face detection and attribute estimation unit 120 outputs the estimated output text.

[0060] Second Embodiment Next, a second embodiment will be described. The configurations of the learning device 100 and the inference device 200 in the second embodiment are the same as the configurations of the learning device 100 and the inference device 200 in the first embodiment. Furthermore, the processing flow of the learning device 100 and the inference device 200 in the second embodiment is the same as the processing flow of the learning device 100 and the inference device 200 in the first embodiment.

[0061] The second embodiment differs from the first embodiment in the input text and output text: in the first embodiment, the input text and output text are each expressed in a natural language, but in the second embodiment, the input text and output text are each expressed by a sequence of special tokens.

[0062] An example of the operation of the training data generation unit 110 in the second embodiment will be described with reference to Fig. 7. As shown in Fig. 7, the training data generation unit 110 receives training data for all tasks as input and generates a set of "input image, input text, and correct output text" shown in Fig. 7.

[0063] The input text in the second embodiment is, for example, "<FD_on><AE_on><GE_on> " The input text is always a fixed sequence of tokens.

[0064] where:<FD_on> ,<AE_on> ,<GE_on> are special tokens (hereinafter referred to as switching tokens) that indicate that face detection, age estimation, and gender estimation are to be performed, respectively.<FD_on> ,<AE_on> ,<GE_on> For example,

[0065] The correct output text is a sequence of tokens representing the rectangular area of ​​the face and tokens indicating the age and gender for each face. In the example shown in Figure 7, when a 13-year-old male appears in the rectangular area with the top left and top right coordinates (48, 97) and (80, 119) in the input image, an 8-year-old female appears in the rectangular area with the coordinates (155, 126) and (183, 150), and a 26-year-old male appears in the rectangular area with the coordinates (239, 49) and (272, 75), the output text is " <s><FD_48> <FD_97> <FD_80> <FD_119> <AE_13> <GE_male> <FD_155> <FD_126> <FD_183> <FD_150> <AE_8> <GE_female> <FD_239> <FD_49> <FD_272> <FD_75> <AE_26> <GE_male> <e>" It has been shown that

[0066] where:<FD_n> is a token indicating that the result of face detection is coordinate n (hereinafter referred to as face detection result token).<AE_n> is a token indicating that the result of age estimation is n years old (hereinafter referred to as age estimation result token).<GE_male> ,<GE_female> are tokens indicating that the gender estimation result is male or female (hereinafter, gender estimation result tokens).

[0067] The dictionary used in the face detection and attribute estimation model 121 registers and uses a switching token, a start token, an end token, a face detection result token, an age estimation result token, and a gender estimation result token.

[0068] Third Embodiment Next, a third embodiment will be described. The configurations of the learning device 100 and the inference device 200 in the third embodiment are the same as the configurations of the learning device 100 and the inference device 200 in the first embodiment. Furthermore, the processing flow of the learning device 100 and the inference device 200 in the third embodiment is the same as the processing flow of the learning device 100 and the inference device 200 in the first embodiment.

[0069] The third embodiment differs from the first embodiment in the teacher data that is input to the teacher data generation unit 110. In the first embodiment, only teacher data for all tasks is used as teacher data, but in the third embodiment, teacher data for some tasks is additionally used.

[0070] A "partial task" is a part of the "total tasks." For example, if the "total tasks" are a face detection task, a gender estimation task, and an age estimation task, then the face detection task, the gender estimation task, and the age estimation task are each examples of a "partial task." Also, for example, a task that performs both face detection and gender estimation is also an example of a "partial task."

[0071] For example, if the "partial task" is a gender estimation task, the training data for the partial task is, for example, a collection of "images of a person with their gender annotated."

[0072] Figure 8 shows an example configuration of learning device 100 in the third embodiment. As shown in Figure 8, in the third embodiment, learning device 100 has access to teacher data DB 150, which stores teacher data for all tasks, as well as teacher data DB 160, which stores teacher data for some tasks. The internal configuration of learning device 100 is the same as that in the first embodiment.

[0073] Teacher data for all tasks is input from teacher data DB 150 to teacher data generation unit 110, and teacher data for some tasks is input from teacher data DB 160 to teacher data generation unit 110.

[0074] The training data generation unit 110 in the third embodiment receives training data for all tasks and training data for some tasks as input, and generates and outputs training data that can be used for training the face detection and attribute estimation model 121. The training data that can be used for training the face detection and attribute estimation model 121 is a set of "input image, input text, and correct output text."

[0075] An example of the operation of the training data generation unit 110 in the third embodiment will be described with reference to Fig. 9. Fig. 9 shows an example in which the "partial tasks" are a face detection task, a gender estimation task, an age estimation task, and an age / gender estimation task.

[0076] As shown in FIG. 9, the method of generating sets of "input image, input text, and correct output text" for training data for all tasks is the same as in the first embodiment.

[0077] The input text for the training data for face detection is, for example, text such as "Please perform face detection" as shown in FIG.

[0078] The correct output text is a sequence of coordinate tokens that represent the rectangular area of ​​a face. In the example shown in Figure 9, if a face appears in the rectangular area whose upper left and upper right coordinates in the input image are (30, 121) and (113, 207), another face appears in the rectangular area whose upper right coordinates are (132, 31), (197, 103), and another face appears in the rectangular area whose upper right coordinates are (219, 62), (320, 160), the output text is " <s> <30> <121> <113> <207> <132> <31> <197> <103> <219> <62> <320> <160> <e>" It has been shown that

[0079] The input text and output correct answer text for each of the training data for age estimation, gender estimation, and age / gender estimation are as shown in the example of Fig. 9. The input text and correct answer output text for the training data for some tasks have the format of partially extracted text for the training data for all tasks.

[0080] As described above, the processing flows of the learning device 100 and the inference device 200 are basically the same as those in the first embodiment.

[0081] In the third embodiment, when the learning device 100 performs learning, for example, parameters are updated using a plurality of "input images, input texts, and correct output texts" generated from the training data for all tasks, and then parameters are updated using a plurality of "input images, input texts, and correct output texts" generated from the training data for face detection. In this way, learning is performed using the training data for all tasks, the training data for face detection, the training data for age estimation, the training data for gender estimation, and the training data for age and gender estimation in this order. Note that the order may be any. It is also possible to perform learning using training data for only some tasks, rather than using training data for all tasks.

[0082] 9, when learning is performed using training data such as that shown in Figure 9, the input text for inference by inference device 200 is always "Please perform face detection, age estimation, and gender estimation" in order to solve all tasks. However, inference may be performed using input text other than "Please perform face detection, age estimation, and gender estimation."

[0083] (Fourth Embodiment) Next, a fourth embodiment will be described. The configurations of the learning device 100 and the inference device 200 in the fourth embodiment are the same as the configurations of the learning device 100 and the inference device 200 in the third embodiment. Furthermore, the processing flow of the learning device 100 and the inference device 200 in the fourth embodiment is the same as the processing flow of the learning device 100 and the inference device 200 in the third embodiment. In the fourth embodiment, as in the third embodiment ( FIG. 8 ), a teacher data DB 150 that stores teacher data for all tasks and a teacher data DB 160 that stores teacher data for some tasks are provided outside the learning device 100.

[0084] The fourth embodiment differs from the third embodiment in that the input text and correct output text output by the training data generation unit 110. In the third embodiment, the input text and correct output text are expressed in natural language, but in the fourth embodiment, the input text and correct output text are expressed by a sequence of special tokens.

[0085] An example of the operation of the teacher data generation unit 110 in the fourth embodiment will be described with reference to Fig. 10. As shown in Fig. 10, the teacher data generation unit 110 receives teacher data for all tasks and some tasks as input, and generates a set of "input image, input text, and correct output text" for each teacher data, as shown in Fig. 10.

[0086] The input text in the fourth embodiment is, for example, "<FD_on><AE_on><GE_on> "or"<FD_on><AE_off><GE_off> " is a series of switching talks like "<FD_on><AE_on><GE_on> " is as explained in the second embodiment.

[0087] <FD_off> ,<AE_off> ,<GE_off> are switching tokens indicating that face detection, age estimation, and gender estimation are not performed, respectively.

[0088] The correct output text is a sequence of a token representing a rectangular region of a face, a token indicating age, and a token indicating gender. The method of generating the correct output text for the training data for all tasks is the same as in the second embodiment. The correct output text for the training data for some tasks is generated by using the correct output text for tasks that are not performed.<FD_null> ,<AE_null> ,<GE_null> The absence of a detection / estimation result is expressed by a face detection result token, an age estimation result token, and a gender estimation result token, as shown in the figure. Note that "absence of a detection / estimation result" can be considered as one of the "detection / estimation results."

[0089] As described above, the processing flows of the learning device 100 and the inference device 200 are the same as those in the third embodiment.

[0090] In the fourth embodiment, when the learning device 100 performs learning, for example, parameter updates are performed using multiple "input images, input texts, and correct output texts" generated from the training data for all tasks, and then parameter updates are performed using multiple "input images, input texts, and correct output texts" generated from the training data for face detection. In this way, learning is performed using the training data for all tasks, the training data for face detection, the training data for age estimation, the training data for gender estimation, and the training data for age and gender estimation in this order. Note that the order may be any. It is also possible to perform learning using training data for only some tasks, rather than using training data for all tasks.

[0091] Furthermore, when learning is performed using the training data shown in FIG. 10, the input text is always "<FD_on><AE_on><GE_on> " However, "<FD_on><AE_on><GE_on> Inference may be made using input text other than ".

[0092] (Hardware Configuration Example) Any of the devices described in this embodiment (the learning device 100 and the inference device 200) can be realized, for example, by causing a computer to execute a program. This computer may be a physical computer or a virtual machine on the cloud.

[0093] That is, the device can be realized by executing a program corresponding to the processing performed by the device using hardware resources such as a CPU and memory built into a computer. The program can be recorded on a computer-readable recording medium (such as a portable memory) and stored or distributed. The program can also be provided via a network such as the Internet or email.

[0094] Fig. 11 is a diagram showing an example of the hardware configuration of the computer. The computer in Fig. 11 includes a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, and the like, all of which are interconnected via a bus B. The computer may further include a GPU.

[0095] The program that realizes the processing on the computer is provided by a recording medium 1001, such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.

[0096] The memory device 1003 reads and stores a program from the auxiliary storage device 1002 when an instruction to start the program is received. The CPU 1004 realizes functions related to the device in accordance with the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network, etc. The display device 1006 displays a GUI (Graphical User Interface) or the like according to the program. The input device 1007 is composed of a keyboard, mouse, buttons, a touch panel, etc., and is used to input various operation instructions. The output device 1008 outputs the results of calculations.

[0097] (Summary of Technology According to Embodiments) As described above, in all embodiments, the learning device 100 receives an input image and an input text, and learns a face detection and attribute estimation model 121 (conditional text generation model) so as to continuously output the results of face detection and attribute estimation for each face in text format.

[0098] In this way, by handling face detection and multiple attribute estimation using a conditional text generation model, it is possible to prevent errors in face detection from propagating to subsequent processing, and to improve accuracy by using face area information in attribute estimation, compared to when face detection and attribute estimation are cascaded. In the first and second embodiments, the face area and its attributes are expressed in a single text format, and this is output continuously for each face, making it possible to model using a conditional text generation model.

[0099] In the fourth embodiment, the face detection and attribute estimation model 121 (conditional text generation model) is simultaneously trained with training data for some tasks using switching tokens.

[0100] When handling face detection and multiple attribute estimation using a conditional text generation model, it is usually necessary to collect training data to which annotations for all tasks are simultaneously added, which results in high collection costs. In the third and fourth embodiments, this is reduced by simultaneously learning training data for some tasks.

[0101] In particular, in the fourth embodiment, as described above, a switching token is used as the input text, and the output text also clearly indicates that the task is not the target. This makes it possible to explicitly tell the model which tasks are being treated as the population, more so than in the third embodiment, in which the input text and output text are given in natural language, thereby enabling efficient learning of a model that can solve all tasks.

[0102] (Effects of the Technology According to the Embodiment) The technology according to the embodiment makes it possible to realize the process of detecting human faces from an image containing multiple people and estimating their attributes using a single conditional text generation model.

[0103] This has the effect of preventing errors in face detection from propagating to subsequent processing, compared to when face detection and attribute estimation are cascaded, and it has the effect of improving accuracy by using information about the face area in attribute estimation.

[0104] Furthermore, since it is possible to use data annotated with facial regions and some of the attribute estimation annotations to train the model, the model can be trained without collecting a large amount of training data annotated with facial regions and all of the desired attributes.

[0105] For the face detection and attribute estimation model having the structure described in Non-Patent Document 1, conditional text generation models were constructed using the second embodiment and the fourth embodiment, and comparative verification experiments were conducted.

[0106] As training data, approximately 16,000 images of training data for all tasks were used, and in the fourth embodiment, approximately 128,000 images of training data for face detection, approximately 205,000 images of training data for age estimation, approximately 203,000 images of training data for gender estimation, and approximately 234,000 images of training data for age and gender estimation were also used. As evaluation data, approximately 4,000 images of data for all tasks that were not included in the training data were used.

[0107] The results of the verification experiment are shown in Fig. 12. In Fig. 12, the cumulative score for age estimation is the ratio of areas in which face detection was successful and the age error was within 5 years to all face areas in the evaluation data. The accuracy for gender estimation is the ratio of areas in which face detection was successful and gender estimation was also correct to all face areas in the evaluation data.

[0108] As shown in FIG. 12, it was confirmed that the fourth embodiment made it possible to utilize training data for some tasks, and that accuracy was improved.

[0109] The following additional notes are provided regarding the above-described embodiments.

[0110] <Additional Notes> (Additional Item 1) A learning device that executes learning of a model that performs face detection and attribute estimation, comprising: a memory; and at least one processor connected to the memory, wherein the processor trains the model to continuously output results of the face detection and the attribute estimation in text format from an input image. (Additional Item 2) The learning device according to Additional Item 1, wherein the processor generates an input image, input text, and correct output text from partial task training data that is training data to which annotations for only some of all of the face detection and attribute estimation tasks are added, and outputs the correct output text generated from the partial task training data from the input image generated from the partial task training data and the input text generated from the partial task training data. (Supplementary Item 3) The processor further generates an input image, input text, and correct output text from all-task teacher data, which is teacher data annotated with all of the tasks, and trains the model to output the correct output text generated from the all-task teacher data from the input image generated from the all-task teacher data and the input text generated from the all-task teacher data, wherein the input text generated from the some-task teacher data has a format of text extracted from the input text generated from the all-task teacher data, and the correct output text generated from the some-task teacher data has a format of text extracted from the correct output text generated from the all-task teacher data. (Supplementary Item 4) The learning device of Supplementary Item 2, wherein the input text generated from the some-task teacher data has information indicating that the some of the tasks will be performed and information indicating that tasks other than the some of the tasks will not be performed out of all the tasks. (Appendix 5) The learning device described in Appendix 4, wherein the correct output text generated from the training data for the partial tasks includes information indicating that no results exist for any of the tasks other than the partial tasks among all the tasks.(Supplementary Item 6) An inference device that performs face detection and attribute estimation, comprising: a memory; and at least one processor connected to the memory, wherein the processor takes a training data image as input, inputs the image to a model that has been trained to continuously output the face detection results and the attribute estimation results in text format, and obtains the face detection results for the image and the attribute estimation results for the image from the model. (Supplementary Item 7) A learning method executed by a learning device that executes learning of a model that performs face detection and attribute estimation, comprising a learning step of training the model to continuously output the face detection results and the attribute estimation results in text format from an input image. (Supplementary Item 8) A non-transitory storage medium storing a program that causes a computer to function as a learning device that executes learning of a model that performs face detection and attribute estimation, the non-transitory storage medium storing a program that causes the computer to function as a learning unit that trains the model to continuously output the face detection results and the attribute estimation results in text format from an input image.

[0111] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims.

[0112] REFERENCE SIGNS LIST 100 Learning device 110 Teacher data generation unit 120 Face detection and attribute estimation unit 121 Face detection and attribute detection model 130 Parameter update unit 150, 160 Teacher data DB 1000 Drive device 1001 Recording medium 1002 Auxiliary storage device 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input device 1008 Output device< / e> < / s> < / e> < / s> < / e> < / s> < / n> < / e> < / s> < / e> < / s> < / e> < / s>

Claims

1. A learning device that performs learning of a model that performs face detection and attribute estimation, comprising a learning unit that trains the model so that the results of the face detection and the results of the attribute estimation are continuously output in text format from an input image.

2. The learning device of claim 1 further comprises a training data generation unit that generates an input image, an input text, and a correct output text from partial task training data, which is training data to which annotations for only some of all tasks of the face detection and the attribute estimation are added, and the learning unit trains the model so as to output the correct output text generated from the partial task training data from the input image generated from the partial task training data and the input text generated from the partial task training data.

3. The learning device described in claim 2, wherein the teacher data generation unit generates an input image, an input text, and a correct output text from all-task teacher data, which is teacher data to which annotations for all of the tasks have been added; the learning unit trains the model so as to output the correct output text generated from the all-task teacher data from the input image generated from the all-task teacher data and the input text generated from the all-task teacher data; the input text generated from the partial task teacher data has the format of text extracted from the input text generated from the all-task teacher data, and the correct output text generated from the partial task teacher data has the format of text extracted from the correct output text generated from the all-task teacher data.

4. The learning device described in claim 2, wherein the input text generated from the partial task training data includes information indicating that the partial tasks will be performed and information indicating that all of the tasks except the partial tasks will not be performed.

5. A learning device as described in claim 4, wherein the correct output text generated from the training data for the partial tasks includes information indicating that there are no results for tasks other than the partial tasks among all the tasks.

6. An inference device that performs face detection and attribute estimation, comprising an estimation unit that inputs a training data image into a model that has been trained to continuously output the results of the face detection and the results of the attribute estimation in text format, and obtains the results of face detection for the image and the results of attribute estimation for the image from the model.

7. A learning method executed by a learning device that executes learning of a model that performs face detection and attribute estimation, the learning method comprising a learning step of training the model so as to continuously output the results of the face detection and the results of the attribute estimation in text format from an input image.

8. A program that causes a computer to function as a learning device that executes learning of a model that performs face detection and attribute estimation, and causes the computer to function as a learning unit that trains the model so that the results of the face detection and the results of the attribute estimation are continuously output in text format from an input image.

Images