Data collection method and data collection device

Abstract

Provided is a data collection method for use in learning for estimation based on a human facial expression, the data collection method comprising: a step for acquiring arousal and valence for each data item contained in a dataset of human facial expressions; a step for plotting the acquired arousal and valence in an arousal-valence coordinate space; a step for determining a data-deficient region in the coordinate space; a step for synthesizing new data corresponding to the data-deficient region by using a plurality of data items that are contained in the dataset and for which the arousal and valence are known; and a step for adding the new synthesized data to the dataset.

Description

Data collection method and data collection device

[0001] This application claims priority to Japanese Patent Application No. 2024-151748, filed on September 3, 2024, the contents of which are incorporated herein by reference.

[0002] Humans can interact with each other using multiple communication channels. For example, facial expressions and gestures are commonly used as means of conveying intentions and emotions. In the field of human-robot interaction (HRI), research is underway on estimation models that estimate human intentions and emotions by analyzing such human non-verbal information (cues). One approach proposed is to predict arousal and valence (AV) from human facial expressions (see, for example, Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2023-129152

[0004] To achieve smooth communication between humans and robots, it is important to improve the accuracy of the estimation model, which requires the preparation of sufficient training data for the estimation model. However, human facial expressions vary widely and differ depending on the subject (race, gender, etc.). Furthermore, the appearance of facial expressions varies depending on the environment, such as lighting conditions. It has not been easy to prepare such a wide variety of training data for human facial expressions.

[0005] The present invention has been made in consideration of the above-mentioned problems, and aims to provide a data collection method and a data collection device that enable the collection of diverse data to be used for learning for estimation based on human facial expressions.

[0006] (1) In order to achieve the above-mentioned object, a data collection method according to one aspect of the present invention is a data collection method used for learning to make inferences based on human facial expressions, and includes the steps of acquiring the arousal and valence for each piece of data included in a dataset of human facial expressions, plotting the acquired arousal and valence in a coordinate space of arousal and valence, determining areas in the coordinate space where there is a lack of data, using a plurality of pieces of data included in the dataset with known arousal and valence to synthesize new data corresponding to the areas where there is a lack of data, and adding the synthesized new data to the dataset.

[0007] (2) In the data collection method of (1) above, the step of synthesizing the new data may include a step of synthesizing facial expression information contained in a plurality of data whose arousal levels and valence levels are known, and a step of synthesizing the numerical values ​​of arousal levels and valence levels contained in a plurality of data whose arousal levels and valence levels are known.

[0008] (3) In the data collection method of (2) above, the facial expression information may include facial expression coefficients extracted from three-dimensional information of a human face.

[0009] (4) The data collection methods (1) to (3) above may further include a step of obtaining human facial expression coefficients by performing three-dimensional model fitting on the data included in the data set.

[0010] (5) In the data collection methods (1) to (3) above, the step of determining the area where data is insufficient may further include a step of classifying the data included in the coordinate space based on the arousal level and emotional valence of each piece of data included in the data set, and a step of determining, as the area where data is insufficient, the area corresponding to the classification with the smallest number of pieces of data as a result of classifying the data included in the coordinate space.

[0011] (6) In the data collection method of (5) above, the step of synthesizing the new data may include a step of synthesizing first data included in an area corresponding to a classification with the smallest number of data and second data different from the first data included in the data set.

[0012] (7) In the data collection method of (6) above, the second data may be randomly selected from data included in the data set.

[0013] (8) In the data collection method of (6) above, the second data may be selected from data included in an area in the coordinate space corresponding to a classification having the smallest number of data.

[0014] (9) In the data collection method of (6) above, the second data may be data of a second facial expression similar to a first facial expression corresponding to the first data selected from data included in the data set.

[0015] (10) In the data collection method of (6) above, the step of synthesizing the new data may include a step of fixing coefficients randomly selected from the facial expression coefficients included in the first data, and replacing the remaining coefficients that were not selected with coefficients of the second data.

[0016] (11) In the data collection method of (6) above, the step of synthesizing the new data may include a step of fixing coefficients randomly selected from the facial expression coefficients included in the first data, and weighting and synthesizing the remaining coefficients that were not selected and the coefficients of the second data.

[0017] (12) In the data collection method of (6) above, the step of synthesizing the new data may include a step of weighting and synthesizing all of the facial expression coefficients included in the first data and the facial expression coefficients included in the second data.

[0018] (13) Another aspect of the present invention provides a data collection device used for learning to make inferences based on human facial expressions, and includes an acquisition unit that acquires the arousal and valence of each piece of data included in a data set of human facial expressions; a generation unit that plots the acquired arousal and valence in a coordinate space of arousal and valence; a determination unit that determines areas in the coordinate space where there is a lack of data; and a synthesis unit that uses multiple pieces of data included in the data set with known arousal and valence to synthesize new data corresponding to the areas where there is a lack of data, and adds the synthesized new data to the data set.

[0019] According to (1) to (13), it is possible to collect a variety of data to be used for learning for estimation based on human facial expressions.

[0020] FIG. 1 is a diagram illustrating an overview of a data collection method according to an embodiment. FIG. 2 is a diagram illustrating input / output data of a 3DMM according to an embodiment. FIG. 3 is a graph illustrating a distribution state of video data plotted in a coordinate space of arousal and valence according to an embodiment. FIG. 4 is a diagram illustrating how extended data according to an embodiment is generated. FIG. 5 is a graph illustrating a distribution state of video data (including extended data) plotted in a coordinate space of arousal and valence according to an embodiment. FIG. 6 is a diagram illustrating an example of a data collection system according to an embodiment. FIG. 7 is a flowchart illustrating an example of the processing flow of a data collection device according to an embodiment.

[0021] A data collection method and a data collection device of the present invention will be described below with reference to the drawings. The data collection method and the data collection device of the present invention enable the collection of diverse data (training data) used for learning estimation based on human facial expressions. A training model generated by training the collected data is used, for example, in HRI setup between a robot and a human (e.g., a group of multiple people).

[0022] <Overview of Data Collection Method> Fig. 1 is a diagram illustrating an overview of a data collection method according to an embodiment. The data collection method includes a feature extraction process and a 3D morphable model (3DMM)-based data augmentation process. The feature extraction process generates a three-dimensional data source (referred to as a "3D source") from a two-dimensional data source (referred to as a "2D source") that serves as the source of data augmentation, and extracts features of the generated 3D source. Meanwhile, the 3DMM-based data augmentation process determines areas (referred to as "augmented areas") where data is insufficient (data that should be augmented), and performs processing to augment the data in the determined augmented areas.

[0023] [Feature Extraction Process] (1) A video-based dataset is used as the source data for 2D source data augmentation. This video data is obtained, for example, from a video database publicly available via a network. The video database is, for example, the SEWA database (see Reference 1). This SEWA database stores a comprehensive dataset in which people appearing in each frame of video data are labeled (annotated) with continuous labels of arousal and emotional valence (referred to as "AV value"). Arousal is an indicator of emotional intensity. Meanwhile, emotional valence is an indicator of how good (pleasant, enjoyable) or bad (unpleasant, unpleasant) the emotion is. These labels are manually assigned by multiple administrators for targets of various ages, genders, cultures (nationalities), etc. These labels are, for example, continuous numerical values ​​ranging from -1 to 1.

[0024] Reference 1: Kossaifi, Jean et al., “Sewa DB: A rich database for audio-visual emotion and sentiment research in the wild.” IEEE transactions on pattern analysis and machine intelligence 43.3 (2019): 1022-1040.

[0025] (2) 3D Source: Using the above 2D source, a three-dimensional model of a face (referred to as a "3D model") is generated by performing three-dimensional model fitting (referred to as "3D model fitting"). For example, an EMOCA (Emotion-driven monocular face capture and animation) model, which is a 3D model model (see Reference 2), is used for the 3D model fitting. The 3D model model is a statistical model that represents facial expressions and shapes. The 3D model model is known to be highly robust against different lighting conditions and head positions. The parameters of a specific face are a set of coefficients that modify the model face so that it matches the target face. The EMOCA model incorporates the perceptual emotion of the face into its loss function. This allows the EMOCA model to fit the 3D model to the face under conditions where facial emotions are preserved.

[0026] Reference 2: Danecek, Radek, Michael J. Black, and Timo Bolkart. “EMOCA: Emotion-driven monocular face capture and animation.” Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2022.

[0027] (3) Facial Expression Coefficients (Facial Expression Information) FIG. 2 is a diagram illustrating input and output data of a 3DMM according to an embodiment. As shown in FIG. 2, two-dimensional data (referred to as "2D data"), which is a facial image extracted from a 2D source, is input to the 3DMM, and three-dimensional model data (referred to as "3D model data" or "3D information") and facial expression coefficients (FECo) representing the facial expressions of the 3D model data are obtained as output. The facial expression coefficients FECo represent a set of coefficients that indicate how facial muscles change when a person expresses an expression. The facial expression coefficients FECo are represented, for example, by a 50-dimensional vector with values ​​ranging from -4 to 4. The output of the 3DMM includes the above-mentioned facial expression coefficients as well as the position and shape of the head.

[0028] [3DMM-Based Data Augmentation Process] (4) Classification (Clustering) A set of video data acquired from the video database is classified (clustered). This clustering is performed based on the average values ​​(referred to as "average AV values") of arousal and valence calculated for each video data and expressed in a coordinate space of arousal and valence (referred to as "AV space"). FIG. 3 is a graph showing the distribution of video data plotted in the coordinate space of arousal and valence according to an embodiment. As shown in FIG. 3, an average AV value is calculated and plotted for each video data. The average AV value can be calculated, for example, by dividing the sum of the arousal and valence values ​​labeled for each frame included in the video data by the number of frames. Video data with the same (or similar) average AV value is represented by a color map (or grayscale) according to the frequency of their occurrence. By checking this graph, a user can identify areas of arousal and valence where data is insufficient in the video data set.

[0029] (5) Expansion Area Determination, (6) AV Space Based on the above classification results, the video data (referred to as "source video data") that will be used for synthesis is selected. For example, in the AV space shown in Figure 3, areas of insufficient arousal and emotional valence are identified. Identification methods include, for example, a frame-based method and a video-based method. The source video data is an example of "first data."

[0030] Frame-based blending involves combining frames of source video data, determining the most similar frame in AV space based on AV values, and then combining these two frames to generate a new frame that represents the expanded sample frame.

[0031] In the video-based blending method, the blending is performed on a source video data basis. Here, the most similar video data in the AV space is determined based on the average AV value. The two determined video data are then blended to generate new video data that will serve as the extended sample.

[0032] (7) Augmented Space Video data that are close to each other in the AV space are determined, and the determined video data are combined. This allows for the expansion of video data corresponding to various facial expressions. K-means clustering and discrete K-bin clustering are used to select source video data. The K-means clustering method uses the K-means clustering algorithm. The average AV value of each video data is used as input. A greedy method is used to select source video data from the cluster with the smallest number of video data. Discrete K-bin clustering separates the AV space into K bins based on the average AV value of each video data. That is, arousal and emotional valence are divided into K bins. Then, a greedy method is used to select source video data from the bin with the smallest number of video data.

[0033] (8) and (9) Selection of Target Video Data: Here, target video data to be synthesized is determined. The target video data is data to be synthesized with the source video data. The target video data is selected for each source video data. For selection, a random selection method, a proximity selection method, and a similarity selection method are used. The random selection method randomly selects video data from a dataset. That is, the random selection method randomly selects two video data to be synthesized from the dataset. The proximity selection method selects video data that is similar to the source video data. "Similar" means, for example, belonging to the same cluster as the source video data. That is, the proximity selection method selects two video data that have similar values ​​in the AV space (arousal and emotional valence). The similarity selection method selects video data that is similar to the source video data. "Similar" means, for example, that the mean and variance distances between the source video data and the target video data are minimized. The similarity selection method selects target video data that is similar in the feature space of the 3DMM. This feature space of the 3DMM is represented, for example, by facial expression coefficients extracted using an EMOCA model. In other words, the similarity selection method selects video data showing similar facial expressions. The similarity selection method calculates the distance between two video data by subtracting and resting the mean and variance of a series of facial expression coefficients, and selects the video data with the smallest distance. The target video data is an example of "second data."

[0034] (10) Combining Facial Expression Coefficients and AV Values: Here, faces (facial expression coefficients) and AV values ​​(AV labels) are combined for the source video data and target video data selected as described above. The combination of facial expression coefficients is performed for each frame using a sequence of consecutive facial expression coefficients (FECSeq) included in the source video data and a sequence of consecutive facial expression coefficients (FECSeq) included in the target video data. The combination of facial expression coefficients is performed using a random blending method, a selective weighted blending method, and a full weighted blending method. The random blending method randomly fixes a subset of the facial expression coefficients of the source video data and replaces the remaining facial expression coefficients with the facial expression coefficients of the target video data. The selective weighted blending method randomly fixes a subset of the facial expression coefficients of the source video data and combines the remaining facial expression coefficients with the facial expression coefficients of the target video data by weighting them. The weight value is, for example, a random value between 0.25 and 0.75. The full weighted blending method weights and blends all the expression coefficients of the source video data with those of the target video data, where the weights are random values ​​between 0.25 and 0.75, for example.

[0035] The synthesis of AV values ​​(AV labels) is performed by calculating the average values ​​(weighted average values) of the arousal levels and emotional valences of the source video data and the target video data for each frame.

[0036] (11) Video Data with New AV Values ​​As a result of combining the facial expression coefficients and AV values, video data (extended data) with new AV values ​​(continuous AV value labels) is generated. Figure 4 is a diagram illustrating how extended data according to an embodiment is generated. As shown in Figure 4, source video data (average arousal level = 0.04, average valence level = 0.58) and target video data (average arousal level = 0.00, average valence level = 0.55) are combined to generate extended data (average arousal level = 0.02, average valence level = 0.57).

[0037] The newly generated extended data is added to a set (data set) of video data acquired from the video database. Figure 5 is a graph showing the distribution of video data (including extended data) plotted in a coordinate space of arousal and valence according to an embodiment. The graph in Figure 5 shows that new data has been added corresponding to areas where data is lacking in the graph in Figure 3. In this way, it is possible to acquire new data corresponding to areas where data is lacking by using multiple data with known arousal and valence included in video data acquired from the video database.

[0038] (12) Consistent Sequence Query The Consistent Sequence Query is the result of combining the above (5) selection of an extended region, (8) and (9) selection of target video data, and (10) synthesis of facial expression coefficients and AV values. In particular, in this embodiment, the smoothest extended data can be obtained when the combination of (5) selection of an extended region is the "video-based method," (8) and (9) selection of target video data is the "similar selection method," and (10) blending of facial expression coefficients and AV values ​​is the "full-weighted blending method" is adopted.

[0039] <System Configuration> Next, a system configuration for realizing the data collection method of this embodiment will be described. Fig. 6 is a diagram showing an example of a data collection system S according to this embodiment. The data collection system S includes, for example, a data collection device 1 and a video database 3. These components can communicate with each other via a network NW. The network NW includes the Internet, a WAN (Wide Area Network), a LAN (Local Area Network), a public line, a provider device, a dedicated line, a wireless base station, etc.

[0040] [Data Collection Device] The data collection device 1 performs data extension processing. The data collection device 1 is operated by a user who performs data extension processing. The data collection device 1 is, for example, a personal computer. The data collection device 1 may be configured as a server device that can be accessed from a user's terminal device (not shown) via a network NW.

[0041] The data collection device 1 includes, for example, a control unit 10, a communication unit 12, an input interface 14, a display unit 16, and a storage unit 18. The communication unit 12 communicates with an external device such as a video database 3 via a network NW. The communication unit 12 is, for example, a network card for connecting to the network NW.

[0042] The input interface 14 receives various input operations from the user, converts the received input operations into electrical signals, and outputs the electrical signals to the control unit 10. The input interface 14 includes, for example, a keyboard, a mouse, a touch panel, and the like.

[0043] The display unit 16 displays various types of information. For example, the display unit 16 displays various images generated by the control unit 10, a GUI (Graphical User Interface) for receiving various input operations from the user, etc. The display unit 16 is an LCD (Liquid Crystal Display) or the like.

[0044] The storage unit 18 stores various types of information. The storage unit 18 is, for example, a random access memory (RAM), a semiconductor memory element such as a flash memory, or a storage device (a storage device having a non-transitory storage medium) such as a hard disk. The storage unit 18 may also be realized by an external storage device connected via the network NW.

[0045] The control unit 10 includes, for example, an acquisition unit 101, a fitting unit 103, a graph generation unit 105, a determination unit 107, a selection unit 109, a synthesis unit 111, and a display control unit 113. Each functional unit of the control unit 10 is realized by, for example, a hardware processor such as a CPU (Central Processing Unit) executing a program (software).

[0046] The acquisition unit 101 acquires video data from the video database 3 via the network NW. The acquisition unit 101 also acquires the arousal level and valence for each frame included in the acquired video data (a data set of human facial expressions). The acquisition unit 101 also acquires user instructions input via the input interface 14. The acquisition unit 101 is an example of an "acquisition unit."

[0047] The fitting unit 103 performs 3D model fitting on the 2D data of a human face for each frame included in the acquired video data, thereby acquiring 3D model data and the facial expression coefficient FECo of the 3D model data. For example, the fitting unit 103 inputs the 2D data of a human face into a 3DMM, and acquires the 3D model data and the facial expression coefficient FECo of the 3D model data as its output. The fitting unit 103 is an example of a "fitting unit."

[0048] The graph generating unit 105 plots the acquired arousal levels and valence levels in a coordinate space of arousal levels and valence levels to generate a graph showing the distribution of video data, for example, as shown in Fig. 3. The graph generating unit 105 is an example of a "generating unit."

[0049] The determination unit 107 determines an extended region (a region lacking data) in the coordinate space of arousal and valence. The determination unit 107 classifies the data included in the coordinate space based on the arousal and valence of each piece of data, and determines the region corresponding to the classification with the smallest number of pieces of data as the region lacking data. For example, the determination unit 107 determines the region lacking data using a K-means clustering method or a discrete K-bin clustering method. The determination unit 107 is an example of a "determination unit."

[0050] The selection unit 109 selects source video data from the data included in the area corresponding to the classification with the smallest number of data, and selects target video data different from the source video data from the data set included in the coordinate space of arousal and valence. For example, the selection unit 109 selects the target video data using a random selection technique, a proximity selection technique, or a similarity selection technique. The selection unit 109 is an example of a "selection unit."

[0051] The synthesis unit 111 uses multiple pieces of data (i.e., source video data and target video data) with known arousal levels and valence levels included in a data set in the coordinate space of arousal levels and valence levels to synthesize new data corresponding to areas where data is lacking, and adds the newly synthesized data to the data set. The synthesis unit 111 synthesizes facial expression information included in the multiple pieces of data with known arousal levels and valence levels, and synthesizes the numerical values ​​of arousal levels and valence levels included in the multiple pieces of data with known arousal levels and valence levels. For example, the synthesis unit 111 performs the synthesis process using a random blending method, a selective weighted blending method, or a full weighted blending method. The synthesis unit 111 is an example of a "synthesizing unit."

[0052] The display control unit 113 controls the display of the display unit 16. The display control unit 113 causes the display unit 16 to display images generated by the control unit 10 (such as the data expansion results and a graph showing the distribution status of video data as shown in FIG. 3 ), a GUI for accepting various input operations from the user, and the like. The display control unit 113 is an example of a "display control unit."

[0053] [Video Database] The video database 3 is a storage device that stores multiple video data. Each video data is assigned a consecutive label indicating the arousal level and emotional valence of the person appearing in each frame of the video. This video data is made public, for example, via a network NW. The video database 3 is, for example, a SEWA database. The video database 3 is a storage device (a storage device with a non-transitory storage medium) such as a RAM, a semiconductor memory element such as a flash memory, or a hard disk. The video database 3 may be incorporated into the data collection device 1.

[0054] <Processing Flow> Next, the processing flow of the data collection method according to the embodiment will be described. Fig. 7 is a flowchart showing an example of the processing flow of the data collection device according to the embodiment. The processing shown in Fig. 7 is started, for example, when a user operates the input interface 14 of the data collection device 1 to input a data collection instruction.

[0055] First, the acquisition unit 101 acquires video data from the video database 3 (step S101). The acquisition unit 101 also acquires the arousal level and valence for each frame included in the acquired video data.

[0056] Next, the fitting unit 103 performs 3D model fitting on the 2D data of a human face for each frame included in the acquired video data, thereby acquiring 3D model data and the facial expression coefficient FECo of the 3D model data (step S103). For example, the fitting unit 103 inputs the 2D data of a human face into a 3DMM, and acquires the 3D model data and the facial expression coefficient FECo of the 3D model data as its output.

[0057] Next, the graph generation unit 105 plots the acquired arousal levels and valence levels in a coordinate space of arousal levels and valence levels to generate a graph showing the distribution of video data, for example, as shown in Figure 3 (step S105).

[0058] Next, the determination unit 107 classifies the data contained in the coordinate space based on the arousal level and emotional valence of each piece of data, and determines the area corresponding to the classification with the smallest number of pieces of data as an expanded area (area with insufficient data) (step S107).

[0059] Next, the selection unit 109 selects source video data from the data included in the area corresponding to the classification with the smallest number of data (step S109). The selection unit 109 also selects target video data different from the source video data from the data set included in the coordinate space of arousal and valence (step S111). For example, the selection unit 109 selects the target video data using the random selection method, the proximity selection method, or the similarity selection method.

[0060] Next, the combining unit 111 combines the source video data and the target video data and adds the combined new data to the data set (step S113). For example, the combining unit 111 combines the source video data and the target video data using the random blending method, the selective weighted blending method, or the full weighted blending method.

[0061] Next, the display control unit 113 displays information indicating the obtained data collection results (data contained in the coordinate space of arousal and valence) on the display unit 16 (step S115). The user can understand the data collection information by checking the information displayed on the display unit 16.

[0062] Next, the acquisition unit 101 determines whether or not an instruction to end data collection has been received from the user (step S117). If the acquisition unit 101 determines that an instruction to end data collection has not been received (S117; NO), the acquisition unit 101 returns to step S107 and repeats the subsequent processing. On the other hand, if the acquisition unit 101 determines that an instruction to end data collection has been received (S117; YES), the acquisition unit 101 ends the processing of this flowchart.

[0063] According to the embodiment described above, it is possible to collect a variety of data (learning data) to be used for learning estimation based on human facial expressions. A learning model generated by learning the collected data can be used, for example, in HRI setup between a robot and a human (e.g., a group of multiple people).

[0064] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0065] 1...data collection device, 3...video database, 10...control unit, 12...communication unit, 14...input interface, 16...display unit, 18...storage unit, 101...acquisition unit, 103...fitting unit, 105...graph generation unit, 107...determination unit, 109...selection unit, 111...synthesis unit, 113...display control unit, NW...network, S...data collection system

Claims

1. A data collection method used in learning to make inferences based on human facial expressions, comprising: a step of acquiring arousal and valence for each piece of data included in a data set of human facial expressions; a step of plotting the acquired arousal and valence in a coordinate space of arousal and valence; a step of determining areas in the coordinate space where there is a lack of data; a step of synthesizing new data corresponding to the areas where there is a lack of data using a plurality of pieces of data included in the data set whose arousal and valence are known; and a step of adding the synthesized new data to the data set.

2. The data collection method of claim 1, wherein the step of synthesizing new data includes the steps of: synthesizing facial expression information contained in a plurality of data sets whose arousal levels and valence levels are known; and synthesizing the numerical values ​​of arousal levels and valence levels contained in a plurality of data sets whose arousal levels and valence levels are known.

3. The data collection method according to claim 2, wherein the facial expression information includes facial expression coefficients extracted from three-dimensional information of a human face.

4. The data collection method according to any one of claims 1 to 3, further comprising the step of performing three-dimensional model fitting on the data included in the data set to obtain human facial expression coefficients.

5. A data collection method according to any one of claims 1 to 3, wherein the step of determining an area where data is insufficient further includes the steps of: classifying the data contained in the coordinate space based on the arousal and emotional valence of each piece of data contained in the data set; and determining, as the area where data is insufficient, the area corresponding to the classification with the smallest number of pieces of data as a result of classifying the data contained in the coordinate space.

6. The data collection method according to claim 5, wherein the step of synthesizing the new data includes a step of synthesizing first data included in an area corresponding to the classification with the smallest number of data and second data different from the first data included in the data set.

7. The data collection method according to claim 6, wherein the second data is randomly selected from the data included in the data set.

8. The data collection method according to claim 6, wherein the second data is selected from among data included in an area in the coordinate space corresponding to a classification having the smallest number of data.

9. The data collection method according to claim 6, wherein the second data is data of a second facial expression similar to a first facial expression corresponding to the first data selected from data included in the data set.

10. The data collection method according to claim 6, wherein the step of synthesizing new data includes the step of fixing coefficients randomly selected from the facial expression coefficients included in the first data, and replacing the remaining coefficients that were not selected with coefficients of the second data.

11. The data collection method according to claim 6, wherein the step of synthesizing the new data includes the step of fixing coefficients randomly selected from the facial expression coefficients included in the first data, and weighting and synthesizing the remaining coefficients that were not selected and the coefficients of the second data.

12. The data collection method according to claim 6, wherein the step of synthesizing the new data includes a step of weighting and synthesizing all of the facial expression coefficients included in the first data and the facial expression coefficients included in the second data.

13. A data collection device used for learning to make inferences based on human facial expressions, comprising: an acquisition unit that acquires the arousal and valence of each piece of data included in a data set of human facial expressions; a generation unit that plots the acquired arousal and valence in a coordinate space of arousal and valence; a determination unit that determines areas in the coordinate space where there is a lack of data; and a synthesis unit that uses a plurality of pieces of data included in the data set with known arousal and valence to synthesize new data corresponding to the areas where there is a lack of data, and adds the synthesized new data to the data set.

Images