JUDGMENT PROGRAM, JUDGMENT METHOD, AND JUDGMENT DEVICE

Abstract

To generate training data for use in estimation of an expression.SOLUTION: A determination program according to the present invention makes a computer carry out processing formed of: a step of acquiring an image including a face attached with a first marker and a second marker; a step of specifying a first position of the first marker and a second position of the second marker both included in the image; a step of determining, based on difference between the first position and a reference position of the first marker, and difference between a positional relationship between the first position and the second position and a reference positional relationship between the first marker and the second marker, generation strength of a specific expression on a face; and a step of outputting the generation strength of the determined specific expression in association with the image.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a determination technique. [Background technology]

[0002] Facial expressions play an important role in non-verbal communication. Facial expression estimation technology is important for understanding and sensing people. A method called AU (Action Unit) is known as a tool for estimating facial expressions. AU is a method for breaking down and quantifying facial expressions based on facial parts and facial muscles.

[0003] The AU estimation engine is based on machine learning using a large amount of training data, which consists of facial expression image data and the occurrence and intensity of each AU. The occurrence and intensity of the training data are also annotated by experts called coders. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] US Patent Application Publication No. 2019 / 0294868 [Patent Document 2] US Patent Application Publication No. 2017 / 0286759 [Patent Document 3] JP 2020-160792 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, the conventional methods have a problem that it may be difficult to generate training data for facial expression estimation. For example, annotation by a coder is expensive and time-consuming, so it is difficult to create a large amount of data. In addition, it is difficult to accurately capture small changes in the movement measurement of each part of the face by image processing of the face image, and it is difficult for a computer to determine AUs from a face image without human judgment. Therefore, it is difficult for a computer to generate training data in which AU labels are attached to face images without human judgment.

[0006] In one aspect, an object of the present invention is to provide a determination program, a determination method, and a determination device that can generate training data for facial expression estimation. [Means for solving the problem]

[0007] A judgment program according to one embodiment causes a computer to execute a process of acquiring an image including a face with a first marker and a second marker, identifying a first position of the first marker and a second position of the second marker included in the image, judging an occurrence intensity of a specific facial expression in the face based on a difference between the first position and a reference position of the first marker and a difference between the positional relationship between the first position and the second position and a reference positional relationship between the first marker and the second marker, and outputting the determined occurrence intensity of the specific facial expression in association with the image. Effect of the Invention

[0008] According to one embodiment, training data for facial expression estimation can be generated. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a machine learning system. [Diagram 2] FIG. 2 is a diagram showing an example of camera arrangement. [Diagram 3] FIG. 3 is a block diagram illustrating an example of a functional configuration of the determination device. [Figure 4] FIG. 4 is a diagram illustrating an example of the movement of the marker. [Diagram 5] FIG. 5 is a diagram for explaining an example of a method for determining the occurrence intensity. [Figure 6] FIG. 6 is a diagram for explaining an example of a method for determining the occurrence intensity. [Figure 7] FIG. 7 is a diagram showing an example of the AU4 definition vector. [Figure 8A] FIG. 8A is a diagram showing an example of a marker position. [Figure 8B] FIG. 8B is a diagram showing an example of the marker positions. [Figure 8C] FIG. 8C is a diagram showing an example of the marker positions. [Figure 9] FIG. 9 is a diagram showing an example of the movement amount of the marker. [Figure 10] FIG. 10 is a diagram showing an example of the movement amount of the marker. [Figure 11] FIG. 11 is a diagram showing an example of an estimated value of the occurrence intensity of AUs. [Figure 12] FIG. 12 is a diagram showing an example of an estimated value of the occurrence intensity of AUs. [Figure 13] FIG. 13 is a diagram for explaining an example of a method for creating a mask image. [Figure 14] FIG. 14 is a diagram for explaining an example of a method for deleting a marker. [Figure 15] FIG. 15 is a flowchart showing a procedure for determining the occurrence intensity. [Figure 16] FIG. 16 is a flowchart showing a procedure for generating training data. [Figure 17] FIG. 17 is a diagram illustrating an example of a hardware configuration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the judgment program, judgment method, and judgment device according to the present application will be described with reference to the attached drawings. Each embodiment merely shows one example or aspect, and the numerical values, range of functions, usage scenes, etc. are not limited by such examples. Each embodiment can be appropriately combined within a range that does not cause contradiction in the processing contents.

[0011] An example of the configuration of a machine learning system according to an embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the configuration of a machine learning system. As shown in Fig. 1, a machine learning system 1 includes an RGB (Red, Green, Blue) camera 31, an IR (infrared) camera 32, a determination device 10, a generation device 20, and a machine learning device 50.

[0012] As shown in FIG. 1, the RGB camera 31 and the IR camera 32 may be arranged facing the face of a person to which a marker is attached. For example, the RGB camera 31 is a general digital camera that receives visible light and generates an image. For example, the IR camera 32 senses infrared light. The marker is, for example, an IR reflective (retroreflective) marker. The IR camera 32 can perform motion capture by utilizing IR reflection by the marker. In the following description, the person to be imaged is called a subject.

[0013] The determination device 10 acquires an image 110 captured by the RGB camera 31 and a result 120 of motion capture by the IR camera 32. Then, the determination device 10 determines an AU generation intensity 121 based on the amount of movement of a marker assigned to each AU. Then, the determination device 10 outputs the AU generation intensity 121 to the generation device 20 in association with the image 110 captured by the RGB camera 31.

[0014] The "occurrence intensity" referred to here may be, for example only, data in which the intensity at which each AU occurs is expressed in a five-level rating from A to E, and annotations are performed such as "AU1:2, AU2:5, AU4:1, ...". Note that the occurrence intensity is not limited to being expressed in a five-level rating, and may be expressed, for example, in a two-level rating (presence or absence of occurrence). In this case, for example only, a rating of 2 or more on the five-level rating may be expressed as "present", while a rating of less than 2 may be expressed as "absent".

[0015] The generating device 20 acquires the image 110 of the RGB camera 31 and the occurrence intensities 121 of the AUs output by the determining device 10. Then, the generating device 20 generates an image 111 in which the markers are deleted by image processing from the image 110 captured by the RGB camera 31. Furthermore, the generating device 20 generates a dataset including training data TR in which the image 111 from which the markers have been deleted is associated with the occurrence intensities 121 of the AUs that are taken as correct labels. Then, the generating device 20 outputs the dataset of the training data TR to the machine learning device 50.

[0016] The machine learning device 50 executes machine learning using a data set of the training data TR output from the generation device 20. For example, the machine learning device 50 sets the image 111 from which the markers have been deleted as an explanatory variable of the machine learning model m, sets the occurrence intensity 121 of the AU that is the correct label as the objective variable of the machine learning model m, and trains the machine learning model m according to a machine learning algorithm such as deep learning. In this way, a machine learning model M for estimating the occurrence intensity of the AU from an image is generated.

[0017] Next, the arrangement of the cameras will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of the arrangement of the cameras. As shown in Fig. 2, a plurality of IR cameras 32 may constitute a marker tracking system. In this case, the marker tracking system can detect the position of the IR reflective marker by stereo photography. In addition, the relative positional relationship between each of the plurality of IR cameras 32 is assumed to be corrected in advance by camera calibration.

[0018] Furthermore, multiple markers are attached to the subject's face to be imaged so as to cover the target AUs (e.g., AU1 to AU28). The positions of the markers change according to changes in the subject's facial expression. For example, marker 401 is placed near the base of the eyebrows. Furthermore, markers 402 and 403 are placed near the facial line. The markers may be placed on one or more AUs and on skin corresponding to the movement of facial muscles. Furthermore, the markers may be placed to avoid skin where texture changes are significant due to wrinkling, etc.

[0019] Furthermore, the subject wears the device 40 with a reference point marker attached. Even if the facial expression of the subject changes, the position of the reference point marker attached to the device 40 does not change. Therefore, the determination device 10 can detect a change in the position of the marker attached to the face based on a change in the relative position from the reference point marker. Furthermore, by setting the number of reference markers to three or more, the determination device 10 can identify the position of the marker in three-dimensional space.

[0020] The device 40 is, for example, a headband, and a reference point marker is placed outside the contour of the face. The device 40 may also be a VR headset and a mask made of a hard material. In this case, the determination device 10 can use the rigid surface of the device 40 as a reference point marker.

[0021] Next, an example of a functional configuration of the determination device 10 will be described with reference to Fig. 3. Fig. 3 is a block diagram showing an example of a functional configuration of the determination device 10. As shown in Fig. 3, the determination device 10 has a communication control unit 11, a storage unit 13, and a control unit 14. Note that the determination device 10 may further have functional units of the generation device 20 and the machine learning device 50 in addition to the functional units shown in Fig. 3, such as a function of generating training data and a function of executing machine learning.

[0022] The communication control unit 11 is a functional unit that controls communication with other devices, such as cameras and the generating device 20. For example, the communication control unit 11 can be realized by a network interface card such as a LAN (Local Area Network) card. In one aspect, the communication control unit 11 receives an image 110 captured by the RGB camera 31, or receives a result 120 of motion capture by the IR camera 32. In another aspect, the communication control unit 11 outputs the image 110 of the RGB camera 31 associated with the generation intensity 121 of the AU to the generating device 20.

[0023] The storage unit 13 is a functional unit that stores various data. As just one example, the storage unit 13 is realized by a storage, for example, an internal, external, or auxiliary storage. For example, the storage unit 13 stores an image 131 of the RGB camera 31 associated with the generation intensity of an AU. In addition to such an image 131, the storage unit 13 can store various data such as correspondence relationship information that indicates the correspondence relationship between a marker and an AU.

[0024] The control unit 14 is a processing unit that performs overall control of the determination device 10. For example, the control unit 14 is realized by a hardware processor. Alternatively, the control unit 14 may be realized by hardwired logic. As shown in FIG. 3, the control unit 14 includes an acquisition unit 141, an identification unit 142, and a determination unit 143.

[0025] The acquisition unit 141 acquires a captured image including a face. For example, the acquisition unit 141 acquires a captured image including a face with multiple markers attached to multiple positions corresponding to multiple AUs. The acquisition unit 141 acquires an image captured by the RGB camera 31.

[0026] Here, when the IR camera 32 and the RGB camera 31 are photographed, the subject changes his / her facial expression. This allows the determination device 10 to obtain images of the changes in facial expression over time. The RGB camera 31 may also capture a moving image. The moving image may be regarded as a plurality of still images arranged in a time series. The subject may also freely change his / her facial expression, or may change his / her facial expression according to a predetermined scenario.

[0027] The identification unit 142 identifies the position of a marker included in the captured image. The identification unit 142 identifies the position of each of a plurality of markers included in the captured image. Furthermore, when a plurality of images are acquired in time series, the identification unit 142 identifies the position of the marker for each image. Furthermore, the identification unit 142 can identify the coordinates of each marker on a plane or in space based on the positional relationship with a reference marker attached to the instrument 40. Note that the identification unit 142 may determine the position of the marker from a reference coordinate system or from a projected position of a reference plane.

[0028] The determination unit 143 determines whether each of the multiple AUs occurs based on the AU determination criteria and the positions of the multiple markers. The determination unit 143 determines the occurrence intensity for one or more AUs that occur among the multiple AUs. At this time, when the determination unit 143 determines that an AU corresponding to a marker among the multiple AUs occurs based on the determination criteria and the positions of the marker, the determination unit 143 can select the AU corresponding to the marker.

[0029] For example, the determination unit 143 determines the generation intensity of the first AU based on the movement amount of the first marker calculated based on the distance between the reference position of the first marker associated with the first AU included in the determination criterion and the position of the first marker identified by the identification unit 142. Note that the first marker can be one or more markers corresponding to a specific AU.

[0030] The AU determination criteria indicate, for example, one or more markers among the multiple markers used to determine the occurrence intensity of the AU for each AU. The AU determination criteria may include reference positions of the multiple markers. The AU determination criteria may include, for each of the multiple AUs, a relationship (conversion rule) between the movement amount of the marker used to determine the occurrence intensity and the occurrence intensity. Note that the reference positions of the markers may be determined according to the positions of the multiple markers in a captured image in which the subject is in an expressionless state (no AUs are occurring).

[0031] Here, the movement of the marker will be described with reference to Fig. 4. Fig. 4 is a diagram for explaining an example of the movement of the marker. Reference numerals 41 to 43 in Fig. 4 are images captured by the RGB camera 31. The images are assumed to be captured in the order of reference numerals 41, 42, and 43. For example, image 41 is an image when the subject has a neutral expression. The determination device 10 can regard the position of the marker in image 41 as a reference position where the amount of movement is zero.

[0032] As shown in Fig. 4, the subject is frowning. At this time, the position of the marker 401 moves downward in accordance with the change in facial expression. At that time, the distance between the position of the marker 401 and the reference marker attached to the device 40 becomes larger.

[0033] Moreover, the variation value of the distance of the marker 401 from the reference marker in the X direction and the Y direction is expressed as shown in Fig. 5. Fig. 5 is a diagram for explaining an example of a method for determining the occurrence intensity. As shown in Fig. 5, the determination unit 143 can convert the variation value into the occurrence intensity. Note that the occurrence intensity may be quantized into five stages according to FACS (Facial Action Coding System), or may be defined as a continuous amount based on the amount of variation.

[0034] There are various possible rules for the determination unit 143 to convert the amount of variation into the occurrence intensity. The determination unit 143 may perform the conversion according to one predetermined rule, or may perform the conversion according to a plurality of rules and adopt the largest occurrence intensity.

[0035] For example, the determination unit 143 may acquire in advance a maximum variation amount, which is the amount of variation when the subject changes the facial expression to the maximum extent, and convert the occurrence intensity based on the ratio of the amount of variation to the maximum variation amount. The determination unit 143 may also determine the maximum variation amount using data tagged by a coder using a conventional method. The determination unit 143 may also convert the amount of variation into the occurrence intensity linearly. The determination unit 143 may also perform the conversion using an approximation formula created from prior measurements of multiple subjects.

[0036] Also, for example, the determination unit 143 can determine the occurrence intensity based on a movement vector of the first marker calculated based on a position previously set as a determination criterion and the position of the first marker identified by the identification unit 142. In this case, the determination unit 143 determines the occurrence intensity of the first AU based on the degree of match between the movement vector of the first marker and a defined vector previously defined for the first AU. The determination unit 143 may also correct the correspondence between the magnitude of the vector and the occurrence intensity using an existing AU estimation engine.

[0037] FIG. 6 is a diagram for explaining an example of a method for determining the generation intensity. For example, it is assumed that the AU4 defined vector corresponding to AU4 is previously determined as (-2 mm, -6 mm). In this case, the determination unit 143 calculates the inner product of the movement vector of the marker 401 and the AU4 defined vector, and normalizes it by the magnitude of the AU4 defined vector. Here, if the inner product coincides with the magnitude of the AU4 defined vector, the determination unit 143 determines the generation intensity of AU4 to be 5 out of 5 levels. On the other hand, if the inner product is half the AU4 defined vector, for example, in the case of the linear conversion rule described above, the determination unit 143 determines the generation intensity of AU4 to be 3 out of 5 levels.

[0038] FIG. 7 is a diagram showing an example of an AU4 definition vector. In FIG. 7, the AU4 definition vector (-2 mm, -6 mm) corresponding to AU4 is shown by a thick line. Furthermore, in FIG. 7, the individual movement vector of the marker 401 corresponding to AU4 is shown by a thin line, and the range in which the individual movement vector may be dispersed is shown by hatching. As shown by hatching in FIG. 7, the AU4 movement vector of each individual has dispersion, but by calculating the inner product with respect to a single AU4 definition vector, it is possible to extract only the AU4 definition vector component. Furthermore, since the dispersion angle of individual differences is small, the accuracy of determining the generation intensity of AU4 is not greatly affected even if the setting accuracy of the AU4 definition vector is not high.

[0039] Also, for example, as shown in Fig. 6, it is assumed that the magnitude of the AU11 defining vector corresponding to AU11 is previously determined to be 3 mm. In this case, if the amount of change in the distance between the marker 402 and the marker 403 matches the magnitude of the AU11 vector, the determination unit 143 determines the generation intensity of AU11 to be 5 out of 5 levels. On the other hand, if the amount of change in the distance is half that of the AU4 defining vector, for example, in the case of the linear conversion rule described above, the determination unit 143 determines the generation intensity of AU11 to be 3 out of 5 levels. In this way, the determination unit 143 can determine the generation intensity based on the change in the distance between the position of the first marker and the position of the second marker identified by the identification unit 142.

[0040] In this way, in addition to the method of determining the generation intensity of an AU based on the change in the amount of movement of the marker from a reference position based on the position of the marker when the face is expressionless, the determination unit 143 can apply a different method of determining the generation intensity of an AU to a specific AU.

[0041] For example, in a specific AU, there may be a difference in the appearance of the change in the amount of movement of the marker between when the specific AU is expressed alone and when the specific AU is expressed simultaneously with other AUs. Figs. 8A to 8C are diagrams showing an example of the marker position. Figs. 8A to 8C show the position of the marker 405 used for observing AU15 as an example only. Fig. 8A shows the position of the marker 405 when the subject has a neutral expression. Fig. 8B shows the position of the marker 405 when AU15 is expressed alone. Fig. 8C shows the position of the marker 405 when AU15 is expressed simultaneously with AU17. Figs. 9 and 10 are diagrams showing an example of the amount of movement of the marker. Figs. 9 and 10 show the amount of movement of the marker 405 for observing AU15. Figs. 9 and 10 show the silhouette of the marker 405 when the subject has a neutral expression with a dashed line, while the silhouette of the marker 405 when AU15 is expressed with a solid line. 9 and 10, the movement of the marker 405 during the expression of AU15 is indicated by a solid arrow, and in FIG. 10, the movement of the marker 405 during the expression of AU17 is indicated by a dashed arrow.

[0042] As shown in Fig. 8A to Fig. 8C, a marker 405 placed beside the mouth is used to estimate AU15. For example, when AU15 is expressed alone, the marker 405 for observing AU15 moves from the reference position of the marker 405 shown in Fig. 8A to the position of the marker 405 shown in Fig. 8B. That is, as shown in Fig. 9, the marker 405 for observing AU15 moves downward due to the movement of the marker 405 that appears with the expression of AU15, so that the displacement from the reference position in the neutral state in the negative direction of the Y axis is observed as the amount of movement of the marker 405. On the other hand, when AU15 and AU17 are expressed simultaneously, the marker 405 for observing AU15 transitions from the reference position of the marker 405 shown in Fig. 8A to the position of the marker 405 shown in Fig. 8C. In this case, as shown in Figure 10, the movement of marker 405 (solid arrow) that appears with the expression of AU15 is influenced by the movement (dashed arrow) that occurs with the expression of AU17, and it may be difficult to observe the displacement in the negative direction of the Y axis as the amount of movement.

[0043] In this case, even if the similarity, for example, the inner product, between the AU15 specified vector (0 mm, -1 mm) corresponding to AU15 and the movement vector of the marker 405 for observing AU15 is calculated, the similarity when AU15 and AU17 co-occur may be low.

[0044] From this, when determining the generation intensity of a specific marker including AU15, the determination unit 143 determines the generation intensity of the specific marker based on the relative positional relationship of a plurality of markers in a non-expression state and based on the amount of change in the relative positional relationship.

[0045] Hereinafter, as an example of a specific AU, AU15 is taken as an example. For example, for determining the generation intensity of AU15, as an example of the first marker, the marker 405 attached to the side of the mouth is used, and as an example of the second marker, the marker 406 attached to the lower lip is used. Here, an example in which two markers are used is given, but three or more markers may be used.

[0046] The above relative positional relationship can be defined in any number of dimensions. For example, when the position of the marker is represented in three dimensions, the relative positional relationship can be defined in up to three dimensions, but it is not limited to this, and it can also be narrowed down and defined to one dimension or two dimensions. As just an example, different numbers of dimensions can be set for each specific AU for the relative positional relationship, and the number of dimensions for defining the relative positional relationship can be narrowed down based on the magnitude of each component included in the specified vector of the AU. For example, in the case of the AU15 specified vector shown in the following formula (1), since the X component of the AU15 specified vector < the Y component of the AU15 specified vector, it can be narrowed down to one dimension in the Y direction. In this case, as shown in the following formula (2), the relative positional relationship can be defined by the difference between the Y coordinate of the marker 405 at the side of the mouth and the Y coordinate of the marker 406 on the lower lip. Here, an example in which the number of dimensions is narrowed down to the component with the largest magnitude among the components of the AU specified vector is given, but the number of dimensions may be narrowed down to the component with a magnitude equal to or greater than the threshold value, or the component with a magnitude less than the threshold value can also be excluded. AU15 specified vector: (0, -1) ··· (1) Relative position: Y [side of mouth-lower lip] (2)

[0047] With such a definition of the relative positional relationship, the determination unit 143 calculates the similarity with the AU15 defined vector, for example, the inner product, based on the amount of change in the relative positional relationship between the expressionless state and the determination state. That is, as shown in the following formula (3), the generation intensity of AU15 is determined by calculating the inner product of the amount of change in the relative positional relationship and the AU15 defined vector. For example, according to the following formula (3), the amount of change ΔY in the relative positional relationship is calculated as follows. For example, the amount of change ΔY can be calculated as the difference between the amount of movement in the Y direction of the marker 405 beside the mouth from the reference position, i.e., the displacement in the Y direction of the marker 405 beside the mouth, and the amount of movement in the Y direction of the marker 406 on the lower lip from the reference position, i.e., the displacement in the Y direction of the marker 406 on the lower lip. ΔY[side of mouth-lower lip]·(0,-1)···(3)

[0048] This allows the occurrence intensity of AU15 to be determined even when AU15 and AU17 are simultaneously expressed. For example, when AU15 is expressed alone, the marker 405 beside the mouth and the marker 406 on the lower lip change from the position shown in FIG. 8A to the position shown in FIG. 8B. In this case, the displacement of the marker 405 beside the mouth in the negative direction of the Y axis is observed, while the displacement of the marker 406 on the lower lip is not observed. Therefore, when AU15 is expressed alone, the inner product calculation shown in the following formula (4) is performed. In this case, the displacement in the negative direction of the Y axis is used as the change amount ΔY of the relative positional relationship in the inner product calculation, so that the inner product is greater than zero, and as a result, the detection of the occurrence of AU15 does not occur, and the occurrence intensity of AU15 can be determined. On the other hand, when AU15 is expressed together with AU17, the marker 405 beside the mouth and the marker 406 on the lower lip change from the position shown in FIG. 8A to the position shown in FIG. 8C. In this case, the displacement of the marker 405 beside the mouth is not observed, while the displacement of the marker 406 on the lower lip in the positive direction on the Y axis is observed. Therefore, when AU15 and AU17 are simultaneously expressed, the inner product calculation shown in the following formula (5) is executed. In this case as well, the displacement in the negative direction on the Y axis is used in the inner product calculation as the amount of change ΔY in the relative positional relationship, so that the inner product becomes larger than zero, and as a result, the expression of AU15 is not missed, and the occurrence intensity of AU15 can be determined. ΔY[side of mouth]·(0,-1)···(4) ΔY[side of mouth-lower lip]·(0,-1)···(5)

[0049] 11 and 12 are diagrams showing an example of an estimated value of the generation intensity of AUs. In these Figs. 11 and 12, the estimated value of the generation intensity of AUs is shown when the subject repeats the generation intensity of AU15 in the order of "low", "medium", and "high" as photographed by the RGB camera 31 and the IR camera 32. Figs. 11 and 12 show graphs with the horizontal axis representing time and the vertical axis representing the generation intensity of AUs. In the graph g1 shown in Fig. 11 and the graph G1 shown in Fig. 12, the generation intensity of the correct AU15 is plotted as a rectangular mark as a reference.

[0050] Graph g1 shown in FIG. 11 shows an estimated value of the generation intensity of AU15 when the generation intensity of AU15 is determined based on the inner product of the movement vector of the marker 405 beside the mouth and the AU15 definition vector (0, -1). According to the graph g1 shown in FIG. 11, when the generation intensity of AU15 is "small" and "medium", the generation intensity of the reference AU15 and the estimated value of the generation intensity of AU15 are almost the same, so it can be seen that the generation intensity of AU15 can be estimated even with the marker 405 beside the mouth alone. On the other hand, when the generation intensity of AU15 is "large", the generation intensity of the reference AU15 and the estimated value of the generation intensity of AU15 are deviated. That is, the estimated value of the generation intensity of AU15 is lower than the generation intensity of AU15 of the reference, and the occurrence of AU15 is not detected. Furthermore, graph g2 shown in FIG. 11 shows an estimated value of the generation intensity of AU17. Comparing graphs g1 and g2 in FIG. 11, it is clear that the time period when the generation intensity of AU15 is "high" is synchronized with the time period when AU17 is expressed. This means that when the generation intensity of AU15 is "high", the expression of AU17 is likely to be accompanied. From these facts, it can be inferred that it is difficult to estimate the generation intensity of AU15 when AU15 and AU17 are simultaneously expressed, using only the marker 405 beside the mouth.

[0051] Graph G1 in Fig. 12 shows an estimated value of the generation intensity of AU15 when the generation intensity of AU15 is determined based on the amount of change in the relative positional relationship between the marker 405 beside the mouth and the marker 406 on the lower lip. According to graph G1 in Fig. 12, the generation intensity of AU15 in the reference and the estimated value of the generation intensity of AU15 are approximately the same regardless of whether the generation intensity of AU15 is "small," "medium," or "large." Therefore, it can be said that it is clear that the generation intensity of AU15 can be determined even when AU15 and AU17 are simultaneously expressed.

[0052] The AU generation intensity determined as described above is output to the generating device 20 in association with the image captured by the RGB camera 31. In addition, the determination device 10 can also store in the storage unit 13 the image captured by the RGB camera 31 associated with the AU generation intensity determined by the determination unit 143 as the image 131 with generation intensity.

[0053] Next, the generation of training data executed by the generating device 20 will be described. The generating device 20 can delete markers using a mask image. FIG. 13 is a diagram for explaining an example of a method for creating a mask image. Reference numeral 110 in FIG. 13 is an image captured by the RGB camera 31. First, the generating device 20 extracts the color of a marker that has been intentionally added in advance and defines it as a representative color. Then, as indicated by reference numeral 110B in FIG. 13, the generating device 20 generates an area image of a color near the representative color. Furthermore, as indicated by reference numeral 110C in FIG. 7, the generating device 20 performs processing such as contraction and expansion on the area of ​​the color near the representative color to generate a mask image for deleting the marker. Also, the accuracy of extracting the color of the marker may be improved by setting the color of the marker to a color that is unlikely to exist as a face color.

[0054] FIG. 14 is a diagram for explaining an example of a method for deleting a marker. As shown in FIG. 14, first, the generating device 20 applies a mask image to a still image acquired from a video. Furthermore, the generating device 20 inputs the image to which the mask image has been applied, for example, into a neural network to obtain a processed image. Note that the neural network is assumed to have been trained using images of the subject with and without a mask. Note that by acquiring a still image from a video, there are advantages in that intermediate data of facial expression changes can be obtained and a large amount of data can be obtained in a short period of time. In addition, the generating device 20 may use GMCNN (Generative Multi-column Convolutional Neural Networks) or GAN (Generative Adversarial Networks) as the neural network.

[0055] The method by which the generating device 20 deletes the marker is not limited to the above. For example, the generating device 20 may detect the position of the marker based on a predetermined shape of the marker and generate a mask image. Furthermore, the relative positions of the IR camera 32 and the RGB camera 31 may be calibrated in advance. In this case, the generating device 20 can detect the position of the marker from information on marker tracking by the IR camera 32.

[0056] The generating device 20 may also employ different detection methods depending on the marker. For example, since the marker on the nose moves little and its shape is easy to recognize, the generating device 20 may detect its position by shape recognition. Furthermore, since the marker beside the mouth moves a lot and its shape is difficult to recognize, the generating device 20 may detect its position by a method of extracting a representative color.

[0057] After the markers are deleted in this manner, the generating device 20 generates training data for machine learning by adding information about the first AU to the generated image. For example, the generating device 20 generates training data for machine learning by adding the occurrence intensity of the AU determined by the determining unit 143 to the generated image. In addition, the machine learning device 50 may perform machine learning by adding the training data generated by the generating device 20 to existing training data.

[0058] For example, the training data can be used for machine learning of an estimation model that estimates an occurring AU using an image as an input. The estimation model may be a model specialized for each AU. When the estimation model is specialized for a specific AU, the generating device 20 may change the generated training data to training data in which only information related to the specific AU is used as a teacher label. In other words, for an image in which another AU different from the specific AU occurs, the generating device 20 can delete information related to the other AU and add information indicating that the specific AU does not occur as a teacher label.

[0059] According to this embodiment, it is possible to estimate the required training data. In general, performing machine learning requires a huge computation cost. The computation cost includes time and the amount of usage of a GPU, etc.

[0060] As the quality and quantity of the dataset improves, the accuracy of the model obtained by machine learning improves. Therefore, if the quality and quantity of the dataset required for the target accuracy can be roughly estimated in advance, the calculation cost can be reduced. Here, for example, the quality of the dataset is the marker deletion rate and deletion accuracy. Also, for example, the amount of the dataset is the number of datasets and the number of subjects.

[0061] Among the combinations of AUs, there are combinations that are highly correlated with each other. Therefore, it is considered that an estimation performed for a certain AU can be applied to other AUs that are highly correlated with the AU. For example, it is known that AU18 and AU22 are highly correlated, and the corresponding markers may be common. Therefore, if it is possible to estimate the quality and quantity of a data set that achieves the target estimation accuracy for AU18, it is possible to roughly estimate the quality and quantity of a data set that achieves the target estimation accuracy for AU22.

[0062] The machine learning model M generated by the machine learning device 50 may be provided to an estimation device (not shown) that executes estimation of the occurrence intensity of AUs. The estimation device actually performs estimation using the machine learning model M generated by the machine learning device 50. The estimation device acquires an image in which a person's face is captured and the occurrence intensity of each AU is unknown, and inputs the acquired image to the machine learning model M, thereby outputting the occurrence intensity of AUs output by the machine learning model M as an estimation result of AUs to an arbitrary output destination. Such an output destination may be, by way of example only, a device, program, or service that estimates a facial expression using the occurrence intensity of AUs or calculates a level of understanding or satisfaction.

[0063] Next, the flow of processing by the determination device 10 will be described. Fig. 15 is a flowchart showing the procedure of the generation intensity determination processing. As shown in Fig. 15, first, the determination device 10 acquires a captured image of the face of the subject (step S101). Next, the determination device 10 identifies the position of a marker included in the captured image acquired in step S101 (step S102).

[0064] Then, the determination device 10 executes a loop process 1 in which the processes from step S102 to step S107 are repeated a number of times corresponding to the number K of AUs.

[0065] For example, the determination device 10 selects one AU (step S103). At this time, if the selected AU is not a specific AU (No in step S104), the determination device 10 executes the following process. That is, the determination device 10 calculates a movement vector of the marker based on the position of the marker assigned to the estimation of the selected AU among the positions of the markers identified in step S102 and the reference position (step S105). Then, the determination device 10 determines the generation intensity of the AU based on the movement vector (step S106).

[0066] On the other hand, if the selected AU is a specific AU (Yes in step S104), the determination device 10 executes the following process. That is, the determination device 10 calculates the amount of change in the relative positional relationship of the multiple markers based on the positions of each of the multiple markers assigned to the estimation of the specific AU among the positions of the markers identified in step S102 (step S107). Next, the determination device 10 determines the occurrence intensity of the specific AU based on the inner product of the amount of change in the relative positional relationship calculated in step S107 and the defined vector of the specific AU (step S108).

[0067] The occurrence intensity can be determined for each AU by repeating such loop process 1. Note that, although an example in which the processes from step S103 to step S108 are executed as a loop process has been given in the flowchart shown in Fig. 15, the present invention is not limited to this, and the processes may be executed in parallel for each AU.

[0068] Next, the flow of processing by the generating device 20 will be described. Fig. 16 is a flowchart showing the procedure of the training data generation processing. As shown in Fig. 16, first, the generating device 20 identifies the position of the marker in the captured image (step S301). The generating device 20 deletes the marker from the image (step S302). Then, the generating device 20 assigns the generation intensity of the AU to the image from which the marker has been deleted (step S303).

[0069] As described above, the determination device 10 according to this embodiment uses the relative positional relationship of the multiple markers when the user has no facial expression as a reference and determines the occurrence strength of the specific AU based on the amount of change in the relative positional relationship at the time of execution of the determination. Therefore, the determination device 10 according to this embodiment can determine the occurrence strength of the specific AU when the specific AU and another AU are simultaneously present.

[0070] In the above embodiment, the judgment device 10, the generation device 20, and the machine learning device 50 are each individual devices, but the judgment device 10 may have the functions of the generation device 20, or the functions of both the generation device 20 and the machine learning device 50.

[0071] In the above embodiment, the determination unit 143 determines the generation intensity of an AU based on the amount of movement of the marker. However, the fact that the marker has not moved can also be a criterion for the determination unit 143 to determine the generation intensity.

[0072] In addition, a color that is easy to detect may be arranged around the marker. For example, a round green adhesive sticker with an IR marker in the center may be attached to the subject. In this case, the generating device 20 can detect the round green area from the captured image and delete the area together with the IR marker.

[0073] The information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. In addition, the specific examples, distributions, numerical values, etc. described in the embodiments are merely examples and can be changed as desired.

[0074] In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure. In other words, all or a part of them can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc. Furthermore, each processing function performed by each device can be realized in whole or in any part by a CPU and a program analyzed and executed by the CPU, or can be realized as hardware using wired logic.

[0075] Fig. 17 is a diagram for explaining an example of a hardware configuration. As shown in Fig. 17, the determination device 10 includes a communication interface 10a, a hard disk drive (HDD) 10b, a memory 10c, and a processor 10d. The components shown in Fig. 17 are connected to each other via a bus or the like.

[0076] The communication interface 10a is a network interface card or the like, and communicates with other servers. The HDD 10b stores programs and DBs that operate the functions shown in FIG.

[0077] The processor 10d is a hardware circuit that operates a process that executes each function described in FIG. 3 and the like by reading a program that executes the same processing as each processing unit shown in FIG. 3 from the HDD 10b or the like and expanding it in the memory 10c. That is, this process executes a function similar to that of each processing unit possessed by the determination device 10. Specifically, the processor 10d reads a program having functions similar to those of the acquisition unit 141, the identification unit 142, and the determination unit 143 from the HDD 10b or the like. Then, the processor 10d executes a process that executes the same processing as the acquisition unit 141, the identification unit 142, the determination unit 143, and the like.

[0078] In this way, the determination device 10 operates as an information processing device that executes a determination method by reading and executing a program. The determination device 10 can also realize functions similar to those of the above-mentioned embodiment by reading the program from a recording medium using a media reading device and executing the read program. Note that the program in these other embodiments is not limited to being executed by the determination device 10. For example, the present invention can be similarly applied to cases where another computer or server executes a program, or where these cooperate to execute a program.

[0079] This program can be distributed via a network such as the Internet. This program can also be recorded on a computer-readable recording medium such as a hard disk, a flexible disk (FD), a CD-ROM, a magneto-optical disk (MO), or a digital versatile disk (DVD), and can be executed by being read out from the recording medium by a computer.

[0080] The following supplementary notes are further disclosed regarding the embodiments including the above examples.

[0081] (Appendix 1) Obtain an image including a face with a first marker and a second marker; Identifying a first position of the first marker and a second position of the second marker included in the image; determining an occurrence intensity of a specific facial expression on the face based on a difference between the first position and a reference position of the first marker, and a difference between a positional relationship between the first position and the second position and a reference positional relationship between the first marker and the second marker; outputting the determined intensity of occurrence of the particular facial expression in association with the image; A determination program that causes a computer to execute a process.

[0082] (Additional Note 2) The reference position corresponds to a position of the first marker when the facial expression is neutral. 2. The judgment program according to claim 1,

[0083] (Additional Note 3) The difference between the positional relationship and the reference positional relationship corresponds to a difference between a movement amount of the first marker from a reference position and a movement amount of the second marker from a reference position. 3. The judgment program according to claim 2,

[0084] (Appendix 4) The computer further executes a process of generating training data to be used for training a machine learning model by assigning the occurrence intensity of the specific facial expression determined in the determination process as a correct label to an image obtained by deleting the first marker and the second marker from the image. 2. The judgment program according to claim 1,

[0085] (Appendix 5) The computer is further caused to execute a process of executing machine learning in which the image included in the training data is used as an explanatory variable of the machine learning model, and the correct answer label included in the training data is used as a target variable of the machine learning model. 5. The judgment program according to claim 4,

[0086] (Appendix 6) Acquire an image including a face with a first marker and a second marker; Identifying a first position of the first marker and a second position of the second marker included in the image; determining an occurrence intensity of a specific facial expression on the face based on a difference between the first position and a reference position of the first marker, and a difference between a positional relationship between the first position and the second position and a reference positional relationship between the first marker and the second marker; outputting the determined intensity of occurrence of the particular facial expression in association with the image; A determination method characterized in that the processing is executed by a computer.

[0087] (Additional Note 7) The reference position corresponds to a position of the first marker when the facial expression is neutral. The method for determining whether or not a substance is oxidized is described in Appendix 6.

[0088] (Additional Note 8) The difference between the positional relationship and the reference positional relationship corresponds to a difference between a movement amount of the first marker from a reference position and a movement amount of the second marker from a reference position. The method for determining whether or not a compound is present is described in Appendix 7.

[0089] (Appendix 9) The computer further executes a process of generating training data to be used for training a machine learning model by assigning the occurrence intensity of the specific facial expression determined in the process of determining as a correct label to an image obtained by deleting the first marker and the second marker from the image. The method for determining whether or not a substance is oxidized is described in Appendix 6.

[0090] (Appendix 10) The computer further executes a process of executing machine learning in which the image included in the training data is used as an explanatory variable of the machine learning model, and the correct answer label included in the training data is used as a target variable of the machine learning model. 10. The method of claim 9,

[0091] (Appendix 11) Acquire an image including a face with a first marker and a second marker; Identifying a first position of the first marker and a second position of the second marker included in the image; determining an occurrence intensity of a specific facial expression on the face based on a difference between the first position and a reference position of the first marker, and a difference between a positional relationship between the first position and the second position and a reference positional relationship between the first marker and the second marker; outputting the determined intensity of occurrence of the particular facial expression in association with the image; A determination device including a control unit that executes processing.

[0092] (Additional Note 12) The reference position corresponds to a position of the first marker when the facial expression is neutral, that is, when the facial expression is expressionless. 12. The determination device according to claim 11,

[0093] (Supplementary Note 13) The difference between the positional relationship and the reference positional relationship corresponds to a difference between a movement amount of the first marker from a reference position and a movement amount of the second marker from a reference position. 13. The determination device according to claim 12.

[0094] (Appendix 14) The control unit further executes a process of generating training data to be used for training a machine learning model by assigning the occurrence intensity of the specific facial expression determined in the determining process as a correct label to an image obtained by deleting the first marker and the second marker from the image. 12. The determination device according to claim 11,

[0095] (Appendix 15) The control unit further executes a process of executing machine learning with the image included in the training data as an explanatory variable of the machine learning model and the correct answer label included in the training data as a target variable of the machine learning model. 15. The determination device according to claim 14. [Explanation of symbols]

[0096] 1. Machine Learning Systems 10 Judgment device 11 Communication control section 13 Storage section 131 Image with generated intensity 14 Control section 141 Acquisition Department 142 Specific part 143 Judgment section 20 Generator 50 Machine Learning Device

Claims

1. acquiring an image including a face with a first marker and a second marker; Identifying a first position of the first marker and a second position of the second marker included in the image; determining an occurrence intensity of a specific facial expression on the face based on a difference between the first position and a reference position of the first marker, and a difference between a positional relationship between the first position and the second position and a reference positional relationship between the first marker and the second marker; outputting the determined intensity of occurrence of the particular facial expression in association with the image; A determination program that causes a computer to execute a process.

2. the reference position corresponds to a position of the first marker when the facial expression is neutral, that is, when the facial expression is expressionless.

2. The determination program according to claim 1.

3. the difference between the positional relationship and the reference positional relationship corresponds to a difference between a movement amount of the first marker from a reference position and a movement amount of the second marker from a reference position.

3. The determination program according to claim 2.

4. and further causing the computer to execute a process of generating training data to be used for training a machine learning model by assigning the occurrence intensity of the specific facial expression determined in the determining process as a correct label to an image obtained by deleting the first marker and the second marker from the image.

4. The determination program according to claim 1, wherein the determination program is a program for determining whether or not a plurality of objects are included in the plurality of objects.

5. and further causing the computer to execute a process of executing machine learning in which the image included in the training data is used as an explanatory variable of the machine learning model, and the correct label included in the training data is used as a target variable of the machine learning model.

5. The determination program according to claim 4.

6. acquiring an image including a face with a first marker and a second marker; Identifying a first position of the first marker and a second position of the second marker included in the image; determining an occurrence intensity of a specific facial expression on the face based on a difference between the first position and a reference position of the first marker, and a difference between a positional relationship between the first position and the second position and a reference positional relationship between the first marker and the second marker; outputting the determined occurrence intensity of the particular facial expression in association with the image; A determination method characterized in that the processing is executed by a computer.

7. acquiring an image including a face with a first marker and a second marker; Identifying a first position of the first marker and a second position of the second marker included in the image; determining an occurrence intensity of a specific facial expression on the face based on a difference between the first position and a reference position of the first marker, and a difference between a positional relationship between the first position and the second position and a reference positional relationship between the first marker and the second marker; outputting the determined intensity of occurrence of the particular facial expression in association with the image; A determination device including a control unit that executes processing.

Images