Evaluation method, evaluation system, and program

Abstract

The computer of the present invention evaluates the expression of a subject. More specifically, the computer acquires a criteria for biological information on a facial area of the subject in response to the occurrence of a given incident (steps S14 and S20). The computer further acquires a measurement result pertaining to the biological information on the facial area of the subject at evaluation target timing. The computer then uses the criteria and the measurement result to generate evaluation information pertaining to the expression of the subject and outputs the evaluation information.

Description

Technical Field

[0001] The present invention relates to a method for evaluating, an evaluation system, and a program for the facial expressions of a subject.

Background Art

[0002] Since human emotions such as pleasure and displeasure are likely to appear in facial expressions, conventionally, techniques for evaluating human expressions in real time have been proposed. For example, Japanese Patent Application Laid-Open No. 2018-010305 (Patent Document 1) discloses a technique for acquiring a facial image of a user (subject) who is playing a game and scoring the subject's expression according to a given standard such as a sample image of a smiling face.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, since there are individual differences in how emotions appear in expressions, it is difficult to accurately evaluate the expressions of subjects using conventional techniques.

[0005] The present invention has been conceived in view of such circumstances, and an object thereof is to provide a technique for accurately evaluating the expressions of subjects.

Means for Solving the Problems

[0006] An evaluation method according to an aspect of the present disclosure is a method for evaluating the expression of a subject, including: obtaining a standard of biometric information of the subject's face region; obtaining a measurement result of biometric information of the subject's face region at the timing of the evaluation target; generating evaluation information regarding the subject's expression using the standard and the measurement result; and outputting the evaluation information.

[0007] An evaluation system according to an aspect of the present disclosure is a system for evaluating a subject's facial expression, comprising a processor and an interface for acquiring biometric information of the subject's face region. The processor generates evaluation information regarding the subject's facial expression using a standard of the biometric information of the subject's face region and a measurement result of the biometric information of the subject's face region acquired at the timing of evaluation, and outputs the evaluation information.

[0008] A program according to an aspect of the present disclosure is a program for causing a subject to evaluate a facial expression. When the program is executed by a processor of a computer, the computer is caused to perform steps of acquiring a standard of biometric information of the subject's face region, acquiring a measurement result of the biometric information of the subject's face region at the timing of evaluation, generating evaluation information regarding the subject's facial expression using the standard and the measurement result, and outputting the evaluation information.

Advantages of the Invention

[0009] According to the present disclosure, a technique for accurately estimating an individual's emotion is provided.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Embodiments for Carrying Out the Invention

[0011] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the drawings, the same or corresponding parts are denoted by the same reference numerals and their description will not be repeated.

[0012] [Embodiment 1] <System Configuration> FIG. 1 is a diagram schematically showing the overall configuration of the facial expression evaluation system according to Embodiment 1. The facial expression evaluation system 100 shown in FIG. 1 acquires an electromyogram signal from a subject and evaluates the facial expression of the subject based on the muscle activity represented by the acquired electromyogram signal.

[0013] The facial expression evaluation system 100 includes a wearable terminal 10 and a fixed terminal 90. The wearable terminal 10 is worn by the subject. The fixed terminal 90 is installed in the environment around the subject. The wearable terminal 10 and the fixed terminal 90 are configured to enable two-way communication. Hereinafter, each configuration will be described.

[0014] <Wearable Terminal 10> The wearable terminal 10 includes a sensor unit 1, a signal processing circuit 2, a controller 3, a communication module 4, a battery 5, and a housing 6. The housing 6 houses the signal processing circuit 2, the controller 3, the communication module 4, and the battery 5 inside thereof.

[0015] The sensor unit 1 includes a first myoelectric potential sensor 11 and a second myoelectric potential sensor 12. The first myoelectric potential sensor 11 and / or the second myoelectric potential sensor 12 is mounted on the face of the subject to detect the myoelectric potential signal at the mounting site. In Embodiment 1, the myoelectric potential signal is an example of biological information.

[0016] The myoelectric potential signal means a weak electrical signal generated when moving a muscle. In the present disclosure, the "face" of the subject is not limited to the facial surface (front or side of the face), and may include the neck of the subject. For example, when the sensor unit 1 is mounted on the throat of the subject, the myoelectric potential change associated with the swallowing motion of the subject may be detected.

[0017] The sensor unit 1 may include a plurality of sensors. The plurality of sensors may be mounted on different types of muscles. The type of muscle can be specified by the site where the sensor is mounted. In Embodiment 1, when specifying the type of muscle, it is not necessary to be limited to the composition or structure of the muscle. If the sites where the sensors are mounted are different, myoelectric potential signals of different types of muscles can be acquired.

[0018] FIG. 2 is a diagram for explaining the mounting sites of the first myoelectric potential sensor 11 and the second myoelectric potential sensor 12 in the present embodiment. Referring to FIG. 2, a configuration and an aspect of utilization of the first myoelectric potential sensor 11 and the second myoelectric potential sensor 12 will be described.

[0019] The first myoelectric potential sensor 11 is mounted on the cheek of the subject. The first myoelectric potential sensor 11 includes a working electrode 111 and a reference electrode 112. The working electrode 111 is incorporated into the pad 151, and the reference electrode 112 is incorporated into the pad 152.

[0020] The first myoelectric potential sensor 11 detects the myoelectric potential signals of the muscles near the cheek by mounting the pad 151 directly above the zygomaticus minor muscle and the pad 152 directly above the masseter muscle. More specifically, the first myoelectric potential sensor 11 detects the potential of the working electrode 111 with reference to the potential of the reference electrode 112 as the myoelectric potential signal of the muscles near the cheek. The mounting positions of each of the pad 151 and the pad 152 (the working electrode 111 and the reference electrode 112) may deviate slightly from directly above those muscles as long as they are in the vicinity of the zygomaticus minor muscle and the masseter muscle respectively. The first myoelectric potential sensor 11 outputs the above myoelectric potential signal to the signal processing circuit 2 of the wearable terminal 10 as a myoelectric potential signal indicating the activity of the facial muscles in the cheek portion.

[0021] The second myoelectric potential sensor 12 is mounted on the eyebrows of the subject. The second myoelectric potential sensor 12 includes a working electrode 121 and a reference electrode 122. The working electrode 121 is incorporated in the pad 171, and the reference electrode 122 is incorporated in the pad 172.

[0022] The second myoelectric potential sensor 12 detects the myoelectric potential signals of the muscles near the eyebrows (e.g., the corrugator supercilii muscle) by mounting the pad 171 and the pad 172 directly above the corrugator supercilii muscle. More specifically, the second myoelectric potential sensor 12 detects the potential of the working electrode 121 with reference to the potential of the reference electrode 122 as the myoelectric potential signal of the muscles near the eyebrows. The mounting positions of the pad 171 and the pad 172 (the working electrode 121 and the reference electrode 122) may deviate slightly from directly above the corrugator supercilii muscle as long as they are in the vicinity of the corrugator supercilii muscle. The second myoelectric potential sensor 12 outputs the above myoelectric potential signal to the signal processing circuit 2 as a myoelectric potential signal indicating the activity of the corrugator supercilii muscle.

[0023] In the example of FIG. 2, each of the zygomaticus minor muscle, the masseter muscle, and the corrugator supercilii muscle is an example of facial muscles. More specifically, each of the zygomaticus minor muscle and the masseter muscle is an example of the muscles near the cheek. The corrugator supercilii muscle is an example of the muscles near the eyebrows.

[0024] Referring to FIG. 1 again, the wearable terminal 10 further includes a controller 3. Although not shown, the signal processing circuit 2 includes a filter, an amplifier, and an A / D converter. The signal processing circuit 2 performs predetermined signal processing (such as noise removal, rectification, amplification, digitization, etc.) on each of the myoelectric potential signals acquired from the sensor unit 1, and outputs each processed signal to the controller 3.

[0025] In FIG. 1, the myoelectric potential signal from the first myoelectric potential sensor 11 processed by the signal processing circuit 2 is described as "myoelectric potential signal MS1", and the myoelectric potential signal from the second myoelectric potential sensor 12 processed by the signal processing circuit 2 is described as "myoelectric potential signal MS2".

[0026] The controller 3 is an arithmetic unit including a processor 31, a memory 32, and an input / output port 33. The processor 31 is realized by, for example, a CPU (Central Processing Unit). The memory 32 is realized by, for example, a ROM (Read Only Memory) and a RAM (Random Access Memory). The input / output port 33 is an interface for data input / output in the controller 3. The controller 3 executes arithmetic processing for evaluating the expression of the subject based on the myoelectric potential signals MS1 and MS2.

[0027] The wearable terminal 10 further includes a communication module 4. The communication module 4 enables the wearable terminal 10 to communicate with an external device, and is realized by, for example, a communication device compliant with a short-range wireless communication standard. The controller 3 controls the communication module 4 to control the exchange of information between the wearable terminal 10 and the external (such as the fixed terminal 90).

[0028] The wearable terminal 10 further includes a battery 5. The battery 5 is a secondary battery such as a lithium-ion secondary battery. The battery 5 supplies an operating voltage to each device in the wearable terminal 10.

[0029] The wearable terminal 10 further includes a speaker 21 and an infrared sensor 22. The controller 3 outputs a control signal (the "signal SS" in FIG. 1) to the speaker 21, thereby outputting voice through the speaker 21. Also, the controller 3 determines whether the wearable terminal 10 is worn by a person based on the detection output (the "signal WS" in FIG. 1) of the infrared sensor 22.

[0030] <Fixed terminal 90> The fixed terminal 90 is, for example, a PC (Personal Computer) or a server. The fixed terminal 90 communicates with the wearable terminal 10 via a communication module (not shown) and receives a signal indicating the calculation result of the controller 3. The fixed terminal 90 includes a controller 91 and a communication module 92.

[0031] Similar to the controller 3, the controller 91 includes a processor 95, a memory 96, and an input / output port 97, and executes various arithmetic processes. Similar to the communication module 4, the communication module 92 is realized by, for example, a communication device compliant with a short-range wireless communication standard. That is, the communication module 92 enables the fixed terminal 90 to communicate with other devices.

[0032] The controller 91 is connected to a display 90A and a speaker 90B. The controller 91 displays a screen that is the calculation result on the display 90A and outputs voice that is the calculation result through the speaker 90B.

[0033] <Structure of the wearable terminal 10> FIG. 3 is a diagram schematically showing the structure of the wearable terminal 10. As shown in FIG. 3, the outer shell of the wearable terminal 10 is mainly constituted by a housing 6. The housing 6 includes a first arm 10A and a second arm 10B.

[0034] The housing 6 houses various elements including the signal processing circuit 2 (see FIG. 1). The signal processing circuit 2 and the first myoelectric potential sensor 11 (active electrode 111 and reference electrode 112) are connected by lines 141 and 142. The signal processing circuit 2 and the second myoelectric potential sensor 12 (active electrode 121 and reference electrode 122) are connected by lines 161 and 162.

[0035] The speaker 21 and the infrared sensor 22 (see FIG. 1) are installed, for example, on the first arm 10A and / or the second arm 10B. When the wearable terminal 10 is worn on the head of the subject (see FIG. 2), the infrared sensor 22 detects infrared rays from the subject.

[0036] <Emotion Index> Generally, it is considered that unpleasant emotions are manifested in the activities of the muscles near the eyebrows, as the expression of an unpleasant or worried expression is "frowning". On the other hand, it is considered that pleasant emotions are manifested in the activities of the muscles near the cheeks, as the expression of a happy or relieved expression is "the cheeks relax". The facial expression evaluation system 100 generates information for evaluating the facial expression of the subject by monitoring the activities of these muscles.

[0037] The emotion indices E1 and E2 are examples of the information generated for evaluating facial expressions. Each of the emotion indices E1 and E2 is calculated according to each of the following formulas (1) and (2).

[0038] E1 = k11·A1 + k12·A2 …(1) E2 = k21·A1 + k22·A2 …(2) In formulas (1) and (2), A1 represents the activity amount of the muscles near the cheeks. A2 represents the activity amount of the muscles near the eyebrows. Each of k11, k12, k21, and k22 represents a count prepared in advance. That is, the emotion index E1 is calculated by adding the product of the activity amount A1 and the coefficient k11 and the product of the activity amount A2 and the coefficient k12. Also, the emotion index E2 is calculated by adding the product of the activity amount A1 and the coefficient k21 and the product of the activity amount A2 and the coefficient k22.

[0039] In one implementation example, the activity amount A1 of the muscles near the cheek is specified as the sum of the potentials of the working electrode 111 with reference to the potential of the reference electrode 112 over a given period. Also, in one implementation example, the activity amount A2 of the muscles near the eyebrows is specified as the sum of the potentials of the working electrode 121 with reference to the potential of the reference electrode 122 over a given period. The emotion index E1 may be an index representing the strength of positive emotions, and the emotion index E2 may be an index representing the strength of negative emotions.

[0040] <Evaluation information> The facial expression evaluation system 100 generates evaluation information for the facial expression of the subject using the above-described emotion indexes E1 and E2. FIG. 4 is a diagram for explaining the generation of evaluation information using the emotion indexes E1 and E2.

[0041] In FIG. 4, a map MP is shown. In the map MP, the horizontal axis represents the emotion index E1, and the vertical axis represents the emotion index E2. The map MP includes eight regions, namely regions AR11 to 14 and regions AR21 to 24. Each of the regions AR11 to 14 corresponds to each of the evaluation information "calm", "60% positivity", "80% positivity", and "100% positivity". Each of the regions AR21 to 24 corresponds to each of the evaluation information "calm", "60% negativity", "80% negativity", and "100% negativity".

[0042] In Embodiment 1, the combination of the emotion index E1 and the emotion index E2 is specified, one of the regions AR11 to 14 and AR21 to 24 corresponding to the specified combination is specified, and the evaluation information corresponding to the specified region is specified, whereby the evaluation information corresponding to the emotion indexes E1 and E2 is generated (specified).

[0043] <Processing flow> Figures 5 and 6 are flowcharts of the processes implemented by the fixed terminal 90 to output evaluation information in the facial expression evaluation system 100. In one implementation example, in the fixed terminal 90, this process is implemented by the processor 95 executing a given program. With reference to Figures 5 and 6, the flow of this process will be described.

[0044] First, with reference to Figure 5, in step S10, the fixed terminal 90 determines whether the wearable terminal 10 is worn by the subject. In one implementation example, the fixed terminal 90 determines whether the wearable terminal 10 is worn by the subject based on whether the detection signal from the infrared sensor 22 installed in the first arm 10A and / or the second arm 10B indicates that the wearable terminal 10 is worn on the subject's head.

[0045] The fixed terminal 90 repeats the control of step S10 until it determines that the wearable terminal 10 is worn by the subject (NO in step S10). When the fixed terminal 90 determines that the wearable terminal 10 is worn by the subject (YES in step S10), it proceeds with the control to step S12.

[0046] In step S12, the fixed terminal 90 outputs a first message. The first message is a message that prompts the subject to make a positive facial expression. In one implementation example, the fixed terminal 90 outputs the voice "Please smile." via the speaker 21.

[0047] In step S14, the fixed terminal 90 obtains a first reference value X1. The first reference value X1 is a reference value for the activity amount of the muscles near the cheeks. In one implementation example, the fixed terminal 90 obtains, as the first reference value X1, the sum of the potentials of the working electrode 111 with respect to the potential of the reference electrode 112 over a given period. Then, the fixed terminal 90 stores the first reference value X1 in the memory 96.

[0048] In step S16, the fixed terminal 90 notifies that it has acquired the first reference value X1. In one implementation example, in step S16, the fixed terminal 90 outputs, via the speaker 21, the voice "The acquisition of the first reference value X1 has been completed."

[0049] In step S18, the fixed terminal 90 outputs a second message. The second message is a message that prompts the subject to make a negative expression. In one implementation example, the fixed terminal 90 outputs, via the speaker 21, the voice "Please frown."

[0050] In step S20, the fixed terminal 90 acquires a second reference value X2. The second reference value X2 is a reference value regarding the activity amount of the muscles near the eyebrows. In one implementation example, the fixed terminal 90 acquires, as the second reference value X2, the sum of the potentials of the working electrode 121 with respect to the potential of the reference electrode 122 during a given period. Then, the fixed terminal 90 stores the second reference value X2 in the memory 96.

[0051] In step S22, the fixed terminal 90 notifies that it has acquired the second reference value X2. In one implementation example, in step S22, the fixed terminal 90 outputs, via the speaker 21, the voice "The acquisition of the second reference value X2 has been completed."

[0052] In step S24, the fixed terminal 90 notifies that it starts the evaluation of the subject's expression. In one implementation example, in step S24, the fixed terminal 90 outputs, via the speaker 21, the voice "The evaluation will start."

[0053] Referring to FIG. 6, in step S26, the fixed terminal 90 acquires a first measurement value x1. The first measurement value x1 is the activity amount of the muscles near the cheeks. In one implementation example, the fixed terminal 90 acquires the sum of the potentials of the working electrode 111 with respect to the potential of the reference electrode 112 during a given period after the control in step S24. Then, the fixed terminal 90 acquires the sum as the first measurement value x1 and stores it in the memory 96. The above "given period" that is the acquisition target of the first measurement value x1 is an example of the timing of the evaluation target.

[0054] In step S28, the fixed terminal 90 determines whether the first measured value x1 is less than or equal to the first reference value X1. If the fixed terminal 90 determines that the first measured value x1 is less than or equal to the first reference value X1 (YES in step S28), the control proceeds to step S32. Also, if the fixed terminal 90 determines that the first measured value x1 is not less than or equal to the first reference value X1 (NO in step S28), the control proceeds to step S30.

[0055] In step S30, the fixed terminal 90 updates the value of the first reference value X1 stored in the memory 96 with the value of the first measured value x1, and proceeds with the control to step S32.

[0056] In step S32, the fixed terminal 90 acquires the second measured value x2. The second measured value x2 is the activity amount of the muscles near the eyebrows. In one implementation example, the fixed terminal 90 acquires the sum of the potentials of the working electrode 121 with reference to the potential of the reference electrode 122 during a given period after the control of step S24. Then, the fixed terminal 90 acquires the sum as the second measured value x2 and stores it in the memory 96. The above "given period" for which the second measured value x2 is acquired is an example of the timing of the evaluation target.

[0057] In step S34, the fixed terminal 90 determines whether the second measured value x2 is less than or equal to the second reference value X2. If the fixed terminal 90 determines that the second measured value x2 is less than or equal to the second reference value X2 (YES in step S34), the control proceeds to step S38. Also, if the fixed terminal 90 determines that the second measured value x2 is not less than or equal to the second reference value X2 (NO in step S34), the control proceeds to step S36.

[0058] In step S36, the fixed terminal 90 updates the value of the second reference value X2 stored in the memory 96 with the value of the second measured value x2, and proceeds with the control to step S38.

[0059] In step S38, the fixed terminal 90 calculates the emotion indices E1 and E2 according to equations (1) and (2). Note that the activity amount A1 used in equations (1) and (2) is calculated according to the following equation (3), and the activity amount A2 is calculated according to the following equation (4).

[0060] A1 = x1 / X1 …(3) A2 = x2 / X2 …(4) In step S40, the fixed terminal 90 generates evaluation information using the emotion indices E1 and E2 calculated in step S38. In one implementation example, the fixed terminal 90 generates evaluation information by applying the emotion indices E1 and E2 to the map MP (Figure 4).

[0061] In step S42, the fixed terminal 90 outputs the evaluation information generated in step S40. An example of the output of the evaluation information is to directly display the character string (such as "positive degree 60%") that constitutes the evaluation information. Another example is to output an image and / or voice corresponding to the character string (such as "positive degree 60%") that constitutes the evaluation information.

[0062] In step S44, the fixed terminal 90 determines whether the subject has removed the wearable terminal 10. In one implementation example, the fixed terminal 90 determines whether the subject has removed the wearable terminal 10 based on whether the detection signal from the infrared sensor 22 indicates that the wearable terminal 10 has been removed from the subject.

[0063] When the fixed terminal 90 determines that the subject has removed the wearable terminal 10 (YES in step S44), it returns control to step S10. On the other hand, when the fixed terminal 90 determines that the subject has not removed the wearable terminal 10 (NO in step S44), it returns control to step S26.

[0064] Accordingly, if the subject has not removed the wearable terminal 10, the fixed terminal 90 continues to generate and output evaluation information (steps S26 to S42). On the other hand, when the subject removes the wearable terminal 10, if the fixed terminal 90 detects that the wearable terminal 10 has been worn by the subject (YES in step S10), after obtaining the first reference value X1 and the second reference value X2 (steps S12 to S24), it generates and outputs evaluation information (steps S26 to S42).

[0065] In the first embodiment described above, for generating evaluation information regarding the expression of the subject, not only the measured value of the timing of the evaluation target but also the reference value of the subject obtained in response to the occurrence of a given incident is used. Thereby, individual differences in how emotions appear in expressions are reflected in the evaluation information. Therefore, if the generated evaluation information is used, an individual's emotion can be accurately estimated.

[0066] In addition, in the processes described with reference to FIGS. 5 and 6, in steps S14 and S20, the reference value was obtained in the process of generating evaluation information, but the timing at which the reference value is obtained is not limited to this. The reference value may be obtained in advance for each subject before the above process is executed. Also, the reference value may be obtained after the measured value is obtained.

[0067] [Embodiment 2] <System Configuration> FIG. 7 is a diagram schematically showing the overall configuration of the expression evaluation system according to the second embodiment. The expression evaluation system 200 shown in FIG. 7 acquires a face image of a subject (the subject) and generates evaluation information of the subject's expression based on the feature amount obtained from the face image. In the second embodiment, the feature amount obtained from the face image is an example of biometric information.

[0068] The facial expression evaluation system 200 includes a fixed terminal 90, a display 90A, a speaker 90B, and an imaging device 90X. The facial expression evaluation system 200 captures a face image of a subject located in the imaging target area of the imaging device 90X, and outputs evaluation information regarding the facial expression of the subject using the captured face image.

[0069] In the example of FIG. 7, the facial expression evaluation system 200 displays evaluation information on the display 90A together with the face image of the subject captured by the imaging device 90X. The “smile degree 80%” displayed as the evaluation information corresponds to the evaluation information “positivity degree 80%” shown in FIG. 4. That is, in the example of FIG. 7, the “positivity degree” constituting the evaluation information is output as the “smile degree”. Also, the “negativity degree” constituting the evaluation information is output as the “regret degree”. However, how the evaluation information is output can be changed as appropriate.

[0070] <Feature quantity> In Embodiment 2, in order to calculate the emotion indices E1 and E2 described in Embodiment 1, feature quantities F1 and F2 are used instead of the activity amounts A1 and A2. That is, in Embodiment 2, each of Equation (1) and Equation (2) is changed to each of the following Equation (5) and Equation (6).

[0071] E1 = k11·F1 + k12·F2 …(5) E2 = k21·F1 + k22·F2 …(6) The feature quantity F1 is a feature quantity that changes according to the movement of the cheeks, and is a feature quantity representing the features of an expression corresponding to a positive emotion. The feature quantity F2 is a feature quantity that changes according to the movement of the eyebrows, and is a feature quantity representing the features of an expression corresponding to a negative emotion. The emotion index E1 is calculated as the sum of the product of the feature quantity F1 and the coefficient k11 and the product of the feature quantity F2 and the coefficient k12. Also, the emotion index E2 is calculated as the sum of the product of the feature quantity F1 and the coefficient k21 and the product of the feature quantity F2 and the coefficient k22.

[0072] The feature quantity F1 is, for example, the distance between the center of the nose and the corners of the mouth. The feature quantity F2 is, for example, the distance between the inner sides of both eyebrows.

[0073] <Flow of processing> FIGS. 8 and 9 are flowcharts of the processing performed by the fixed terminal 90 to output evaluation information (evaluation information) in the facial expression evaluation system 200. In the processing shown in FIGS. 8 and 9, the same reference numerals are assigned to the steps common to the processing shown in FIGS. 5 and 6. Hereinafter, the changes from the processing shown in FIGS. 5 and 6 in the processing shown in FIGS. 8 and 9 will be mainly described.

[0074] Referring to FIG. 8, in step S11, the fixed terminal 90 determines whether a person is present in the imaging target area of the imaging device 90X. In one implementation example, the fixed terminal 90 determines whether a person is present in the imaging target area by determining whether the image captured by the imaging device 90X includes a pattern that is identified as a human face. Since the recognition of the pattern identified as a human face can be realized using known techniques, a detailed description will not be repeated here.

[0075] The fixed terminal 90 repeats the control of step S11 until it determines that a person is present in the imaging target area (NO in step S11). Then, when the fixed terminal 90 determines that a person is present in the imaging target area (YES in step S11), it advances the control to step S12.

[0076] In step S12, the fixed terminal 90 outputs a first message (a message prompting the subject to make a positive facial expression).

[0077] In step S14, the fixed terminal 90 acquires a first reference value X1. In Embodiment 2, the first reference value X1 is a reference value for the feature quantity F1. In one implementation example, the fixed terminal 90 acquires the feature quantity F1 in the face image captured at a given timing as the first reference value X1. Then, the fixed terminal 90 stores the first reference value X1 in the memory 96.

[0078] In step S16, the fixed terminal 90 notifies that it has acquired the first reference value X1. In step S18, the fixed terminal 90 outputs a second message (a message prompting the subject to make a negative expression).

[0079] In step S20, the fixed terminal 90 acquires a second reference value X2. In Embodiment 2, the second reference value X2 is a reference value for the feature quantity F2. In one implementation example, the fixed terminal 90 acquires the feature quantity F2 in the face image captured at a given timing as the second reference value X2, and stores the second reference value X2 in the memory 96.

[0080] Thereafter, in step S22, the fixed terminal 90 notifies that it has acquired the second reference value X2, and in step S24, notifies that the evaluation of the subject's expression is started.

[0081] Referring to FIG. 9, in step S26, the fixed terminal 90 acquires a first measurement value x1. In Embodiment 2, the first measurement value x1 is a measurement value of the feature quantity F1. In one implementation example, the fixed terminal 90 acquires the feature quantity F1 from the face image captured at a given timing after step S24 as the first measurement value x1, and stores it in the memory 96. The above "given timing" is an example of the timing of the evaluation target.

[0082] In step S28, the fixed terminal 90 determines whether the first measurement value x1 is less than or equal to the first reference value X1. If the fixed terminal 90 determines that the first measurement value x1 is less than or equal to the first reference value X1 (YES in step S28), the control proceeds to step S32. Also, if the fixed terminal 90 determines that the first measurement value x1 is not less than or equal to the first reference value X1 (NO in step S28), the control proceeds to step S30.

[0083] In step S30, the fixed terminal 90 updates the value of the first reference value X1 stored in the memory 96 with the value of the first measurement value x1, and the control proceeds to step S32.

[0084] In step S32, the fixed terminal 90 acquires the second measurement value x2. In Embodiment 2, the second measurement value x2 is a measurement value of the feature quantity F2. In one implementation example, the fixed terminal 90 acquires the feature quantity F2 as the second measurement value x2 from a face image captured at a given timing after step S24, and stores it in the memory 96. The above "given timing" is an example of the timing of the evaluation target.

[0085] In step S34, the fixed terminal 90 determines whether the second measurement value x2 is less than or equal to the second reference value X2. When the fixed terminal 90 determines that the second measurement value x2 is less than or equal to the second reference value X2 (YES in step S34), the control proceeds to step S38. Also, when the fixed terminal 90 determines that the second measurement value x2 is not less than or equal to the second reference value X2 (NO in step S34), the control proceeds to step S36.

[0086] In step S36, the fixed terminal 90 updates the value of the second reference value X2 stored in the memory 96 with the value of the second measurement value x2, and proceeds with the control to step S38.

[0087] In step S38, the fixed terminal 90 calculates the emotion indices E1 and E2 according to formulas (5) and (6). Note that the feature quantity F1 used in formulas (5) and (6) is calculated according to the following formula (7), and the feature quantity F2 is calculated according to the following formula (8).

[0088] F1 = x1 / X1 …(7) F2 = x2 / X2 …(8) In step S40, the fixed terminal 90 generates evaluation information using the emotion indices E1 and E2 calculated in step S38.

[0089] In step S42, the fixed terminal 90 outputs the evaluation information generated in step S40.

[0090] In step S45, the fixed terminal 90 determines whether there is a person in the imaging target area of the imaging device 90X. In one implementation example, the fixed terminal 90 determines whether there is a person in the imaging target area by determining whether the image captured by the imaging device 90X contains a pattern that can be identified as a human face.

[0091] If the fixed terminal 90 determines that there is no person in the imaging target area (YES in step S45), it returns the control to step S11. On the other hand, if the fixed terminal 90 determines that there is a person in the imaging target area (NO in step S45), it returns the control to step S26.

[0092] Thereby, when the same subject continuously positions in the imaging target area, the fixed terminal 90 continues to generate and output evaluation information (steps S26 to S42). On the other hand, after a certain subject leaves the imaging target area (YES in step S45), when the fixed terminal 90 detects that the next subject exists in the imaging target area (YES in step S11), after obtaining the first reference value X1 and the second reference value X2 (steps S12 to S24), it generates evaluation information (steps S26 to S40).

[0093] [Specific implementation example] A specific implementation example of the technology disclosed in the present disclosure will be described. Note that the following description is for illustrative purposes and does not limit the implementation modes of the technology disclosed in the present disclosure.

[0094] [Evaluation of product] In one implementation example, the technology disclosed in the present disclosure is used for estimating the user's emotion in a series of experiences of purchasing and using a product. More specifically, the emotion of the user in the above experience is estimated from the evaluation information of the expression, and the estimation result of the emotion is used as feedback information in product development. Such an estimation result of the emotion can also be used in marketing applications such as neuromarketing.

[0095] [Evaluation of content] In one implementation example, the technology disclosed in the present disclosure is used for estimating the emotions of a user who is provided with content such as a game. More specifically, from the expression evaluation information, the emotions of the user who is receiving the content are estimated, and the estimation result of the emotions is used as feedback information in content production. More specifically, the user's impression regarding the difficulty level of game content and / or the impression received by the user from video content are obtained as feedback information and can be utilized in the production of new content.

[0096] [Modification Example] <The number of types of activity amounts or feature amounts used> In the above-described Embodiment 1, the activity amount A1 of the muscles near the cheeks and the activity amount A2 of the muscles near the eyebrows were used to generate evaluation information regarding the expression of the subject. Also, in Embodiment 2, the feature amount F1 representing the features of an expression corresponding to a positive emotion and the feature amount F2 representing the features of an expression corresponding to a negative emotion were used to generate evaluation information regarding the expression of the subject.

[0097] Note that the number of types of activity amounts or feature amounts used for generating the evaluation information regarding the expression of the subject is not limited to 2. The number of types of activity amounts or feature amounts used may be 1 or may be 3 or more.

[0098] For example, n types of activity amounts A1 to An may be used to generate n types of emotion indices E1 to En according to the following formula (9), and the emotion indices E1 to En may be used to generate evaluation information. In formula (9), the coefficients k11 to knn constitute a coefficient matrix of the coefficients used for generating the evaluation information, which is represented by an n×n matrix.

[0099] [Equation]

[0100] <Usage mode of the reference value> As shown in Formula (3) and Formula (4), the activity amount used for calculating the emotion index is calculated as the ratio of the measured value to the reference value (x1 / X1 or x2 / X2). Note that the activity amount does not necessarily have to be the ratio of the measured value to the reference value. As long as the reference value is reflected as an individual difference for each subject in the activity amount, the activity amount may be calculated as a function of the reference value and the measured value. Similarly, as long as the reference value is reflected as an individual difference for each subject in the feature amount, the feature amount used for calculating the emotion index may be calculated as a function of the reference value and the measured value.

[0101] <Message Output> In step S12, a first message prompting the subject to make a positive expression is output. Also, in step S18, a second message prompting the subject to make a negative expression is output. Each of these message outputs is an example of a "given incident" that defines the timing for obtaining the first reference value X1 and the second reference value X2, and these messages are examples of the content output for obtaining the reference value. Then, in response to the output of each of these messages, the first reference value X1 and the second reference value X2 are each obtained (steps S14, S20).

[0102] Instead of outputting these messages, content for prompting the subject to make a given expression may be output.

[0103] For example, in step S12, content (video and / or audio prepared to make the subject laugh) that prompts the subject to make a positive expression may be output. Also, in step S18, content (video and / or audio prepared to make the subject uncomfortable or give a sense of horror) that prompts the subject to make a negative expression may be output.

[0104] Furthermore, the timing at which the above-described content is output may not be notified to the subject. That is, when the fixed terminal 90 outputs the above-described content, the first reference value X1 and the second reference value X2 may be generated without notifying the subject. As a result, based on each of the positive expression and the negative expression made by the subject without being conscious, each of the first reference value X1 and the second reference value X2 can be obtained.

[0105] <Detection of a new subject> In step S44 of the first embodiment, after the acquisition of the first reference value X1 in step S14 and the acquisition of the second reference value X2 in step S20, it is determined whether the subject continues to wear the wearable terminal 10. When it is determined that the subject has removed the wearable terminal 10, after it is again determined that the subject has removed the wearable terminal 10, the acquisition of the first reference value X1 in step S14 and the acquisition of the second reference value X2 in step S20 are performed again.

[0106] In step S45 of the second embodiment, after the acquisition of the first reference value X1 in step S14 and the acquisition of the second reference value X2 in step S20, it is determined whether the subject continues to be located in the imaging target area of the imaging device 90X. When it is determined that the subject is no longer located in the imaging target area of the imaging device 90X, after it is again determined that the subject is located in the imaging target area of the imaging device 90X, the acquisition of the first reference value X1 in step S14 and the acquisition of the second reference value X2 in step S20 are performed again.

[0107] According to the above control, when a new subject is detected as an evaluation target of the expression, the first reference value X1 and the second reference value X2 are newly obtained. As a result, even when the subjects are changed, the first reference value X1 and the second reference value X2 can be obtained for each subject.

[0108] In this sense, step S10 is an example of control for detecting a new subject. Also, step S11 is another example of control for detecting a new subject. And the acquisition of the first reference value X1 in step S14 and the acquisition of the second reference value X2 in step S20 are performed in response to the detection of a new subject.

[0109] Note that the method for detecting a new subject can be detected by any method other than the wearing of the wearable terminal 10 in the first embodiment and the detection of the presence of a person in the imaging target area in the second embodiment. For example, a new subject may be detected by a human sensor or by an operation on a dedicated button.

[0110] Also, in the above-described embodiment, an example in which the reference value is updated when the measured value exceeds the reference value has been described. However, the present invention is not limited to this, and includes an example in which the reference value is updated when the measured value is lower than the reference value depending on the measurement target.

[0111] For example, when evaluating an expression such as frowning eyebrows by the electromyogram near the eyebrows, the larger the degree of frowning, the larger the measured value. Therefore, it is appropriate to update the reference value when the measured value exceeds the reference value. In this case, the measured value exceeding the reference value can be an example of the measurement result exceeding the standard.

[0112] On the other hand, when capturing the same frowning expression by the distance between the inner sides of both eyebrows, the larger the degree of frowning, the smaller the measured value. Therefore, it is appropriate to update the reference value when the measured value is lower than the reference value. In this case, the measured value being lower than the reference value can be an example of the measurement result exceeding the standard.

[0113] Also, the term "reference value" is used as an example of a reference regarding a subject, and this reference may be expressed in a form other than a value. Further, the term "measured value" is used as an example of a measurement result regarding a subject, and this measurement result may be expressed in a form other than a value. That is, the reference and the measurement result in this paragraph may be expressed in a form other than a value, such as image information. And when information regarding the subject's facial expression is generated using the reference and the measurement result, the information may be generated by comparing images or the like.

[0114] [Aspect] Those skilled in the art will understand that the above-described plurality of exemplary embodiments are specific examples of the following aspects.

[0115] (Item 1) An evaluation method according to one aspect is a method for evaluating a subject's facial expression, the method including: obtaining a reference value of biometric information of the subject's face region; obtaining a measured value of biometric information of the subject's face region at a timing of an evaluation target; generating evaluation information regarding the subject's facial expression using the reference value and the measured value; and outputting the evaluation information.

[0116] According to the evaluation method described in Item 1, a technique for accurately estimating an individual's emotion is provided.

[0117] (Item 2) In the evaluation method described in Item 1, a reference value and a measured value may be obtained as the reference and the measurement result of the biometric information of the subject's face region, respectively.

[0118] According to the evaluation method described in Item 2, evaluation information can be easily generated. (Item 3) In the evaluation method described in Item 2, the step of generating evaluation information regarding the subject's facial expression may include calculating the evaluation information using a ratio of the measured value to the reference value.

[0119] According to the evaluation method described in claim 3, evaluation information can be generated more easily. (Claim 4) The evaluation method described in claim 2 or 3 may further include a step of updating the reference value by replacing the reference value with the measured value when the measured value exceeds the reference value.

[0120] According to the evaluation method described in claim 4, personal feelings can be estimated more accurately based on the evaluation information.

[0121] (Claim 5) In the evaluation method described in any one of claims 2 to 4, each of the reference value and the measured value may include one or more types of feature amounts of the subject's face obtained from the face image of the subject.

[0122] According to the evaluation method described in claim 5, personal feelings can be estimated more accurately based on the evaluation information.

[0123] (Claim 6) In the evaluation method described in claim 5, the one or more types of feature amounts may include at least one of a first feature amount that changes according to the movement of the cheeks and a second feature amount that changes according to the movement of both eyebrows.

[0124] According to the evaluation method described in claim 6, personal feelings can be estimated more accurately based on the evaluation information.

[0125] (Claim 7) In the evaluation method described in claim 6, the first feature amount may be the distance between the nose and the corners of the mouth.

[0126] According to the evaluation method described in claim 7, evaluation information can be generated more easily. (Claim 8) In the evaluation method described in claim 6 or 7, the second feature amount may be the distance between both eyebrows.

[0127] According to the evaluation method described in claim 8, evaluation information can be generated more easily. (Item 9) In the evaluation method according to any one of Items 1 to 8, the biological information of the subject's face area may be obtained from the electromyogram signal of the subject's facial expression muscles.

[0128] According to the evaluation method described in Item 9, personal feelings can be estimated more accurately based on the evaluation information.

[0129] (Item 10) In the evaluation method according to Item 9, the facial expression muscles may include at least one of the muscles near the cheeks and the muscles near the eyebrows.

[0130] According to the evaluation method described in Item 10, personal feelings can be estimated more accurately based on the evaluation information.

[0131] (Item 11) In the evaluation method according to any one of Items 1 to 10, the biological information of the subject's face area may be obtained from the subject's face image.

[0132] According to the evaluation method described in Item 11, personal feelings can be estimated more accurately based on the evaluation information.

[0133] (Item 12) The evaluation method described in any one of Items 1 to 11 may further include a step of outputting content for obtaining the reference, and the step of obtaining the reference may be performed in response to the output of the content.

[0134] According to the evaluation method described in Item 12, evaluation information can be easily generated. (Item 13) The evaluation method described in any one of Items 1 to 12 may further include a step of detecting a new subject as an evaluation target of the facial expression, and the step of obtaining the reference may be performed in response to the detection of the new subject.

[0135] According to the evaluation method described in Item 13, even when the subjects are changed, reference values can be obtained for each subject.

[0136] (Item 14) An evaluation system according to one aspect is a system for evaluating a subject's facial expression, comprising a processor and an interface for acquiring biometric information of the subject's face region, wherein the processor uses a standard of the biometric information of the subject's face region and a measurement result of the biometric information of the subject's face region acquired at the timing of the evaluation target to generate evaluation information regarding the subject's facial expression and output the evaluation information.

[0137] According to the evaluation system described in Item 14, a technique for accurately estimating an individual's emotion is provided.

[0138] (Item 15) A program according to one aspect is a program for causing a subject's facial expression to be evaluated. When the program is executed by a processor of a computer, the computer is caused to perform steps of acquiring a standard of biometric information of the subject's face region, acquiring a measurement result of the biometric information of the subject's face region at the timing of the evaluation target, generating evaluation information regarding the subject's facial expression using the standard and the measurement result, and outputting the evaluation information.

[0139] According to the program described in Item 15, a technique for accurately estimating an individual's emotion is provided.

[0140] The embodiments disclosed this time should be considered to be illustrative in all respects and not restrictive. The scope of the present disclosure is indicated by the claims rather than the description of the above-described embodiments, and it is intended that all modifications within the meaning and scope equivalent to the claims be included. Also, each technique in the embodiments can be implemented alone or, if necessary, in combination with other techniques in the embodiments as much as possible.

Description of Reference Numerals

[0141] 1 Sensor unit, 2 Signal processing circuit, 3,91 Controller, 4,92 Communication module, 5 Battery, 6 Housing, 10 Wearable terminal, 10A First arm, 10B Second arm, 11 First electromyogram sensor, 12 Second electromyogram sensor, 14,24,AR11,AR21 Region, 21,90B Speaker, 22 Infrared sensor, 31,95 Processor, 32,96 Memory, 33,97 Input / output port, 90 Fixed terminal, 90A Display, 90X Imaging device, 100,200 Facial expression evaluation system, 111,121 Working electrode, 112,122 Reference electrode, 141,142,161,162 Line, 151,152,171,172 Pad.

Claims

1. A method for evaluating a subject's expression, comprising: obtaining a standard of biometric information of the face region of the subject; obtaining a measurement result of the biometric information of the face region of the subject at the timing of the evaluation target; generating evaluation information regarding the expression of the subject using the standard and the measurement result; outputting the evaluation information, wherein a reference value is obtained as the standard of the biometric information of the face region of the subject, and a measured value is obtained as the measurement result of the biometric information of the face region of the subject, further comprising the step of updating the reference value by replacing the reference value with the measured value when the measured value exceeds the reference value; The biometric information of the face region of the subject is obtained from the electromyogram signal of the facial muscles of the subject.

2. The step of generating evaluation information regarding the expression of the subject includes calculating the evaluation information using the ratio of the measured value to the reference value. The evaluation method according to claim 1.

3. further comprising the step of outputting content for obtaining the standard, The step of obtaining the standard is performed in response to the output of the content. The evaluation method according to claim 1.

4. further comprising the step of detecting a new subject as an evaluation target of the expression, The step of obtaining the standard is performed in response to the detection of a new subject. The evaluation method according to claim 1.

5. A system for evaluating a subject's expression, comprising: a processor; an interface for obtaining biometric information of the face region of the subject, wherein the processor generates evaluation information regarding the expression of the subject using the standard of the biometric information of the face region of the subject and the measurement result of the biometric information of the face region of the subject obtained at the timing of the evaluation target, and is configured to output the evaluation information, wherein a reference value is obtained as the standard of the biometric information of the face region of the subject, and a measured value is obtained as the measurement result of the biometric information of the face region of the subject, the processor updates the reference value by replacing the reference value with the measured value when the measured value exceeds the reference value, The biometric information of the face region of the subject is obtained from the electromyogram signal of the facial muscles of the subject.

6. A program for evaluating the expression of a subject, which, when executed by a processor of a computer, causes the computer to acquire a reference of biometric information of the face region of the subject; acquire a measurement result of biometric information of the face region of the subject at the timing of the evaluation target; generate evaluation information regarding the expression of the subject using the reference and the measurement result; output the evaluation information; a reference value is acquired as a reference of biometric information of the face region of the subject, and a measured value is acquired as a measurement result of biometric information of the face region of the subject; the program causes the computer to perform a step of updating the reference value by replacing the reference value with the measured value when the measured value exceeds the reference value; the biometric information of the face region of the subject is acquired from an electromyogram signal of the facial muscles of the subject.

Images