Facial expression code generation device, facial expression generation device, facial expression generation system, and facial expression code generation method

Abstract

Provided are a facial expression code generation device, a facial expression generation device, a facial expression generation system, and a facial expression code generation method that have a reduced data processing load.　This facial expression code generation device comprises: a plurality of electrodes capable of acquiring myoelectric potentials by coming into contact with a plurality of parts of the face; and a code generation unit that generates code representing the displacement of at least one of the plurality of parts on the basis of the plurality of myoelectric potentials acquired by the plurality of electrodes.

Description

Facial expression code generating device, facial expression generating device, facial expression generating system, and facial expression code generating method

[0001] The present disclosure relates to an expression code generation device, an expression generation system, and an expression code generation method.

[0002] Conventionally, there has been a device that attaches a plurality of electrodes to a human face to measure the myoelectric potential of the face, and generates a facial image such as an avatar with an expression corresponding to the human's facial expression based on the myoelectric potential. This conventional device generates data representing the entire facial expression based on the myoelectric potential (see, for example, Patent Document 1).

[0003] U.S. Pat. No. 10,943,100

[0004] However, when facial expressions change, the shape of only a part of the face, such as the eyebrows or mouth, may change, rather than the entire face. Conventional devices generate data representing the entire facial expression even in such cases, which places a heavy burden on data processing. If data processing takes a long time, a time lag occurs between the time when the actual human facial expression changes and the time when the avatar's facial expression changes, causing viewers to feel uncomfortable.

[0005] Therefore, an object of the present invention is to provide a facial expression code generating device, a facial expression generating system, and a facial expression code generating method that reduce the load of data processing.

[0006] An embodiment of the facial expression code generation device of the present disclosure includes a plurality of electrodes capable of contacting a plurality of parts of the face, respectively, to acquire myoelectric potentials, and a code generation unit that generates a code representing the displacement of at least one part of the plurality of parts based on the myoelectric potentials acquired by the plurality of electrodes.

[0007] It is possible to provide a facial expression code generating device, a facial expression generating system, and a facial expression code generating method that reduce the load of data processing.

[0008] 1 is a diagram illustrating an example of the configuration of the facial expression code generation device 100 according to an embodiment. FIG. 2 is a diagram illustrating an example of the configuration of the facial expression code generation device 100 according to an embodiment. FIG. 3 is a diagram illustrating an example of muscles in a human face. FIG. 4 is a diagram illustrating an example of the configuration of the facial expression code generation device 100 and a facial expression generation system 300 according to an embodiment. FIG. 5 is a diagram illustrating an example of thresholds used by the firmware 132C to generate a code. FIG. 6 is a diagram illustrating an example of the configuration of parts of the signal processing unit 130 relating to the sensor amplifier 131, the common-mode noise calculation unit 135, and the amplifier 136. FIG. 7 is a flowchart (part 1) illustrating an example of processing executed by the firmware 132C. FIG. 8 is a flowchart (part 2) illustrating an example of processing executed by the firmware 132C. FIG. 9 is a diagram illustrating an example of a facial expression generation device 500A according to an embodiment. FIG. 10 is a diagram illustrating an example of a facial expression generation device 500B according to an embodiment. FIG. 11 is a diagram illustrating an example of a facial expression generation device 500C according to an embodiment. FIG. 12 is a diagram illustrating an example of the configuration of the facial expression generation device 500C. FIG. 13 is a diagram illustrating an example of key codes. FIG. 14 is a diagram illustrating an example of facial expressions of an avatar realized by key codes A to J. 4 is a diagram illustrating an example of the flow of data between the facial expression code generation device 100 and the smartphone 200 in FIG. 3. FIG. 5 is a diagram illustrating the configuration of a facial expression code generation device 100M1 according to a modified embodiment. FIG. 6 is a diagram illustrating the configuration of a facial expression code generation device 100M2 according to a modified embodiment. FIG. 7 is a diagram illustrating the configuration of a facial expression code generation device 100M3 according to a modified embodiment. FIG. 8 is a diagram illustrating the configuration of a facial expression code generation device 100M4 according to a modified embodiment. FIG. 9 is a diagram illustrating the configuration of a facial expression code generation device 100M5 according to a modified embodiment. FIG. 10 is a diagram illustrating an example of how the facial expression code generation device 100 according to the embodiment is worn. FIG. 11 is a diagram illustrating an example of how an facial expression code generation device 100M6 according to a modified embodiment is worn. FIG. 12 is a diagram illustrating an example of how an facial expression code generation device 100M7 according to a modified embodiment is worn. FIG. 13 is a diagram illustrating an example of how an facial expression code generation device 100M8 according to a modified embodiment is worn.

[0009] Hereinafter, embodiments of the facial expression code generation device, facial expression generation system, and facial expression code generation method according to the present disclosure will be described. In the following, the same elements will be denoted by the same reference numerals, and duplicated descriptions may be omitted.

[0010] In the following description, an XYZ coordinate system is defined. The direction parallel to the X axis (X direction), the direction parallel to the Y axis (Y direction), and the direction parallel to the Z axis (Z direction) are perpendicular to one another. A planar view refers to a view from the XY plane. For ease of explanation, a vertical relationship is used in which the +Z direction side is the upper side and the −Z direction side is the lower side, but this does not represent a universal vertical relationship. In addition to the XYZ coordinate system, the following description will also use left, right, top, and bottom as seen from the user. In the following description, the length, width, thickness, etc. of each part may be exaggerated to make the configuration easier to understand.

[0011] 1A and 1B are diagrams showing an example of the configuration of an expression code generation device 100 according to an embodiment. The expression code generation device 100 is, for example, a goggle-type device that can be worn on the head of a user. In FIGS. 1A and 1B, the -X direction is the left side, and the +X direction is the right side.

[0012] <Facial expression code generation device 100> The facial expression code generation device 100 includes goggles 110, electrodes 120-1 to 120-8, a noise reduction electrode 125, and a signal processing unit 130. The goggles 110 are an example of a wearing device. The electrodes 120-1 to 120-8 form an eight-channel electrode configuration. Hereinafter, when there is no need to distinguish between the electrodes 120-1 to 120-8, they will be simply referred to as electrodes 120.

[0013] <Goggles 110> The goggles 110 include a main body 111, a headband 112, a shield 113, and a cushion 114. The main body 111 is a frame that surrounds the eyes of a user wearing the facial expression code generation device 100. When viewed from a plane (XY plane), both left and right ends of the main body 111 are curved toward the -Y direction to fit the shape of the face. Hereinafter, when the user wears the facial expression code generation device 100, the side of the main body 111 that faces the face will be referred to as the -Y direction side of the main body 111, and the side that faces away from the face will be referred to as the +Y direction side of the main body 111. Note that the main body 111 may be made of resin, metal, or the like, for example.

[0014] The headband 112 is attached to the left and right ends of the main body 111 and forms a loop that surrounds the head together with the main body 111. Note that only the portion of the headband 112 that is connected to the main body 111 is shown in Figures 1A and 1B. The headband 112 is, for example, a belt made of elastic rubber or a belt with a length adjustment function.

[0015] The shield 113 is attached to the +Y direction side of the main body 111, and is a transparent cover that covers the opening of the frame-shaped main body 111. The shield 113 is made of, for example, resin or glass.

[0016] The cushion 114 is attached to the -Y direction side of the main body 111 and is a frame-shaped cushion corresponding to the shape of the main body 111. When the user wears the facial expression code generation device 100, the cushion 114 comes into contact with the skin of the face around both eyes, such as the forehead, temples, cheeks, and nose. The cushion 114 is, for example, an elastic material such as urethane foam or sponge. As with the main body 111, when the user wears the facial expression code generation device 100, the side of the cushion 114 that is located on the face side is referred to as the -Y direction side of the cushion 114.

[0017] Here, a configuration in which the facial expression code generation device 100 includes the goggles 110 will be described, but a full-face mask or the like may be used instead of the goggles 110. Furthermore, the facial expression code generation device 100 may be configured not to include the goggles 110 as an example of a wearing unit, but to use the electrodes 120-1 to 120-8 by directly attaching them to the face with tape or the like.

[0018] <Electrodes 120-1 to 120-8> The electrodes 120-1 to 120-8 are provided on the -Y direction side of the main body 111 of the goggles 110, exposed on the -Y direction side of the cushion 114, and output myoelectric potential signals representing myoelectric potentials. Each of the electrodes 120-1 to 120-8 has two electrode pieces and is capable of acquiring differential myoelectric potential signals. In other words, the myoelectric potential signals acquired by the electrodes 120-1 to 120-8, each of which has two electrode pieces, are differential signals representing myoelectric potentials. Here, the positions of the electrodes 120-1 to 120-8 when the facial expression code generation device 100 is worn by a user will be described. Figure 2 will also be used to explain the positions of the electrodes 120-1 to 120-8. Figure 2 is a diagram showing an example of human facial muscles. Although the configuration in which each of electrodes 120-1 to 120-8 has two electrode pieces is described here, each of electrodes 120-1 to 120-8 may be configured to be a single electrode and to output a single-ended myoelectric potential signal.

[0019] As shown in Figure 2, the left and right corrugator supercilii muscles, the left and right frontalis muscles, the left and right zygomaticus major muscles, and the left and right levator labio labio nas alae muscles are located around the eyes on the face. These facial muscles move to form facial expressions when a person laughs, saddens (crying), gets angry, etc. In order to measure such changes in facial expressions, the facial code generation device 100 includes electrodes 120-1 to 120-8 that measure the myoelectric potentials of the left and right corrugator supercilii muscles, the left and right frontalis muscles, the left and right zygomaticus major muscles, and the left and right levator labio labio nas alae muscles. Note that, hereinafter, the left and right corrugator supercilii muscles, the left and right frontalis muscles, the left and right zygomaticus major muscles, and the left and right levator labio labio nas alae muscles may be referred to as facial muscles when no particular distinction is made.

[0020] Electrode 120-1 is provided at a position corresponding to the left corrugator supercilii. Electrode 120-2 is provided at a position corresponding to the right corrugator supercilii. Electrode 120-3 is provided at a position corresponding to the left frontalis. Electrode 120-4 is provided at a position corresponding to the right frontalis. Electrode 120-5 is provided at a position corresponding to the left levator labii alae naris superioris. Electrode 120-6 is provided at a position corresponding to the right levator labii alae naris superioris. Electrode 120-7 is provided at a position corresponding to the left zygomaticus major. Electrode 120-8 is provided at a position corresponding to the right zygomaticus major. Note that, hereinafter, electrodes 120-1 to 120-8 may be distinguished as channels 1 to 8. For this reason, in FIG. 2, the muscles corresponding to electrodes 120-1 to 120-8 are labeled Ch1 to Ch8.

[0021] By using the electrodes 120-1 to 120-8, it is possible to detect the movements of the left eyebrow, right eyebrow, left eye, right eye, left cheek, right cheek, left upper lip, and right upper lip.

[0022] Here, a configuration in which the facial expression code generation device 100 includes eight electrodes 120-1 to 120-8 will be described, but the facial expression code generation device 100 may be configured to omit any of the eight electrodes 120-1 to 120-8. Furthermore, the myoelectric potentials of facial muscles other than the left and right corrugator supercilii, left and right frontalis muscles, left and right zygomaticus major muscles, and left and right levator labii superioris alae naris may also be measured. The circuitry for acquiring the differential signal will be described later.

[0023] <Noise Cancellation Electrode 125> The noise cancellation electrode 125 is provided to cancel out noise contained in the differential signals acquired by each of the electrodes 120-1 to 120-8. As an example, the facial expression code generation device 100 has two noise cancellation electrodes 125. The two noise cancellation electrodes 125 are provided in the center in the Z direction on both the left and right ends of the −Y direction side of the main body 111 of the goggles 110, respectively, and are exposed on the −Y direction side of the cushion 114.

[0024] The noise filtering electrodes 125 contact the left and right cheeks of the user's face when the user is wearing the facial expression code generation device 100. The noise filtering electrodes 125 output cancellation signals to the left and right cheeks that cancel out noise contained in the differential signal. The positions of the noise filtering electrodes 125 are not limited to positions corresponding to the left and right cheeks of the face, but are preferably positions corresponding to any of the left or right corrugator supercilii, left or right frontalis, left or right levator labii superioris alae naris, or left or right zygomatic major. At least one noise filtering electrode 125 is sufficient, but three or more may be provided. The circuit that outputs the cancellation signal to the noise filtering electrodes 125 is included in the signal processing unit 130, the details of which will be described later.

[0025] <Signal Processing Unit 130> The signal processing unit 130 is provided, for example, at the left end of the main body 111. The signal processing unit 130 includes a signal processing circuit and a battery 130A. The battery 130A is provided, for example, at the lower end of the signal processing unit 130. The position at which the signal processing unit 130 is provided is not limited to the left end of the main body 111, and the signal processing unit 130 may be provided at a location other than the left end of the main body 111. The signal processing unit 130 may be attached to the headband 112 in consideration of weight balance, etc. The battery 130A may be provided separately from the signal processing circuit portion of the signal processing unit 130. The signal processing unit 130 may not include the battery 130A and may instead be configured to receive power from a smartphone 200, a personal computer (PC), a tablet computer, an XR headset (an augmented reality (AR) headset, or a virtual reality (VR) headset), etc., described below, via a USB cable or the like.

[0026] The signal processing circuit generates a code representing the displacement of at least one of the multiple facial parts based on the myoelectric potentials acquired by electrodes 120-1 to 120-8. The signal processing unit 130 outputs the code via wireless communication, wired communication, or the like. The code is output to a device that can change the facial expression of an avatar or character image displayed on a display device such as a monitor, or a device that can physically change the facial expression of a mask or costume. Details of the signal processing circuit and code of the signal processing unit 130 will be described later.

[0027] <Facial Expression Code Generating Device 100 and Facial Expression Generating System 300> FIG. 3 is a diagram showing an example of the configuration of the facial expression code generating device 100 and the facial expression generating system 300 according to the embodiment.

[0028] <Facial Expression Generation System 300> The facial expression generation system 300 includes the facial expression code generation device 100 and the smartphone 200. The facial expression generation system 300 includes the facial expression code generation device 100 and another device. The other device included in the facial expression generation system 300 is a device that can change the facial expression of an avatar or character image displayed on a display device such as a monitor, based on the code output by the facial expression code generation device 100.

[0029] The facial expression generation device of the embodiment is a device such as a mask or a costume integrated with the facial expression code generation device 100, and is a device that can physically change the facial expression of the mask or costume based on the code generated by the facial expression code generation device 100. The facial expression generation device of the embodiment will be described later using the figure ☆. Here, a facial expression generation system 300 including the facial expression code generation device 100 and a smartphone 200 will be described.

[0030] The smartphone 200 is an example of a device that can change the facial expression of an avatar or character image displayed on a display device such as a display, based on a code. Here, a form in which the facial expression generation system 300 includes the facial expression code generation device 100 and the smartphone 200 will be described. However, instead of the smartphone 200, the facial expression generation system 300 may include a PC (Personal Computer), a tablet computer, an XR headset (an AR (Augmented Reality) headset, or a VR (Virtual Reality) headset), or the like.

[0031] Fig. 3 shows a simplified view of the goggles 110 of the facial expression code generation device 100. Also, in Fig. 3, two electrodes are provided for each channel, and each electrode that differentially detects myoelectric potentials is shown simplified as a single electrode. The noise elimination electrode 125 is omitted from Fig. 3. Fig. 3 also shows the portion of the signal processing unit 130 of the facial expression code generation device 100 that generates codes from the myoelectric potentials acquired by the electrodes 120-1 to 120-8, and omits the portion that performs signal processing related to the noise elimination electrode 125.

[0032] The signal processing unit 130 includes a sensor amplifier 131, a CPU (Central Processing Unit) 132, a BLE (Bluetooth (registered trademark) Low Energy) module 133, an antenna 134, and a battery 130A. The BLE module 133 is an example of a code output unit.

[0033] Eight sensor amplifiers 131 are provided corresponding to the electrodes 120-1 to 120-8. That is, eight sensor amplifiers 131 are provided for eight channels. The eight sensor amplifiers 131 are connected to a CPU 132. Each sensor amplifier 131 differentially amplifies a differential signal obtained from the myoelectric potential of each of the two electrode pieces of the electrodes 120-1 to 120-8, and outputs a displacement signal representing the displacement of the facial muscles to the CPU 132.

[0034] Although a configuration in which eight sensor amplifiers 131 are provided corresponding to the electrodes 120-1 to 120-8 will be described here, the number of electrodes 120 may be other than eight. The number of sensor amplifiers 131 may be the same as the number of electrodes 120, and one sensor amplifier 131 may be connected to each electrode 120.

[0035] The CPU 132 is a CPU chip that includes a selector 132A, an analog-to-digital converter (ADC) 132B, and firmware (FW) 132C. The firmware 132C is an example of a code generation unit, and is realized by a digital signal processor (DSP), for example.

[0036] The selector 132A has eight input terminals connected to the output terminals of the eight sensor amplifiers 131, respectively, and one output terminal connected to the input terminal of the ADC 132B. The selector 132A selects the displacement signals input from the eight sensor amplifiers 131 in a time-division manner one by one (one channel at a time), and outputs the selected signal to the ADC 132B.

[0037] The ADC 132B digitally converts the eight displacement signals input from the selector 132A in a time-division manner and outputs the converted signals to the firmware 132C. Hereinafter, the displacement signals digitally converted by the ADC 132B will be referred to as digital displacement signals. The digital displacement signals have, for example, discrete values ​​that represent the signal levels of the analog displacement signals before digital conversion.

[0038] The ADC 132B digitally converts each of the eight displacement signals input in a time-division manner from the eight sensor amplifiers 131 and outputs the eight digital displacement signals to the firmware 132C in sequence. The eight digital displacement signals are output from the selector 132A to the ADC 132B in the order of Ch1 to Ch8, making it possible to identify which facial muscle the signal is based on the myoelectric potential of: the left and right corrugator supercilii, the left and right frontalis, the left and right levator labii superioris alaris, and the left and right zygomatic major. Note that, for example, if there is a channel of higher importance among Ch1 to Ch8, the digital displacement signal of the higher importance channel may be output several times. Furthermore, the order in which the digital displacement signals are output may be other than Ch1 to Ch8.

[0039] The number of ADCs 132B provided may be the same as the number of sensor amplifiers 131, and one ADC 132B may be connected to the output side of each sensor amplifier 131.

[0040] The firmware 132C converts the displacement signal digitized by the ADC 132B into a code and outputs the code. The firmware 132C has a threshold value for determining whether or not facial muscles are moving based on the signal level (discrete value) of the digital displacement signal.

[0041] Next, thresholds will be described with reference to Figures 4A and 4B. Figures 4A and 4B are diagrams showing examples of thresholds used by the firmware 132C to generate codes. The thresholds shown in Figures 4A and 4B are thresholds used for one of the eight digital displacement signals. Figure 4A shows one threshold TH1, and Figure 4B shows two thresholds TH11 and TH12. Note that, as an example, TH12 is greater than TH11.

[0042] For example, the firmware 132C uses the threshold value TH1 shown in FIG. 4A to generate a code "1" if the signal level of the digital displacement signal is equal to or greater than the threshold value TH1, and does not generate a code if the signal level of the digital displacement signal is less than the threshold value TH1. The code "1" indicates that a facial muscle has moved. By generating a code only when the signal level of the digital displacement signal is equal to or greater than the threshold value TH1 and not generating a code when the signal level is less than the threshold value TH1, it is possible to generate a code that indicates only the portion of the facial muscle that has been displaced. This also reduces the load on data processing.

[0043] 4B, the firmware 132C generates a code "11" if the signal level of the digital displacement signal is equal to or greater than the threshold value TH11 and less than the threshold value TH12, and generates a code "12" if the signal level of the digital displacement signal is equal to or greater than the threshold value TH12. The firmware 132C does not generate a code if the signal level of the digital displacement signal is less than the threshold value TH11.

[0044] A code "11" indicates a small movement of the facial muscles, and a code "12" indicates a large movement of the facial muscles.

[0045] 4A and 4B are thresholds used for one of the eight digital displacement signals. By using the same thresholds for the other seven digital displacement signals, the firmware 132C can generate codes that can determine the presence or absence of facial muscle movement for the eight digital displacement signals. The firmware 132C outputs the codes to the BLE module 133.

[0046] <BLE module 133> The BLE module 133 is a communication module capable of wireless communication using Bluetooth (registered trademark) Low Energy. The BLE module 133 is connected to an antenna 134, and outputs an advertising signal including a code from the antenna 134.

[0047] <Smartphone 200> The smartphone 200 includes a control device 210, a BLE module 220, and a display 230. The display 230 is an example of a display device.

[0048] <Control device 210> The control device 210 is a device that performs processing related to the operation of the smartphone 200, and is realized by a computer system having a CPU and memory. As an example, the control device 210 has an application 210A. The application 210A is a functional block that is realized when the CPU of the control device 210 executes a program stored in the memory. Code is input to the control device 210 via the BLE module 220.

[0049] The application 210A is, for example, an application program for a smartphone VTuber. The application 210A generates facial expressions for a facial image based on code input from the BLE module 220. The facial expression generation for a facial image is a process of generating facial expressions and time-series movements of the facial expressions (facial expression / motion script), and as an example, the facial expressions of a two-dimensional (2D) avatar are generated.

[0050] A VTuber wears the facial expression code generation device 100 on their face and uses the application 210A while changing their own facial expression, thereby changing the facial expression of an avatar based on their own face. For example, if the VTuber moves their left eyebrow upward, the left eyebrow of the avatar's face displayed on the display 230 moves upward, and if the VTuber lifts their right cheek, the right cheek of the avatar's face displayed on the display 230 is lifted.

[0051] <Common-mode noise calculation unit> Fig. 5 is a diagram showing an example of the configuration of the portion of the signal processing unit 130 related to the sensor amplifier 131, the common-mode noise calculation unit 135, and the amplifier 136. The common-mode noise calculation unit 135 is an example of a cancellation signal output unit. Fig. 5 also shows two electrode pieces of the electrode 120 and one noise cancellation electrode 125.

[0052] Two electrode pieces of electrode 120 are connected to two input terminals of sensor amplifier 131, and sensor amplifier 131 differentially amplifies a differential signal obtained from the myoelectric potential of the two electrodes and outputs a displacement signal representing the displacement of facial muscles. While Fig. 3 shows a circuit configuration in which CPU 132 is connected to the output side of sensor amplifier 131, in reality, CPU 132 and common-mode noise calculation unit 135 are connected to the output side of sensor amplifier 131. Therefore, Fig. 5 shows wiring connected to CPU 132 on the output side of sensor amplifier 131.

[0053] The common-mode noise calculation unit 135 is connected to the noise removal electrode 125 via an amplifier 136 .

[0054] The in-phase noise calculation unit 135 generates an anti-phase component of the noise contained in the displacement signal and outputs it to the amplifier 136. The anti-phase component of the noise contained in the displacement signal is a signal that has an anti-phase to the in-phase noise contained in the differential signal obtained from the myoelectric potentials of the two electrodes, and therefore can reduce the in-phase noise in the differential signal. Although differential signals have high noise resistance, they cannot remove in-phase noise (in-phase noise) contained in the two signals of the differential signal. Therefore, the in-phase noise calculation unit 135 outputs a signal that is anti-phase to the in-phase noise contained in the differential signal, thereby reducing the in-phase noise in the differential signal obtained by the two electrode pieces of the electrode 120.

[0055] The sensor amplifier 131 connected to the input side of the common-mode noise calculation unit 135 may be at least one of the eight sensor amplifiers 131 connected to the output sides of the eight electrodes 120-1 to 120-8. The sensor amplifier 131 connected to the input side of the common-mode noise calculation unit 135 may be the sensor amplifier 131 connected to the electrode 120 closest to the noise removal electrode 125 among the electrodes 120-1 to 120-8.

[0056] As an example, there are two noise elimination electrodes 125, one on the left and one on the right, and each noise elimination electrode 125 is connected to a common-mode noise calculation unit 135 as shown in FIG. 5 . As an example, the left noise elimination electrode 125, the common-mode noise calculation unit 135, and the amplifier 136 may be connected to any one of the electrodes 120-1, 120-3, 120-5, and 120-7 in the left half. In this case, the right noise elimination electrode 125, the common-mode noise calculation unit 135, and the amplifier 136 may be connected to any one of the electrodes 120-2, 120-4, 120-6, and 120-8 in the right half. Alternatively, eight noise elimination electrodes 125 may be provided, and the noise elimination electrode 125, the common-mode noise calculation unit 135, and the amplifier 136 may be connected to each of the eight electrodes 120-1 to 120-8. Alternatively, the noise eliminating electrode 125, the common-mode noise calculation unit 135, and the amplifier 136 may be connected to any one of the eight electrodes 120-1 to 120-8. Alternatively, the upper noise eliminating electrode 125, the common-mode noise calculation unit 135, and the amplifier 136 may be connected to any one of the upper half electrodes 120-1 to 120-4. In this case, the lower noise eliminating electrode 125, the common-mode noise calculation unit 135, and the amplifier 136 may be connected to any one of the lower half electrodes 120-5 to 120-8.

[0057] 6A is a flowchart (part 1) showing an example of a process executed by the firmware 132C. This process is realized by the facial expression code generation method according to the embodiment.

[0058] When the firmware 132C starts processing, it selects each of the eight channels one by one and executes the subroutine processing of steps S1 to S3 to generate a code.

[0059] The firmware 132C acquires myoelectric potentials by causing the selector 132A to select the sensor amplifiers 131 for channels 1 to 8 one by one in order (step S1).

[0060] The firmware 132C determines whether the myoelectric potential is equal to or greater than the threshold value TH1 (step S2).

[0061] When the firmware 132C determines that the myoelectric potential is equal to or greater than the threshold value TH1 (S2: Yes), it generates a code "1" for that channel, which indicates that a facial muscle has moved (step S3).

[0062] Furthermore, if the firmware 132C determines in step S2 that the myoelectric potential is not equal to or greater than the threshold value TH1 (S2: No), the firmware 132C advances the flow to step S4 without generating a code.

[0063] After completing the process of step S2 or S3, the firmware 132C determines whether to end the series of processes (step S4).

[0064] If the firmware 132C determines that the series of processes should not be ended (S4: No), the flow returns to the start of the subroutine process and executes the processes from step S1 onwards.

[0065] When the firmware 132C determines in step S4 that the series of processes is to be ended (S4: Yes), the firmware 132C ends the series of processes (STOP).

[0066] <Flowchart (Part 2)> Fig. 6B is a flowchart (Part 2) showing an example of processing executed by the firmware 132C. This processing is realized by the facial expression code generation method according to the embodiment.

[0067] When the firmware 132C starts the process, it selects the eight channels one by one and executes the subroutine process of step S1A to acquire the myoelectric potential.

[0068] The firmware 132C acquires myoelectric potentials by causing the selector 132A to select the sensor amplifiers 131 for channels 1 to 8 one by one in order (step S1A).

[0069] The firmware 132C acquires the maximum myoelectric potential among the eight myoelectric potentials from channels 1 to 8 (step S2A).

[0070] The firmware 132C determines whether the maximum myoelectric potential is equal to or greater than the threshold value TH1 (step S3A).

[0071] If the firmware 132C determines that the maximum myoelectric potential is equal to or greater than the threshold value TH1 (S3A: Yes), it generates a code "1" for that channel, which indicates that a facial muscle has moved (step S4A).

[0072] Furthermore, if the firmware 132C determines in step S3A that the maximum myoelectric potential is not equal to or greater than the threshold value TH1 (S3A: No), the firmware 132C advances the flow to step S5A without generating a code.

[0073] After completing the process of step S3A or S4A, the firmware 132C determines whether to end the process (step S4).

[0074] If the firmware 132C determines not to end the process (S5A: No), it returns the flow to the start of the subroutine process and executes the processes from step S1A onwards.

[0075] If the firmware 132C determines in step S5A that the process is to be ended (S5A: Yes), the firmware 132C ends the series of processes (STOP). Note that it may be determined whether the myoelectric potential of each channel is equal to or greater than the threshold value TH1 without specifying the site showing the maximum myoelectric potential.

[0076] <Facial Expression Generation Device 500A> Fig. 7A is a diagram showing an example of the facial expression generation device 500A according to an embodiment. The left side of Fig. 7A shows an example of the state of the facial expression generation device 500A before the facial expression changes, and the right side of Fig. 7A shows an example of the state of the facial expression generation device 500A after the facial expression changes.

[0077] 7A includes the facial expression generating device 500A and the facial expression code generating device 100 and the costume suit 400A. A person wearing the facial expression code generating device 100 on their face enters the facial expression generating device 500A and wears the costume suit 400A. For this reason, the facial expression code generating device 100 worn by a person on their face inside the costume suit 400A is shown by a dashed line.

[0078] The costume 400A is configured so that the shape of the eyes and mouth on the face can be changed by actuators, allowing the facial expression to be changed. A control unit that controls the actuators of the costume 400A is connected to the CPU 132 of the facial expression code generation device 100, and codes output from the CPU 132 are input to the control unit, which activates the actuators and allows the shape of the eyes and mouth on the face to be changed.

[0079] For example, when a person wearing the facial expression code generation device 100 changes the expression of the face 10 (see inside the speech bubble) of the stuffed animal suit 400A to a smile, the facial expression code generation device 100 generates a code representing the displacement of the muscles around the mouth, and the expression of the stuffed animal suit 400A changes to a smile, as shown on the right side of Fig. 7A. In other words, a person wearing the facial expression code generation device 100 can change the facial expression of the stuffed animal suit 400A by changing their own facial expression. In other words, it is possible to change the expression of the stuffed animal suit 400A using the face as an input unit.

[0080] Furthermore, since the code is not data representing the facial expression of the entire face, but merely code data representing the changed parts of the facial muscles that form the expression, the data volume is small, and the load on data processing can be reduced.

[0081] Furthermore, when the costume 400A is used at an event venue, the facial expression of the person inside the costume 400A changes, so that the facial expression of the costume 400A changes, allowing communication with the audience without gestures. The person inside can use their own facial expression as an operating unit for the facial expression of the costume 400A. Note that, although the above description has been given using the costume 400A, instead of the costume 400A, a puppet or the like may be configured so that the shape of the eyes and mouth of the face can be changed by an actuator.

[0082] <Facial Expression Generation Device 500B> Fig. 7B is a diagram showing an example of the facial expression generation device 500B according to an embodiment. The left side of Fig. 7B shows an example of the state of the facial expression generation device 500B before the facial expression changes, and the right side of Fig. 7B shows an example of the state of the facial expression generation device 500B after the facial expression changes.

[0083] The facial expression generating device 500B shown in Fig. 7B includes the facial expression code generating device 100 and a mask 400B. In the facial expression generating device 500B, the facial expression code generating device 100 and the mask 400B are integrated. Fig. 7B shows the face of the mask 400B, and the facial expression code generating device 100 including the goggles 110 shown in Fig. 1A and Fig. 1B is integrally provided on the back side.

[0084] The facial expression generating device 500B is an integrated device of the facial expression code generating device 100 and the mask 400B, and is worn on the face. For this reason, the facial expression code generating device 100 worn by a person on the face is shown by a dashed line within the mask 400B.

[0085] As an example, the mask 400B has LEDs (Light Emitting Diodes) attached to both eyes, and the LEDs are configured to be changeable in color, so that changing the color of the LEDs changes the facial expression. By connecting a control unit that controls the color of the LEDs emitted by the mask 400B to the CPU 132 of the facial expression code generation device 100 and inputting a code output from the CPU 132 to the control unit, the color of the LEDs emitted can be changed based on the code generated by the facial expression code generation device 100 in accordance with the wearer's facial expression. It is also possible to change the color of the light emitted by any part of the mask 400B.

[0086] That is, a person wearing the facial expression code generation device 100 on their face can change the color of the LEDs in the eyes of the mask 400B by changing their facial expression. That is, it is possible to change the facial expression of the mask 400B using the face as an input unit.

[0087] Furthermore, since the code is not data representing the facial expression of the entire face, but merely code data representing the changed parts of the facial muscles that form the expression, the data volume is small, and the load on data processing can be reduced.

[0088] The LEDs attached to the mask 400B light up or change their lighting style (color, lighting pattern, etc.) depending on the facial expression, allowing the wearer of the mask 400B to become part of the decorations that brighten up the event venue. This is particularly useful in amusement parks, etc.

[0089] In addition, by collecting customer reactions via wireless data communication, the promoters (performers) can obtain information on the reactions of the venue.

[0090] Furthermore, although the mask 400B has been described here, instead of the mask 400B, LEDs capable of emitting light in various colors may be attached to the goggles 110 so that they can be illuminated around the face. Penlights available for purchase at live venues have become popular in recent years as a tool for livening up a concert by pressing a button on the goggles to change the color of the performer's choice. However, by using the goggles 110 equipped with LEDs that can be illuminated around the face as described above instead of a penlight, the wearer can change their facial expression to light up the LEDs, allowing their reactions to liven up the concert. Furthermore, by connecting wirelessly, the organizer of the concert can collect reactions and control the colors.

[0091] Instead of or in addition to the LED light emission, sound may be used to express the facial expression. Also, instead of mask 400B, a full-face mask with LEDs attached may be used, or the shape of the eyes and mouth on the mask may be changed by an actuator, thereby changing the facial expression.

[0092] <Facial Expression Generation Device 500C> Fig. 7C is a diagram showing an example of the facial expression generation device 500C according to an embodiment. The facial expression generation device 500C includes the facial expression code generation device 100 and an XR headset 400C. In the facial expression generation device 500C, the facial expression code generation device 100 and the XR headset 400C are integrated together. In Fig. 7C, the XR headset 400C is integrally provided on the +Y direction side of the facial expression code generation device 100, which includes the goggles 110. The facial expression generation device 500C is worn on the face when in use.

[0093] 1A and 1B, the facial expression code generation device 100 is configured so that the display 430 of the XR headset 400C can be seen through the goggles 110. As an example, a two-dimensional or three-dimensional avatar is displayed on the display 430.

[0094] 7D is a diagram showing an example of the configuration of a facial expression generation device 500C. The facial expression code generation device 100 shown in Fig. 7D does not include the BLE module 133, and the CPU 132 is connected to the control device 210 of the XR headset 400C.

[0095] The control device 210 of the XR headset 400C shown in Fig. 7D is a device that performs processing related to the operation of the XR headset 400C, and is realized by a computer system having a CPU and memory. As an example, the control device 210 has an application 210B. The application 210B is a functional block realized by the CPU of the control device 210 executing a program stored in memory. Codes are input to the control device 210 from the CPU 132 of the facial expression code generation device 100.

[0096] As an example, the application 210B is an application for the XR headset 400C.

[0097] Application 210B is an application in which a driver is provided in the preceding stage of Open XR, and an interface compatible with device A is provided in the subsequent stage of Open XR. Device A is, for example, an interface compatible with XR headsets from various companies, and generates facial expressions for a facial image in accordance with the display method of each company's XR headset.

[0098] The code input to the driver is output from OpenXR to one of the devices A, and the facial expression of the facial image is generated by changing the image of the part of the facial image that corresponds to the code. As a result, a two-dimensional (2D) or three-dimensional (3D) avatar is generated and displayed on the display 430. Since only the image of a part of the facial image is changed rather than changing the expression of the entire facial image, the processing load is light.

[0099] By wearing the facial expression generating device 500C, which is an integrated combination of the XR headset 400C and the facial expression code generating device 100, and using the application 210B while changing one's own facial expression, one can change the facial expression of an avatar based on one's own face. For example, if one moves one's left eyebrow upward, the left eyebrow of the avatar's face displayed on the display 430 moves upward, and if one lifts one's right cheek, the right cheek of the avatar's face displayed on the display 230 is lifted.

[0100] That is, a person wearing facial expression generating device 500C on their face can change their own facial expression, thereby changing the facial expression of the avatar displayed on display 430. That is, it is possible to change the facial expression of the avatar using the face as an input unit.

[0101] Furthermore, since the code is not data representing the facial expression of the entire face, but merely code data representing the changed parts of the facial muscles that form the expression, the data volume is small, and the load on data processing can be reduced.

[0102] <Key Code> Here, a key code that can be used instead of the above-mentioned code will be described. The firmware 132C may be configured to convert the digital displacement signal converted by the ADC 132B into a key code and output the key code. In other words, the firmware 132C may use the key code described below instead of the above-mentioned code.

[0103] An example of the key code is a USB (Universal Serial Bus) HID (Human Interface Device) Usage ID (Identifier) ​​key code.

[0104] Furthermore, the key codes used in this embodiment may be key codes other than those assigned to physical keys on a keyboard, among all key codes. The keyboard may be, for example, a Japanese JIS keyboard or a QWERTY keyboard. An example of a key code other than those assigned to physical keys on a keyboard, among all key codes, is F13 to F24, which are assigned as function keys but do not exist as physical keys. Specific examples of keyboards on which F13 to F24 do not exist as physical keys include the English 101 / 102 keyboards of the QWERTY keyboard and the 106 / 109 keyboards of the Japanese JIS keyboard.

[0105] When the facial expression code generation device 100 generates key codes assigned to physical keys of a keyboard as described above, there is a risk that a command based on the key code assigned to the physical key may be input to the smartphone 200 or a PC. For example, if the key code assigned to the physical key is the letter A, a command to input the letter A may be input to the smartphone 200 or a PC due to a change in the myoelectric potential of facial muscles, which may cause the smartphone 200 or the PC to operate. In contrast, if a key code other than the key codes assigned to physical keys of a Japanese JIS keyboard or a QWERTY keyboard is used, the smartphone 200 or the PC will not operate even if the key code generated by the facial expression code generation device 100 is input to the smartphone 200 or the PC. Therefore, it is possible to generate dedicated key codes based on the change in the myoelectric potential of facial muscles, and to stably change the facial expressions of avatars or characters displayed on a display device, or masks or costumes whose facial expressions can be physically changed, without generating unused commands. The facial expression code generation device 100 may generate key codes assigned to physical keys of a keyboard as described above. In this case, for example, commands based on the key codes assigned to the physical keys can operate the smartphone 200 or a PC, and the facial expression of an avatar or character can be changed. For example, the facial expression of an avatar or character can be changed while chatting with characters.

[0106] FIG. 8 is a diagram showing an example of key codes corresponding to types of facial expressions. For example, there are eight types of facial expressions: left eyebrow raised, left eyebrow lowered, left eye wide open, left cheek raised, left eye closed, right eyebrow raised, right eyebrow lowered, right eye wide open, right cheek raised, and right eye closed. FIG. 8 shows Ch1 to Ch8. Ch1 to Ch8 correspond to the facial muscles shown in FIG. 2. FIG. 8 also shows the comparison results between the digital displacement signal based on the myoelectric potential acquired by electrodes 120-1 to 120-8 of Ch1 to Ch8 and a threshold value, using H (High) and L (Low) levels. The H level indicates the comparison result when the digital displacement signal is equal to or greater than the threshold value, and the L level indicates the comparison result when the digital displacement signal is less than the threshold value.

[0107] When the facial expression is left eyebrow raised, the comparison results of the digital displacement signals of Ch1, Ch3, Ch5, and Ch7 are L, H, L, L. This is because the left frontalis muscle is displaced.

[0108] When the facial expression is one in which the left eyebrow is lowered, the comparison results of the digital displacement signals of Ch1, Ch3, Ch5, and Ch7 are H, L, L, L. This is because the left corrugator supercilii muscle is displaced.

[0109] When the facial expression is with the left eye open, the comparison results of the digital displacement signals of Ch1, Ch3, Ch5, and Ch7 are H, H, L, L. This is because the left corrugator supercilii and left frontalis muscles are displaced.

[0110] When the facial expression is left cheek up, the comparison results of the digital displacement signals of Ch1, Ch3, Ch5, and Ch7 are L, L, L, and H. This is because the left zygomatic major muscle is displaced.

[0111] When the facial expression is left eye closed (closed tightly), the comparison results of the digital displacement signals of Ch1, Ch3, Ch5, and Ch7 are L, H, H, L. This is because the left frontalis muscle and the left levator labii superioris alae naris muscle are displaced.

[0112] When the facial expression is right eyebrow raising, the comparison results of the digital displacement signals of Ch2, Ch4, Ch6, and Ch8 are L, H, L, L. This is because the right frontalis muscle is displaced.

[0113] When the facial expression is right eyebrow lowering, the comparison results of the digital displacement signals of Ch2, Ch4, Ch6, and Ch8 are H, L, L, L. This is because the right corrugator supercilii muscle is displaced.

[0114] When the facial expression is right eye wide open, the comparison results of the digital displacement signals of Ch2, Ch4, Ch6, and Ch8 are H, H, L, and L. This is because the right corrugator supercilii and right frontalis muscles are displaced.

[0115] When the facial expression is right cheek up, the comparison results of the digital displacement signals of Ch2, Ch4, Ch6, and Ch8 are L, L, L, and H. This is because the right zygomatic major muscle is displaced.

[0116] When the facial expression is right eye closed (closed tightly), the comparison results of the digital displacement signals of Ch2, Ch4, Ch6, and Ch8 are L, H, H, and L. This is because the right frontalis muscle and the right levator labii superioris alae naris muscle are displaced.

[0117] In such a case, as an example, key codes A to E are assigned to left eyebrow raising, left eyebrow lowering, left eye wide open, left cheek lifting, and left eye closed. Also, as an example, key codes F to J are assigned to right eyebrow raising, right eyebrow lowering, right eye wide open, right cheek lifting, and right eye closed. As an example, firmware 132C may associate H / L combinations of Ch1 to Ch8 with key codes and store them in an internal memory.

[0118] By using such key codes A to J, key codes A to J can be generated by wearing the facial expression code generation device 100 on the face and changing the facial expression. For example, as shown in FIG. 3 , when the facial expression code generation device 100 and the smartphone 200 are used in a manner that allows wireless communication via BLE, the smartphone 200 can install facial expression files representing the following facial expressions in its memory: left eyebrow raised, left eyebrow lowered, left eye wide open, left cheek raised, left eye closed, right eyebrow raised, right eyebrow lowered, right eye wide open, right cheek raised, and right eye closed. The facial expression file corresponding to the key code (any one of A to J) obtained from the facial expression code generation device 100 can then be specified by the key bind function. As a result, the facial expression of the avatar displayed on the display 230 is controlled by the control device 210 according to the key codes A to J.

[0119] Fig. 9 is a diagram showing examples of facial expressions of an avatar realized by key codes A to J. The leftmost part of Fig. 9 shows the default expression.

[0120] To the right of the default facial expression, in two rows and five columns, are shown facial expressions: left eyebrow raised, left eyebrow lowered, left eye wide open, left cheek raised, left eye closed, right eyebrow raised, right eyebrow lowered, right eye wide open, right cheek raised, and right eye closed, along with assigned key codes A to J. That is, Fig. 9 shows 11 facial expressions: default facial expression, left eyebrow raised, left eyebrow lowered, left eye wide open, left cheek raised, left eye closed, right eyebrow raised, right eyebrow lowered, right eye wide open, right cheek raised, and right eye closed.

[0121] Here, as an example, it is assumed that there are ten facial expression files representing ten facial expressions other than the default facial expression shown in FIG. 9 in order to change the default facial expression.

[0122] The facial expression files corresponding to each of the ten facial expressions shown in Fig. 9 are files in text format that represent the movements of each facial expression (left eyebrow raised, left eyebrow lowered, left eye wide open, left cheek raised, left eye closed, right eyebrow raised, right eyebrow lowered, right eye wide open, right cheek raised, right eye closed) as parameters. The file format of the facial expression files is, for example, a JSON file. The file names of the facial expression files may be any name.

[0123] These 10 facial expression files are stored in the memory of smartphone 200. As an example, application 210A of smartphone 200 associates each of the 10 facial expression files with one of 10 key codes (A to J) by a key binding function.

[0124] The application 210A of the smartphone 200 identifies the expression file corresponding to the key code (one of A to J) obtained from the expression code generation device 100, and changes the avatar's expression to the expression represented by the identified expression file. In this way, the facial expression of the avatar displayed on the display 230 can be changed to various expressions shown in FIG.

[0125] Also, here, the file format of the facial expression file is a JSON file, and the application 210A uses a key binding function to identify the facial expression file corresponding to the key code. However, as long as the application 210A can identify the facial expression file corresponding to the key code, the file format of the facial expression file is not limited to a JSON file, and the application 210A may identify the facial expression file corresponding to the key code using a function other than the key binding function.

[0126] Although the embodiment using the key bind function has been described above, table data associating data representing types of facial expressions with key codes may be stored in the memory of the smartphone 200. The application 210A of the smartphone 200 may then specify data of types of facial expressions in the table data that correspond to the key codes acquired from the facial code generation device 100, and change the facial expression of the avatar.

[0127] FIG. 10 is a diagram illustrating an example of the flow of data between the facial expression code generation device 100 and the smartphone 200 in FIG.

[0128] When the host controller of the smartphone 200 acquires a key code from the facial expression code generation device 100, the key code is input to the application 210A via the HID standard driver (kbdhid.sys), the class driver (kbdclass.sys), the kernel, and the API. The application 210A identifies an facial expression file corresponding to the key code using a key bind function, and uses the identified facial expression file to change the facial expression of the avatar, for example, by raising the left eyebrow.

[0129] <Facial Expression Code Generation Devices 100M1 to 100M5 According to Modifications of the Embodiment> FIGS. 11A to 11E are diagrams showing the configurations of facial expression code generation devices 100M1 to 100M5 according to modifications of the embodiment.

[0130] 11A includes only the left half of the electrodes 120-1 to 120-8 of the facial expression code generation device 100 (see FIGS. 1A and 1B), that is, electrodes 120-1, 120-3, 120-5, and 120-7. The facial expression code generation device 100M1 does not include electrodes 120-2, 120-4, 120-6, and 120-8.

[0131] For example, the facial expressions of the avatar may not use expressions that displace only one side, such as an expression in which only one eye is closed, an expression in which only one eyebrow is raised, or an expression in which only one cheek is moved. The facial expressions of the avatar may use expressions in which both sides of the face are displaced equally, such as an expression in which both eyes are closed, an expression in which both eyebrows are raised, or an expression in which both cheeks are moved. Furthermore, the costume 400A (see FIG. 7A ) and the mask 400B may also use expressions in which both sides of the face are displaced equally. In such cases, the number of electrodes 120 need not be eight, but can be reduced to four channels on either the left or right side.

[0132] The facial expression code generating device 100M1 includes only electrodes 120-1, 120-3, 120-5, and 120-7 for Ch1, Ch3, Ch5, and Ch7, but can generate codes for Ch1 to Ch8 as follows.

[0133] In the facial expression code generation device 100M1, the firmware 132C generates codes for Ch1 and Ch2 when the myoelectric potential acquired on Ch1 is equal to or greater than the threshold value TH1. Similarly, the firmware 132C generates codes for Ch3 and Ch4 when the myoelectric potential acquired on Ch3 is equal to or greater than the threshold value TH1. The firmware 132C generates codes for Ch5 and Ch6 when the myoelectric potential acquired on Ch5 is equal to or greater than the threshold value TH1. The firmware 132C generates codes for Ch7 and Ch8 when the myoelectric potential acquired on Ch7 is equal to or greater than the threshold value TH1.

[0134] The codes generated for Ch2, Ch4, Ch6, and Ch8 are symmetric codes that represent displacements that are bilaterally symmetrical to the displacements generated by the codes generated for Ch1, Ch3, Ch5, and Ch7 when viewed from the front of the face.

[0135] In this way, it is possible to generate facial expressions such as left and right eyebrows raised, left and right eyebrows lowered, left and right eyes wide open, left and right cheeks raised, and left and right eyes closed.

[0136] Furthermore, by reducing the number of electrodes 120, the facial expression code generation device 100M1 can be simplified and manufacturing costs can be reduced.

[0137] The facial expression code generation device 100M2 shown in FIG. 11B has a configuration in which the electrodes 120 include electrodes 120-1, 120-3, 120-4, 120-5, 120-7, and 120-8, but do not include electrodes 120-2 and 120-6.

[0138] Of the facial muscles around both eyes, the left and right corrugator supercilii (Ch1, Ch2) and levator labii superioris alae naris (Ch5, Ch6), which are located in the left-right center of the face, are closer to each other and have less movement than the left and right frontalis (Ch3, Ch4) and left and right zygomaticus major (Ch7, Ch8), which are located on the left and right sides of the face.In other words, the left and right corrugator supercilii and levator labii superioris alae naris have less influence on facial expressions than the left and right frontalis and left and right zygomaticus major.

[0139] For this reason, the facial expression code generating device 100M2 omits electrodes 120-2 and 120-6, and the firmware 132C generates codes for Ch1 and Ch2 when the myoelectric potential acquired on Ch1 is equal to or greater than the threshold value TH1, and generates codes for Ch5 and Ch6 when the myoelectric potential acquired on Ch5 is equal to or greater than the threshold value TH1.

[0140] The codes generated for Ch2 and Ch6 are symmetrical codes that represent displacements that are symmetrical to the displacements generated by the codes generated for Ch1 and Ch5 when viewed from the front of the face. Note that the codes for Ch3 to Ch4 and Ch7 to Ch8 are the same as those of the facial expression code generation device 100.

[0141] By doing this, for example, for an expression of raised eyebrows, it is possible to generate an expression of raised eyebrows on both sides instead of just one side, but for expressions of lowered eyebrows, wide-open eyes, raised cheeks, and closed eyes, it is possible to generate expressions of only one side.

[0142] Furthermore, by reducing the number of electrodes 120, the facial expression code generation device 100M1 can be simplified and manufacturing costs can be reduced.

[0143] Furthermore, facial expression code generation devices 100M3 to 100M5, which will be described below with reference to FIGS. 11C to 11E, have a configuration in which some of the electrodes 120-1 to 120-8 (see FIGS. 1A and 1B) are omitted as will be described below.

[0144] The facial expression code generation device 100M3 shown in Figure 11C has a configuration in which the two electrode segments of each of electrodes 120-1 and 120-2 of the facial expression code generation device 100 (see Figures 1A and 1B) are reduced to one, and the remaining two electrode segments are assigned to electrode 120-1 of Ch1. The two electrode segments assigned to electrode 120-1 are arranged symmetrically, one on each side of the left and right halves of the center of the face in the left-right direction when viewed from the front. The facial expression code generation device 100M3 includes electrodes 120-1 and 120-3 to 120-8, but does not include electrode 120-2, but can generate codes for Ch1 to Ch8. Ch3 to Ch8 are the same as those of the facial expression code generation device 100.

[0145] By using such electrodes 120-1, a differential signal for Ch1 can be obtained from the myoelectric potential of the left and right corrugator supercilii muscles. In the facial expression code generation device 100M3, when generating a code for Ch1, a code for Ch2 can also be generated. The code generated for Ch2 is a code that represents a displacement that is symmetrical to the displacement caused by the code generated for Ch1 when the face is viewed from the front. The two codes for Ch1 and Ch2 are symmetrical codes that represent displacement that is symmetrical when the face is viewed from the front.

[0146] The facial expression code generation device 100M4 shown in Figure 11D has a configuration in which the two electrode segments of each of electrodes 120-5 and 120-6 of the facial expression code generation device 100 (see Figures 1A and 1B) are reduced to one, and the remaining two electrode segments are assigned to electrode 120-5 of Ch5. The two electrode segments assigned to electrode 120-5 are arranged symmetrically, one on each side of the left and right halves of the center of the face in the left-right direction when viewed from the front. The facial expression code generation device 100M4 includes electrodes 120-1 to 120-5 and electrodes 120-7 to 120-8, but does not include electrode 120-6, but can generate codes for Ch1 to Ch8. Ch1 to Ch4 and Ch7 to Ch8 are the same as those of the facial expression code generation device 100.

[0147] By using such electrodes 120-5, a differential signal for Ch5 can be obtained from the myoelectric potential of the left and right levator labii superioris alae naris muscles. When generating a code for Ch5, the facial expression code generation device 100M4 also generates a code for Ch6. The code generated for Ch6 is a code that represents a displacement that is symmetrical to the displacement caused by the code generated for Ch5 when viewed from the front of the face. The two codes for Ch5 and Ch6 are symmetrical codes that represent displacements that are symmetrical when viewed from the front of the face.

[0148] The facial expression code generation device 100M5 shown in FIG. 11E has a configuration in which the two electrodes 120-1 and 120-2 of the facial expression code generation device 100 (see FIGS. 1A and 1B) have been reduced to one, and the remaining two electrodes have been assigned to electrode 120-1 of Ch1. The two electrodes assigned to electrode 120-1 are located symmetrically on the left and right halves of the central portion of the face in the left-right direction when viewed from the front. The facial expression code generation device 100M5 also has a configuration in which the two electrodes 120-5 and 120-6 of the facial expression code generation device 100 (see FIGS. 1A and 1B) have been reduced to one, and the remaining two electrodes have been assigned to electrode 120-5 of Ch5. The two electrodes assigned to electrode 120-5 are located symmetrically on the left and right halves of the central portion of the face in the left-right direction when viewed from the front. The facial expression code generation device 100M5 includes electrodes 120-1, 120-3 to 120-5, and 120-7 to 120-8, but does not include electrodes 120-2 and 120-6, and can generate codes for Ch1 to Ch8. Ch3 to Ch4 and Ch7 to Ch8 are the same as those of the facial expression code generation device 100.

[0149] In the facial expression code generation device 100M5, a differential signal for Ch1 is obtained from the myoelectric potential of the left and right corrugator supercilii muscles using electrode 120-1, and a differential signal for Ch5 is obtained from the myoelectric potential of the left and right levator labii superioris alaris naris muscles using electrode 120-5. In the facial expression code generation device 100M5, when generating a code for Ch1, a code for Ch2 may also be generated, and when generating a code for Ch5, a code for Ch6 may also be generated. The code generated for Ch2 is a code representing a displacement that is symmetrical to the displacement by the code generated for Ch1 when viewed from the front of the face, and the code generated for Ch6 is a code representing a displacement that is symmetrical to the displacement by the code generated for Ch5 when viewed from the front of the face. The two codes for Ch1 and Ch2 are symmetrical codes representing displacements that are symmetrical to the displacement by the code generated for Ch5 when viewed from the front of the face. The two codes for Ch5 and Ch6 are symmetrical codes representing displacements that are symmetrical to the displacement by the code generated for Ch6 when viewed from the front of the face.

[0150] Here, the embodiment in which two codes (symmetric codes) for Ch1 and Ch2 are generated has been described. However, the two electrodes assigned to electrode 120-1 in FIG. 11E may be used to control the movement of the left and right corrugator supercilii with a single code. The region where the electrode on the −X direction side of the two electrodes assigned to electrode 120-1 is located (the region corresponding to the left corrugator supercilii) and the region where the electrode on the +X direction side is located (the region corresponding to the right corrugator supercilii) may be treated as a single region and assigned to a single channel (e.g., Ch1), and the movement of the left and right corrugator supercilii may be controlled with a single code. In other words, the region where the electrode on the −X direction side is located (the region corresponding to the left corrugator supercilii) and the region where the electrode on the +X direction side is located (the region corresponding to the right corrugator supercilii) may be treated as a single region, and a single code representing the overall displacement may be generated. The same applies to the two codes (symmetric codes) for Ch5 and Ch6.

[0151] <Wearing state of facial expression code generation device 100 of embodiment and wearing states of facial expression code generation devices 100M6, 100M7, and 100M8 of modified embodiments> Fig. 12A is a diagram showing an example of a wearing state of the facial expression code generation device 100 of the embodiment. Figs. 12B, 12C, and 12D are diagrams showing an example of a wearing state of the facial expression code generation devices 100M6, 100M7, and 100M8 of modified embodiments.

[0152] The facial expression code generation device 100M6 shown in Fig. 12B has a configuration obtained by modifying the goggle-type facial expression code generation device 100 shown in Fig. 12A into a glasses-type configuration. The facial expression code generation device 100M6 has a glasses-type main body 111B, temples 112B, and lenses 113B. The electrodes 120-1 to 120-8, the noise elimination electrode 125, and the signal processing unit 130 are the same as those of the facial expression code generation device 100. As an example, the electrodes 120-1, 120-3, 120-5, and 120-7 and the noise elimination electrode 125 may be attached to the main body 111B, and the signal processing unit 130 may be attached to the temples 112B.

[0153] 12C includes a band-shaped main body 111C extending laterally above and below both eyes, and an ear-hook-type band 112C. The electrodes 120-1 to 120-8, the noise elimination electrode 125, and the signal processing unit 130 are the same as those of the facial expression code generation device 100. As an example, the electrodes 120-1 to 120-8 and the noise elimination electrode 125 may be attached to the main body 111C, and the signal processing unit 130 may be attached to the band 112C.

[0154] The facial expression code generation device 100M8 shown in FIG. 12D is an eyepatch type device that covers one eye and includes a main body 111D for one eye (for example, the left eye) and a headband 112D. For example, the facial expression code generation device 100M8 may be an eyepatch type device that includes only one electrode 120, such as the facial expression code generation device 100M1 shown in FIG. 11A. The electrodes 120-1 to 120-8, the noise elimination electrode 125, and the signal processing unit 130 are the same as those in the facial expression code generation device 100. For example, the electrodes 120-1, 120-3, 120-5, and 120-7 and the noise elimination electrode 125 may be attached to the main body 111D, and the signal processing unit 130 may be attached to the headband 112D.

[0155] The facial expression code generation device 100 is not limited to the goggle-type shown in FIG. 12A, but may have an external appearance similar to that of facial expression code generation devices 100M6 to 100M8 shown in FIGS. 12B to 12D, or may have another external appearance not shown here.

[0156] <Effects> The facial expression code generation device 100 of the present disclosure includes a plurality of electrodes 120 that can contact a plurality of facial regions, respectively, to acquire myoelectric potentials, and a code generation unit (firmware 132C) that generates a code representing the displacement of at least one of the plurality of facial regions based on the plurality of myoelectric potentials acquired by the plurality of electrodes 120. In this way, a code representing the displacement of at least one of the plurality of facial regions is generated based on the plurality of myoelectric potentials, rather than data representing facial expressions. Such a code has a small data volume and imposes a low burden on data processing.

[0157] Therefore, it is possible to provide the facial expression code generating device 100 with a reduced data processing load.

[0158] There are also devices that acquire facial expression data from images captured by a camera, but such camera-based devices require the user's face to be constantly facing the camera in order to acquire facial expressions, making it difficult for the user to move freely. In contrast, the facial expression code generation device 100 acquires facial expressions simply by wearing it on the user's face. This allows the user to broadcast outdoors using a VTuber application, or to use it while engaging in various activities such as cooking and exercising, allowing the user to move freely. In this way, the facial expression code generation device 100 is highly convenient.

[0159] Furthermore, VTubers can change the facial expression of their avatar or character by changing their own actual facial expression. There is no need to operate a keyboard or the like to change the facial expression of their avatar or character; they can use their own face as an operating unit for changing the facial expression of their avatar or character. Furthermore, because there is no need to operate a keyboard or the like, operations to change the facial expression of their avatar or character can be easily performed even outdoors or in a dark place, making it easy to change the facial expression of their avatar when streaming from outdoors or in a dark place.

[0160] Furthermore, in online competitive games, players will be able to exchange facial expressions (equivalent to emoticons) without using a controller, making it easy to communicate with teammates and visualize emotions when defeating or being defeated by an opponent. This is expected to make online competitive games even more exciting.

[0161] Furthermore, devices that acquire facial expression data using an IR camera system tend to be thick and large when including the optical system. Furthermore, devices currently available on the market are expensive, integrated VR headsets, and are unsuitable for communication without entering the metaverse (online meetings, etc.). In contrast, the facial expression code generation device 100 can be realized with a simple configuration in which electrodes 120 are placed in contact with the face, and is suitable for both applications in the metaverse and applications for communication without entering the metaverse. The facial expression code generation device 100 reduces the burden on users and inconvenience, and can be used for both applications in the metaverse and applications for communication without entering the metaverse.

[0162] Furthermore, when using an avatar or character in remote communication in an online conference, the facial expression code generation device 100 can convey realistic emotions to the other party.

[0163] The device may further include a wearing device (goggles 110) that holds multiple electrodes 120 and can be worn on the face with the multiple electrodes 120 in contact with multiple parts of the face. By wearing the goggles 110 on the face, the multiple electrodes 120 come into contact with the face, freeing up both hands. For example, the facial expression of an avatar or character can be changed while performing various activities such as cooking or exercising.

[0164] The wearing equipment (goggles 110) may have openings located in front of both eyes when worn on the face. Since the front field of view is secured even when wearing the wearing equipment (goggles 110), it is possible to change the facial expression of an avatar or character while performing various activities.

[0165] The code generation unit (firmware 132C) may have multiple thresholds corresponding to multiple parts, and generate multiple codes by comparing multiple myoelectric potentials acquired by multiple electrodes 120 with the multiple thresholds. By using multiple thresholds corresponding to multiple parts, it is possible to detect changes in facial muscles in various parts of the face, and to reproduce a wider range of facial expressions.

[0166] The code generation unit (firmware 132C) may have a plurality of threshold levels, and may generate a plurality of types of codes corresponding to the plurality of levels by comparing a plurality of myoelectric potentials acquired by a plurality of sensors with the plurality of threshold levels. By using the plurality of threshold levels, it becomes possible to detect changes in the facial muscles in various parts of the face more precisely, and to reproduce a greater variety of facial expressions.

[0167] Furthermore, each of the plurality of electrodes 120 may have two electrode pieces that output a differential signal representing the acquired myoelectric potential, and the code generation unit (firmware 132C) may generate a code based on a plurality of differential signals representing a plurality of myoelectric potentials acquired by the two electrode pieces of the plurality of electrodes 120. Using a differential signal improves resistance to noise, makes it easier to detect changes in facial muscles, and enables stable reproduction of facial expressions.

[0168] The device may further include a noise elimination electrode 125 that contacts a predetermined part of the face with the plurality of electrodes 120 in contact with a plurality of parts, respectively, and a cancellation signal output unit (common-mode noise calculation unit 135) that outputs a cancellation signal that cancels out noise included in at least one of the plurality of differential signals based on at least one of the differential signals to the noise elimination electrode 125. Since common-mode noise that cannot be canceled out by a differential signal can be reduced, changes in facial muscles can be more easily detected and facial expressions can be reproduced more stably.

[0169] The multiple regions may also be regions where multiple muscles are located, including the left or right corrugator supercilii, the left or right frontalis, the left or right levator labii superioris alae naris, and the left or right zygomaticus major. Detecting changes in any of these facial muscles makes it easier to capture changes in facial expression, enabling a wider range of facial expressions to be reproduced.

[0170] The signal processing unit 130 may further include a code output unit that outputs the code generated by the code generation unit (firmware 132C). An example of the code output unit is the BLE module 133. When the signal processing unit 130 is connected to an electronic device such as the smartphone 200 via wired communication using a cable or the like, an output terminal that outputs a code from the CPU 132 is an example of the code output unit. The inclusion of the code output unit makes it easier to transfer the code to an external device such as the electronic device such as the smartphone 200.

[0171] The facial expression generation devices 500A and 500B of the present disclosure include the facial expression code generation device 100, a physical modification unit capable of physically modifying the facial expression of the mask 400B, mask, or costume 400A, a storage unit that stores a plurality of facial expression files each representing the movement of a plurality of facial parts, an extraction unit that extracts, from the plurality of facial expression files, a facial expression file corresponding to a code generated by a code generation unit (firmware 132C), and a facial expression modification unit that modifies the facial expression of the mask 400B, mask, or costume 400A by modifying the state of the physical modification unit based on the facial expression file extracted by the extraction unit. The actuators of the costume 400A or mask, or the LEDs of the mask 400B are examples of the physical modification unit. The control unit that controls the light color of the actuators of the costume 400A or mask, or the LEDs of the mask 400B, are examples of the storage unit, extraction unit, and facial expression modification unit.

[0172] In this way, a code representing the displacement of at least one of the multiple facial parts is generated based on multiple myoelectric potentials, rather than data representing facial expressions. Such a code has a small data volume and imposes a low data processing burden. Therefore, it is possible to provide facial expression generating devices 500A and 500B with a reduced data processing burden.

[0173] Furthermore, the facial expression generating devices 500A, 500B can physically change the facial expression of the mask 400B, mask, or costume 400A using the code generated by the facial expression code generating device 100. For example, when a fan approaches the costume and the person inside wants to react, they can change the expression of the costume 400A by changing their own facial expression, so a wide range of reactions other than gestures can be easily achieved by changing the facial expression. This is also true for the mask 400B and the mask.

[0174] The facial expression generating device 500C of the present disclosure includes the facial expression code generating device 100, a display device (display 430) capable of displaying a predetermined avatar or character, a storage unit that stores a plurality of facial expression files each representing the movement of a plurality of facial parts, an extraction unit that extracts, from the plurality of facial expression files, a facial expression file corresponding to a code generated by a code generating unit (firmware 132C), and an expression modification unit that modifies the facial expression of the predetermined avatar or character displayed on the display device based on the facial expression file extracted by the extraction unit. The control device 210 of the XR headset 400C is an example of the storage unit, the extraction unit, and the expression modification unit.

[0175] In this way, a code representing the displacement of at least one of the multiple facial parts is generated based on multiple myoelectric potentials, rather than data representing facial expressions. Such a code has a small data volume and imposes a low data processing burden. Therefore, it is possible to provide a facial expression generating device 500C with a reduced data processing burden.

[0176] The facial expression generating device 500C is an integrated device of the facial expression code generating device 100 and the XR headset 400C, so that it is possible to provide a headset-type facial expression generating device 500C that reduces the load of data processing.

[0177] The facial expression generation system 300 of the present disclosure includes the facial expression code generation device 100, a storage unit that stores a plurality of facial expression files each representing the movement of a plurality of facial parts, an extraction unit that extracts, from the plurality of facial expression files, a facial expression file corresponding to a code generated by a code generation unit (firmware 132C), and an expression modification unit that modifies the facial expression of a predetermined avatar or character displayed on a display device based on the facial expression file extracted by the extraction unit. As shown in FIG. 3 , when a smartphone 200 is used, the storage unit is the memory of the smartphone 200. An example of the extraction unit and the expression modification unit is an application 210A.

[0178] In this way, a code representing the displacement of at least one of the multiple facial parts is generated based on multiple myoelectric potentials, rather than data representing facial expressions. Such a code has a small data volume and imposes a low data processing burden. Therefore, it is possible to provide a facial expression generation system 300 with a reduced data processing burden.

[0179] Furthermore, VTubers can change the facial expression of their avatar or character by changing their own actual facial expression in the facial expression generation system 300. There is no need to operate a keyboard or the like to change the facial expression of the avatar or character, and they can use their own face as an operation unit for changing the facial expression of the avatar or character, etc. Furthermore, because there is no need to operate a keyboard or the like, operations for changing the facial expression of the avatar or character, etc. can be easily performed even outdoors or in a dark place, so the facial expression of the avatar can be easily changed when streaming from outdoors or in a dark place.

[0180] The facial expression code generation method of the present disclosure uses an facial expression code generation device including a plurality of electrodes 120 capable of acquiring myoelectric potentials by contacting the plurality of facial regions, respectively, and generates a code representing the displacement of at least one of the plurality of facial regions based on the plurality of myoelectric potentials acquired by the plurality of electrodes 120. In this way, a code representing the displacement of at least one of the plurality of facial regions is generated based on the plurality of myoelectric potentials, rather than data representing facial expressions. Such a code has a small data volume and imposes a low burden on data processing.

[0181] Therefore, it is possible to provide a facial expression code generation method that reduces the load of data processing.

[0182] <Effects> The facial expression code generation device 100 of the present disclosure includes a plurality of electrodes 120 capable of contacting a plurality of facial regions to acquire myoelectric potentials, a wearable device (goggles 110) that holds the plurality of electrodes 120 and can be worn on the face with the plurality of electrodes 120 in contact with the plurality of facial regions, and a code generation unit (firmware 132C) that generates a code representing the displacement of at least one of the plurality of facial regions based on the plurality of myoelectric potentials acquired by the plurality of electrodes 120. In this way, a key code representing the displacement of at least one of the plurality of facial regions is generated based on the plurality of myoelectric potentials, rather than data representing the facial expression of the entire face. The displacement of at least one of the plurality of facial regions corresponds to a change in the facial muscles that form the facial expression. Such a key code has a small data capacity and imposes a low data processing burden.

[0183] Therefore, it is possible to provide the facial expression code generating device 100 with a reduced data processing load.

[0184] There are also devices that acquire facial expression data from images captured by a camera, but such camera-based devices require the user's face to be constantly facing the camera in order to acquire facial expressions, making it difficult for the user to move away from the camera and allowing for free movement. In contrast, the facial expression code generation device 100 acquires facial expressions simply by wearing it on the user's face. This allows the user to broadcast outdoors using a VTuber application, or to use it while engaging in various activities such as cooking and exercising, allowing the user to move freely. In this way, the facial expression code generation device 100 is highly convenient.

[0185] Furthermore, VTubers can change the facial expression of their avatar or character by changing their own actual facial expression. There is no need to operate a keyboard or the like to change the facial expression of their avatar or character; they can use their own face as an operating unit for changing the facial expression of their avatar or character. Furthermore, because there is no need to operate a keyboard or the like, operations to change the facial expression of their avatar or character can be easily performed even outdoors or in a dark place, making it easy to change the facial expression of their avatar when streaming from outdoors or in a dark place.

[0186] Furthermore, in online competitive games, players will be able to exchange facial expressions (equivalent to emoticons) without using a controller, making it easy to communicate with teammates and visualize emotions when defeating or being defeated by an opponent. This is expected to make online competitive games even more exciting.

[0187] Furthermore, devices that acquire facial expression data using an IR camera system tend to be thick and large when including the optical system. Furthermore, devices currently available on the market are expensive, integrated VR headsets, and are unsuitable for communication without entering the metaverse (online meetings, etc.). In contrast, the facial expression code generation device 100 can be realized with a simple configuration in which electrodes 120 are placed in contact with the face, and is suitable for both applications in the metaverse and applications for communication without entering the metaverse. The facial expression code generation device 100 reduces the burden on users and inconvenience, and can be used for both applications in the metaverse and applications for communication without entering the metaverse.

[0188] Furthermore, when using an avatar or character in remote communication in an online conference, the facial expression code generation device 100 can convey realistic emotions to the other party.

[0189] The electrodes 120 may be arranged at positions where the myoelectric potential of the facial muscles around the left and right eyes of the face can be measured. By measuring the myoelectric potential of the facial muscles around the left and right eyes of the face, a key code representing the displacement of the part of the face due to a change in facial expression can be generated.

[0190] Furthermore, the multiple electrodes 120 are arranged at positions where the myoelectric potential of the facial muscles in one of the left and right halves of the face can be measured, and the code generation unit (firmware 132C) may generate a code for one of the left and right halves of the face based on the multiple myoelectric potentials acquired by the multiple electrodes 120, and may also generate a symmetric code for the other of the left and right halves of the face, representing a displacement that is symmetrical to the displacement caused by the code when viewed from the front of the face. For example, if an avatar uses facial expressions in which both sides of the face are displaced equally, the number of electrodes 120 can be reduced by half. Reducing the number of electrodes 120 allows for a simpler configuration.

[0191] Furthermore, the multiple electrodes 120 are arranged in one of the left and right halves of the face at the center in the horizontal direction of the face, and when the code generation unit (firmware 132C) generates a code for one of the left and right halves of the face at the center in the horizontal direction, it may generate a symmetric code for the other half of the face, representing a displacement that is symmetrical to the displacement caused by the code when viewed from the front of the face. For example, if an avatar uses facial expressions in which both sides of the center in the horizontal direction of the face are displaced equally, the number of electrodes 120 in the center in the horizontal direction of the face can be reduced by half. Reducing the number of electrodes 120 allows for a simpler configuration.

[0192] Furthermore, of the multiple electrodes 120, the electrodes 120 on the left and right ends, excluding the central portion in the horizontal direction, are arranged on both the left and right ends. When generating a code on either the left or right end, the code generation unit (firmware 132C) may generate a code representing the displacement of a region located on either the left or right end based on the myoelectric potential acquired by the electrodes 120 arranged on either the left or right end. For example, in a case where an avatar uses facial expressions in which both sides of the horizontal center of the face are displaced equally, separate codes can be generated on the left and right ends of the horizontal center according to the displacement of the facial muscles on the left end and the displacement of one of the facial muscles on the right end. This allows the number of electrodes 120 in the horizontal center of the face to be reduced by half, thereby achieving a simple configuration.

[0193] The code generation unit (firmware 132C) may have multiple thresholds corresponding to multiple body parts, respectively, and generate codes by comparing multiple myoelectric potentials acquired by multiple electrodes 120 with the multiple thresholds. By using multiple thresholds corresponding to multiple body parts, it is possible to detect changes in facial muscles in various parts of the face, and to reproduce a wider range of facial expressions.

[0194] The code generation unit (firmware 132C) may have multiple threshold levels and generate multiple types of codes corresponding to the multiple levels by comparing multiple myoelectric potentials acquired by the multiple electrodes 120 with the multiple threshold levels. By using multiple threshold levels, changes in facial muscles in various parts of the face can be detected more precisely, making it possible to reproduce a wider range of facial expressions.

[0195] Furthermore, each of the plurality of electrodes 120 may have two electrode pieces that output a differential signal representing the acquired myoelectric potential, and the code generation unit (firmware 132C) may generate a code based on the plurality of differential signals representing the plurality of myoelectric potentials acquired by the plurality of electrodes 120. Using differential signals improves noise resistance, makes it easier to detect changes in facial muscles, and enables stable reproduction of facial expressions.

[0196] The device may further include a noise elimination electrode 125 that contacts a predetermined part of the face with the plurality of electrodes 120 in contact with a plurality of parts, respectively, and a cancellation signal output unit (common-mode noise calculation unit 135) that outputs a cancellation signal that cancels out noise included in at least one of the plurality of differential signals based on at least one of the differential signals to the noise elimination electrode 125. Since common-mode noise that cannot be canceled out by a differential signal can be reduced, changes in facial muscles can be more easily detected and facial expressions can be reproduced more stably.

[0197] Furthermore, among the multiple electrodes 120, the electrode 120 located in the horizontal center of the face has two electrode pieces that output a differential signal representing the acquired myoelectric potential, and the two electrode pieces are located symmetrically on the left and right halves of the horizontal center of the face when viewed from the front. When generating codes in the horizontal center, the code generation unit (firmware 132C) generates two codes representing displacements of the two areas where the two electrode pieces are located, based on the differential signals output from the two electrode pieces. The two codes may be symmetric codes representing displacements that are symmetrical in a front view of the face. For example, if an avatar uses facial expressions in which both sides of the horizontal center of the face are displaced equally, the number of electrodes 120 in the horizontal center of the face can be reduced by half. Reducing the number of electrodes 120 allows for a simpler configuration.

[0198] Furthermore, among the multiple electrodes 120, the electrode 120 located in the horizontal center of the face has two electrode pieces that output a differential signal representing the acquired myoelectric potential, and the two electrode pieces are located symmetrically on the left and right halves of the horizontal center of the face when viewed from the front. When generating a code in the horizontal center, the code generation unit (firmware 132C) may generate a code representing the overall displacement of the two entire regions where the two electrode pieces are located based on the differential signals output from the two electrode pieces. For example, if an avatar uses facial expressions in which both sides of the horizontal center of the face are displaced equally, the two entire regions where the two electrode pieces are located can be treated as a single region, thereby reducing the number of electrodes 120 in the horizontal center of the face by half. Reducing the number of electrodes 120 allows for a simpler configuration.

[0199] Furthermore, the central portion in the left-right direction may be the portion of the face where the left and right corrugator supercilii muscles or the left and right levator labio nasalis muscles are located in the left-right direction. The left and right corrugator supercilii muscles and levator labio nasalis muscles located in the left-right direction of the face are less likely to move on one side and have smaller movements than the left and right frontalis muscles and the left and right zygomaticus major muscles located on the left and right sides of the face due to the closer left-right distance between them. Therefore, it is possible to generate facial expressions in which both sides are displaced equally in the left-right central portion of the face, and a simple configuration can be realized by reducing the number of electrodes 120 in the left-right central portion of the face by half.

[0200] <Effects> The facial expression code generation device 100 of the present disclosure includes a plurality of electrodes 120 capable of contacting a plurality of facial regions, respectively, to acquire myoelectric potentials, and a code generation unit (firmware 132C) that generates a key code representing the displacement of at least one of the plurality of facial regions based on the plurality of myoelectric potentials acquired by the plurality of electrodes 120. In this way, a key code representing the displacement of at least one of the plurality of facial regions is generated based on the plurality of myoelectric potentials, rather than data representing the facial expression of the entire face. The displacement of at least one of the plurality of facial regions corresponds to a change in the facial muscles that form the facial expression. Such a key code has a small data capacity and imposes a low burden on data processing.

[0201] Therefore, it is possible to provide the facial expression code generating device 100 with a reduced data processing load.

[0202] There are also devices that acquire facial expression data from images captured by a camera, but such camera-based devices require the user's face to be constantly facing the camera in order to acquire facial expressions, making it difficult for the user to move away from the camera and allowing for free movement. In contrast, the facial expression code generation device 100 acquires facial expressions simply by wearing it on the user's face. This allows the user to broadcast outdoors using a VTuber application, or to use it while engaging in various activities such as cooking and exercising, allowing the user to move freely. In this way, the facial expression code generation device 100 is highly convenient.

[0203] Furthermore, VTubers can change the facial expression of their avatar or character by changing their own actual facial expression. There is no need to operate a keyboard or the like to change the facial expression of their avatar or character; they can use their own face as an operating unit for changing the facial expression of their avatar or character. Furthermore, because there is no need to operate a keyboard or the like, operations to change the facial expression of their avatar or character can be easily performed even outdoors or in a dark place, making it easy to change the facial expression of their avatar when streaming from outdoors or in a dark place.

[0204] Furthermore, in online competitive games, players will be able to exchange facial expressions (equivalent to emoticons) without using a controller, making it easy to communicate with teammates and visualize emotions when defeating or being defeated by an opponent. This is expected to make online competitive games even more exciting.

[0205] Furthermore, devices that acquire facial expression data using an IR camera system tend to be thick and large when including the optical system. Furthermore, devices currently available on the market are expensive, integrated VR headsets, and are unsuitable for communication without entering the metaverse (online meetings, etc.). In contrast, the facial expression code generation device 100 can be realized with a simple configuration in which electrodes 120 are placed in contact with the face, and is suitable for both applications in the metaverse and applications for communication without entering the metaverse. The facial expression code generation device 100 reduces the burden on users and inconvenience, and can be used for both applications in the metaverse and applications for communication without entering the metaverse.

[0206] Furthermore, when using an avatar or character in remote communication in an online conference, the facial expression code generation device 100 can convey realistic emotions to the other party.

[0207] The device may further include a wearing device (goggles 110) that holds multiple electrodes 120 and can be worn on the face with the multiple electrodes 120 in contact with multiple parts of the face. By wearing the goggles 110 on the face, the multiple electrodes 120 come into contact with the face, freeing up both hands. For example, the facial expression of an avatar or character can be changed while performing various activities such as cooking or exercising.

[0208] The wearing equipment (goggles 110) may have openings located in front of both eyes when worn on the face. Since the front field of view is secured even when wearing the wearing equipment (goggles 110), it is possible to change the facial expression of an avatar or character while performing various activities.

[0209] The code generation unit (firmware 132C) may have a plurality of thresholds corresponding to a plurality of parts, respectively, and generate a plurality of key codes by comparing a plurality of myoelectric potentials acquired by a plurality of electrodes 120 with the plurality of thresholds. By using a plurality of thresholds corresponding to a plurality of parts, it is possible to detect changes in facial muscles in various parts of the face, and to reproduce a wider range of facial expressions.

[0210] The code generation unit (firmware 132C) may have a plurality of threshold levels, and may generate a plurality of types of key codes corresponding to the plurality of levels by comparing a plurality of myoelectric potentials acquired by a plurality of sensors with the plurality of threshold levels. By using the plurality of threshold levels, it becomes possible to detect changes in facial muscles in various parts of the face more precisely, and to reproduce a greater variety of facial expressions.

[0211] Furthermore, each of the plurality of electrodes 120 may have two electrode pieces that output a differential signal representing the acquired myoelectric potential, and the code generation unit (firmware 132C) may generate a key code based on a plurality of differential signals representing a plurality of myoelectric potentials acquired by the two electrode pieces of the plurality of electrodes 120. Using differential signals improves resistance to noise, makes it easier to detect changes in facial muscles, and enables stable reproduction of facial expressions.

[0212] The device may further include a noise elimination electrode 125 that contacts a predetermined part of the face with the plurality of electrodes 120 in contact with a plurality of parts, respectively, and a cancellation signal output unit (common-mode noise calculation unit 135) that outputs a cancellation signal that cancels out noise included in at least one of the plurality of differential signals based on at least one of the differential signals to the noise elimination electrode 125. Since common-mode noise that cannot be canceled out by a differential signal can be reduced, changes in facial muscles can be more easily detected and facial expressions can be reproduced more stably.

[0213] Furthermore, the key codes may be key codes other than those assigned to the physical keys of a Japanese JIS keyboard or a QWERTY keyboard. It is possible to provide an expression code generation device 100 that can generate dedicated key codes based on the displacement of the myoelectric potential of facial muscles and can stably change the expressions of avatars or characters displayed on a display device, or masks or costumes whose facial expressions can be physically changed, without generating unused commands.

[0214] The key code may be a key code assigned to a physical key of a Japanese JIS keyboard or a QWERTY keyboard. A command based on the key code assigned to the physical key operates the smartphone 200 or a PC, and the facial expression of an avatar or character can be changed. For example, the facial expression of an avatar or character can be changed while chatting with characters.

[0215] The multiple regions may also be regions where multiple muscles are located, including the left or right corrugator supercilii, the left or right frontalis, the left or right levator labii superioris alae naris, and the left or right zygomaticus major. Detecting changes in any of these facial muscles makes it easier to capture changes in facial expression, enabling a wider range of facial expressions to be reproduced.

[0216] The signal processing unit 130 may further include a code output unit that outputs a key code generated by the code generation unit (firmware 132C). An example of the code output unit is the BLE module 133. When the signal processing unit 130 is connected to an electronic device such as the smartphone 200 via wired communication using a cable or the like, an output terminal that outputs a key code from the CPU 132 is an example of the code output unit. The inclusion of the code output unit makes it easier to transfer the key code to an external device such as the electronic device such as the smartphone 200.

[0217] The facial expression generation devices 500A and 500B of the present disclosure include the facial expression code generation device 100, a physical modification unit capable of physically modifying the facial expression of the mask 400B, mask, or costume 400A, a storage unit that stores a plurality of facial expression files representing the movements of the plurality of facial parts, an extraction unit that extracts, from the plurality of facial expression files, an expression file corresponding to a key code generated by a code generation unit (firmware 132C), and an expression modification unit that modifies the facial expression of the mask 400B, mask, or costume 400A by modifying the state of the physical modification unit based on the expression file extracted by the extraction unit. The actuators of the costume 400A or mask, or the LEDs of the mask 400B are examples of the physical modification unit. The control unit that controls the light color of the actuators of the costume 400A or mask, or the LEDs of the mask 400B are examples of the storage unit, extraction unit, and expression modification unit.

[0218] In this way, a key code representing the displacement of at least one of the facial parts is generated based on multiple myoelectric potentials, rather than data representing the facial expression of the entire face. Such a key code has a small data volume and imposes a low data processing burden. Therefore, it is possible to provide facial expression generating devices 500A and 500B with a reduced data processing burden.

[0219] Furthermore, the facial expression generating devices 500A, 500B can physically change the facial expression of the mask 400B, the mask, or the costume 400A using the key code generated by the facial expression code generating device 100. For example, when a fan approaches the costume and the person inside wants to react, the person inside can change the expression of the costume 400A by changing their own facial expression, so a wide range of reactions other than gestures can be easily achieved by changing the facial expression. This is also true for the mask 400B and the mask.

[0220] The facial expression generating device 500C of the present disclosure includes the facial expression code generating device 100, a display device (display 430) capable of displaying a predetermined avatar or character, a storage unit that stores a plurality of facial expression files each representing the movement of a plurality of facial parts, an extraction unit that extracts, from the plurality of facial expression files, an expression file corresponding to a key code generated by a code generating unit (firmware 132C), and an expression modification unit that modifies the facial expression of the predetermined avatar or character displayed on the display device based on the expression file extracted by the extraction unit. The control device 210 of the XR headset 400C is an example of the storage unit, the extraction unit, and the expression modification unit.

[0221] In this way, a key code representing the displacement of at least one of the facial parts is generated based on multiple myoelectric potentials, rather than data representing the facial expression of the entire face. Such a key code has a small data volume and imposes a low data processing burden. Therefore, it is possible to provide a facial expression generating device 500C with a reduced data processing burden.

[0222] The facial expression generating device 500C is an integrated device of the facial expression code generating device 100 and the XR headset 400C, so that it is possible to provide a headset-type facial expression generating device 500C that reduces the load of data processing.

[0223] The facial expression generation system 300 of the present disclosure includes the facial expression code generation device 100, a storage unit that stores a plurality of facial expression files each representing the movement of a plurality of facial parts, an extraction unit that extracts, from the plurality of facial expression files, a facial expression file corresponding to a code generated by a code generation unit (firmware 132C), and an expression modification unit that modifies the facial expression of a predetermined avatar or character displayed on a display device based on the facial expression file extracted by the extraction unit. As shown in FIG. 3 , when a smartphone 200 is used, the storage unit is the memory of the smartphone 200. An example of the extraction unit and the expression modification unit is an application 210A.

[0224] In this way, a key code representing the displacement of at least one of the facial parts is generated based on multiple myoelectric potentials, rather than data representing the facial expression of the entire face. Such a key code has a small data volume and imposes a low data processing burden. Therefore, it is possible to provide a facial expression generation system 300 with a reduced data processing burden.

[0225] Furthermore, VTubers can change the facial expression of their avatar or character by changing their own actual facial expression in the facial expression generation system 300. There is no need to operate a keyboard or the like to change the facial expression of the avatar or character, and they can use their own face as an operation unit for changing the facial expression of the avatar or character, etc. Furthermore, because there is no need to operate a keyboard or the like, operations for changing the facial expression of the avatar or character, etc. can be easily performed even outdoors or in a dark place, so the facial expression of the avatar can be easily changed when streaming from outdoors or in a dark place.

[0226] The facial expression code generation method of the present disclosure uses an facial expression code generation device including a plurality of electrodes 120 capable of acquiring myoelectric potentials by contacting the plurality of facial regions, respectively, and generates a key code representing the displacement of at least one of the plurality of facial regions based on the plurality of myoelectric potentials acquired by the plurality of electrodes 120. In this way, a key code representing the displacement of at least one of the plurality of facial regions is generated based on the plurality of myoelectric potentials, rather than data representing the facial expression of the entire face. Such a key code has a small data capacity and imposes a low burden on data processing.

[0227] Therefore, it is possible to provide a facial expression code generation method that reduces the load of data processing.

[0228] The above describes exemplary embodiments of the facial expression code generation device, facial expression generation system, and facial expression code generation method of the present disclosure. However, the present disclosure is not limited to the specifically disclosed embodiments, and various modifications and changes are possible without departing from the scope of the claims.

[0229] This international application claims priority based on Japanese Patent Application Nos. 2024-149938, 2024-149939, and 2024-149940, all filed on August 30, 2024. The entire contents of which are incorporated herein by reference.

[0230] The following supplementary notes are further disclosed in relation to the above embodiments. (Supplementary Note 1A) A facial expression code generation device including: a plurality of electrodes capable of contacting a plurality of regions of the face, respectively, to acquire myoelectric potentials; a wearing device that holds the plurality of electrodes and can be worn on the face with the plurality of electrodes in contact with the plurality of regions, respectively; and a code generation unit that generates a code representing a displacement of at least one region of the plurality of regions, based on the plurality of myoelectric potentials acquired by the plurality of electrodes. (Supplementary Note 2A) The facial expression code generation device according to Supplementary Note 1A, wherein the plurality of electrodes are arranged at positions that allow measurement of the myoelectric potentials of facial muscles around the left and right eyes. (Supplementary Note 3A) The facial expression code generation device according to Supplementary Note 1A or Supplementary Note 2A, wherein the plurality of electrodes are arranged at positions where myoelectric potentials of facial muscles in one of the left and right halves of the face can be measured, and the code generation unit generates the code for the one of the left and right halves of the face based on the plurality of myoelectric potentials acquired by the plurality of electrodes, and generates a symmetric code for the other of the left and right halves of the face that represents a displacement that is symmetrical to the displacement caused by the code when viewed from the front of the face. (Supplementary Note 4A) The facial expression code generation device according to Supplementary Note 1A or Supplementary Note 2A, wherein the plurality of electrodes are arranged at one of the left and right halves of the face at a center in the left-right direction of the face, and when generating the code for the one of the left and right halves of the face at the center in the left-right direction, the code generation unit generates a symmetric code for the other of the left and right halves of the face that represents a displacement that is symmetrical to the displacement caused by the code when viewed from the front of the face. (Appendix 5A) The facial expression code generation device described in Appendix 4A, wherein, of the plurality of electrodes, the electrodes on the left and right ends excluding the center portion in the left-right direction are arranged on both the left and right ends, and when the code generation unit generates the code on one of the left and right ends, the code generation unit generates a code representing a displacement of the part located on one of the left and right ends based on the myoelectric potential acquired by the electrodes arranged on one of the left and right ends.(Supplementary Note 6A) The facial expression code generation device according to any one of Supplementary Note 1A to Supplementary Note 5A, wherein the code generation unit has a plurality of thresholds corresponding to the plurality of body parts, and generates the code by comparing a plurality of myoelectric potentials acquired by the plurality of electrodes with the plurality of thresholds. (Supplementary Note 7A) The facial expression code generation device according to Supplementary Note 6A, wherein the code generation unit has a plurality of levels of the threshold and generates a plurality of types of the code according to the plurality of levels by comparing a plurality of myoelectric potentials acquired by the plurality of electrodes with the plurality of thresholds. (Supplementary Note 8A) The facial expression code generation device according to any one of Supplementary Note 1A to Supplementary Note 7A, wherein each of the plurality of electrodes has two electrode pieces that output differential signals representing the acquired myoelectric potentials, and the code generation unit generates the code based on a plurality of differential signals representing the plurality of myoelectric potentials acquired by the two electrode pieces of the plurality of electrodes, respectively. (Supplementary Note 9A) The facial expression code generation device according to Supplementary Note 8A, further comprising: a noise elimination electrode that contacts predetermined regions of the face with the plurality of electrodes in contact with the plurality of regions, respectively; and a cancellation signal output unit that outputs to the noise elimination electrode a cancellation signal that cancels out noise included in at least one of the plurality of differential signals based on at least one differential signal among the plurality of differential signals. (Supplementary Note 10A) The facial expression code generation device according to Supplementary Note 1A, wherein an electrode of the plurality of electrodes that is located in a central portion in the left-right direction of the face has two electrode pieces that output a differential signal that represents the acquired myoelectric potential, the two electrode pieces being located one on each side of the central portion in the left-right direction of the face in a front view, and the code generation unit, when generating the code at the central portion in the left-right direction, generates two codes that represent displacements of the two regions where the two electrode pieces are located, based on the differential signals output from the two electrode pieces, and the two codes are symmetrical codes that represent displacements that are symmetrical in the front view of the face.(Supplementary Note 11A) The facial expression code generation device according to Supplementary Note 1A, wherein an electrode of the plurality of electrodes arranged in a central portion in the left-right direction of the face has two electrode pieces that output a differential signal representing the acquired myoelectric potential, the two electrode pieces being arranged symmetrically on the left and right halves of the central portion in the left-right direction of the face in a front view, and when generating the code at the central portion in the left-right direction, the code generation unit generates a code representing an overall displacement of the two portions for the entire two portions where the two electrode pieces are arranged, based on the differential signals output from the two electrode pieces. (Supplementary Note 12A) The facial expression code generation device according to any one of Supplementary Note 4A, Supplementary Note 5A, Supplementary Note 10A, and Supplementary Note 11A, wherein the central portion in the left-right direction is a portion where the left and right corrugator supercilii muscles or the left and right levator labii superioris alae naris muscles are located in the left-right direction of the face.

[0231] (Supplementary Note 1B) An facial expression code generation device comprising: a plurality of electrodes capable of contacting a plurality of regions of the face, respectively, to acquire myoelectric potentials; and a code generation unit that generates a key code representing a displacement of at least one of the plurality of regions based on the plurality of myoelectric potentials acquired by the plurality of electrodes. (Supplementary Note 2B) The facial expression code generation device according to Supplementary Note 1B, further comprising a wearing device that holds the plurality of electrodes and can be worn on the face with the plurality of electrodes in contact with the plurality of regions, respectively. (Supplementary Note 3B) The facial expression code generation device according to Supplementary Note 2B, wherein the wearing device has openings located in front of both eyes when worn on the face. (Supplementary Note 4B) The facial expression code generation device according to Supplementary Note 1B, wherein the code generation unit has a plurality of thresholds corresponding to the plurality of regions, respectively, and generates the key code by comparing the plurality of myoelectric potentials acquired by the plurality of electrodes with the plurality of thresholds. (Supplementary Note 5B) The facial expression code generation device according to Supplementary Note 4B, wherein the code generation unit has a plurality of levels of the threshold value, and generates a plurality of types of the key code according to the plurality of levels by comparing a plurality of myoelectric potentials acquired by the plurality of electrodes with the plurality of threshold values. (Supplementary Note 6B) The facial expression code generation device according to any one of Supplementary Notes 1B to 5B, wherein each of the plurality of electrodes has two electrode pieces that output a differential signal representing the acquired myoelectric potentials, and the code generation unit generates the code based on a plurality of differential signals representing the plurality of myoelectric potentials acquired by the two electrode pieces of the plurality of electrodes. (Supplementary Note 7B) The facial expression code generation device according to Supplementary Note 6B, further including: a noise elimination electrode that contacts a predetermined portion of a face with the plurality of electrodes in contact with the plurality of portions, respectively; and a cancellation signal output unit that outputs to the noise elimination electrode a cancellation signal that cancels out noise included in at least one differential signal of the plurality of differential signals, based on at least one differential signal of the plurality of differential signals. (Appendix 8B) The facial expression code generation device according to any one of appendices 1B to 7B, wherein the key code is a key code other than a key code assigned to a physical key of a Japanese JIS keyboard or a QWERTY keyboard.(Supplementary Note 9B) The facial expression code generation device according to any one of Supplementary Notes 1B to 7B, wherein the key codes are key codes assigned to physical keys of a Japanese JIS keyboard or a QWERTY keyboard. (Supplementary Note 10B) The facial expression code generation device according to any one of Supplementary Notes 1B to 9B, wherein the plurality of regions are regions where any plurality of muscles selected from the group consisting of left or right corrugator supercilii, left or right frontalis, left or right levator labii superioris alae naris, and left or right zygomaticus major are located. (Supplementary Note 11B) The facial expression code generation device according to any one of Supplementary Notes 1B to 10B, further including a code output unit that outputs the key codes generated by the code generation unit. (Appendix 12B) An expression generation device including: an expression code generation device according to any one of Appendices 1B to 11B; a physical modification unit capable of physically modifying the facial expression of a mask, a mask, or a costume; a storage unit that stores a plurality of expression files each representing the movement of a plurality of parts of the face; an extraction unit that extracts, from the plurality of expression files, the expression file corresponding to the key code generated by the code generation unit; and an expression modification unit that modifies the facial expression of the mask, the mask, or the costume by modifying the mode of the physical modification unit based on the expression file extracted by the extraction unit. (Appendix 13B) A facial expression generation device comprising: a facial expression code generation device according to any one of Appendices 1B to 12B; a display device capable of displaying a predetermined avatar or character; a storage unit for storing a plurality of facial expression files each representing the movement of a plurality of parts of a face; an extraction unit for extracting, from the plurality of facial expression files, the facial expression file corresponding to the key code generated by the code generation unit; and an expression modification unit for modifying the facial expression of the predetermined avatar or character displayed on the display device based on the facial expression file extracted by the extraction unit.(Appendix 14B) An expression generation system comprising: the expression code generation device according to any one of Appendices 1B to 11B; a storage unit that stores a plurality of expression files that respectively represent movements of the plurality of parts of a face; an extraction unit that extracts, from the plurality of expression files, the expression file that corresponds to the key code generated by the code generation unit; and an expression modification unit that modifies the facial expression of a predetermined avatar or character displayed on a display device based on the expression file extracted by the extraction unit. (Appendix 15B) An expression code generation method in an expression code generation device including a plurality of electrodes that can be in contact with a plurality of parts of a face, respectively, to acquire myoelectric potentials, the method comprising: generating a key code that represents a displacement of at least one part of the plurality of parts based on the plurality of myoelectric potentials acquired by the plurality of electrodes.

[0232] 100, 100M1 to 100M8 Facial expression code generation device 110 Goggles (an example of a wearing device) 111 Main body 112 Headband 113 Shield 114 Cushion 120, 120-1 to 120-8 Electrode 125 Noise removal electrode 130 Signal processing unit 130A Battery 131 Sensor amplifier 132 CPU 132C Firmware (an example of a code generation unit) 133 BLE module (an example of a code output unit) 134 Antenna 135 In-phase noise calculation unit (an example of a cancellation signal output unit) 136 Amplifier 200 Smartphone 210 Control device 210A, 210B Application 220 BLE module 230 Display (an example of a display device) 300 Facial expression generation system 400A Costume 400B Mask 400C XR headset 430 Display (an example of a display device) 500A, 500B, 500C Facial expression generation device

Claims

1. A facial expression code generation device comprising: a plurality of electrodes capable of contacting a plurality of parts of the face, respectively, to acquire myoelectric potentials; and a code generation unit that generates a code representing the displacement of at least one part of the plurality of parts based on the myoelectric potentials acquired by the plurality of electrodes.

2. The facial expression code generating device according to claim 1, further comprising a wearing device that holds the plurality of electrodes and can be worn on the face with the plurality of electrodes in contact with the plurality of regions, respectively.

3. The facial expression code generating device according to claim 2, wherein the wearing device has openings positioned in front of both eyes when worn on the face.

4. The facial expression code generating device according to claim 1, wherein the code generating unit has a plurality of thresholds corresponding to the plurality of body parts, and generates the code by comparing a plurality of myoelectric potentials acquired by the plurality of electrodes with the plurality of thresholds.

5. The facial expression code generating device according to claim 4, wherein the code generating unit has a plurality of levels of thresholds, and generates a plurality of types of codes according to the plurality of levels by comparing a plurality of myoelectric potentials acquired by the plurality of electrodes with the plurality of thresholds.

6. A facial expression code generation device according to any one of claims 1 to 5, wherein each of the plurality of electrodes has two electrode pieces that output a differential signal representing the acquired myoelectric potential, and the code generation unit generates the code based on a plurality of differential signals representing the plurality of myoelectric potentials acquired by the two electrode pieces of the plurality of electrodes, respectively.

7. The facial expression code generating device according to claim 6, further comprising: a noise elimination electrode that contacts predetermined parts of the face with the plurality of electrodes in contact with the plurality of parts, respectively; and a cancellation signal output unit that outputs a cancellation signal to the noise elimination electrode based on at least one differential signal among the plurality of differential signals, to cancel out noise contained in the at least one differential signal.

8. A facial expression code generation device according to any one of claims 1 to 7, wherein the plurality of regions are regions where any plurality of muscles among the left or right corrugator supercilii, the left or right frontalis, the left or right levator labii superioris alae naris, or the left or right zygomaticus major are located.

9. The facial expression code generating device according to any one of claims 1 to 8, further comprising a code output unit that outputs the code generated by the code generating unit.

10. A facial expression generation device comprising: a facial expression code generation device according to any one of claims 1 to 9; a physical modification unit capable of physically modifying the facial expression of a mask, a mask, or a costume; a storage unit for storing a plurality of facial expression files each representing the movement of a plurality of parts of the face; an extraction unit for extracting, from the plurality of facial expression files, the facial expression file corresponding to the code generated by the code generation unit; and an facial expression modification unit for modifying the aspect of the physical modification unit based on the facial expression file extracted by the extraction unit to modify the facial expression of the mask, the mask, or the costume.

11. A facial expression generation device comprising: a facial expression code generation device according to any one of claims 1 to 9; a display device capable of displaying a predetermined avatar or character; a storage unit for storing a plurality of facial expression files each representing the movement of a plurality of parts of a face; an extraction unit for extracting, from the plurality of facial expression files, the facial expression file corresponding to the code generated by the code generation unit; and an expression modification unit for modifying the facial expression of the predetermined avatar or character displayed on the display device based on the facial expression file extracted by the extraction unit.

12. A facial expression generation system comprising: a facial expression code generation device according to any one of claims 1 to 9; a storage unit for storing a plurality of facial expression files each representing the movement of a plurality of parts of the face; an extraction unit for extracting, from the plurality of facial expression files, the facial expression file corresponding to the code generated by the code generation unit; and an expression modification unit for modifying the facial expression of a predetermined avatar or character displayed on a display device based on the facial expression file extracted by the extraction unit.

13. A facial expression code generation method in which a facial expression code generation device includes a plurality of electrodes capable of contacting a plurality of parts of the face, respectively, to acquire myoelectric potentials, and a code representing the displacement of at least one part of the plurality of parts is generated based on the plurality of myoelectric potentials acquired by the plurality of electrodes.

14. A facial expression code generation device comprising: a plurality of electrodes capable of contacting a plurality of parts of the face, respectively, to acquire myoelectric potentials; a wearing device that holds the plurality of electrodes and can be worn on the face with the plurality of electrodes in contact with the plurality of parts; and a code generation unit that generates a code representing the displacement of at least one part of the plurality of parts based on the plurality of myoelectric potentials acquired by the plurality of electrodes.

15. A facial expression code generation device comprising: a plurality of electrodes capable of contacting a plurality of parts of the face, respectively, to acquire myoelectric potentials; and a code generation unit that generates a key code representing the displacement of at least one part of the plurality of parts based on the myoelectric potentials acquired by the plurality of electrodes.

Images