Facial expression code generation device, facial expression generation device, facial expression generation system, and facial expression code generation method

Abstract

A facial expression code generation device, facial expression generation device, facial expression generation system, and facial expression code generation method that reduce the load of data processing. To provide. [Solution] The facial expression code generation device includes a plurality of electrodes that can contact a plurality of parts of the face to acquire electromyography, and a code generation unit that generates a code representing the displacement of at least one of the plurality of parts based on the plurality of electromyography acquired by the plurality of electrodes.

Description

Technical Field

[0001] The present disclosure relates to an expression code generation device, an expression generation device, an expression generation system, and an expression code generation method.

Background Art

[0002] Conventionally, there is a device that attaches a plurality of electrodes for measuring the myoelectric potential of a human face to the face and generates an image of a face, such as an avatar, having an expression corresponding to the human expression based on the myoelectric potential. In this conventional device, data representing the overall expression of the face is generated based on the myoelectric potential (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, when the expression changes, only the shape of a part of the face, such as the eyebrows or the mouth, may change, rather than the entire face. Since the conventional device generates data representing the overall expression of the face even in such a case, the burden of data processing is large. If it takes time for data processing, a time difference occurs between the time when the human expression actually changes and the time when the expression of an avatar or the like changes, which causes the viewer to feel a sense of discomfort.

[0005] Therefore, an object is to provide an expression code generation device, an expression generation device, an expression generation system, and an expression code generation method with a reduced data processing load.

Means for Solving the Problems

[0006] The facial expression code generation device of the embodiment of the present disclosure includes a plurality of electrodes that can contact a plurality of parts of the face to acquire myoelectric potentials, and a code generation unit that generates a key code representing the displacement of at least one of the plurality of parts based on a plurality of myoelectric potentials acquired by the plurality of electrodes. [Effects of the Invention]

[0007] A facial expression code generation device, facial expression generation device, facial expression generation system, and facial expression code generation method that reduce the load of data processing. We can provide this. [Brief explanation of the drawing]

[0008] [Figure 1A] This figure shows an example of the configuration of the facial expression code generation device 100 according to the embodiment. [Figure 1B] This figure shows an example of the configuration of the facial expression code generation device 100 according to the embodiment. [Figure 2] This is a diagram showing an example of the muscles of the human face. [Figure 3] This figure shows an example of the configuration of the facial expression code generation device 100 and facial expression generation system 300 according to the embodiment. [Figure 4A] This figure shows an example of a threshold used by firmware 132C for code generation. [Figure 4B] This figure shows an example of a threshold used by firmware 132C for code generation. [Figure 5] This figure shows an example of the configuration of the sensor amplifier 131, common-mode noise calculation unit 135, and amplifier 136 within the signal processing unit 130. [Figure 6A] This is a flowchart (Part 1) showing an example of the process performed by firmware 132C. [Figure 6B] This is a flowchart (Part 2) showing an example of the process performed by firmware 132C. [Figure 7A] This figure shows an example of the facial expression generation device 500A according to the embodiment. [Figure 7B]It is a diagram showing an example of the expression generation device 500B of the embodiment. [Figure 7C] It is a diagram showing an example of the expression generation device 500C of the embodiment. [Figure 7D] It is a diagram showing an example of the configuration of the expression generation device 500C. [Figure 8] It is a diagram showing an example of a key code. [Figure 9] It is a diagram showing an example of the facial expressions of an avatar realized by key codes A to J. [Figure 10] It is a diagram for explaining an example of the data flow between the expression code generation device 100 and the smartphone 200 in FIG. 3. [Figure 11A] It is a diagram showing the configuration of the expression code generation device 100M1 according to a modification of the embodiment. [Figure 11B] It is a diagram showing the configuration of the expression code generation device 100M2 according to a modification of the embodiment. [Figure 11C] It is a diagram showing the configuration of the expression code generation device 100M3 according to a modification of the embodiment. [Figure 11D] It is a diagram showing the configuration of the expression code generation device 100M4 according to a modification of the embodiment. [Figure 11E] It is a diagram showing the configuration of the expression code generation device 100M5 according to a modification of the embodiment. [Figure 12A] It is a diagram showing an example of the wearing state of the expression code generation device 100 of the embodiment. [Figure 12B] It is a diagram showing an example of the wearing state of the expression code generation device 1OO M6 of a modification of the embodiment. [Figure 12C] It is a diagram showing an example of the wearing state of the expression code generation device 100M7 of a modification of the embodiment. [Figure 12D] It is a diagram showing an example of the wearing state of the expression code generation device 100M8 of a modification of the embodiment.

Mode for Carrying Out the Invention

[0009] The following describes embodiments to which the facial expression code generation device, facial expression generation device, facial expression generation system, and facial expression code generation method of this disclosure are applied. In the following, the same elements may be denoted by the same reference numerals, and redundant explanations may be omitted.

[0010] Furthermore, the following will define and explain the XYZ coordinate system. The directions parallel to the X-axis (X direction), the directions parallel to the Y-axis (Y direction), and the directions parallel to the Z-axis (Z direction) are mutually orthogonal. Also, a planar view refers to viewing from the XY plane. Furthermore, for the sake of explanation, we will use an up-down relationship where the +Z direction side is the top and the -Z direction side is the bottom, but this does not represent a universal up-down relationship. In addition to the XYZ coordinate system, the following will also use left, right, up, and down as seen from the user's perspective. Furthermore, in the following, the length, width, thickness, etc. of each part may be exaggerated to make the structure easier to understand.

[0011] <Embodiment 1> Figures 1A and 1B show an example of the configuration of the facial expression code generation device 100 according to the embodiment. The facial expression code generation device 100 is, for example, goggle-type and can be worn on the user's head. In Figures 1A and 1B, the -X direction is the left side and the +X direction is the right side.

[0012] <Expression Code Generator 100> The facial expression code generation device 100 includes goggles 110, electrodes 120-1 to 120-8, a noise reduction electrode 125, and a signal processing unit 130. The goggles 110 are an example of a wearable device. Electrodes 120-1 to 120-8 constitute an 8-channel electrode configuration. Hereafter, electrodes 120-1 to 120-8 will be simply referred to as electrode 120 unless otherwise specified.

[0013] <Goggles 110> The goggles 110 consist of a main body 111, a headband 112, a shield 113, and a cushion 114. The main body 111 is a frame-shaped structure that surrounds both eyes on the face when the user is wearing the facial expression code generation device 100. In planar view (XY plane view), both ends of the main body 111 are curved toward the -Y direction to conform to the shape of the face. Hereinafter, the side of the main body 111 that is on the face side when the user is wearing the facial expression code generation device 100 will be referred to as the -Y direction side of the main body 111, and the side that is on the opposite side from the face will be referred to as the +Y direction side of the main body 111. For example, such a main body 111 may be made of resin or metal.

[0014] The headband 112 is attached to the left and right ends of the main body 111 and is a band that forms a loop around the head together with the main body 111. Note that in Figures 1A and 1B, only the portion of the headband 112 that connects to the main body 111 is shown. The headband 112 may be, for example, an elastic rubber belt or a belt with a length adjustment function.

[0015] The shield 113 is attached to the +Y side of the main body 111 and is a transparent cover that covers the opening of the frame-shaped main body 111. The shield 113 is made of resin or glass, for example.

[0016] The cushion 114 is attached to the -Y direction side of the main body 111 and is a frame-shaped cushion corresponding to the shape of the main body 111. When the user is wearing the facial expression code generation device 100, the cushion 114 comes into contact with the skin of the face, around both eyes, including the forehead, temples, cheeks, and nose. The cushion 114 is made of an elastic material such as foamed urethane or sponge, for example. With respect to the cushion 114, as with the main body 111, the side of the cushion 114 that is on the face side when the user is wearing the facial expression code generation device 100 is referred to as the -Y direction side of the cushion 114.

[0017] Here, we describe a configuration in which the facial expression code generation device 100 includes goggles 110, but a full-face mask or the like may be used instead of goggles 110. Furthermore, the facial expression code generation device 100 may also be used in a configuration where electrodes 120-1 to 120-8 are directly attached to the face with tape or the like, without including goggles 110 as an example of a mounting part.

[0018] <Electrodes 120-1~120-8> Electrodes 120-1 to 120-8 are provided on the -Y side of the main body 111 of the goggles 110 and are exposed on the -Y side of the cushion 114, outputting electromyographic signals representing electromyography. Each of electrodes 120-1 to 120-8 has two electrode pieces and is capable of acquiring differential electromyographic signals. That is, the electromyographic signals acquired by electrodes 120-1 to 120-8, each having two electrode pieces, are differential signals representing electromyography. Here, we will describe the positions of electrodes 120-1 to 120-8 when the facial expression code generation device 100 is worn by the user. Figure 2 will be used to explain the positions of electrodes 120-1 to 120-8. Figure 2 is a diagram showing an example of the muscles of the human face. Here, we will describe a configuration in which each of electrodes 120-1 to 120-8 has two electrode pieces, but each of electrodes 120-1 to 120-8 may consist of one electrode and output a single-ended electromyographic signal.

[0019] As shown in Figure 2, the facial muscles surrounding both eyes are the left and right corrugator supercilii muscles, left and right frontalis muscles, left and right zygomaticus major muscles, and left and right levator labii superioris alaeque nasi muscles. These facial muscles move when a person smiles, cries, gets angry, etc., forming facial expressions. The facial expression code generation device 100 includes electrodes 120-1 to 120-8 to measure the electromyographic activity of the left and right corrugator supercilii muscles, left and right frontalis muscles, left and right zygomaticus major muscles, and left and right levator labii superioris alaeque nasi muscles in order to measure such changes in facial expressions. In the following, the left and right corrugator supercilii muscles, left and right frontalis muscles, left and right zygomaticus major muscles, and left and right levator labii superioris alaeque nasi muscles may be referred to as facial muscles without special distinction.

[0020] Electrode 120-1 is positioned to correspond to the left corrugator supercilii muscle. Electrode 120-2 is positioned to correspond to the right corrugator supercilii muscle. Electrode 120-3 is positioned to correspond to the left frontalis muscle. Electrode 120-4 is positioned to correspond to the right frontalis muscle. Electrode 120-5 is positioned to correspond to the left levator labii superioris alaeque nasi muscle. Electrode 120-6 is positioned to correspond to the right levator labii superioris alaeque nasi muscle. Electrode 120-7 is positioned to correspond to the left zygomaticus major muscle. Electrode 120-8 is positioned to correspond to the right zygomaticus major muscle. Note that in the following, electrodes 120-1 to 120-8 may be distinguished as channels 1 to 8. Therefore, in Figure 2, the muscles corresponding to electrodes 120-1 to 120-8 are labeled Ch1 to Ch8.

[0021] By using electrodes 120-1 to 120-8, it is possible to detect the movement of the left eyebrow, right eyebrow, left eye, right eye, left cheek, right cheek, left upper lip, and right upper lip.

[0022] Here, a configuration in which the facial expression code generation device 100 includes eight electrodes 120-1 to 120-8 is described, but the facial expression code generation device 100 may be configured with one or more of the eight electrodes 120-1 to 120-8 omitted. In addition, the electromyography (EMG) of facial muscles other than the left and right corrugator supercilii muscles, left and right frontalis muscles, left and right zygomaticus major muscles, and left and right levator labii superioris alaeque nasi muscles may be measured. The circuit for acquiring differential signals will be described later.

[0023] <Noise reduction electrode 125> The noise reduction electrodes 125 are provided to cancel out noise contained in the differential signals acquired by each of the electrodes 120-1 to 120-8. As an example, the facial expression code generation device 100 has two noise reduction electrodes 125. The two noise reduction electrodes 125 are provided at the center in the Z direction at both the left and right ends of the body 111 of the goggles 110 on the -Y direction side, and are exposed on the -Y direction side of the cushion 114.

[0024] The noise reduction electrodes 125 come into contact with the left and right cheeks of the user's face while the user is wearing the facial expression code generation device 100. The noise reduction electrodes 125 output cancellation signals to the left and right cheeks to cancel out noise contained in the differential signal. The position of the noise reduction electrodes 125 is not limited to the positions corresponding to the left and right cheeks of the face, but is preferably a position corresponding to one of the following: the left or right corrugator supercilii muscle, the left or right frontalis muscle, the left or right levator labii superioris alaeque nasi muscle, or the left or right zygomaticus major muscle. In addition, there only needs to be at least one noise reduction electrode 125, and there may be three or more. The circuit that outputs the cancellation signals to the noise reduction electrodes 125 is included in the signal processing unit 130, and its details will be described later.

[0025] <Signal processing unit 130> The signal processing unit 130 is, for example, located at the left end of the main unit 111. The signal processing unit 130 includes a signal processing circuit and a battery 130A. The battery 130A is, for example, located at the bottom end of the signal processing unit 130. The location of the signal processing unit 130 is not limited to the left end of the main unit 111; it may be located at a different part of the main unit 111, or it may be attached to the headband 112, taking into consideration weight balance, etc. Furthermore, the battery 130A may be provided separately from the signal processing circuit portion of the signal processing unit 130. In addition, the signal processing unit 130 may be configured to receive power via a USB cable or the like from a smartphone 200, PC (Personal Computer), tablet computer, or XR headset (AR (Augmented Reality) headset or VR (Virtual Reality) headset), etc., as described later, without having a battery 130A.

[0026] The signal processing circuit generates a code representing the displacement of at least one of several facial regions based on the electromyographic potential acquired by electrodes 120-1 to 120-8. The signal processing unit 130 outputs the code via wireless or wired communication. The output destination of the code is a device that can change the facial expression of an avatar or character image displayed on a display device such as a display, or a device that can physically change the facial expression of a mask or costume based on the code. Details of the signal processing circuit and code of the signal processing unit 130 will be described later.

[0027] <Expression code generation device 100 and expression generation system 300> Figure 3 shows an example of the configuration of the facial expression code generation device 100 and facial expression generation system 300 according to the embodiment.

[0028] <Facial Expression Generation System 300> The facial expression generation system 300 includes a facial expression code generation device 100 and a smartphone 200. The facial expression generation system 300 includes the facial expression code generation device 100 and another device. The other device included in the facial expression generation system 300 is a device that can change the facial expression of an avatar or character image displayed on a display device such as a display, based on the code output by the facial expression code generation device 100.

[0029] The facial expression generation device of this embodiment is a device such as a mask or costume integrated with the facial expression code generation device 100, and is a device that can physically change the facial expression of the mask or costume based on the code generated by the facial expression code generation device 100. The facial expression generation device of this embodiment will be described later with reference to Figure ☆. Here, we will describe the facial expression generation system 300 including the facial expression code generation device 100 and the smartphone 200.

[0030] The smartphone 200 is an example of a device capable of changing the facial expressions of avatars or character images displayed on a display device such as a screen, based on a code. Here, we describe a form in which the facial expression generation system 300 includes the facial expression code generation device 100 and the smartphone 200, but the smartphone 200 may be replaced with a PC (Personal Computer), a tablet computer, or an XR headset (AR (Augmented Reality) headset, or VR (Virtual Reality) headset), etc.

[0031] Figure 3 shows a simplified representation of the goggles 110 of the facial expression code generation device 100. Also, in Figure 3, each of the two electrodes provided for each channel, which detect electromyography differentially, is simplified and shown as a single electrode. In Figure 3, the noise reduction electrode 125 is omitted. Furthermore, Figure 3 shows the part of the signal processing unit 130 of the facial expression code generation device 100 that generates codes from electromyography acquired by electrodes 120-1 to 120-8, while the part that performs signal processing related to the noise reduction electrode 125 is omitted.

[0032] The signal processing unit 130 includes a sensor amplifier 131, a CPU (Central Processing Unit) 132, a BLE (Bluetooth® Low Energy) module 133, an antenna 134, and a battery 130A. The BLE module 133 is an example of a code output unit.

[0033] There are eight sensor amplifiers 131, corresponding to electrodes 120-1 to 120-8. In other words, there are eight channels of sensor amplifiers 131. The eight sensor amplifiers 131 are connected to the CPU 132. Each sensor amplifier 131 differentially amplifies the differential signal obtained from the electromyographic potential of each pair of electrode pieces 120-1 to 120-8 and outputs a displacement signal representing the displacement of the facial muscles to the CPU 132.

[0034] In this description, we will explain a configuration in which eight sensor amplifiers 131 are provided, corresponding to electrodes 120-1 to 120-8. However, the number of electrodes 120 may be other than eight. The number of sensor amplifiers 131 should match the number of electrodes 120, and it is sufficient for one sensor amplifier 131 to be connected to each electrode 120.

[0035] The CPU 132 is a CPU chip comprising a selector 132A, an ADC (Analog to Digital Converter) 132B, and firmware (FW) 132C. Firmware 132C is an example of a code generation unit, and is implemented, for example, by a DSP (Digital Signal Processor).

[0036] The selector 132A has eight input terminals, each connected to the output terminals of the eight sensor amplifiers 131, and one output terminal connected to the input terminal of the ADC 132B. The selector 132A selects one displacement signal at a time from each of the eight sensor amplifiers 131 (one channel at a time) and outputs it to the ADC 132B.

[0037] The ADC132B digitally converts the eight displacement signals input in time division from the selector 132A and outputs them to the firmware 132C. Hereafter, the displacement signals digitally converted by the ADC132B will be referred to as digital displacement signals. The digital displacement signals have discrete values ​​that represent the signal levels of the analog displacement signals before digital conversion, for example.

[0038] The ADC132B digitally converts eight displacement signals, each input in time division from eight sensor amplifiers 131, and outputs the eight digital displacement signals sequentially to the firmware 132C. The eight digital displacement signals are output from the selector 132A to the ADC132B in the order of Ch1 to Ch8, making it possible to identify which of the facial muscles—the left and right corrugator supercilii, left and right frontalis, left and right levator labii superioris alaeque nasi, and left and right zygomaticus major—they represent signals based on electromyography. For example, if there is a channel of higher importance among Ch1 to Ch8, the digital displacement signal of the higher-importance channel may be output repeatedly. Furthermore, the output order of the digital displacement signals does not have to be Ch1 to Ch8.

[0039] Furthermore, the same number of ADC132B units may be provided as the number of sensor amplifiers 131, and one ADC132B may be connected to the output side of each sensor amplifier 131.

[0040] Firmware 132C converts the displacement signal, which has been digitally converted by ADC132B, into a code and outputs it. Firmware 132C has a threshold for determining the presence or absence of facial muscle movement based on the signal level (discrete value) of the digital displacement signal.

[0041] Furthermore, the thresholds will be explained using Figures 4A and 4B. Figures 4A and 4B show examples of thresholds used by firmware 132C for code generation. The thresholds shown in Figures 4A and 4B are thresholds used for one of the eight digital displacement signals. Figure 4A shows one threshold TH1, and Figure 4B shows two thresholds TH11 and TH12. For example, TH12 > TH11.

[0042] Firmware 132C, for example, uses the threshold TH1 shown in Figure 4A to generate code "1" if the signal level of the digital displacement signal is equal to or greater than threshold TH1, and does not generate a code if the signal level of the digital displacement signal is less than threshold TH1. Code "1" indicates that the facial muscles have moved. By generating a code only when the signal level of the digital displacement signal is equal to or greater than threshold TH1, and not generating a code when it is less than threshold TH1, it is possible to generate a code that represents only the portion of the facial muscles that has been displaced. This also reduces the data processing load.

[0043] Furthermore, firmware 132C, as an example, uses thresholds TH11 and TH12 shown in Figure 4B to generate code "11" if the signal level of the digital displacement signal is greater than or equal to threshold TH11 and less than TH12, and generates code "12" if the signal level of the digital displacement signal is greater than or equal to threshold TH12. Firmware 132C does not generate a code if the signal level of the digital displacement signal is less than threshold TH11.

[0044] Code "11" indicates slight facial muscle movement, while code "12" indicates significant facial muscle movement.

[0045] As mentioned above, the thresholds shown in Figures 4A and 4B are used for one of the eight digital displacement signals. By using similar thresholds for the other seven digital displacement signals, firmware 132C can generate a code that can determine the presence or absence of facial muscle movement for each of the eight digital displacement signals. Firmware 132C outputs the code to the BLE module 133.

[0046] <BLEモジュール133> The BLE module 133 is a communication module that enables wireless communication using Bluetooth® Low Energy. The BLE module 133 is connected to the antenna 134 and outputs an advertisement signal including a code from the antenna 134.

[0047] <Smartphone 200> The smartphone 200 includes a control device 210, a BLE module 220, and a display 230. The display 230 is an example of a display device.

[0048] <Control device 210> The control device 210 is a device that performs processing related to the operation of the smartphone 200, and is implemented by a computer system having a CPU and memory. For example, the control device 210 has an application 210A. Application 210A is a functional block implemented by the CPU of the control device 210 executing a program stored in memory. Codes are input to the control device 210 via the BLE module 220.

[0049] Application 210A is, as an example, an application program for a smartphone VTuber. Application 210A generates facial expressions based on a code input from the BLE module 220. Facial expression generation is a process that generates facial expressions and their time-series movements (Facial-expression / Motion script), and as an example, it generates facial expressions for a 2D avatar.

[0050] VTubers can change the facial expressions of their avatars, which are based on their own faces, by attaching an expression code generation device 100 to their faces and using application 210A while changing their own facial expressions. For example, if they move their left eyebrow upwards, the left eyebrow of the avatar's face displayed on the display 230 will move upwards, and if they pull up their right cheek, the right cheek of the avatar's face displayed on the display 230 will be pulled up.

[0051] <Common-mode noise calculation unit> Figure 5 shows an example of the configuration of the sensor amplifier 131, common-mode noise calculation unit 135, and amplifier 136 parts of the signal processing unit 130. The common-mode noise calculation unit 135 is an example of a cancellation signal output unit. Figure 5 also shows the two electrode pieces of electrode 120 and one noise rejection electrode 125.

[0052] Two electrode pieces of electrode 120 are connected to the two input terminals of sensor amplifier 131. Sensor amplifier 131 differentially amplifies the differential signal obtained from the electromyographic potential of the two electrode pieces and outputs a displacement signal representing the displacement of the facial muscles. Figure 3 shows a circuit configuration in which CPU 132 is connected to the output side of sensor amplifier 131. In reality, CPU 132 and common-mode noise calculation unit 135 are connected to the output side of sensor amplifier 131. Therefore, Figure 5 shows the wiring in which CPU 132 is connected to the output side of sensor amplifier 131.

[0053] The noise reduction electrode 125 is connected to the common-mode noise calculation unit 135 via the amplifier 136.

[0054] The common-mode noise calculation unit 135 generates an out-of-phase component of the noise contained in the displacement signal and outputs it to the amplifier 136. Since the out-of-phase component of the noise contained in the displacement signal is a signal that has the opposite phase to the common-mode noise contained in the differential signal obtained from the electromyographic potentials of the two electrode pieces, it can reduce the common-mode noise of the differential signal. Differential signals have high noise immunity, but it is not possible to remove the common-mode noise contained in the two signals of a differential signal. For this reason, the common-mode noise calculation unit 135 outputs a signal that has the opposite phase to the common-mode noise contained in the differential signal, thereby reducing the common-mode noise of the differential signal acquired by the two electrode pieces of electrode 120.

[0055] Furthermore, the sensor amplifier 131 connected to the input side of the common-mode noise calculation unit 135 may be at least one of the eight sensor amplifiers 131 connected to the output side of the eight electrodes 120-1 to 120-8. The sensor amplifier 131 connected to the input side of the common-mode noise calculation unit 135 may also be the sensor amplifier 131 connected to the electrode 120 closest to the noise rejection electrode 125 among the electrodes 120-1 to 120-8.

[0056] As an example, there are two noise reduction electrodes 125, one on the left and one on the right. As shown in Figure 5, a common-mode noise calculation unit 135 is connected to each noise reduction electrode 125. For example, the left noise reduction electrode 125, the common-mode noise calculation unit 135, and the amplifier 136 can be connected to any one of the left half electrodes 120-1, 120-3, 120-5, or 120-7. In this case, the right noise reduction electrode 125, the common-mode noise calculation unit 135, and the amplifier 136 can be connected to any one of the right half electrodes 120-2, 120-4, 120-6, or 120-8. Alternatively, eight noise reduction electrodes 125 may be provided, and each of the eight electrodes 120-1 to 120-8 may be connected to a noise reduction electrode 125, a common-mode noise calculation unit 135, and an amplifier 136. Alternatively, the noise reduction electrode 125, the common-mode noise calculation unit 135, and the amplifier 136 may be connected to any one of the eight electrodes 120-1 to 120-8. Alternatively, the upper noise reduction electrode 125, the common-mode noise calculation unit 135, and the amplifier 136 may be connected to any one of the upper electrodes 120-1 to 120-4. In this case, the lower noise reduction electrode 125, the common-mode noise calculation unit 135, and the amplifier 136 may be connected to any one of the lower electrodes 120-5 to 120-8.

[0057] <Flowchart (Part 1)> Figure 6A is a flowchart (Part 1) showing an example of the process performed by firmware 132C. This process is implemented by the facial expression code generation method of the embodiment.

[0058] When the firmware 132C starts processing, it generates code by selecting one of the eight channels at a time and executing the subroutine processing steps S1 to S3.

[0059] Firmware 132C acquires electromyographic potentials by having selector 132A sequentially select sensor amplifiers 131 from channel 1 to channel 8 (step S1).

[0060] Firmware 132C determines whether the electromyographic potential is above the threshold TH1 (step S2).

[0061] When firmware 132C determines that the electromyographic potential is above the threshold TH1 (S2: Yes), it generates a code "1" for that channel, indicating that the facial muscles have moved (step S3).

[0062] Furthermore, if firmware 132C determines in step S2 that the electromyographic potential is not above the threshold TH1 (S2: No), it proceeds to step S4 without generating any code.

[0063] After completing the processing in step S2 or S3, the firmware 132C determines whether to terminate the series of processes (step S4).

[0064] If firmware 132C determines that the series of processes will not be terminated (S4: No), it returns the flow to the start of the subroutine processing and executes the processes from step S1 onwards.

[0065] If firmware 132C determines in step S4 that it is time to terminate the series of processes (S4:Yes), it terminates the series of processes (STOP).

[0066] <Flowchart (Part 2)> Figure 6B is a flowchart (Part 2) showing an example of a process performed by firmware 132C. This process is implemented by the facial expression code generation method of the embodiment.

[0067] When firmware 132C starts processing, it acquires electromyographic signals by selecting one of the eight channels at a time and executing the subroutine process in step S1A.

[0068] Firmware 132C acquires electromyographic potentials (step S1A) by having selector 132A sequentially select sensor amplifiers 131 from channel 1 to channel 8.

[0069] Firmware 132C acquires the maximum electromyographic potential among the eight electromyographic potentials from channels 1 to 8 (step S2A).

[0070] Firmware 132C determines whether the maximum electromyographic potential is above the threshold TH1 (step S3A).

[0071] When firmware 132C determines that the maximum electromyographic potential is above the threshold TH1 (S3A: Yes), it generates a code "1" for that channel, indicating that the facial muscles have moved (step S4A).

[0072] Furthermore, if firmware 132C determines in step S3A that the maximum electromyographic potential is not equal to or greater than the threshold TH1 (S3A: No), it proceeds to step S5A without generating any code.

[0073] After completing the processing in step S3A or S4A, firmware 132C determines whether to terminate the process (step S4).

[0074] If firmware 132C determines that it will not terminate the process (S5A: No), it returns the flow to the start of the subroutine processing and executes the processes from step S1A onward.

[0075] If firmware 132C determines in step S5A that processing should be terminated (S5A:Yes), it terminates the series of processes (STOP). Alternatively, it may be determined whether the electromyographic potential of each channel is above the threshold TH1 without identifying the site showing the maximum electromyographic potential.

[0076] <Facial expression generation device 500A> Figure 7A shows an example of the facial expression generating device 500A according to the embodiment. The left side of Figure 7A shows an example of the state of the facial expression generating device 500A before the facial expression changes, and the right side of Figure 7A shows an example of the state of the facial expression generating device 500A after the facial expression has changed.

[0077] The facial expression generation device 500A shown in Figure 7A includes a facial expression code generation device 100 and a mascot costume 400A. The facial expression generation device 500A is used by a person who wears the facial expression code generation device 100 on their face, gets inside, and wears the mascot costume 400A. For this reason, the facial expression code generation device 100 worn on the face by the person inside the mascot costume 400A is shown with a dashed line.

[0078] The 400A mascot costume is configured so that the shape of the eyes and mouth on its face can be changed using actuators, allowing for changes in facial expression. By connecting the control unit that controls the actuators of the 400A mascot costume to the CPU 132 of the facial expression code generation device 100, and inputting the code output from the CPU 132 into the control unit, the actuators are activated, and the shape of the eyes and mouth on the face can be changed.

[0079] For example, when the facial expression code generation device 100 is worn on the face and the facial expression of the person wearing the mascot costume 400A (see inside the speech bubble) changes to a smile, the facial expression code generation device 100 generates a code representing the displacement of the muscles around the mouth, and as shown on the right side of Figure 7A, the facial expression of the mascot costume 400A changes to a smile. In other words, a person wearing the facial expression code generation device 100 on their face can change the facial expression of the mascot costume 400A by changing the facial expression of their own face. That is, it is possible to change the facial expression of the mascot costume 400A using the face as an input.

[0080] Furthermore, since the code does not represent data representing the entire facial expression, but rather only code data representing the changed parts of the facial muscles that make up the expression, the data size is small, and the data processing load can be reduced.

[0081] Furthermore, by using the 400A mascot costume at an event venue, the mascot costume's expression changes as the person inside changes their facial expression, allowing for communication with the audience through means other than gestures. The person inside can use their own facial expressions as a control unit for the 400A mascot costume's expression. Although the above explanation uses the 400A mascot costume, instead of the 400A mascot costume, a puppet or other facial structure could be configured so that the shape of the eyes and mouth can be changed using actuators.

[0082] <Facial expression generation device 500B> Figure 7B shows an example of the facial expression generating device 500B according to the embodiment. The left side of Figure 7B shows an example of the state of the facial expression generating device 500B before the facial expression changes, and the right side of Figure 7B shows an example of the state of the facial expression generating device 500B after the facial expression has changed.

[0083] The facial expression generating device 500B shown in Figure 7B includes a facial expression code generating device 100 and a mask 400B. In the facial expression generating device 500B, the facial expression code generating device 100 and the mask 400B are integrated. Figure 7B shows the face of the mask 400B, and on the back side, the facial expression code generating device 100, including the goggles 110 shown in Figures 1A and 1B, is integrally provided.

[0084] The facial expression generation device 500B is used by wearing the facial expression generation device 500B, which is an integrated unit of the facial expression code generation device 100 and the mask 400B, on the face. For this reason, the facial expression code generation device 100, which is worn on the face by the human, is shown with a dashed line within the mask 400B.

[0085] Mask 400B, for example, has LEDs (Light Emitting Diodes) attached to both eyes, and is configured to allow the LED light color to be changed. By changing the LED light color, the facial expression can be altered. The control unit that controls the LED light color of Mask 400B is connected to the CPU 132 of the facial expression code generation device 100. By inputting the code output from the CPU 132 into the control unit, the LED light color can be changed based on the code generated by the facial expression code generation device 100 according to the wearer's facial expression. It is also possible to change the light color of any part of Mask 400B.

[0086] In other words, a person wearing the facial expression code generation device 100 on their face can change the color of the LEDs in both eyes of the mask 400B by changing the facial expression of their own face. That is, it is possible to change the facial expression of the mask 400B by using the face as an input.

[0087] Furthermore, since the code does not represent data representing the entire facial expression, but rather only code data representing the changed parts of the facial muscles that make up the expression, the data size is small, and the data processing load can be reduced.

[0088] The LEDs attached to the 400B mask light up in response to the wearer's facial expression, or the way they light up (color, pattern, etc.) changes, allowing the wearer to become part of the decorations that adorn the event venue. This makes it particularly valuable for use in amusement parks and similar venues.

[0089] Furthermore, by collecting audience reactions via wireless data communication, the organizers (performers) can obtain feedback on the venue's response.

[0090] While the mask 400B was described here, instead of the mask 400B, LEDs capable of emitting various colors could be attached to the goggles 110, allowing them to light up around the face. Penlights, which can be purchased at live venues, have become popular in recent years as a tool to liven up live performances by changing the color with the press of a button. However, by using the goggles 110 with LEDs that can light up around the face as described above as a substitute for a penlight, the wearer can change their facial expressions to light up the LEDs, thereby livening up the live performance through reactions. Furthermore, if connected wirelessly, the live event organizer can collect reactions and control the colors.

[0091] Alternatively, sound may be used instead of, or in addition to, LED illumination. Furthermore, instead of mask 400B, a full-face mask with LEDs attached may be used, or the shape of the eyes and mouth on the mask may be changed using actuators to alter facial expressions.

[0092] <Facial expression generation device 500C> Figure 7C shows an example of the facial expression generation device 500C according to the embodiment. The facial expression generation device 500C includes a facial expression code generation device 100 and an XR headset 400C. In the facial expression generation device 500C, the facial expression code generation device 100 and the XR headset 400C are integrated. In Figure 7C, the XR headset 400C is integrally provided on the +Y direction side of the facial expression code generation device 100, which includes goggles 110. The facial expression generation device 500C is used by being worn on the face.

[0093] The facial expression code generation device 100 does not include the shield 113 shown in Figures 1A and 1B, and is configured to allow viewing of the display 430 of the XR headset 400C via the goggles 110. The display 430 displays, for example, a two-dimensional or three-dimensional avatar.

[0094] Figure 7D shows an example of the configuration of the facial expression generation device 500C. The facial expression code generation device 100 shown in Figure 7D does not include the BLE module 133, and the CPU 132 is connected to the control device 210 of the XR headset 400C.

[0095] The control device 210 of the XR headset 400C shown in Figure 7D is a device that performs processing related to the operation of the XR headset 400C and is implemented by a computer system having a CPU and memory. As an example, the control device 210 has an application 210B. Application 210B is a functional block implemented by the CPU of the control device 210 executing a program stored in memory. The control device 210 receives codes from the CPU 132 of the facial expression code generation device 100.

[0096] Application 210B is an example application for the XR headset 400C.

[0097] Application 210B is an application that has a driver placed in front of OpenXR, and an interface compatible with Device A is placed behind OpenXR. Device A is, for example, an interface compatible with XR headsets from various companies, and generates facial expressions for face images according to the display method of each company's XR headset.

[0098] The code entered into the driver is output from OpenXR to one of the devices A, and the image of the part of the face corresponding to the code is modified, thereby generating facial expressions. As a result, a 2D or 3D avatar is generated and displayed on display 430. Since only the image of a part of the face is modified, rather than the entire facial expression, the processing load is light.

[0099] By wearing the facial expression generator 500C, which integrates the XR headset 400C and the facial expression code generation device 100, and using the application 210B while changing one's own facial expressions, it is possible to change the facial expressions of an avatar based on one's own face. For example, if the user moves their left eyebrow upward, the left eyebrow of the avatar's face displayed on the display 430 will move upward, and if the user pulls up their right cheek, the right cheek of the avatar's face displayed on the display 230 will be pulled up.

[0100] In other words, a person wearing the facial expression generation device 500C on their face can change the facial expression of the avatar displayed on the display 430 by changing the facial expression of their own face. That is, it is possible to change the facial expression of the avatar by using the face as an input.

[0101] Furthermore, since the code does not represent data representing the entire facial expression, but rather only code data representing the changed parts of the facial muscles that make up the expression, the data size is small, and the data processing load can be reduced.

[0102] <Keycode> This section describes key codes that can be used instead of the codes mentioned above. Firmware 132C may be configured to convert the digital displacement signal, digitally converted by ADC132B, into a key code and output it. In other words, firmware 132C may use the key codes described below instead of the codes mentioned above.

[0103] A key code is, for example, the key code for a USB (Universal Serial Bus) HID (Human Interface Device) Usage ID (Identifier).

[0104] Furthermore, the key codes used in this embodiment may be any key codes other than those assigned to physical keys on the keyboard. A keyboard, for example, is a Japanese-layout JIS keyboard or a QWERTY-layout keyboard. An example of a key code other than those assigned to physical keys on the keyboard is F13-F24, etc., which are assigned to function keys but do not exist as physical keys. Specific examples of keyboards in which F13-F24 etc. do not exist as physical keys include English-layout 101 / 102 keyboards within the QWERTY layout, and 106 / 109 keyboards within the Japanese-layout JIS keyboard.

[0105] If the facial expression code generation device 100 generates key codes that are assigned to the physical keys of a keyboard as described above, there is a risk that commands based on the key codes assigned to the physical keys may be input to the smartphone 200 or PC. For example, if the key code assigned to a physical key is the letter A, the displacement of the electromyography of the facial muscles may cause a command to input the letter A to the smartphone 200 or PC, potentially causing the smartphone 200 or PC to operate. In contrast, if a key code other than those assigned to the physical keys of a Japanese JIS keyboard or a QWERTY keyboard is used, even if the key code generated by the facial expression code generation device 100 is input to the smartphone 200 or PC, the smartphone 200 or PC will not operate. Therefore, it is possible to generate dedicated key codes based on the displacement of the electromyography of the facial muscles, and to stably change the expressions of avatars or characters displayed on a display device, or masks or costumes that can physically change facial expressions, without generating unused commands. The facial expression code generation device 100 may also generate key codes that are assigned to the physical keys of a keyboard as described above. In this case, for example, the smartphone 200 or PC can be operated by commands based on key codes assigned to physical keys, and the facial expressions of avatars or characters can be changed. For example, it becomes possible to change the facial expressions of avatars or characters while chatting with text input.

[0106] Figure 8 shows an example of key codes corresponding to different types of facial expressions. There are eight types of facial expressions, as an example: left eyebrow raised, left eyebrow lowered, left eye wide open, left cheek raised, left eye closed, right eyebrow raised, right eyebrow lowered, right eye wide open, right cheek raised, and right eye closed. Figure 8 shows Ch1 to Ch8. Ch1 to Ch8 correspond to the facial muscles as shown in Figure 2. Figure 8 also shows the comparison results between the digital displacement signals based on electromyography acquired by electrodes 120-1 to 120-8 of Ch1 to Ch8 and the threshold, shown in H (High) and L (Low) levels. The H level represents the comparison result when the digital displacement signal is above the threshold, and the L level represents the comparison result when the digital displacement signal is below the threshold.

[0107] When the facial expression involves raising the left eyebrow, the comparison results of the digital displacement signals in Ch1, Ch3, Ch5, and Ch7 are L, H, L, L. This is because the left frontal muscle is displaced.

[0108] When the facial expression involves lowering the left eyebrow, the comparison results of the digital displacement signals in Ch1, Ch3, Ch5, and Ch7 are H, L, L, L. This is because the left corrugator supercilii muscle is displaced.

[0109] When the facial expression shows the left eye wide open, the comparison results of the digital displacement signals in Ch1, Ch3, Ch5, and Ch7 are H, H, L, L. This is because the left corrugator supercilii muscle and the left frontalis muscle are displaced.

[0110] When the facial expression involves raising the left cheek, the comparison results of the digital displacement signals in Ch1, Ch3, Ch5, and Ch7 are L, L, L, H. This is because the left zygomaticus major muscle is displaced.

[0111] When the facial expression is with the left eye closed (tightly closed), the comparison results of the digital displacement signals of Ch1, Ch3, Ch5, and Ch7 are L, H, H, L. This is because the left frontalis muscle and the left levator labii superioris alaeque nasi muscle are displaced.

[0112] When the facial expression involves raising the right eyebrow, the comparison results of the digital displacement signals in Ch2, Ch4, Ch6, and Ch8 are L, H, L, L. This is because the right frontal muscle is displaced.

[0113] When the facial expression involves lowering the right eyebrow, the comparison results of the digital displacement signals in Ch2, Ch4, Ch6, and Ch8 are H, L, L, L. This is because the right corrugator supercilii muscle is displaced.

[0114] When the facial expression shows the right eye wide open, the comparison results of the digital displacement signals in Ch2, Ch4, Ch6, and Ch8 are H, H, L, L. This is because the right corrugator supercilii muscle and the right frontalis muscle are displaced.

[0115] When the facial expression involves raising the right cheek, the comparison results of the digital displacement signals in Ch2, Ch4, Ch6, and Ch8 are L, L, L, H. This is because the right zygomaticus major muscle is displaced.

[0116] When the facial expression is one of right eye closure (tightly closed eye), the comparison results of the digital displacement signals of Ch2, Ch4, Ch6, and Ch8 are L, H, H, L. This is because the right frontalis muscle and the right levator labii superioris alaeque nasi muscle are displaced.

[0117] In such cases, as an example, key codes A to E are assigned to raising the left eyebrow, lowering the left eyebrow, opening the left eye, raising the left cheek, and closing the left eye. Also, as an example, key codes F to J are assigned to raising the right eyebrow, lowering the right eyebrow, opening the right eye, raising the right cheek, and closing the right eye. As an example, it is sufficient for firmware 132C to store in its internal memory the association between the H / L combinations of Ch1 to Ch8 and the key codes.

[0118] By using key codes A to J, the facial expression code generation device 100 can be attached to the face, and by changing facial expressions, key codes A to J can be generated. For example, as shown in Figure 3, when the facial expression code generation device 100 and smartphone 200 are used with wireless communication via BLE, the smartphone 200 can install facial expression files representing the types of facial expressions such as left eyebrow raised, left eyebrow lowered, left eye wide open, left cheek raised, left eye closed, right eyebrow raised, right eyebrow lowered, right eye wide open, right cheek raised, and right eye closed into its memory, and then use the key binding function to identify the facial expression file corresponding to the key code (any one of A to J) obtained from the facial expression code generation device 100. As a result, the facial expressions of the avatar displayed on the display 230 are controlled by the control device 210 according to the key codes A to J.

[0119] Figure 9 shows examples of avatar facial expressions that can be realized using key codes A through J. The leftmost part of Figure 9 shows the default expression.

[0120] To the right of the default expression, in a 2x5 grid, are shown the expressions for left eyebrow raised, left eyebrow lowered, left eye wide open, left cheek raised, left eye closed, right eyebrow raised, right eyebrow lowered, right eye wide open, right cheek raised, and right eye closed, along with the assigned key codes A through J. In other words, Figure 9 shows 11 expressions: the default expression, left eyebrow raised, left eyebrow lowered, left eye wide open, left cheek raised, left eye closed, right eyebrow raised, right eyebrow lowered, right eye wide open, right cheek raised, and right eye closed.

[0121] Here, as an example, we will explain by assuming that there are 10 expression files, each representing a different expression from the default expression shown in Figure 9, in order to change the default expression.

[0122] The expression file corresponding to each of the 10 expressions shown in Figure 9 is a text file that represents the actions (left eyebrow raised, left eyebrow lowered, left eye wide open, left cheek raised, left eye closed, right eyebrow raised, right eyebrow lowered, right eye wide open, right cheek raised, right eye closed) in each expression as parameters. The expression file format is, for example, a JSON file. The file name of each expression file can be any name.

[0123] The memory of smartphone 200 stores these 10 facial expression files. For example, application 210A on smartphone 200 associates each of the 10 facial expression files with one of the 10 key codes (A to J) using its keybinding function.

[0124] The application 210A on the smartphone 200 identifies an expression file corresponding to a key code (one of A to J) obtained from the expression code generation device 100, and changes the avatar's expression to the expression represented by the identified expression file. In this way, the facial expression of the avatar displayed on the display 230 can be changed to one of the various expressions shown in Figure 9.

[0125] Furthermore, this explanation describes a configuration where the expression file format is a JSON file, and application 210A uses a keybinding function to identify the expression file corresponding to the key code. However, as long as application 210A can identify the expression file corresponding to the key code, the expression file format is not limited to a JSON file, and the expression file corresponding to the key code may be identified using a function other than the keybinding function.

[0126] Furthermore, although this description focuses on a configuration using the keybinding function, table data associating data representing different types of facial expressions with key codes may also be stored in the memory of the smartphone 200. The application 210A on the smartphone 200 may then identify the data corresponding to the type of facial expression in the table data based on the key code obtained from the facial expression code generation device 100, and change the avatar's facial expression.

[0127] Figure 10 is a diagram illustrating an example of the data flow between the facial expression code generation device 100 and the smartphone 200 in Figure 3.

[0128] When the host controller of the smartphone 200 obtains a key code from the facial expression code generation device 100, the key code is input to application 210A via the HID standard driver (kbdhid.sys), class driver (kbdclass.sys), kernel, and API. Application 210A identifies the facial expression file corresponding to the key code using its key binding function, and uses the identified facial expression file to change the avatar's facial expression, for example, by raising the left eyebrow.

[0129] <Modified Expression Code Generators 100M1~100M5 of the Embodiment> Figures 11A to 11E show the configurations of the facial expression code generation devices 100M1 to 100M5 according to modified embodiments.

[0130] The facial expression code generation device 100M1 shown in Figure 11A has a configuration that includes only the left half of electrodes 120-1, 120-3, 120-5, and 120-7 of the electrodes 120-1 to 120-8 of the facial expression code generation device 100 (see Figures 1A and 1B). The facial expression code generation device 100M1 does not include electrodes 120-2, 120-4, 120-6, and 120-8.

[0131] For example, regarding the facial expressions of an avatar, there may be cases where expressions that distort only one side of the face are not used, such as an expression with only one eye closed, an expression with only one eyebrow raised, or an expression with only one cheek moved. Alternatively, the facial expressions of an avatar may utilize expressions that distort both sides of the face equally, such as an expression with both eyes closed, an expression with both eyebrows raised, or an expression with both cheeks moved. Furthermore, in the case of the mascot costume 400A (see Figure 7A) and the mask 400B, it may also be possible to use expressions that distort both sides of the face equally. In such cases, the electrodes 120 do not need to be 8 channels, and can be reduced to 4 channels on either the left or right side.

[0132] The facial expression code generation device 100M1 includes only electrodes 120-1, 120-3, 120-5, and 120-7 for Ch1, Ch3, Ch5, and Ch7, but can generate codes for Ch1 to Ch8 as follows.

[0133] In the facial expression code generation device 100M1, firmware 132C generates codes for Ch1 and Ch2 if the electromyogram acquired in Ch1 is greater than or equal to the threshold TH1. Similarly, firmware 132C generates codes for Ch3 and Ch4 if the electromyogram acquired in Ch3 is greater than or equal to the threshold TH1. Firmware 132C generates codes for Ch5 and Ch6 if the electromyogram acquired in Ch5 is greater than or equal to the threshold TH1. Firmware 132C generates codes for Ch7 and Ch8 if the electromyogram acquired in Ch7 is greater than or equal to the threshold TH1.

[0134] The codes generated for Ch2, Ch4, Ch6, and Ch8 are symmetrical codes that represent displacements that are symmetrical to the displacements generated for Ch1, Ch3, Ch5, and Ch7 in a frontal view of the face.

[0135] By doing this, it is possible to generate expressions such as those with both eyebrows raised, both eyebrows lowered, both eyes wide open, both cheeks raised, and both eyes closed.

[0136] Furthermore, by reducing the number of electrodes 120, the facial expression code generation device 100M1 can be simplified and its manufacturing costs reduced.

[0137] The facial expression code generation device 100M2 shown in Figure 11B has a configuration in which electrodes 120 include electrodes 120-1, 120-3, 120-4, 120-5, 120-7, and 120-8, but do not include electrodes 120-2 and 120-6.

[0138] Of the facial muscles surrounding both eyes, the left and right corrugator supercilii muscles (Ch1, Ch2) and the left and right levator labii superioris alaeque nasi muscles (Ch5, CH6), located in the central part of the face, are closer together than the left and right frontalis muscles (Ch3, Ch4) and left and right zygomaticus major muscles (Ch7, Ch8), which are located on the left and right ends of the face. This is because they are closer together, making it difficult to move only one side of the face, and their movement is smaller. In other words, the left and right corrugator supercilii muscles and levator labii superioris alaeque nasi muscles have less influence on facial expressions compared to the left and right frontalis muscles and left and right zygomaticus major muscles.

[0139] Therefore, the facial expression code generation device 100M2 omits electrodes 120-2 and 120-6, and firmware 132C generates codes for Ch1 and Ch2 if the electromyogram acquired in Ch1 is above the threshold TH1, and generates codes for Ch5 and Ch6 if the electromyogram acquired in Ch5 is above the threshold TH1.

[0140] The codes generated for Ch2 and Ch6 are symmetrical codes representing displacements that are symmetrical to the displacements generated for Ch1 and Ch5 when viewed from the front of the face. Note that Ch3-Ch4 and Ch7-Ch8 are the same as those of the facial expression code generation device 100.

[0141] In this way, for example, an eyebrow-raising expression will result in both eyebrows raised instead of just one, but expressions with both eyebrows lowered, both eyes wide open, both cheeks raised, and both eyes closed can be generated with only one side.

[0142] Furthermore, by reducing the number of electrodes 120, the facial expression code generation device 100M1 can be simplified and its manufacturing costs reduced.

[0143] Furthermore, the facial expression code generation devices 100M3 to 100M5, described below using Figures 11C to 11E, have a configuration in which some of the electrodes 120-1 to 120-8 (see Figures 1A and 1B) are omitted as described below.

[0144] The facial expression code generation device 100M3, shown in Figure 11C, has a configuration in which the two electrode pieces of electrodes 120-1 and 120-2 of the facial expression code generation device 100 (see Figures 1A and 1B) are reduced to one, and the remaining two electrode pieces are assigned to electrode 120-1 of Ch1. The two electrode pieces assigned to electrode 120-1 are positioned symmetrically, one on the left half and one on the right half of the central part of the face in a front view. The facial expression code generation device 100M3 includes electrodes 120-1 and 120-3 to 120-8, but does not include electrode 120-2, but can generate codes for Ch1 to Ch8. For Ch3 to Ch8, it is the same as the facial expression code generation device 100.

[0145] Using electrodes 120-1, a differential signal of Ch1 can be obtained from the electromyographic signals of the left and right corrugator supercilii muscles. In the facial expression code generation device 100M3, when generating the code for Ch1, a code for Ch2 should also be generated. The code generated for Ch2 represents a displacement that is symmetrical to the displacement generated for Ch1 when viewed from the front of the face. The two codes for Ch1 and Ch2 are symmetrical codes that represent displacements that are symmetrical when viewed from the front of the face.

[0146] The facial expression code generation device 100M4, shown in Figure 11D, has a configuration in which the two electrode pieces of electrodes 120-5 and 120-6 of the facial expression code generation device 100 (see Figures 1A and 1B) are reduced to one, and the remaining two electrode pieces are assigned to electrode 120-5 of Ch5. The two electrode pieces assigned to electrode 120-5 are positioned symmetrically, one on the left half and one on the right half of the central part of the face in a front view. The facial expression code generation device 100M4 includes electrodes 120-1 to 120-5 and electrodes 120-7 to 120-8, but does not include electrode 120-6, but can generate codes for Ch1 to Ch8. For Ch1 to Ch4 and Ch7 to Ch8, it is the same as the facial expression code generation device 100.

[0147] Using electrodes 120-5, differential signals for Ch5 can be obtained from the electromyographic signals of the left and right levator labii superioris alaeque nasi muscles. In the facial expression code generation device 100M4, when generating the code for Ch5, a code for Ch6 should also be generated. The code generated for Ch6 represents a displacement that is symmetrical to the displacement generated for Ch5 in a frontal view of the face. The two codes for Ch5 and Ch6 are symmetrical codes that represent displacements that are symmetrical in a frontal view of the face.

[0148] The facial expression code generation device 100M5, shown in Figure 11E, has a configuration in which the two electrode pieces of electrodes 120-1 and 120-2 of the facial expression code generation device 100 (see Figures 1A and 1B) are reduced to one, and the remaining two electrode pieces are assigned to electrode 120-1 of Ch1. The two electrode pieces assigned to electrode 120-1 are positioned symmetrically, one on the left half and one on the right half of the central part of the face in a front view. Furthermore, the facial expression code generation device 100M5 has a configuration in which the two electrode pieces of electrodes 120-5 and 120-6 of the facial expression code generation device 100 (see Figures 1A and 1B) are reduced to one, and the remaining two electrode pieces are assigned to electrode 120-5 of Ch5. The two electrode pieces assigned to electrode 120-5 are positioned symmetrically, one on the left half and one on the right half of the central part of the face in a front view. The facial expression code generation device 100M5 includes electrodes 120-1, 120-3 to 120-5, and 120-7 to 120-8, but does not include electrodes 120-2 and 120-6, and is capable of generating codes for Ch1 to Ch8. For Ch3 to Ch4 and Ch7 to Ch8, it is the same as the facial expression code generation device 100.

[0149] In the facial expression code generation device 100M5, the differential signal of Ch1 is obtained from the electromyographic potential of the left and right corrugator supercilii muscles using electrode 120-1, and the differential signal of Ch5 is obtained from the electromyographic potential of the left and right levator labii superioris alaeque nasi muscles using electrode 120-5. In the facial expression code generation device 100M5, when generating the code for Ch1, it is sufficient to also generate a code for Ch2, and when generating the code for Ch5, it is sufficient to also generate a code for Ch6. The code generated for Ch2 represents a displacement that is symmetrical to the displacement caused by the code generated for Ch1 when viewed from the front of the face, and the code generated for Ch6 represents a displacement that is symmetrical to the displacement caused by the code generated for Ch5 when viewed from the front of the face. The two codes for Ch1 and Ch2 are symmetrical codes that represent displacements that are symmetrical to the

[0150] Here, we have described a method for generating two codes (symmetrical codes) for Ch1 and Ch2. However, the movement of the left and right corrugator supercilii muscles may be controlled with a single code using the two electrode pieces assigned to electrode 120-1 in Figure 11E. The area where the electrode piece on the -X direction side of the two electrode pieces assigned to electrode 120-1 is placed (corresponding to the left corrugator supercilii muscle) and the area where the electrode piece on the +X direction side is placed (corresponding to the right corrugator supercilii muscle) may be treated as a single area, and the movement of the left and right corrugator supercilii muscles may be controlled with a single code by combining them into a single channel (e.g., Ch1). In other words, the entire area where the electrode piece on the -X direction side is placed (corresponding to the left corrugator supercilii muscle) and the area where the electrode piece on the +X direction side is placed (corresponding to the right corrugator supercilii muscle) may be treated as a single area, and a single code representing the overall displacement may be generated. The same applies to the two codes (symmetrical codes) for Ch5 and Ch6.

[0151] <Wearing state of the facial expression code generation device 100 in the embodiment, and wearing state of the facial expression code generation devices 100M6, 100M7, and 100M8 in modified examples of the embodiment> Figure 12A shows an example of the wearing state of the facial expression code generation device 100 of the embodiment. Figures 12B, 12C, and 12D show examples of the wearing state of modified facial expression code generation devices 100M6, 100M7, and 100M8 of the embodiment.

[0152] The facial expression code generation device 100M6 shown in Figure 12B has a configuration that is a modified version of the goggle-type facial expression code generation device 100 shown in Figure 12A, adapted to a spectacle type. The facial expression code generation device 100M6 has a spectacle-type body 111B, temples 112B, and lenses 113B. The electrodes 120-1 to 120-8, the noise reduction electrode 125, and the signal processing unit 130 are the same as those of the facial expression code generation device 100. For example, electrodes 120-1, 120-3, 120-5, 120-7, and the noise reduction electrode 125 can be attached to the body 111B, and the signal processing unit 130 can be attached to the temple 112B.

[0153] The facial expression code generation device 100M7 shown in Figure 12C has a band-type main body 111C that extends horizontally above and below both eyes, and an ear-hook type band 112C. The electrodes 120-1 to 120-8, the noise reduction electrode 125, and the signal processing unit 130 are the same as those of the facial expression code generation device 100. For example, electrodes 120-1 to 120-8 and the noise reduction electrode 125 can be attached to the main body 111C, and the signal processing unit 130 can be attached to the band 112C.

[0154] The facial expression code generation device 100M8 shown in Figure 12D is an eye patch type that covers one eye and has a main body 111D for one eye (for example, for the left eye) and a headband 112D. For example, if the facial expression code generation device 100M1 shown in Figure 11A has a configuration that has only one electrode 120 on the left or right side, then the eye patch type facial expression code generation device 100M8 may be used. The electrodes 120-1 to 120-8, the noise reduction electrode 125, and the signal processing unit 130 are the same as those of the facial expression code generation device 100. For example, electrodes 120-1, 120-3, 120-5, 120-7 and the noise reduction electrode 125 can be attached to the main body 111D, and the signal processing unit 130 can be attached to the headband 112D.

[0155] The facial expression code generation device 100 is not limited to the goggle type shown in Figure 12A, but may have an appearance like the facial expression code generation devices 100M6 to 100M8 shown in Figures 12B to 12D, or it may have a different appearance not shown here.

[0156] <Effects> The facial expression code generation device 100 of this disclosure includes a plurality of electrodes 120 capable of contacting a plurality of parts of the face to acquire electromyographic potentials, and a code generation unit (firmware 132C) that generates a key code representing the displacement of at least one of the plurality of parts based on the plurality of electromyographic potentials acquired by the plurality of electrodes 120. In this way, based on the plurality of electromyographic potentials, a key code is generated that represents the displacement of at least one of the plurality of parts of the face, rather than data representing the facial expression of the entire face. The displacement of at least one of the plurality of parts of the face corresponds to the changed part of the facial muscle that forms the expression. Such a key code has a small data size and a low data processing burden.

[0157] Therefore, it is possible to provide an expression code generation device 100 that reduces the data processing load.

[0158] Furthermore, while there are devices that acquire facial expression data from images captured by a camera, such camera-based devices require the user to constantly face the camera in order to acquire facial expressions, making it difficult to move freely as the user cannot move away from the camera. In contrast, the facial expression code generation device 100 acquires facial expressions as long as it is worn on the face, so it can be used for outdoor streaming using VTuber applications, or while engaging in various activities such as cooking or exercising, allowing the user to move freely. Thus, the facial expression code generation device 100 offers high convenience.

[0159] Furthermore, VTubers can change the expressions of their avatars or characters by changing their own facial expressions. There is no need to operate a keyboard or other controls to change the expressions of the avatar or character; their own face can be used as the control unit for changing the expressions of the avatar or character. In addition, because there is no need to operate a keyboard or other controls, it is easy to change the expressions of the avatar or character even outdoors or in dark places, so it is easy to change the expressions of the avatar when streaming from outdoors or in dark places.

[0160] Furthermore, in online competitive games, players can exchange facial expressions (equivalent to emoticons) without using a controller, making it easy to communicate with teammates and visualize emotions when defeating or being defeated by an opponent. This is expected to make online competitive games more exciting.

[0161] Furthermore, devices that acquire facial expression data using an IR camera tend to be thick and large when the optical system is included. Also, the devices currently available on the market are expensive VR headset integrated units, making them unsuitable for applications that require communication without entering the metaverse space (such as online meetings). In contrast, the facial expression code generation device 100 can be implemented with a simple configuration that involves contacting electrodes 120 to the face, making it suitable for both applications in the metaverse space and applications that require communication without entering the metaverse space. The facial expression code generation device 100 reduces the burden and inconvenience for the user and can accommodate both applications in the metaverse space and applications that require communication without entering the metaverse space.

[0162] Furthermore, the facial expression code generation device 100 can convey realistic emotions to the other party when using an avatar or character in remote communication during online meetings.

[0163] The system may also include a wearable device (goggles 110) that holds multiple electrodes 120 and can be worn on the face with the electrodes 120 in contact with multiple parts of the face. By wearing the goggles 110 on the face, the multiple electrodes 120 are in contact with the face, freeing up both hands. For example, it is possible to change the facial expressions of an avatar or character while performing various activities such as cooking or exercising.

[0164] Furthermore, the goggles (goggles 110) may have openings that are positioned in front of both eyes when worn on the face. Since a clear view of the front is maintained even when wearing the goggles (goggles 110), it is possible to change the facial expressions of the avatar or character while performing various activities.

[0165] Furthermore, the code generation unit (firmware 132C) may have multiple thresholds corresponding to multiple body parts, and may generate multiple key codes by comparing multiple electromyographic signals acquired by multiple electrodes 120 with the multiple thresholds. By utilizing multiple thresholds corresponding to multiple body parts, changes in facial muscles in various parts of the face can be detected, enabling the reproduction of richer facial expressions.

[0166] Furthermore, the code generation unit (firmware 132C) may have multiple threshold levels and generate multiple types of key codes corresponding to multiple levels by comparing multiple electromyographic signals acquired by multiple sensors with multiple threshold levels. By utilizing multiple threshold levels, changes in facial muscles in various parts of the face can be detected more precisely, making it possible to reproduce even richer facial expressions.

[0167] Furthermore, each of the multiple electrodes 120 has two electrode pieces that output a differential signal representing the acquired electromyographic potential, and the code generation unit (firmware 132C) may generate a key code based on the multiple differential signals representing multiple electromyographic potentials acquired by the two electrode pieces of the multiple electrodes 120. By using differential signals, resistance to noise is improved, changes in facial muscles are easier to detect, and facial expressions can be reproduced stably.

[0168] Furthermore, the system may further include a noise reduction electrode 125 that contacts a predetermined area of ​​the face with multiple electrodes 120 in contact with multiple areas, and a cancellation signal output unit (common-mode noise calculation unit 135) that outputs a cancellation signal to the noise reduction electrode 125 that cancels out noise contained in at least one of the differential signals based on at least one of the multiple differential signals. Since common-mode noise that cannot be canceled out by differential signals can be reduced, it becomes easier to detect changes in facial muscles and facial expressions can be reproduced more stably.

[0169] Furthermore, the key codes may be other key codes than those assigned to the physical keys of a Japanese-layout JIS keyboard or a QWERTY-layout keyboard. The expression code generation device 100 can generate dedicated key codes based on the displacement of the electromyographic activity of facial muscles, and can stably change the expressions of avatars or characters displayed on a display device, or the expressions of masks, costumes, etc., that can physically change facial expressions, without generating unused commands.

[0170] Furthermore, the key codes may be those assigned to physical keys on a Japanese JIS keyboard or a QWERTY keyboard. Commands based on the key codes assigned to physical keys will control the operation of the smartphone 200 or PC, and allow for the changing of the facial expressions of avatars or characters. For example, it will be possible to change the facial expressions of avatars or characters while chatting with text input.

[0171] Furthermore, multiple locations may include areas where any multiple muscles are located, such as the left or right corrugator supercilii muscle, the left or right frontalis muscle, the left or right levator labii superioris alaeque nasi muscle, or the left or right zygomaticus major muscle. By detecting changes in any of these facial muscles, it becomes easier to capture changes in facial expressions, enabling the reproduction of even richer expressions.

[0172] Furthermore, the system may also include a code output unit that outputs the key code generated by the code generation unit (firmware 132C). An example of a code output unit is a BLE module 133. Also, if the signal processing unit 130 is connected to an electronic device such as a smartphone 200 via wired communication using a cable, an example of a code output unit is an output terminal that outputs the key code from the CPU 132. Including a code output unit makes it easier to transfer the key code to an external device such as a smartphone 200.

[0173] The facial expression generating devices 500A and 500B of this disclosure include the facial expression code generating device 100, a physical modification unit capable of physically changing the facial expression of the mask 400B, face mask, or mascot costume 400A, a storage unit that stores a plurality of facial expression files representing the movements of the plurality of parts of the face, an extraction unit that extracts a facial expression file corresponding to a key code generated by the code generation unit (firmware 132C) from the plurality of facial expression files, and a facial expression changing unit that changes the facial expression of the mask 400B, face mask, or mascot costume 400A by changing the configuration of the physical modification unit based on the facial expression file extracted by the extraction unit. The actuators of the mascot costume 400A or face mask, or the LEDs of the mask 400B, are examples of the physical modification unit. The control unit that controls the light emission color of the actuators of the mascot costume 400A or face mask, or the LEDs of the mask 400B, is an example of the storage unit, the extraction unit, and the facial expression changing unit.

[0174] In this way, based on multiple electromyographic signals, a key code is generated that represents the displacement of at least one of several facial regions, rather than data representing the expression of the entire face. Such key codes have a small data size and reduce the burden of data processing. Therefore, it is possible to provide facial expression generation devices 500A and 500B that reduce the burden of data processing.

[0175] Furthermore, the facial expression generators 500A and 500B can physically change the facial expression of the mask 400B, face mask, or mascot costume 400A using key codes generated by the facial expression code generator 100. For example, when a fan approaches a mascot costume, the person inside can change the expression of the mascot costume 400A by changing their own facial expression, allowing for a wide range of reactions beyond gestures to be easily performed through changes in facial expression. This is also true for the mask 400B and face mask.

[0176] The facial expression generation device 500C of this disclosure includes the facial expression code generation device 100, a display device (display 430) capable of displaying a predetermined avatar or character, a storage unit for storing a plurality of facial expression files, each representing the movement of a plurality of parts of the face, an extraction unit for extracting a facial expression file corresponding to a key code generated by a code generation unit (firmware 132C) from the plurality of facial expression files, and a facial expression modification unit for changing the facial expression of a predetermined avatar or character displayed on the display device based on the facial expression file extracted by the extraction unit. The control device 210 of the XR headset 400C is an example of the storage unit, the extraction unit, and the facial expression modification unit.

[0177] In this way, based on multiple electromyographic signals, a key code is generated that represents the displacement of at least one of several facial regions, rather than data representing the expression of the entire face. Such a key code has a small data size and reduces the burden of data processing. Therefore, it is possible to provide a facial expression generation device 500C that reduces the burden of data processing.

[0178] Since the facial expression generation device 500C integrates the facial expression code generation device 100 and the XR headset 400C, it is possible to provide a headset-type facial expression generation device 500C that reduces the data processing load.

[0179] The facial expression generation system 300 of this disclosure includes the facial expression code generation device 100, a storage unit for storing multiple facial expression files, each representing the movement of multiple parts of the face, an extraction unit for extracting facial expression files from the multiple facial expression files that correspond to codes generated by the code generation unit (firmware 132C), and a facial expression modification unit for changing the facial expression of a predetermined avatar or character displayed on a display device based on the facial expression files extracted by the extraction unit. As shown in Figure 3, when a smartphone 200 is used, the storage unit is the memory of the smartphone 200. An example of the extraction unit and the facial expression modification unit is application 210A.

[0180] In this way, based on multiple electromyographic signals, a key code is generated that represents the displacement of at least one of several facial regions, rather than data representing the expression of the entire face. Such a key code has a small data size and reduces the burden of data processing. Therefore, it is possible to provide an expression generation system 300 with reduced data processing load.

[0181] Furthermore, VTubers can change the expressions of their avatars or characters by changing their own facial expressions in the facial expression generation system 300. There is no need to operate a keyboard or other device to change the expressions of the avatars or characters; their own faces can be used as the control unit for changing the expressions of the avatars or characters. In addition, because there is no need to operate a keyboard or other device, it is easy to change the expressions of avatars or characters even outdoors or in dark places, so it is easy to change the expressions of avatars when streaming from outdoors or in dark places.

[0182] The facial expression code generation method of this disclosure is a facial expression code generation device that includes a plurality of electrodes 120 capable of acquiring electromyographic potentials by contacting a plurality of parts of the face, and generates a key code representing the displacement of at least one of the plurality of parts based on a plurality of electromyographic potentials acquired by the plurality of electrodes 120. In this way, a key code representing the displacement of at least one of the plurality of parts of the face is generated based on a plurality of electromyographic potentials, rather than data representing the facial expression of the entire face. Such a key code has a small data size and a low data processing burden.

[0183] Therefore, it is possible to provide a method for generating facial expression codes that reduces the burden of data processing.

[0184] The above describes exemplary embodiments of the facial expression code generation device, facial expression generation device, facial expression generation system, and facial expression code generation method of this disclosure. However, this disclosure is not limited to the specifically disclosed embodiments, and various modifications and changes are possible without departing from the scope of the claims.

[0185] The following additional information is disclosed regarding the embodiments described above. (Note 1) Multiple electrodes capable of acquiring electromyographic activity by contacting multiple areas of the face, A code generation unit generates a key code representing the displacement of at least one of the multiple sites based on multiple electromyographic potentials obtained by the multiple electrodes. A device that generates facial expression codes, including the above. (Note 2) The facial expression code generation device according to Appendix 1, further comprising a device that holds the plurality of electrodes and can be attached to the face while the plurality of electrodes are in contact with the plurality of parts. (Note 3) The aforementioned attachment is the facial expression code generating device described in Appendix 2, which has openings located in front of both eyes when attached to the face. (Note 4) The facial expression code generation device according to Appendix 1, wherein the code generation unit has a plurality of thresholds corresponding to each of the plurality of body parts, and generates the key code by comparing a plurality of electromyographic signals obtained by the plurality of electrodes with the plurality of thresholds. (Note 5) The facial expression code generation device according to Appendix 4, wherein the code generation unit has multiple levels of thresholds, and generates multiple types of key codes corresponding to the multiple levels by comparing multiple electromyographic signals acquired by the multiple electrodes with the multiple levels of thresholds. (Note 6) Each of the plurality of electrodes has two electrode pieces that output a differential signal representing the acquired electromyographic potential, The facial expression code generating device according to any one of Appendix 1 to Appendix 5, wherein the code generation unit generates the code based on a plurality of differential signals representing the plurality of electromyographic potentials acquired by the two electrode pieces of the plurality of electrodes, respectively. (Note 7) A noise reduction electrode that contacts a predetermined area of ​​the face while the plurality of electrodes are in contact with the plurality of areas, A cancellation signal output unit outputs a cancellation signal to the noise reduction electrode that cancels out noise contained in at least one of the plurality of differential signals based on that differential signal. The facial expression code generation device described in Appendix 6, further including the device described in Appendix 6. (Note 8) The aforementioned key code is a key code other than those assigned to the physical keys of a Japanese-layout JIS keyboard or a QWERTY-layout keyboard, as described in any one of the items in Appendix 1 to Appendix 7 of the facial expression code generating device. (Note 9) The aforementioned key code is a key code assigned to a physical key on a Japanese layout JIS keyboard or a QWERTY layout keyboard, as described in any one of the items in Appendix 7 of the facial expression code generating device. (Note 10) The facial expression code generating device according to any one of the following appendices 1 to 9, wherein the aforementioned multiple areas are the areas where any multiple muscles among the left or right corrugator supercilii muscle, the left or right frontalis muscle, the left or right levator labii superioris alaeque nasi muscle, or the left or right zygomaticus major muscle are located. (Note 11) The facial expression code generation device according to any one of the appendices 1 to 10, further comprising a code output unit that outputs the key code generated by the code generation unit. (Note 12) A facial expression code generation device described in any one of the items from Appendix 1 to Appendix 11, A physical modification unit that can physically change the facial expression of a mask, face covering, or costume, A storage unit that stores multiple facial expression files, each representing the movement of the aforementioned multiple parts of the face, An extraction unit extracts the facial expression file corresponding to the key code generated by the code generation unit from the plurality of facial expression files, A facial expression changing unit that changes the facial expression of the mask, the face mask, or the mascot costume by changing the configuration of the physical modification unit based on the facial expression file extracted by the extraction unit. A facial expression generating device, including... (Note 13) A facial expression code generation device as described in any one of the items from Appendix 1 to Appendix 12, A display device capable of displaying a predetermined avatar or character, A storage unit that stores multiple facial expression files, each representing the movement of the aforementioned multiple parts of the face, An extraction unit extracts the facial expression file corresponding to the key code generated by the code generation unit from the plurality of facial expression files, A facial expression changing unit that changes the facial expression of the predetermined avatar or character displayed on the display device based on the facial expression file extracted by the extraction unit. A facial expression generating device, including... (Note 14) A facial expression code generation device described in any one of the items from Appendix 1 to Appendix 11, A storage unit that stores multiple facial expression files, each representing the movement of the aforementioned multiple parts of the face, An extraction unit extracts the facial expression file corresponding to the key code generated by the code generation unit from the plurality of facial expression files, A facial expression changing unit that changes the facial expression of a predetermined avatar or character displayed on a display device based on the facial expression file extracted by the extraction unit. A facial expression generation system, including... (Note 15) In a facial expression code generation device that includes multiple electrodes capable of acquiring electromyographic potential by contacting multiple parts of the face, A method for generating facial expression codes, which generates a key code representing the displacement of at least one of the multiple body parts based on multiple electromyographic signals obtained by the multiple electrodes. [Explanation of symbols]

[0186] 100, 100M1~100M8 Expression Code Generator 110 Goggles (Example of a piece of equipment) 111 Main unit 112 Headbands 113 Shield 114 Cushions 120, 120-1~120-8 electrode 125 Noise reduction electrode 130 Signal Processing Unit 130A Battery 131 Sensor Amplifier 132 CPU 132C Firmware (Example of Code Generation Section) 133 BLE module (example of code output section) 134 Antenna 135 Common-mode noise calculation unit (an example of a cancellation signal output unit) 136 Amplifier 200 Smartphones 210 Control device 210A, 210B Applications 220 BLE modules 230 Display (Example of a display device) 300 Facial Expression Generation System 400A Mascot Costume 400B Mask 400C XR Headset 430 Display (Example of a display device) 500A, 500B, 500C facial expression generation device

Claims

1. Multiple electrodes capable of acquiring electromyographic activity by contacting multiple areas of the face, A code generation unit generates a key code representing the displacement of at least one of the multiple sites based on multiple electromyographic potentials obtained by the multiple electrodes. A device that generates facial expression codes, including the above.

2. The facial expression code generation device according to claim 1, further comprising a mounting device that can be attached to the face while holding the plurality of electrodes and with the plurality of electrodes in contact with the plurality of parts.

3. The facial expression code generating device according to claim 2, wherein the attachment has openings located in front of both eyes when attached to the face.

4. The facial expression code generation device according to claim 1, wherein the code generation unit has a plurality of thresholds corresponding to each of the plurality of body parts, and generates the key code by comparing a plurality of electromyographic signals obtained by the plurality of electrodes with the plurality of thresholds.

5. The facial expression code generation device according to claim 4, wherein the code generation unit has multiple levels of thresholds, and generates multiple types of key codes corresponding to the multiple levels by comparing multiple electromyographic signals acquired by the multiple electrodes with the multiple levels of thresholds.

6. Each of the plurality of electrodes has two electrode pieces that output a differential signal representing the acquired electromyographic potential, The facial expression code generation device according to any one of claims 1 to 5, wherein the code generation unit generates the key code based on a plurality of differential signals representing the plurality of electromyographic potentials acquired by the two electrode pieces of the plurality of electrodes, respectively.

7. A noise reduction electrode that contacts a predetermined area of ​​the face while the plurality of electrodes are in contact with the plurality of areas, A cancellation signal output unit outputs a cancellation signal to the noise reduction electrode that cancels out noise contained in at least one of the plurality of differential signals based on that differential signal. The facial expression code generation device according to claim 6, further comprising:

8. The facial expression code generating device according to claim 1, wherein the key code is a key code other than those assigned to the physical keys of a Japanese-layout JIS keyboard or a QWERTY-layout keyboard.

9. The facial expression code generating device according to claim 1, wherein the key code is a key code assigned to a physical key of a Japanese layout JIS keyboard or a QWERTY layout keyboard.

10. The facial expression code generating device according to claim 1, wherein the aforementioned multiple areas are areas where any multiple of the following muscles are located: the left or right corrugator supercilii muscle, the left or right frontalis muscle, the left or right levator labii superioris alaeque nasi muscle, or the left or right zygomaticus major muscle.

11. The facial expression code generation device according to claim 1, further comprising a code output unit that outputs the key code generated by the code generation unit.

12. The facial expression code generation device according to claim 1, A physical modification unit that can physically change the facial expression of a mask, face covering, or costume, A storage unit that stores multiple facial expression files, each representing the movement of the aforementioned multiple parts of the face, An extraction unit extracts the facial expression file corresponding to the key code generated by the code generation unit from the plurality of facial expression files, A facial expression changing unit that changes the facial expression of the mask, the face mask, or the mascot costume by changing the configuration of the physical modification unit based on the facial expression file extracted by the extraction unit. A facial expression generating device, including...

13. The facial expression code generation device according to claim 1, A display device capable of displaying a predetermined avatar or character, A storage unit that stores multiple facial expression files, each representing the movement of the aforementioned multiple parts of the face, An extraction unit extracts the facial expression file corresponding to the key code generated by the code generation unit from the plurality of facial expression files, A facial expression changing unit that changes the facial expression of the predetermined avatar or character displayed on the display device based on the facial expression file extracted by the extraction unit. A facial expression generating device, including...

14. The facial expression code generation device according to claim 1, A storage unit that stores multiple facial expression files, each representing the movement of the aforementioned multiple parts of the face, An extraction unit extracts the facial expression file corresponding to the key code generated by the code generation unit from the plurality of facial expression files, A facial expression changing unit that changes the facial expression of a predetermined avatar or character displayed on a display device based on the facial expression file extracted by the extraction unit. A facial expression generation system, including...

15. In a facial expression code generation device that includes multiple electrodes capable of acquiring electromyographic potential by contacting multiple parts of the face, A method for generating facial expression codes, which generates a key code representing the displacement of at least one of the multiple body parts based on multiple electromyographic signals obtained by the multiple electrodes.

Images