A data annotation system and method based on remote facial electromyography

Abstract

The present invention discloses a data annotation system and method based on remote facial electromyography. The system includes an electromyographic facial band, an electromyographic signal acquisition device, a player, a camera, a digital tube, and a host computer. The electromyographic facial band is worn around the subject's face and is provided with a plurality of spaced electromyographic electrodes. The output end of each electromyographic electrode is connected to the electromyographic signal acquisition device via a lead, and the electromyographic signal acquisition device transmits the collected electromyographic data to the host computer. The player is used to play an emotion-inducing video for the subject to watch, the camera is used to record the changes in the subject's face while watching the video, and the digital tube is used to synchronize the electromyographic signal acquisition with the video acquisition in time. The present invention uses an electromyographic facial band that surrounds the subject's face, eliminating the need to block the subject's facial area and preventing the recognition and detection of facial expressions. Based on the propagation of electromyographic signals from muscle sources, the movement state of facial muscles can be remotely acquired.

Description

Technical Field

[0001] The present invention belongs to the technical field of micro-expression, and in particular relates to a data annotation system and method based on remote facial electromyography. Background Art

[0002] Facial expression is the best way to express personal emotions. Judging a person's current inner emotional state through facial expressions is of great significance to current social life. In recent years, it has gradually become a hot topic of research in the intersection of psychology and computer science. However, for data-driven deep learning methods, the sample size of facial expression databases is obviously insufficient, which to a certain extent restricts the application of facial expression recognition and detection technology in real life. Expanding the sample size of the database has become an important issue that needs to be solved urgently. The process of collecting expression databases generally requires subjects to watch videos with emotional colors to induce the subject's facial expressions and record them through a camera. After that, trained annotators observe the video frame by frame to determine the starting frame, peak frame and end frame of the facial action. This method is very time-consuming and labor-intensive, and is also an important factor related to the sample size of the expression database. Therefore, the present invention proposes to use facial electromyography signals to assist annotators in semi-automatic annotation of the expression database.

[0003] Facial electromyography (EMG) is the electrical signal generated by muscle cells when facial muscles contract. Any facial movement is typically the result of the coordinated action of one or more muscles. Therefore, changes in facial EMG signals can be used to determine the movement state of the relevant muscles, and thus the start, peak, and end frames of the facial expression. Currently, facial EMG is typically acquired by placing measurement electrodes on the surface of the facial muscle source to maximize signal amplitude. However, due to their size and weight, these electrodes can hinder facial muscle movement. From a computer vision perspective, this can also affect facial expression recognition and detection, hindering the annotation of micro-expressions. Summary of the Invention

[0004] In order to avoid the impact of facial expression recognition and detection during facial electromyography acquisition, the present invention provides a data annotation system and method based on remote facial electromyography, which uses a remote method to obtain the movement signals of facial muscles without blocking the face and can obtain the movement status of facial muscles.

[0005] The specific scheme adopted is as follows:

[0006] On the one hand, the present invention provides a data labeling system based on remote facial electromyography, which includes an electromyographic face belt, an electromyographic signal acquisition device, a player, a camera device, a digital tube and a host computer. The electromyographic face belt is worn around the face of the subject and is provided with a plurality of electromyographic electrodes arranged at intervals. The electromyographic electrodes are in contact with the skin of the subject, and the output end of each of the electromyographic electrodes is connected to the electromyographic signal acquisition device through a lead. The electromyographic signal acquisition device sends the collected electromyographic data to the host computer; the player is used to play emotion-inducing videos for the subject to watch, and the camera device is used to record the changes in the face of the subject while watching the video; the digital tube is used to keep the electromyographic signal acquisition of the electromyographic signal acquisition device and the video acquisition of the camera device synchronized in time.

[0007] Furthermore, the myoelectric facial band is a ring-shaped flexible narrow facial band structure, and is formed with a number of circular holes with the same aperture distributed at intervals along its length direction. Electrode caps for inserting myoelectric electrodes are installed in the circular holes, and the inner side of the electrode cap is coated with a conductive medium for fitting the subject's face with the myoelectric electrodes.

[0008] Preferably, the myoelectric electrodes are Ag / AgCl electrodes.

[0009] Furthermore, the myoelectric face band is provided with a left-right symmetrical upper area, a middle area and a lower area along the surrounding area of ​​the subject's face, and the myoelectric electrodes installed in the upper area of ​​the myoelectric face band are respectively located on the motor ends of the subject's internal and external frontal muscles and the extension line of the corrugator supercilii muscle, and the corresponding reference electrode I is located at the forehead of the skull; the myoelectric electrodes installed in the middle area of ​​the myoelectric face band are respectively located at the edge of the marginal part of the orbicularis oculi muscle and the motor end of the zygomatic muscle of the subject, and the corresponding reference electrode II is located above the ear of the skull; the myoelectric electrodes installed in the lower area of ​​the myoelectric face band are distributed along the edge of the subject's mandible, and the corresponding reference electrode III is located at the mastoid process of the ear.

[0010] Further preferably, the myoelectric electrodes in the same area on the myoelectric face band adopt a monopolar configuration, while the myoelectric electrodes between adjacent areas adopt a bipolar configuration.

[0011] Furthermore, the electromyographic signal acquisition device includes: an electromyographic signal acquisition module, a power management module, a controller and a wireless transmission module; the electromyographic signal acquisition module is connected to all the electromyographic electrodes, and is used to obtain the electromyographic analog signal of the surface around the subject's face, and perform signal conditioning and analog-to-digital conversion, and output the converted digital electromyographic signal to the controller through the SPI interface; the controller is used to control the electromyographic signal acquisition module and the wireless transmission module, and forward the processed electromyographic data to the wireless transmission module, and the wireless transmission module sends the electromyographic data wirelessly to the host computer for display; the power management module supplies power to the electromyographic signal acquisition module, the controller and the wireless transmission module.

[0012] Furthermore, the electromyographic signal acquisition module is a 32-channel electromyographic acquisition module composed of four analog-to-digital conversion chips connected in series in daisy-chain mode. It has a built-in right leg drive circuit. The differential input end of each analog-to-digital conversion chip is provided with an RC passive low-pass filter, and a voltage regulator diode is connected in parallel at its front-stage input end.

[0013] On the other hand, the present invention also provides a data annotation method based on remote facial electromyography, the annotation method comprising the following steps:

[0014] Step 1: The subject wears the myoelectric faceband equipped with myoelectric electrodes in an interactive scene;

[0015] Step 2, reset the digital tube to 0;

[0016] Step 3: Turn on the video recording mode of the camera device to record the changes in the subject's facial expressions throughout the whole process;

[0017] Step 4: Turn on the electromyographic signal acquisition device and establish a data transmission connection with the host computer;

[0018] Step 5: Open the player and play the emotion-inducing video for the subjects to watch;

[0019] Step 6: After the emotion-inducing video is played, the collected electromyographic data is processed to obtain a linear envelope of the electromyographic signal;

[0020] Step 7, combining the linear envelope of the electromyographic signal to determine the time point when the electromyographic signal fluctuates;

[0021] Step 8: According to the time point when the electromyographic signal appears, directly locate the facial movements near the corresponding moment in the recorded facial expression video of the subject and annotate the relevant data.

[0022] The technical solution of the present invention has the following advantages:

[0023] A. The system provided by the present invention adopts a method of recording the video of the subject while collecting electromyographic signals. The electromyographic facial band used is a flexible belt structure that is fixed around the subject's face. It does not need to block the subject's facial area, and therefore will not affect the recognition and detection of facial expressions. At the same time, the electromyographic electrodes installed on the electromyographic facial band adopt a remote method to obtain the movement signals of the facial muscles. The electromyographic electrodes are not placed directly on the surface of the facial muscle source. Instead, all the electromyographic electrodes are distributed around the face. When the facial muscle source moves, the electromyographic signals generated will propagate to the adjacent muscles and then propagate to the remote end. Therefore, the movement state of the facial muscles can be obtained.

[0024] B. The myoelectric face band in the system provided by the present invention is equipped with a number of bilaterally symmetrical myoelectric electrodes. After the subject wears it, the myoelectric electrodes correspond to the upper, middle and lower areas around the face of the subject, which are symmetrical on the left and right. The myoelectric electrodes in the same area of ​​the myoelectric face band that fits the area around the subject's face adopt an independent monopole configuration, while the myoelectric electrodes between adjacent areas form a bipolar configuration. The present invention adopts a single-stage configuration in the same area to collect the desired "crosstalk" signal of the area as much as possible, and adopts a bipolar configuration between areas. At the same time, the number of channels is not reduced, and the areas have lower interference and crosstalk sensitivity.

[0025] C. The present invention uses a digital tube to synchronize video data with electromyographic data in time. When electromyographic and video acquisition is completed and data analysis is performed, based on the temporal consistency, by slowing down the video, it can be found that when the subject makes certain facial movements, a large fluctuation can be captured in the electromyographic signal at the same time. Even very subtle facial movements that are almost invisible to the naked eye can be captured by the electromyographic signal. Therefore, using the system and method provided by the present invention for lie identification is a feasible solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the specific embodiments of the present invention, the following will briefly introduce the drawings required for use in the specific embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0027] Figure 1 This is a structural diagram of the data annotation system provided by the present invention;

[0028] Figure 2 yes Figure 1 Schematic diagram of the structure of the electromyographic signal acquisition device;

[0029] Figure 3 This is a schematic diagram of the structure of the myoelectric facial belt provided by the present invention;

[0030] Figure 4a It is a diagram of a monopolar electrode configuration;

[0031] Figure 4b is a diagram of a bipolar electrode configuration;

[0032] Figure 4c It is a diagram of the grouped monopole configuration provided by the present invention;

[0033] Figure 5a It is a diagram of the electrode configuration in the upper left area provided by the present invention;

[0034] Figure 5b It is a diagram of the electrode configuration in the left middle area provided by the present invention;

[0035] Figure 5c is a diagram illustrating the electrode configuration in the lower left region provided by the present invention;

[0036] Figure 6 It is a pre-low-pass filter circuit in the electromyographic signal acquisition module;

[0037] Figure 7 The electromyographic signal acquisition module provided by the present invention is a daisy chain configuration structure diagram formed by four ADS1299 chips;

[0038] Figure 8 This is a diagram of the data annotation method based on remote facial electromyography provided by the present invention;

[0039] Figure 9 This is a scene interaction diagram when wearing an electromyography mask for data annotation;

[0040] Figure 10 It is a diagram of the electromyographic signal collected by the electromyographic signal acquisition device.

[0041] The meanings of the icons are as follows:

[0042] 1a-reference electrode I, 1b-myoelectric electrode I; 2a-reference electrode II, 2b-myoelectric electrode II; 3a-reference electrode III, 3b-myoelectric electrode III; 4-myoelectric facial band, 4a-round hole; 5-electrode cap; 6-myoelectric signal acquisition device. DETAILED DESCRIPTION

[0043] The technical solutions of the present invention will be described clearly and completely below with reference to the accompanying drawings. It is obvious that the embodiments described are only some of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort shall fall within the scope of protection of the present invention.

[0044] like Figure 1 As shown, the present invention provides a data labeling system based on remote facial electromyography, including an electromyographic face belt, an electromyographic signal acquisition device, a player, a camera device, a digital tube and a host computer. The electromyographic face belt is worn around the face of the subject and is provided with a plurality of electromyographic electrodes arranged at intervals. The electromyographic electrodes are in contact with the skin of the subject. The output end of each electromyographic electrode is connected to the electromyographic signal acquisition device through a lead. The electromyographic signal acquisition device sends the collected electromyographic data to the host computer; the player is used to play emotion-inducing videos for the subject to watch, and the camera device is used to record changes in the face of the subject while watching the video; the digital tube is used to maintain the temporal synchronization between the electromyographic signal acquisition device and the video acquisition of the camera device.

[0045] like Figure 3 As shown, the myoelectric facial band 4 is a ring-shaped flexible narrow facial band structure, and is formed with a number of circular holes 4a with the same aperture that are spaced apart along its length. The preferred number of circular holes 4a in the present invention is 32, and an electrode cap 5 for inserting myoelectric electrodes is installed in the circular holes 4a. The myoelectric electrodes here are preferably Ag / AgCl electrodes, which can ensure that the measured values ​​are less sensitive to the relative movement between the electrode surface and the skin. When collecting data, the inner side of the electrode cap 5 is coated with a conductive medium for fitting the subject's face with the myoelectric electrodes and reducing the impedance between the skin and the electrodes. The myoelectric facial band worn on the subject's face used in the present invention can be fixed to the head by a rope or the like, such as Figure 9 As shown, the electromyographic signals on the skin surface are collected after stabilization.

[0046] Traditional electromyographic electrode configuration schemes are generally divided into single-pole and bipolar. The single-pole configuration is to connect the positive input of each differential amplifier to an electrode close to the muscle tissue to be measured; the negative input of all differential amplifiers is connected to one electrode, called the reference electrode, which is usually an area with less muscle and more bones (such as the forehead). The single-pole configuration actually measures the potential relative to the reference electrode, such as Figure 4a As shown, it is easy to collect crosstalk signals from other muscle sources during the measurement process. Figure 4b As shown, the bipolar configuration measures the potential difference between two electrodes. The electrical signal generated in the distant muscle is weakened by the differential, and the bipolar configuration is less sensitive to interference and crosstalk.

[0047] Since the area around the face is relatively small, if only a single-stage configuration is used, although more EMG electrodes can be arranged around the face, the signal acquisition quality is poor; if only a bipolar configuration is used, compared with the single-stage configuration, although the signal quality can be better guaranteed, the number of EMG electrodes is reduced by half. The present invention proposes a grouped single-stage configuration scheme, such as Figure 4cAs shown, based on the anatomical structure of facial muscles and the correspondence between muscles and facial action units (AUs), the present invention divides the face into six regions: three on the left and three on the right, distributed symmetrically. Within each region, a single-stage configuration is used to maximize the capture of the desired signal, which is the "crosstalk" of that region. Figure 4c The figure shows the electrode grouping configuration of three areas on one side of the face. Each group obtains the "crosstalk" of the area where it is located and is not affected by its adjacent areas, so the collected electromyographic signals are more accurate.

[0048] The electrodes in the three areas on the left side of the face are arranged as follows: Figure 5a 、 5b and 5c.

[0049] The upper area of ​​the face includes the frontalis and corrugator muscles, corresponding to AU1, AU2, and AU4. The myoelectric electrodes I1b are located on the motor terminals of the internal and external frontalis muscles and the extension line of the corrugator muscles, and are evenly distributed. The reference electrode I1a is located on the forehead of the skull, as shown in Figure 2. Figure 5a As shown in Figure 1. The middle area of ​​the face includes the orbicularis oculi and zygomaticus muscles, corresponding to AU6, AU7, AU12, AU42, AU43, and AU44. The myoelectric electrode II2b is located at the edge of the orbicularis oculi muscle and the motor end of the zygomaticus muscle, respectively. The reference electrode II2a is located above the ear of the skull, as shown in Figure 1. Figure 5b The middle area of ​​the face includes the orbicularis oris, depressor anguli oris, and mentalis muscles, corresponding to AU16, AU17, AU22, AU23, AU24, AU25, and AU28. The electromyographic electrodes III3b are evenly distributed along the edge of the mandible, and the reference electrode III3a is located at the mastoid process of the ear, as shown in Figure 1. Figure 5c shown.

[0050] In addition, if Figure 2 As shown, the electromyographic signal acquisition device in the present invention includes: an electromyographic signal acquisition module, a power management module, a controller and a wireless transmission module; the electromyographic signal acquisition module is connected to all the electromyographic electrodes, and is used to obtain the electromyographic analog signal of the surface around the subject's face, and perform signal conditioning and analog-to-digital conversion, and output the converted digital electromyographic signal to the controller through the SPI interface; the controller is used to control the electromyographic signal acquisition module and the wireless transmission module, and forward the processed electromyographic data to the wireless transmission module, and the wireless transmission module sends the electromyographic data wirelessly to the host computer for display; the power management module supplies power to the electromyographic signal acquisition module, the controller and the wireless transmission module.

[0051] The controller is preferably composed of a microcontroller STM32F429, and the ESP32C3-mini1 serves as a wireless transmission module to wirelessly transmit the data forwarded by the controller to the host computer (computer). The power management module uses a rechargeable polymer lithium battery to power each functional unit and manage power consumption.

[0052] After experimental testing, it was found that the more channels used in the electromyographic signal acquisition module, the more accurate the information that can be obtained. Therefore, the invention uses four ADS1299 analog-to-digital conversion chips in a daisy-chain configuration to form a 32-channel multi-channel data acquisition module. It has a built-in right leg drive circuit. The differential input end of each analog-to-digital conversion chip is equipped with an RC passive low-pass filter, and a voltage-stabilizing diode is connected in parallel at its front-stage input end. Finally, the electromyographic signal acquisition results are displayed and analyzed through Matlab code.

[0053] Since the original electromyographic signal is weak and easily interfered by the high-frequency signal in the experimental environment, a passive low-pass filter composed of RC is designed at the differential input end of the ADS1299 signal acquisition to filter the electromyographic signal. Generally, the frequency of the electromyographic signal is in the range of 20Hz-450Hz, so the resistance and capacitance values ​​are selected as 4.99k ohms and 4.7nF respectively. At the same time, in order to prevent electrostatic breakdown from damaging the chip, the input signal needs to be voltage-limited. A voltage-stabilizing diode is connected in parallel to the front-end input end and connected to the analog ground. The circuit design is as follows: Figure 6 shown.

[0054] The present invention selects ADS1299 as the analog-to-digital conversion chip, whose short-circuit noise is only 1uVpp, supports daisy chain mode and has a built-in right leg drive circuit. The daisy chain mode can minimize the occupation of MCU pins and greatly improve the scalability of the device. The right leg drive circuit can suppress common-mode interference to a certain extent when collecting electromyography. In addition, the ADS1299 chip also has 8 low-noise 24-bit ∑-△ADCs, built-in EMI filters, programmable gain amplifiers (PGA), input multiplexers, clock oscillators and bias amplifier circuits with internal test sources and lead-off detection circuits, and the data sampling rate ranges from 250SPS-16kSPS. In the present invention, four ADS1299 chips are connected in series in a daisy chain mode to form a 32-channel electromyography acquisition module, such as Figure 7As shown in the figure. In daisy-chain mode, the four chips share the following pins: START, CS, SCLK, DIN, and CLK. START provides a start signal for the chip, CS is the chip select signal, and SCLK is the clock signal used when the MCU and ADS1299 communicate using the SPI bus. The MCU writes data to the ADS1299 registers via DIN to control the chip's operating mode. CLK is also a clock signal used to synchronize the four ADS1299 chips. In this invention, a 2.048 MHz active crystal oscillator is introduced to provide the CLK signal. DRDY is an interrupt pin; when the acquisition chip generates data, the DRDY pin level changes. The DOUT pin is used to output the collected data. When four ADS1299 chips are connected in daisy-chain mode, the DAISY_IN pin of the fourth ADC chip is unused and grounded. The collected data is output from the DOUT pin and sent via a wire to the DAISY_IN pin of the third ADC chip. Similarly, the data collected by the last four chips is output to the MCU via the DOUT pin of the first ADS1299 chip.

[0055] The controller is the core component that makes the entire electromyographic signal acquisition device run, and it is mainly responsible for two parts: First, the controller sends instructions via SPI to control the registers of the ADS1299 chip to implement operations such as the operating mode, sampling rate, and gain of each channel. Second, it processes the data collected by the ADS1299 chip, encapsulates it in the format of TCP / IP data frames, and sends it to the wireless transmission module via SPI. Each frame of processed data is sent to the PC for real-time display of the waveform. In the present invention, the communication method between the controller and the ADS1299 and the wireless transmission module is all via SPI. In addition, due to the use of more ADC chips, a larger data buffer space is required during the data acquisition process. Based on the above considerations, the present invention selects the STM32F429 core board as the controller. It contains a higher-performance Cortex M4 core with an operating frequency of up to 180Mhz, 256KB of on-chip SRAM, 6 SPIs, two DMA controllers (a total of 16 channels), etc. In addition, the board has 32MB of SDRAM and is compact, only 65mm*45mm, making it convenient for application in various projects. Meet the needs of data cache space and fast data conversion.

[0056] The wireless transmission module used in this invention is a common and necessary method for achieving remote data transmission. The wireless transmission module has two functions:

[0057] 1) Receive and respond to commands from the controller;

[0058] 2) The data processed by the controller is forwarded to the PC for real-time display of the waveform.

[0059] The formula for calculating the amount of data generated per second by the electromyographic signal acquisition device in this system is as follows:

[0060] Data = N chonnel ×fs×R

[0061] Data is the amount of data collected by the device per second, in bits; N channel is the number of channels; fs is the sampling frequency; R is the amount of data generated by one sampling of each channel, in bits. Since ADS1299 uses a 24-bit ADC, but converts the 24 bits of each channel into 32 bits during data processing to facilitate subsequent data conversion, the value of R is 32. According to the above formula, when there are 32 channels and a sampling rate of 1KHz, the amount of data collected by the device per second is 1.024MKbit. In actual transmission, there are also overheads such as the frame header, increment bit, check bit in the data frame structure, and packet retransmission in TCP transmission. The required transmission rate will be higher. With a sampling rate of 1KHz for 32 channels, the actual transmission speed is about 1.2Mbit / s. The present invention uses ESP32C3-MINI1 as a wireless transmission module. In actual applications, the maximum stable data transmission speed of this wireless transmission module can reach 3Mbit / s.

[0062] In addition, the power management module preferably uses a polymer lithium battery to power each module. This approach improves portability and enhances system immunity. The hardware circuit board uses a 3.7V input voltage. The wireless transmission module ESP32 is powered by 3.3V, the controller STM32 is powered by 3.3V, and the lithium battery charging voltage is 5V. The analog front end requires separate analog and digital power supplies. The power supply circuit is designed according to the design requirements of each module. When the lithium battery is connected to the circuit, it generates a 3.7V voltage. The HX4002 boost chip generates a 5V voltage, and the LDO-TPS73201 generates a 3.3V power supply. The 5V and 3.3V power supplies are filtered and de-noised using resistors and capacitors, and then converted into analog and digital power supplies to power the controller chip. When the lithium battery is low, it can be charged by connecting to a USB port via the TP4056 charging circuit.

[0063] After powering on the system, the main program initializes the system, setting registers to daisy-chain mode to ensure the four chips can be connected in series. It also configures the system to use an external clock signal, introducing a crystal oscillator to synchronize each chip. The sampling rate for each channel is 1kHz, the gain is 24, and the resolution is 24-bit. The ESP32 is set to STA mode, connected to the local area network, and connected to the remote server. To prevent data loss, the digital EMG signals converted by the ADC are placed in a circular queue rather than forwarded directly via the Wi-Fi module. Finally, interrupts are enabled for data acquisition. When the ADS1299 chip is ready for data, it sets the DRDY pin low, generating a hardware interrupt in the microcontroller. This interrupt continuously reads 32 channels of EMG data via SPI, and stores the data in the queue buffer if the queue is full. Once a connection is established between the wireless transmission module and the remote host computer, if there is data in the queue, it is popped from the queue, encapsulated into a TCP packet with a header, packet sequence number, and parity bit, and then sent to the wireless transmission module via SPI for forwarding to the remote host computer.

[0064] like Figure 8 and Figure 9 As shown, the present invention also provides a data annotation method based on remote facial electromyography, which includes the following steps:

[0065]

S01

[0066]

S02

[0067] [S03] Turn on the video recording mode of the camera device to record the changes in the subject's facial expressions throughout the whole process;

[0068]

S04

[0069]

S05

[0070] [S06] After the emotion-inducing video is played, the collected electromyographic data is processed to obtain the linear envelope of the electromyographic signal;

[0071]

S07

[0072]

S08

[0073] During the EMG data acquisition process, the EMG and video data are synchronized using a digital tube. The time consumed by EMG data acquisition is 1ms. 19 sampling data are combined into a data packet, and each data packet is sent to the host computer via Wi-Fi. Every time the host computer receives four data packets, it increments the mark by 1 and sends the data to the digital tube via the serial port, causing the number on the digital tube to increase by 1. The camera records at a frame rate of 60 frames, and the digital changes on the digital tube can be observed (for reference). Furthermore, the camera recording time precedes the time when the EMG signal occurs. Once the host computer receives the data, it sends a mark to the digital tube, causing it to change from dark to bright and begin displaying the number.

[0074] After the EMG data is collected, the collected EMG data is band-pass filtered at 20-450Hz by MATLAB, and then the DC is removed and full-wave rectified to obtain the linear envelope of the signal. Figure 10 As shown in the figure, the horizontal axis is time and the vertical axis is the channel number. From the figure, we can see the change amplitude of the curve in each channel over time. Figure 10 The changes in the EMG signal when producing eyebrow raising, frowning, grinning, and surprise movements can be clearly seen in the figure. The envelope signal clearly shows the moments when the EMG signal fluctuates, allowing for quick location of the waveform's start time t1 and end time t2. Since the time t3, when the digital tube goes from dark to bright, is determined in the video, the start and end times of the subject's facial movements in the video are t1+t3 and t2+t3, respectively. This allows the annotator to directly locate the corresponding moment in the video, determine the subject's specific facial movements, and annotate the relevant data, significantly reducing the annotator's workload.

[0075] Anything not described in the present invention is applicable to the prior art.

[0076] Obviously, the above embodiments are merely examples for clarity of explanation and are not intended to limit the implementation methods. Those skilled in the art will readily appreciate that other variations or modifications based on the above descriptions are possible. It is not necessary and impossible to enumerate all implementation methods here. Obvious variations or modifications arising therefrom remain within the scope of protection of the present invention.

Claims

1. A data annotation system based on remote facial electromyography, characterized in that: The system includes an electromyographic facial band, an electromyographic signal acquisition device, a player, a camera, a digital tube, and a host computer. The electromyographic facial band is worn around the face of the subject and is provided with a plurality of electromyographic electrodes arranged at intervals. The electromyographic electrodes are in contact with the subject's skin. The output end of each electromyographic electrode is connected to the electromyographic signal acquisition device via a lead. The electromyographic signal acquisition device sends the collected electromyographic data to the host computer. The player is used to play an emotion-inducing video for the subject to watch, and the camera is used to record the changes in the subject's face while watching the video. The digital tube is used to maintain the temporal synchronization between the electromyographic signal acquisition device and the video acquisition by the camera. The myoelectric face band is provided with a bilaterally symmetrical upper area, middle area and lower area along the surrounding area of ​​the subject's face. The myoelectric electrodes I installed in the upper area of ​​the myoelectric face band are respectively located on the motor ends of the subject's internal and external frontal muscles and the extension line of the corrugator supercilii muscle, and the corresponding reference electrode I is located at the forehead of the skull; the myoelectric electrodes II installed in the middle area of ​​the myoelectric face band are respectively located at the edge of the marginal part of the orbicularis oculi muscle and the motor end of the zygomatic muscle of the subject, and the corresponding reference electrode II is located above the ear of the skull; the myoelectric electrodes III installed in the lower area of ​​the myoelectric face band are distributed along the edge of the subject's mandible, and the corresponding reference electrode III is located at the mastoid process of the ear.

2. The data annotation system based on remote facial electromyography according to claim 1 is characterized in that: The myoelectric facial band is a flexible narrow annular facial band structure, with several circular holes of the same aperture distributed at intervals along its length. Electrode caps for inserting myoelectric electrodes are installed in the circular holes. The inner side of the electrode cap is coated with a conductive medium for fitting the subject's face with the myoelectric electrodes.

3. The data annotation system based on remote facial electromyography according to claim 2 is characterized in that: The myoelectric electrodes are Ag / AgCl electrodes.

4. The data annotation system based on remote facial electromyography according to claim 1, characterized in that: The myoelectric electrodes in the same area on the myoelectric face belt adopt a monopolar configuration, while the myoelectric electrodes between adjacent areas adopt a bipolar configuration.

5. The remote facial electromyography-based data annotation system according to any one of claims 1 to 4, characterized in that: The electromyographic signal acquisition device includes: an electromyographic signal acquisition module, a power management module, a controller and a wireless transmission module; the electromyographic signal acquisition module is connected to all the electromyographic electrodes, is used to obtain the electromyographic analog signal from the surface around the subject's face, and performs signal conditioning and analog-to-digital conversion, and outputs the converted digital electromyographic signal to the controller through the SPI interface; the controller is used to control the electromyographic signal acquisition module and the wireless transmission module, and forward the processed electromyographic data to the wireless transmission module, and the wireless transmission module sends the electromyographic data to the host computer in a wireless manner for display; the power management module supplies power to the electromyographic signal acquisition module, the controller and the wireless transmission module.

6. The data annotation system based on remote facial electromyography according to claim 5 is characterized in that: The electromyographic signal acquisition module is a 32-channel electromyographic acquisition module composed of four analog-to-digital conversion chips connected in series in daisy-chain mode. It has a built-in right leg drive circuit. The differential input end of each analog-to-digital conversion chip is provided with an RC passive low-pass filter, and a voltage regulator diode is connected in parallel at its front-stage input end.

7. A data annotation method based on remote facial electromyography, characterized in that: The annotation method adopts the remote facial electromyography-based data annotation system according to claim 1, comprising the following steps: Step 1: The subject wears the myoelectric faceband equipped with myoelectric electrodes in an interactive scene; Step 2, reset the digital tube to 0; Step 3: Turn on the video recording mode of the camera device to record the changes in the subject's facial expressions throughout the whole process; Step 4: Turn on the electromyographic signal acquisition device and establish a data transmission connection with the host computer; Step 5: Open the player and play the emotion-inducing video for the subjects to watch; Step 6: After the emotion-inducing video is played, the collected electromyographic data is processed to obtain a linear envelope of the electromyographic signal; Step 7, combining the linear envelope of the electromyographic signal to determine the time point when the electromyographic signal fluctuates; Step 8: According to the time point when the electromyographic signal appears, directly locate the facial movements near the corresponding moment in the recorded facial expression video of the subject and annotate the relevant data.

Images