A facial expression recognition method and system based on sound perception

Abstract

The present invention belongs to the field of facial expression recognition technology, and specifically relates to a facial expression recognition method and system based on sound perception. The method uses acoustic signals as a sensing medium, utilizes a microphone array to capture sound reflection signals, and generates a spectrogram containing facial features through signal processing. Features are extracted from the spectrogram, and a deep learning network is used to classify facial expressions to achieve expression recognition. The facial expression recognition method provided by the present invention can effectively achieve facial expression recognition even with relatively small amounts of data, without relying on expensive hardware equipment, utilizing commercially available speakers and microphone arrays.

Description

Technical Field

[0001] The present invention belongs to the technical field of facial expression recognition, and in particular relates to a facial expression recognition method and system based on sound perception. Background Art

[0002] With the continuous advancement of computer technology, users' requirements for applications are also increasing. They not only expect them to complete their intended functions, but also to be more intelligent and user-friendly. By recognizing user emotions, intelligent systems can interact with users more naturally and efficiently, improving the overall experience. Facial expressions, as the most direct way to express emotions, can clearly reflect an individual's psychological and physiological changes, and have broad application prospects in various human-computer interaction systems. Facial expression recognition (FER), as a key research direction in the computer field, has attracted widespread attention from academia and industry in recent years due to its wide application value in social robotics, intelligent healthcare, driver fatigue monitoring, and numerous human-computer interaction systems.

[0003] Existing facial expression recognition systems are primarily based on three technical solutions: wearable devices, cameras, and wireless signals. While wearable devices offer high recognition accuracy, they require users to wear additional equipment for extended periods, making them invasive and potentially uncomfortable, impacting the user experience. Many studies have explored using cameras to capture facial images or videos for precise, non-contact monitoring. However, these systems are sensitive to lighting conditions and pose potential privacy risks. In recent years, wireless signals (such as WiFi, millimeter waves, and ultrasound) have provided new approaches for facial expression recognition, demonstrating significant advantages in environmental adaptability, contactless sensing, and privacy protection. Despite this, WiFi and millimeter wave technologies remain difficult to achieve widespread adoption due to their high equipment costs. Summary of the Invention

[0004] To address the above technical issues, the present invention provides a method and system for facial expression recognition based on sound perception. Using acoustic signals as the perception medium, a microphone array captures sound reflection signals. A series of signal processing steps generate a spectrogram that directly captures facial features. Key features are then extracted from the spectrogram, and efficient expression recognition is achieved using a deep learning network. Compared with traditional visual methods, the method provided by the present invention not only effectively avoids the risk of privacy leakage but also overcomes the interference of ambient lighting on recognition performance, providing new possibilities for the practical application of facial expression recognition (FER) technology.

[0005] The present invention is achieved through the following technical solutions:

[0006] A facial expression recognition method based on sound perception uses acoustic signals as the perception medium, utilizes a microphone array to capture sound reflection signals, generates a spectrogram containing facial features through signal processing, extracts features from the spectrogram, and uses a deep learning network to classify facial expressions to achieve expression recognition.

[0007] Furthermore, the method comprises the following steps:

[0008] (1) Signal transmission: The speaker transmits an interval FMCW waveform sound signal. The sound signal is reflected after reaching the face, generating a sound reflection signal, which is received by the microphone array.

[0009] (2) Signal processing: The signal processing process includes an audio frame segmentation sub-step, a distance measurement sub-step, a spectrogram generation sub-step, and a coordinate system conversion sub-step;

[0010] The audio frame segmentation sub-step is used to extract the facial reflected sound signal from the recording of the microphone array;

[0011] The distance measurement sub-step is used to calculate the distance between the face and the microphone array;

[0012] The spectrogram generating sub-step generates a spectrogram containing facial information by using the SRP-PHAT algorithm;

[0013] The coordinate system rotation step is used to convert the spectrogram from a spherical coordinate system to a Cartesian coordinate system;

[0014] (3) Expression classification: Features are extracted from the spectrogram through a pre-trained deep learning model, and the extracted features are input into a fully connected deep neural network. The fully connected deep neural network outputs the user's facial expression classification results by learning the mapping relationship between features and expression categories.

[0015] Furthermore, in step (2), the audio frame segmentation sub-step includes:

[0016] localizing directly propagated signals from raw recordings of microphone arrays;

[0017] Positioning the main reflected signal after the directly propagated signal, thereby determining the location of the facial reflected signal;

[0018] After determining the location of the facial reflection signal from the recording of the microphone array, the facial reflection sound signal is segmented and extracted from the recording of the microphone array.

[0019] Furthermore, in step (2), the distance measurement sub-step specifically includes:

[0020] Use the FMCW radar ranging algorithm to mix the received signal with the transmitted signal to generate an intermediate frequency signal;

[0021] When measuring distance, it is assumed that there is a continuous FMCW signal in the blank period between two FMCW pulses. The sliding window is used to search backward step by step until a significant amplitude is detected at a certain frequency in the intermediate frequency signal.

[0022] The distance between the face and the microphone array d Calculated by the following formula:

[0023] ;

[0024] in, N represents the number of sliding windows, Indicates the maximum ranging distance of a single pulse, Indicates the distance measured by the current window.

[0025] Furthermore, in step (2), the spectrogram generation sub-step uses the SRP-PHAT algorithm to generate the spectrogram;

[0026] When generating a spectrogram using the SRP-PHAT algorithm, the azimuth angle is calculated in the range of [-180°, 180°], and the elevation angle is calculated in the range of: ;

[0027] in, ; d is the distance between the face and the microphone array; is the diagonal length of the drawn image; It is the minimum pitch angle of the SRP-PHAT scanning range of space;

[0028] The minimum resolution res of the pitch angle is:

[0029] .

[0030] Furthermore, in step (2), the coordinate system rotation step is used to convert the spectrogram from a spherical coordinate system to a Cartesian coordinate system;

[0031] The coordinates of the intersection of a certain direction represented by the direction angle and pitch angle in the spherical coordinate system and the projection plane are converted into specific coordinates in the Cartesian coordinate system, so as to map the orientation information in the polar coordinate system to the Cartesian coordinate system and generate an image that can clearly display the contour.

[0032] Furthermore, step (3) specifically includes:

[0033] (3.1) Data preprocessing: Align the spectrogram with the camera image using a two-dimensional affine transformation. Use a face localization algorithm to locate the face in the camera image, and then obtain the face location in the spectrogram.

[0034] (3.2) Expression recognition: Use the pre-trained encoder DINOv2 to encode the facial image in the spectrogram; after encoding the image into a feature vector, use a fully connected deep neural network to classify the feature vector and output the facial expression category.

[0035] Furthermore, in step (3.1) of the present invention, the spectrogram is aligned with the camera image through a two-dimensional affine transformation, specifically:

[0036] Four microphones are used to align the coordinate system of the microphone array with the camera coordinate system;

[0037] Combined with the sound source localization method, the positions of the four microphones in the spectrogram are calculated;

[0038] Determine the corresponding positions of the four microphones in the camera image using an image matching algorithm;

[0039] Based on the corresponding points of the four microphones, any three corresponding points are selected to calculate the affine transformation matrix, and the average of the four obtained affine transformation matrices is taken to obtain the final transformation matrix.

[0040] A facial expression recognition system based on sound perception adopts the facial expression method, and the system comprises:

[0041] Signal transmission module: The signal transmission module transmits an interval FMCW waveform sound signal through a speaker. The sound signal is reflected after reaching the face of the person, generating a sound reflection signal, which is received by the microphone array;

[0042] Signal processing module: The signal processing module includes an audio frame segmentation submodule, a distance measurement submodule, a spectrogram generation submodule, and a coordinate system conversion submodule; the audio frame segmentation submodule is used to extract the sound signal reflected by the face from the recording of the microphone array; the distance measurement submodule is used to calculate the distance between the face and the sensor; the spectrogram generation submodule generates a spectrogram containing facial information using the SRP-PHAT algorithm; and the coordinate system conversion module is used to convert the spectrogram from a spherical coordinate system to a Cartesian coordinate system;

[0043] Expression classification module: Extracts features from the spectrogram through a pre-trained deep learning model and inputs the extracted features into a fully connected deep neural network. The fully connected deep neural network outputs the user's facial expression classification results by learning the mapping relationship between features and expression categories.

[0044] Beneficial technical effects of the present invention:

[0045] 1) The facial expression recognition method provided by the present invention utilizes the multipath effect and adopts the SPR-PHAT algorithm to generate a spectrogram containing facial information, extracting facial features from multi-channel sound signals. This method can effectively and intuitively capture subtle facial dynamics while effectively avoiding the deception problem of traditional image or video recognition methods.

[0046] 2) The facial expression recognition method provided by the present invention can effectively realize facial expression recognition even with a small amount of data, without relying on expensive hardware equipment, by utilizing commercial speakers and microphone arrays.

[0047] 3) The facial expression recognition method provided by this invention combines acoustic imaging with a pre-trained visual model for the first time and has broad development prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a flow chart of a facial expression recognition method based on sound perception in an embodiment of the present invention;

[0049] Figure 2 This is a physical diagram of an acoustic transceiver in an embodiment of the present invention;

[0050] Figure 3 is the signal positioning result in an embodiment of the present invention;

[0051] Figure 4 Schematic diagram of converting from a spherical coordinate system to a Cartesian coordinate system in an embodiment of the present invention;

[0052] Figure 5 The recognition results of the five basic emotional expressions by the facial expression recognition method based on sound perception in an embodiment of the present invention;

[0053] Figure 6 This is a graph showing changes in Loss during the training process of the facial expression recognition method based on sound perception in an embodiment of the present invention;

[0054] Figure numerals: 1. Development board; 2. Driver board; 3. Microphone array; 4. Speaker. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0056] On the contrary, the present invention covers any alternatives, modifications, equivalents, and solutions that fall within the spirit and scope of the present invention as defined by the claims. Furthermore, to facilitate a better understanding of the present invention, certain specific details are described in detail below in the detailed description of the present invention. Those skilled in the art will be able to fully understand the present invention without these details.

[0057] In the field of wireless sensing, adapting to the needs of different devices and application scenarios often requires custom data collection, which is not only costly but also cumbersome. Therefore, achieving efficient emotion recognition at low cost and with minimal data is a significant challenge. The facial expression recognition method provided by this invention achieves this goal even with scarce data.

[0058] Example 1: A facial expression recognition method based on sound perception, which uses acoustic signals as perception media, utilizes a microphone array to capture sound reflection signals, generates a spectrogram containing facial features through signal processing, extracts features from the spectrogram, and utilizes a deep learning network to classify facial expressions to achieve expression recognition.

[0059] like Figure 1 As shown, the method in this embodiment includes the following steps:

[0060] (1) Signal transmission: The speaker transmits an FMCW waveform sound signal, which is reflected after reaching the face and generates a sound reflection signal, which is received by the microphone array;

[0061] (2) Signal processing: The signal processing process includes an audio frame segmentation sub-step, a distance measurement sub-step, a spectrogram generation sub-step, and a coordinate system conversion sub-step;

[0062] The audio frame segmentation sub-step is used to extract the facial reflected sound signal from the recording of the microphone array;

[0063] The distance measurement sub-step is used to calculate the distance between the face and the sensor;

[0064] The spectrogram generating sub-step generates a spectrogram containing facial information by using the SRP-PHAT algorithm;

[0065] The coordinate rotation step is used to transform the spectrogram from a spherical coordinate system to a Cartesian coordinate system.

[0066] (3) Expression classification: Features are extracted from the spectrogram through a pre-trained deep learning model, and the extracted features are input into a fully connected deep neural network. The fully connected deep neural network outputs the user's facial expression classification results by learning the mapping relationship between features and expression categories.

[0067] In step (1) of this embodiment, the length of a single pulse of the FMCW waveform sound signal emitted by the loudspeaker is 2 ms, and the frequency increases linearly in the range of 14 kHz to 15 kHz; an interval of 8 ms is set between adjacent pulses.

[0068] In the present invention, the method for designing the sound signal is: if the distance from the face to the microphone array is d, the time required for the signal to return is:

[0069] ;

[0070] When d=0.8m, t is approximately 0.0047s, or 4.7ms; when d=1m, t is approximately 6ms.

[0071] To reduce overlap between background and facial reflections, the pulse duration is set to 2ms. To minimize the impact of one reflection on the next, the interval is set to 8ms. The choice of the 14kHz to 15kHz frequency range is primarily influenced by the spectral response of the speaker and microphone array. Since these are inexpensive devices, their response is weak and unstable at high frequencies. Experiments have shown that the 14-15kHz frequency range is relatively effective. This frequency range avoids common ambient noise (typically below 8kHz) and can be removed using a bandpass filter. Furthermore, considering the use of inexpensive speakers and microphone arrays, when signal frequencies exceed 20kHz, the power attenuates significantly, reducing the signal-to-noise ratio and making it difficult to separate facial reflections from background noise. Taking these factors into consideration, the present invention selects a 2ms pulse signal with a linearly increasing frequency between 14kHz and 15kHz. Each such signal segment is called a pulse, and an 8ms interval is set between adjacent pulses.

[0072] In step (1) of this embodiment, an acoustic transceiver consisting of a speaker and a microphone array is used for acoustic sensing; Figure 2 As shown, the acoustic transceiver includes a development board 1, a driver board 2, a microphone array 3, and a speaker 4;

[0073] Specifically, the speaker measures 5 cm × 5 cm and supports both a 3.5 mm audio jack and a USB port, making it compatible with a variety of sound output devices. The speaker is used for audio signals. The microphone array consists of eight microphone modules, along with a driver board and a development board. The eight microphone modules are arranged in a circular pattern, with an array aperture of 42.5 mm, an outer diameter of 102 mm, and an inner diameter of 68.7 mm. All microphone modules are synchronized with an internal clock and can simultaneously receive sound signals. The microphone array connects to the driver board via a 28P reverse-phase flexible cable, sampling analog signals at a rate of 192 kHz. The driver board is equipped with a 12V / 2A power supply, and the power indicator illuminates upon power-up. The development board can simultaneously control all modules of the acoustic transceiver. The development board also includes a 24GB memory card for storing long-term sound data.

[0074] In step (2) of this embodiment, the directly propagated signal refers to the signal that is directly propagated from the loudspeaker to the microphone array. Ideally, the directly propagated signal should be exactly the same as the transmitted signal and have the highest intensity.

[0075] The primary reflection signal is a mixture of reflections from the primary facial surfaces (such as the cheeks and forehead). Due to the varying distances between other facial surfaces (such as the nose and chin) and the microphone array, the echoes from these areas arrive at the microphone array at different times, potentially arriving earlier or later than the primary reflection signal. Therefore, the reflection signal from the face region actually includes a combination of all these reflections, capturing the full range of facial information.

[0076] In order to minimize the interference of other reflected signals and effectively extract the target information, it is crucial to accurately segment the reflected signals in the face area.

[0077] In step (2), the audio frame segmentation sub-step includes:

[0078] localizing directly propagated signals from raw recordings of microphone arrays;

[0079] Positioning the main reflected signal after the directly propagated signal, thereby determining the location of the facial reflected signal;

[0080] After determining the location of the facial reflection signal from the recording of the microphone array, the facial reflection sound signal is segmented and extracted from the recording of the microphone array.

[0081] An intuitive approach to locating directly transmitted signals is to play and record the sound simultaneously; however, because the playback and recording processes require multiple layers of hardware and software processing in the operating system, many of which have unpredictable delays, using a constant delay to locate directly transmitted signals is not feasible. Since directly transmitted signals typically have the highest amplitude, their location can be determined by their amplitude. The present invention uses a coarse-to-fine approach to locate signals directly transmitted from the speaker to the microphone.

[0082] First, a sliding window approach is used to locate the sound's starting point within the signal. The signal is divided into two adjacent, non-overlapping windows, A and B, each with a length of L. The two windows are traversed through the signal, sliding forward one sample point at a time until window B exceeds the total length of the signal. At each sliding step, the energy ratio of window B to window A is calculated. By comparing the energy ratios of the two windows, a sudden change in signal energy can be detected, allowing a preliminary determination of the sound's starting point.

[0083] After preliminarily determining the candidate starting point of the sound signal through the sliding window method, further fine search is performed around the candidate point to more accurately locate the starting position of the sound. For each candidate point, a signal of length L is intercepted starting from the candidate point, and the intercepted signal is mixed to obtain an intermediate frequency signal. Next, the intermediate frequency signal is subjected to fast Fourier transform (FFT), and the resulting spectrum is centered to obtain the frequency domain signal. The DC component in the frequency domain signal is extracted, and the point with the largest DC component among all candidate points is selected as the starting position of the sound signal. By analyzing the changes in these DC components, the starting position of the sound signal can be determined more accurately. Through the above method, the direct propagation signal found is as follows Figure 3 As shown in the 0.095-0.097s part.

[0084] When the direct propagation signal is obtained, the main reflection signal in the signal is identified by calculating the distance between the face and the microphone array. The main reflection signal obtained is as follows: Figure 3 The 1-1.002s portion of the image is shown in Figure 1. Furthermore, because the human face is three-dimensional rather than two-dimensional, the present invention extends the main reflection signal by 50 sampling points before and after it, ensuring that the intercepted reflection signal covers the entire facial depth (approximately 9cm). The main reflection signal and the extended sampling points serve as input for the next step of signal processing: filtering and spectrogram generation (SRP-PHAT).

[0085] In step (2) of this embodiment, the distance measurement sub-step specifically includes:

[0086] Use the FMCW radar ranging algorithm to mix the received signal with the transmitted signal to generate an intermediate frequency signal;

[0087] According to the relationship formula between distance and difference frequency:

[0088] ;

[0089] Where R represents the target distance, c is the speed of light, Δf is the frequency difference, and μ is the slope. This formula can be used to calculate the distance between the target and the sensor. Directly using this formula requires the device to send a continuous FMCW signal. In this invention, a blank period is introduced between two FMCW pulses.

[0090] When measuring distance, it is assumed that there is a continuous FMCW signal in the blank period between two FMCW pulses. The sliding window is used to search backward step by step until a significant amplitude is detected at a certain frequency in the intermediate frequency signal.

[0091] The distance between the face and the microphone array d Calculated by the following formula:

[0092] ;

[0093] in, N represents the number of sliding windows, Indicates the maximum ranging distance of a single pulse, Indicates the distance measured in the current window. d Specifically, the horizontal distance from the face to the center of the circular microphone array;

[0094] In step (2) of this embodiment, the spectrogram generation sub-step uses the SRP-PHAT algorithm to generate the spectrogram;

[0095] Facial expressions are minute, irregular deformations. Sound signals, on the other hand, are continuous signals that vary over time, typically in the form of sound waves. In real-world environments, sound signals are often affected by various noises, echoes, and other environmental interference, making weak reflected signals more difficult to discern and preventing subtle facial changes from being recognized. To achieve expression classification, it is necessary to extract subtle facial movements from the sound signal. The facial expression recognition method provided by this invention uses a speaker to emit a specific sound signal to scan the face, generating an acoustic representation that preserves rich facial features and thus captures dynamic facial information. When a sound wave reaches the face, the reflection and scattering in different areas vary depending on the face's shape. Sound signals from a single incident direction are scattered at different reflection angles, generating multiple multipath components that ultimately superimpose at each microphone. The energy of the reflected sound in each direction reflects the three-dimensional surface features of the face. This invention uses the Steered Response Power with Phase Transform (SRP-PHAT) algorithm and the Phase Weighted Delay Accumulation (PHAT) algorithm to calculate the scattered signal intensity in a specific direction. The signal strength of each microphone pair at that location is accumulated to generate a fine-grained facial spectrogram. This spectrogram retains rich facial features and can be effectively used for facial recognition. This process accumulates the signal strength of all microphone pairs at the facial location.

[0096] The present invention only requires the spectrogram within the projection plane, eliminating the need to scan the entire space. Therefore, by dynamically adjusting the scanning angle range and resolution based on the distance from the microphone array to the subject and the size of the mapping area, computational complexity is reduced and efficiency is improved. Setting the spatial angle range and resolution of the spectrogram too large may result in blank spots on the canvas; setting them too small may result in a significant amount of repeated calculations.

[0097] When generating a spectrogram using the SRP-PHAT algorithm, the azimuth angle is calculated in the range of [-180°, 180°], and the elevation angle is calculated in the range of: ;

[0098] in, ; d is the distance between the face and the microphone array; is the diagonal length of the drawn image; It is the minimum pitch angle of the SRP-PHAT scanning range of the space.

[0099] In the current experimental scenario, the drawing range of the azimuth angle θ is [-180°, 180°], so there is no need to dynamically adjust its range. Here we mainly consider the pitch angle The effect on coordinates. When the pitch angle When it decreases, that is, when the angle between the ray and the z-axis increases, As the slope decreases, the coordinate value of point M changes at a faster rate; in order to ensure that every pixel in the image is covered, a minimum resolution is required.

[0100] Obviously, the pixel point farthest from the diagonal direction of the square canvas requires the smallest resolution; the minimum resolution res for the pitch angle is:

[0101] .

[0102] In step (2) of this embodiment, the coordinate system rotation step is used to convert the spectrogram from a spherical coordinate system to a Cartesian coordinate system;

[0103] The coordinates of the intersection of a certain direction represented by the direction angle and pitch angle in the spherical coordinate system and the projection plane are converted into specific coordinates in the Cartesian coordinate system, so as to map the orientation information in the polar coordinate system to the Cartesian coordinate system and generate an image that can clearly display the contour.

[0104] The spectrogram generated in the spectrogram generation substep uses direction and azimuth as coordinates. Based on the measured distance between the face and the microphone array, a coordinate system transformation is performed, converting the spectrogram from polar coordinates to Cartesian coordinates. The horizontal and vertical axes of the converted spectrogram are in centimeters, which more intuitively displays the spatial information of facial expressions.

[0105] To achieve this conversion, we first need to clarify the relative positions of the Cartesian coordinate system and the spherical coordinate system. In the present invention, the Cartesian coordinate system is obtained by horizontally translating the spherical coordinate system along the positive direction of the Z axis by a known distance. In the conversion process, the xy plane of the Cartesian coordinate system is regarded as the projection plane. In order to generate the corresponding image on this projection plane, it is necessary to convert the spherical coordinate system into the xy plane by the direction angle θ and the pitch angle θ. The coordinates of the intersection of a certain direction and the projection plane are converted to specific coordinates in the Cartesian coordinate system. The conversion process is based on geometric relationships. With the known direction angle, pitch angle, and distance between the projection plane and the microphone array, the orientation information in the polar coordinate system can be mapped to the Cartesian coordinate system.

[0106] The relationship between the two coordinate systems is as follows Figure 4 As shown, the plane on the right represents the projection plane. O 1 is the coordinate system where the microphone array is located, O 2 is the coordinate system of the imaging plane. O 2 The coordinate system is set by O 1The coordinate system is translated horizontally along the distance R between the projection plane and the microphone array, so x 1 Axis and x 2 axis parallel, y 1 Axis and y 2 Axis parallel. Rays The azimuth angle is θ, and the elevation angle is , and the plane intersects at point m, so the straight line At the same time perpendicular to Plane and From this we can see that the horizontal and vertical coordinates of point m are equal to the horizontal and vertical coordinates of point n respectively.

[0107] ;

[0108] ;

[0109] In the triangle Middle, angle is a right angle, so The length is:

[0110] ;

[0111] According to the geometric relationship, the coordinates of point m can be obtained as:

[0112] ;

[0113] This coordinate transformation allows spatial information, originally based on azimuth and elevation angles, to be accurately projected onto the projection plane, generating an image that clearly displays contours. This allows the spatial data generated by beamforming to be visualized in a Cartesian coordinate system, producing an intuitive image that aligns with human spatial perception. Because the projection plane is assumed to be far from the microphone array plane, slight distance deviations caused by facial fluctuations are ignored.

[0114] Step (3) of this embodiment specifically includes:

[0115] (3.1) Data preprocessing: Align the spectrogram with the camera image using a two-dimensional affine transformation. Use a face localization algorithm to locate the face in the camera image, and then obtain the face location in the spectrogram.

[0116] (3.2) Facial expression recognition: Use the pre-trained encoder DINOv2 to encode the image; after encoding the image into a feature vector, use a fully connected deep neural network to classify the feature vector and output the facial expression category.

[0117] In the spectrogram, similar to the imaging principle of a camera, it contains information in all directions within an area. Currently, the spectrogram does not have enough information to accurately distinguish between the face and areas very close to the face, such as the forehead and neck. In step (3.1) of the present invention, in order to obtain the position of the face, the pixels in the generated spectrogram are aligned and matched with the pixels in the camera photo. This process requires the use of camera calibration technology to achieve accurate spatial mapping between the spectrogram coordinate system and the camera coordinate system in the same reference frame. In the present invention, the coordinate system where the microphone array is located is set to the world coordinate system. By fixing the relative spatial position of the camera and the microphone array, the camera posture and position are ensured to remain constant throughout the experiment, thereby maintaining the stability of the external parameter matrix. At the same time, the distance between the camera and the object being photographed is kept basically consistent to ensure the consistency of the camera's internal parameters in multiple shots. The present invention uses affine transformation to map the xy coordinates in the world coordinate system to the xy coordinates in the camera coordinate system, and the present invention sets the z-direction coordinates of all points to a uniform value, that is, the z value in the world coordinate corresponding to the point represented by each pixel in the spectrogram is the same. This assumption is based on the fact that the camera and microphone array are located in the same plane, and the distance between the camera and the face is much larger than the local unevenness of the face's surface. Therefore, changes in facial geometry have a negligible effect on the z value of each point in the image. Based on this, the present invention further assumes that the depth information of all image points remains consistent in the camera coordinate system, that is, the z value in the camera coordinate system is the same. In the experiment, the camera and microphone array are fixed in the same plane to ensure that their relative position and orientation remain stable. Therefore, the relative transformation between the world coordinate system and the camera coordinate system only includes rotation about the z axis and translation along the xy plane.

[0118] The intrinsic parameter matrix satisfies the form of the affine transformation matrix, and the process of projecting from the camera coordinate system to the image plane and sampling can also be regarded as an affine transformation. Therefore, the mapping process from three-dimensional space to image can be regarded as the multiplication of two affine transformation matrices, and the final result is still an affine transformation matrix. The affine transformation has 6 degrees of freedom. Each pair of two-dimensional coordinate points can provide two linear equations, and three pairs of two-dimensional points can determine an affine transformation matrix. In order to reduce experimental errors and improve transformation accuracy, four microphones are used in the present invention to align the coordinate system of the microphone array with the camera coordinate system;

[0119] Combined with the sound source localization method, the positions of the four microphones in the spectrogram are calculated;

[0120] Determine the corresponding positions of the four microphones in the camera image using an image matching algorithm;

[0121] Based on the corresponding points of the four microphones, any three corresponding points are selected to calculate the affine transformation matrix, and the average of the four obtained affine transformation matrices is taken to obtain the final transformation matrix.

[0122] This step-by-step approach effectively reduces the deviation caused by single measurement errors and improves the stability and accuracy of the correction algorithm in practical applications. After aligning the spectrogram with the camera image, a sophisticated face localization algorithm is used to determine the position of the face in the camera image, thereby obtaining the position of the face in the spectrogram.

[0123] In step (3.2) of the present invention, a pre-trained model DINOv2 is used as a feature extractor. DINOv2 is trained through self-supervised learning on a dataset of approximately 1 billion images and can generate highly versatile feature representations suitable for a variety of image tasks, such as depth detection, image segmentation, and image generation.

[0124] In this paper, the decoder in the fully connected deep neural network consists of three alternating fully connected layers and two ReLU activation layers. This decoder is used to classify the 384-dimensional features output by DINOv2. The decoder consists of three alternating fully connected layers and two ReLU activation layers. The specific structure is as follows: the first fully connected layer has a dimension of 384×256, the second layer has a dimension of 256×128, and the third layer has a dimension of 128×n, where n represents the number of expression categories. Each fully connected layer is followed by a ReLU activation layer to enhance the network's nonlinear expression capabilities.

[0125] The present invention experimentally evaluates the classification performance of the facial expression recognition method based on sound perception provided by the present invention for five basic emotional expressions (i.e., happiness, anger, disgust, neutrality, and surprise). The experimental results are as follows: Figure 5 The experimental results show that the method proposed in this invention can effectively recognize the user's emotional expressions, and the overall average recognition accuracy reaches 74%. The change of Loss during the training process is shown in Figure 6 As shown, it can be seen that with the increase in the number of training rounds, the Loss value gradually decreases and tends to be stable, indicating that the facial expression recognition method based on sound perception provided by the present invention can effectively learn the characteristics of emotional expressions during the training process and gradually converge to a better state.

[0126] Example 2: A facial expression recognition system based on sound perception, using the facial expression method described in Example 1, the system comprising:

[0127] Signal transmission module: The signal transmission module transmits a sound signal with an FMCW waveform through a speaker. The sound signal is reflected after reaching the face of the person, generating a sound reflection signal, which is received by a microphone array.

[0128] Signal processing module: The signal processing module includes an audio frame segmentation submodule, a distance measurement submodule, a spectrogram generation submodule and a coordinate system conversion submodule;

[0129] The audio frame segmentation submodule is used to extract the facial reflected sound signal from the recording of the microphone array;

[0130] The distance measurement submodule is used to calculate the distance between the face and the sensor;

[0131] The spectrogram generation submodule generates a spectrogram containing facial information through the SRP-PHAT algorithm;

[0132] The coordinate system rotator module is used to convert the spectrogram from a spherical coordinate system to a Cartesian coordinate system.

[0133] Expression classification module: Extracts features from the spectrogram through a pre-trained deep learning model and inputs the extracted features into a fully connected deep neural network. The fully connected deep neural network outputs the user's facial expression classification results by learning the mapping relationship between features and expression categories.

[0134] The facial expression recognition system provided by the present invention aims to achieve high-precision facial expression recognition through inexpensive equipment in scenarios where data is scarce, while protecting user privacy.

[0135] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A facial expression recognition method based on sound perception, characterized in that: The method uses acoustic signals as a perception medium, uses a microphone array to capture sound reflection signals, and generates a spectrogram containing facial features through signal processing. Features are extracted from the spectrogram, and a deep learning network is used to classify facial expressions to achieve expression recognition. The method comprises the following steps: (1) Signal transmission: The speaker transmits an interval FMCW waveform sound signal, which is reflected after reaching the face of the person, generating a sound reflection signal, which is received by the microphone array; (2) Signal processing: The signal processing process includes an audio frame segmentation sub-step, a distance measurement sub-step, a spectrogram generation sub-step, and a coordinate system conversion sub-step; The audio frame segmentation sub-step is used to extract the facial reflected sound signal from the recording of the microphone array; The distance measurement sub-step is used to calculate the distance between the face and the microphone array; The spectrogram generating sub-step generates a spectrogram containing facial information by using the SRP-PHAT algorithm; The coordinate system rotation step is used to convert the spectrogram from a spherical coordinate system to a Cartesian coordinate system; (3) Expression classification: Extract features from the spectrogram using a pre-trained deep learning model, input the extracted features into a deep neural network, and the deep neural network outputs the user's facial expression classification results by learning the mapping relationship between features and expression categories; In step (2), the audio frame segmentation sub-step includes: localizing directly propagated signals from raw recordings of microphone arrays; Positioning the main reflected signal after the directly propagated signal, thereby determining the location of the facial reflected signal; After determining the location of the facial reflection signal from the recording of the microphone array, segmenting and extracting the facial reflection sound signal from the recording of the microphone array; In step (2), the distance measurement sub-step specifically includes: Use the FMCW radar ranging algorithm to mix the received signal with the transmitted signal to generate an intermediate frequency signal; When measuring distance, it is assumed that there is a continuous FMCW signal in the blank period between two FMCW pulses. The sliding window is used to search backward step by step until a significant amplitude is detected at a certain frequency in the intermediate frequency signal. The distance d between the face and the microphone array is calculated using the following formula: d=N*d max +d win Where N represents the number of sliding windows, d max Indicates the maximum ranging distance of a single pulse, d win Indicates the distance measured by the current window; In step (2), the spectrogram generation sub-step uses the SRP-PHAT algorithm to generate the spectrogram; When generating a spectrogram using the SRP-PHAT algorithm, the azimuth angle is calculated in the range of [-180°, 180°], and the elevation angle is calculated in the range of: in, d is the distance between the face and the microphone array; d diag is the diagonal length of the drawn image; φ min It is the minimum pitch angle of the SRP-PHAT scanning range of space; The minimum resolution res of the pitch angle is:

2. A facial expression recognition method based on sound perception according to claim 1, characterized in that In step (2), the coordinate system rotation step is used to convert the spectrogram from a spherical coordinate system to a Cartesian coordinate system; The coordinates of the intersection of a certain direction represented by the direction angle and pitch angle in the spherical coordinate system and the projection plane are converted into specific coordinates in the Cartesian coordinate system, so as to map the orientation information in the polar coordinate system to the Cartesian coordinate system and generate an image that can clearly display the contour.

3. A facial expression recognition method based on sound perception according to claim 1, characterized in that Step (3) specifically includes: (3.1) Data preprocessing: Align the spectrogram with the camera image using a two-dimensional affine transformation. Use a face localization algorithm to locate the face in the camera image, and then obtain the face location in the spectrogram. (3.2) Expression recognition: Use the pre-trained encoder DINOv2 to encode the facial image in the spectrogram; after encoding the image into a feature vector, use a fully connected deep neural network to classify the feature vector and output the facial expression category.

4. A facial expression recognition method based on sound perception according to claim 3, characterized in that, In step (3.1) of the present invention, the spectrogram is aligned with the camera image by a two-dimensional affine transformation, specifically: Four microphones are used to align the coordinate system of the microphone array with the camera coordinate system; Combined with the sound source localization method, the positions of the four microphones in the spectrogram are calculated; Determine the corresponding positions of the four microphones in the camera image through image matching algorithm; Based on the corresponding points of the four microphones, any three corresponding points are selected to calculate the affine transformation matrix, and the average of the four obtained affine transformation matrices is taken to obtain the final transformation matrix.

5. A facial expression recognition system based on sound perception, using the facial expression recognition method according to any one of claims 1 to 4, characterized in that: The system comprises: Signal transmission module: The signal transmission module transmits an interval FMCW waveform sound signal through a speaker. The sound signal is reflected after reaching the face of the person, generating a sound reflection signal, which is received by the microphone array; Signal processing module: The signal processing module includes an audio frame segmentation submodule, a distance measurement submodule, a spectrogram generation submodule, and a coordinate system conversion submodule; the audio frame segmentation submodule is used to extract the sound signal reflected by the face from the recording of the microphone array; the distance measurement submodule is used to calculate the distance between the face and the sensor; the spectrogram generation submodule generates a spectrogram containing facial information using the SRP-PHAT algorithm; and the coordinate system conversion submodule is used to convert the spectrogram from a spherical coordinate system to a Cartesian coordinate system; Expression classification module: Extracts features from the spectrogram through a pre-trained deep learning model and inputs the extracted features into a fully connected deep neural network. The fully connected deep neural network outputs the user's facial expression classification results by learning the mapping relationship between features and expression categories.

Images