Facial information acquisition method and electronic equipment

Abstract

The embodiment of the invention provides a facial information acquisition method and electronic equipment. The method is applied to the electronic equipment, in particular to the electronic equipment without a camera. In the method, the electronic equipment can perform facial state feature detection through ultrasonic waves to generate facial information matched with an actual facial state. The face information comprises a face state and / or a simulated face picture. By implementing the method, the electronic equipment without a camera can acquire the face state and / or the simulated face picture of the user, so that the use scene of the electronic equipment is enriched.

Description

Technical Field

[0001] The present application relates to the fields of vision and terminals, and in particular to a method for acquiring facial information and an electronic device. Background Art

[0002] In daily life, there are many kinds of electronic devices. Among them, there are both electronic devices equipped with cameras (such as mobile phones, computers, etc.) and electronic devices not equipped with cameras (such as smart watches, smart glasses, etc.). Although, compared with electronic devices equipped with cameras, electronic devices without cameras are more limited in functionality and interactivity. However, electronic devices without cameras can show advantages that electronic devices equipped with cameras cannot in specific usage scenarios. For example, taking smart glasses as an example, an electronic device without a camera can be used as glasses to help the wearer see things, and has other functions besides seeing things, including but not limited to: providing navigation prompts, real-time translation, and even displaying email notifications or upcoming schedules in the wearer's line of sight.

[0003] However, electronic devices that are not equipped with cameras cannot obtain facial information, and it is difficult to provide more possibilities in scenarios that require facial information, such as video communication, face detection and recognition. Summary of the invention

[0004] The present application provides a facial information acquisition method and an electronic device for detecting facial state features through ultrasound and generating facial information that matches the actual facial state.

[0005] In a first aspect, the present application provides a facial information acquisition method, which is applied to a first electronic device, wherein the first electronic device includes a speaker and a microphone, and the method includes: acquiring first ultrasonic audio data through the microphone, wherein the first ultrasonic audio data indicates an ultrasonic signal emitted by the speaker and reflected by the face; extracting facial state features of the face through the first ultrasonic audio data; and adjusting a first preset facial image based on the facial state features to generate a simulated facial image.

[0006] In the above embodiment, the facial state feature is obtained based on the first ultrasonic audio data, but the facial state feature can indicate the facial state of the simulated face picture, because the first ultrasonic audio data comes from the ultrasonic signal reflected when the face is in the facial state corresponding to the simulated face picture. Therefore, the electronic device can detect the facial state feature based on the ultrasonic signal, and then generate a simulated face picture (a kind of facial information) that matches the actual facial state. In this way, the electronic device can obtain the face picture without relying on the camera, and can obtain the ultrasonic audio data based on the speaker and microphone carried by the electronic device itself, so as to realize the generation of the face picture.

[0007] In combination with the first aspect, in some embodiments, the first preset face picture is adjusted based on the facial state features to generate a simulated face picture, specifically including: extracting facial features in the first preset face picture through the first preset face; adjusting the facial features through the facial state features; and mapping the adjusted facial features into the simulated face picture.

[0008] In the above embodiment, the facial features are data that can characterize the first preset facial picture and are obtained based on the first preset facial picture. Therefore, adjusting based on the facial features to obtain the simulated facial picture can also be understood as adjusting based on the first preset facial picture to obtain the simulated facial picture. However, the simulated facial picture is obtained by adjusting the facial features corresponding to the first preset facial picture, which improves the adjustment efficiency.

[0009] In combination with the first aspect, in some embodiments, the first electronic device also includes a posture sensor, and the method also includes: acquiring first posture data through the posture sensor, the first posture data indicating the posture of the face; extracting the posture features of the face through the first posture data; adjusting the facial features through the facial state features, specifically including: adjusting the facial features through the facial state features and the posture features to generate a simulated facial image; the simulated facial image has the posture indicated by the posture features.

[0010] In the above embodiment, the posture data is posture data for the head. The introduction of posture data for the head can be used to reduce the negative impact of head movement on facial expression recognition and generate more accurate simulated face images. If the posture data for the head is not introduced, the changes in the posture of the head and the changes in facial expressions will affect the reflection of the ultrasonic signal, and then will be reflected in the ultrasonic audio data. When electronic devices analyze facial information through ultrasonic audio data, there is a problem of inaccuracy. For example, the changes in the posture of the head will be mistaken for changes in facial expressions, resulting in incorrect facial information generated.

[0011] In combination with the first aspect, in some embodiments, the facial state feature is determined by a first feature extraction network in a first facial picture generation model, and the facial feature is determined by a second feature extraction network in the first facial picture generation model; the facial picture generation model also includes a first facial generation network; the first facial generation network includes a first convolutional network, a first deconvolutional network and a second convolutional network; the functions of the first convolutional network and the first deconvolutional network include: adjusting the facial feature based on the facial state feature to obtain the adjusted facial feature; the second convolutional network is used to map the adjusted facial feature into the simulated facial picture; wherein the first facial picture generation model is trained using the second ultrasonic audio data and the second preset facial picture as input data, the predicted facial picture as output data and the sample facial picture as reference data; the sample facial picture matches the facial state feature extracted based on the second ultrasonic audio data.

[0012] In combination with the first aspect, in some embodiments, the facial state feature is determined by a first feature extraction network in a second facial picture generation model, the facial feature is determined by a second feature extraction network in the second facial picture generation model, and the posture feature is determined by a third feature extraction network in the second facial picture generation model; the facial picture generation model also includes a second facial generation network; the second facial generation network includes a third convolutional network, a second deconvolutional network and a fourth convolutional network; the functions of the third convolutional network and the second deconvolutional network include: adjusting the facial feature based on the facial state feature and the posture feature to obtain the adjusted facial feature; the fourth convolutional network is used to map the adjusted facial feature into the simulated facial picture; wherein the second facial picture generation model is trained using the second ultrasonic audio data, the second posture data and the second preset facial picture as input data, the predicted facial picture as output data and the sample facial picture as reference data; the sample facial picture matches the facial state feature extracted based on the second ultrasonic audio data.

[0013] In combination with the first aspect, in some embodiments, before acquiring the first ultrasonic audio data through the microphone, the method also includes: establishing a video call connection with the second electronic device; after generating a simulated human face picture, the method also includes: sending the simulated human face picture to the second electronic device through the video call connection.

[0014] In the above embodiment, the generated simulated face image can be used in a video call scenario. In this way, the video call scenario no longer relies on the camera. By acquiring ultrasonic audio data based on the speaker and microphone carried by the electronic device itself, the face image can be generated, and then the video call can be realized.

[0015] In combination with the first aspect, in some embodiments, obtaining the first ultrasonic audio data through the microphone specifically includes: The microphone collects a sound signal; the sound signal is high-pass filtered to obtain the ultrasonic signal after being reflected by the face; the ultrasonic signal after being reflected by the face is processed to obtain the first ultrasonic audio data.

[0016] In the above embodiment, high-pass filtering is performed because the sound signal may include sound signals in the wearer's environment in addition to the ultrasonic signal. Therefore, high-pass filtering is performed to remove noise, so that the generated first ultrasonic audio data can more accurately indicate the ultrasonic signal emitted by the speaker and reflected by the face, thereby obtaining more accurate facial state features, and finally obtaining a more accurate simulated human face picture.

[0017] In combination with the first aspect, in some embodiments, the method also includes: low-pass filtering the sound signal to obtain an audible sound signal; the frequency of the audible sound signal is lower than the frequency of the ultrasonic signal; processing the audible sound signal to obtain audible audio data; and sending the audible audio data to the second electronic device through the video call connection.

[0018] In the above embodiment, low-pass filtering is performed because the video call microphone not only needs to collect the wearer's voice signal, but also needs to collect the ultrasonic signal reflected by the face, but the voice sent to the other electronic device only needs to reflect the wearer's voice signal.

[0019] In combination with the first aspect, in some embodiments, the first electronic device is a wearable device worn on the head, and at least one speaker, at least one microphone, and at least one gesture sensor are arranged on both sides of the central axis of the first electronic device.

[0020] In the above embodiment, the microphone, the loudspeaker and the posture sensor are arranged in a bilaterally symmetrical manner, so that the acquired ultrasonic audio data and posture data can more accurately feed back facial information and obtain a more accurate simulated human face image.

[0021] In combination with the first aspect, in some embodiments, the first electronic device is an eyeglass device, and the first electronic device includes a first frame, a second frame, a first frame arm connected to the first frame, and a second frame arm connected to the second frame; a speaker is arranged at a first position in the first frame and the second frame, a microphone is arranged at a second position in the first frame and the second frame, and a posture sensor is arranged at a third position in the first frame and the second frame; a speaker is arranged at a first position in the first frame arm and the second frame arm, and a microphone is arranged at a second position in the first frame arm and the second frame arm.

[0022] In the above embodiment, the microphone, the speaker and the posture sensor are symmetrically and evenly arranged in the frame arm and the frame, so that the ultrasonic audio data and posture data acquired by the smart glasses can further more accurately feedback the facial information and obtain a more accurate simulated face image.

[0023] In a second aspect, an embodiment of the present application provides a facial information acquisition method, which is applied to a first electronic device, wherein the first electronic device includes a speaker and a microphone, and the method includes: acquiring first ultrasonic audio data through the microphone, wherein the first ultrasonic audio data indicates an ultrasonic signal emitted by the speaker and reflected by the face; extracting facial state features of the face through the first ultrasonic audio data; and obtaining the facial state of the face based on the facial state feature mapping.

[0024] In the above embodiment, the electronic device can detect facial state features based on ultrasonic signals, and then map to obtain the actual facial state (a type of facial information). In this way, the electronic device can analyze the facial state without relying on the camera, and can obtain ultrasonic audio data based on the speaker and microphone carried by the electronic device itself to achieve facial state analysis.

[0025] In combination with the second aspect, in some embodiments, the first electronic device also includes a posture sensor, and the method also includes: acquiring first posture data through the posture sensor, the first posture data indicating the posture of the face; extracting posture features of the face through the first posture data; based on the facial state features, mapping to obtain the facial state of the face, specifically including: splicing the facial state features and the posture features to obtain splicing features; mapping the splicing features into the facial state.

[0026] In the above embodiment, the posture data is posture data for the head. The introduction of posture data for the head can be used to reduce the negative impact of head movement on facial expression recognition and analyze a more accurate facial state. If the posture data for the head is not introduced, the posture changes of the head and the facial expression changes will affect the reflection of the ultrasonic signal, and then will be reflected in the ultrasonic audio data. When the electronic device analyzes the facial state through ultrasonic audio data, there is a problem of inaccuracy. For example, the posture changes of the head will be mistaken for the facial expression changes, resulting in the generated facial state error.

[0027] In conjunction with the second aspect, in some embodiments, the facial state feature is determined by a first feature extraction network in a first facial state analysis model, and the first facial state analysis model also includes a first facial analysis network; the first facial analysis network is used to map the facial state feature into the facial state of the face; Among them, the first facial state analysis model is trained with the second ultrasonic audio data as input data, the predicted facial state as output data and the sample facial state as reference data; the sample facial state matches the facial state features extracted from the second ultrasonic audio data.

[0028] In combination with the second aspect, in some embodiments, the facial state feature is determined by a first feature extraction network in a second facial state analysis model, and the posture feature is determined by a second feature extraction network in the second facial state analysis model; the second facial state analysis model also includes a second facial state analysis network; the second facial state analysis network is used to map the spliced ​​feature into the facial state of the face, and the spliced ​​feature includes the posture feature and the facial state feature; wherein the first facial state analysis model is trained using the second ultrasonic audio data and the second posture data as input data, the predicted facial state as output data, and the sample facial state as reference data; the sample facial state matches the facial state feature extracted from the second ultrasonic audio data.

[0029] In a third aspect, an embodiment of the present application provides a first electronic device, which includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the first electronic device to execute the method implemented in the first aspect.

[0030] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, comprising instructions, which, when executed on an electronic device, enable the electronic device to execute the method implemented in the first aspect.

[0031] In a fifth aspect, an embodiment of the present application provides a chip system, which is applied to an electronic device. The chip system includes one or more processors, and the processor is used to call computer instructions so that the electronic device executes the method implemented in the first aspect.

[0032] In a sixth aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed on an electronic device, enables the electronic device to execute the method implemented in the first aspect.

[0033] It can be understood that the first electronic device provided in the third aspect, the computer storage medium provided in the fourth aspect, the chip system provided in the fifth aspect, and the computer program product provided in the sixth aspect are all used to execute the method provided in the embodiment of the present application. Therefore, other beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a schematic diagram of smart glasses provided in an embodiment of the present application; Figure 2 An exemplary scenario involved in obtaining training data in an embodiment of the present application is shown; Figure 3 Schematic diagram showing different facial states; Figure 4A A schematic diagram is shown for training a face image generation model a; Figure 4B A schematic diagram showing the steps involved in obtaining a preset face image; Figure 4C A schematic diagram is shown for generating a facial image using a trained model a; Figure 5A A schematic diagram is shown for training the face image generation model 1; Figure 5B A schematic diagram is shown for using the trained face image generation model 1; Fig. 6A A schematic diagram of the training of the facial state analysis model a is shown; Figure 6B A schematic diagram is shown in which the trained facial state analysis model a is used; Fig. 7A A schematic diagram is shown for training the facial state analysis model 1; Figure 7B A schematic diagram is shown for using the trained facial state analysis model 1; Figure 8A schematic diagram of a system structure for implementing the facial information acquisition method in an embodiment of the present application is shown; Fig. 9 A schematic diagram of a module for implementing the facial information acquisition method in an embodiment of the present application is shown; Fig.10 An interactive schematic diagram of implementing the facial information acquisition method in the embodiment of the present application is shown; Fig.11A A schematic diagram is shown for training a facial information generation model; Fig. 11B A schematic diagram is shown involving the use of facial information to generate a model; Fig.12 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] The embodiment of the present application provides a facial information acquisition method, which is applied to electronic devices, and in particular, can be applied to electronic devices that are not equipped with cameras. Implementation of this method can enable the electronic device to detect facial state features through ultrasound (also referred to as ultrasonic signals) and generate facial information that matches the actual facial state. The facial information here includes facial states and / or simulated face images. Among them, the facial state includes facial expressions, or the facial state includes facial expressions and facial directions. Facial expressions can be determined by facial features, such as blinking, opening the mouth, and closing the mouth. The direction of the face is determined by the posture of the head.

[0036] Among them, the solution of obtaining simulated face images can be used in scenarios such as video calls. The solution of obtaining facial status can be used in scenarios where the user's facial status needs to be detected, such as face recognition, fatigue driving detection, etc. Among them, in fatigue driving detection, the facial status can be used to analyze whether the user has his eyes open. If the eyes are not open, it is determined to be fatigue driving. Face recognition can analyze whether the user's face meets the requirements through the facial status. If it meets the requirements, the face recognition passes, otherwise it fails.

[0037] In the facial information acquisition method, a speaker and a microphone are configured in the electronic device. The electronic device can transmit an ultrasonic signal through the speaker, and the ultrasonic signal can be transmitted to the microphone after being reflected by the face. The electronic device can acquire ultrasonic audio data by collecting the ultrasonic signal reflected by the face through the microphone. The ultrasonic audio data carries or indicates the ultrasonic signal sent by the speaker and reflected by the face. Different facial states have different reflection conditions for ultrasonic signals, and the obtained ultrasonic audio data are different. For example, there are different frequency distribution performances. Therefore, by analyzing the ultrasonic audio data, facial state features can be acquired, and then facial information can be obtained. For example, when the face is in an open mouth state and a closed mouth state, the reflection conditions of the same ultrasonic signal are different. When the microphone collects the ultrasonic signal reflected by the face in these two different facial states (open mouth and closed mouth), different ultrasonic audio data can be acquired. The ultrasonic audio data acquired when the face is in an open mouth state includes the reflection conditions of the face to the ultrasonic signal when the mouth is open, which can be used to extract the facial state features when the face is in an open mouth state, and then map the face in an open mouth state. The ultrasonic audio data obtained when the face is in the closed mouth state includes the reflection of the ultrasonic signal by the face when the mouth is closed, which can be used to extract the facial state features when the face is in the closed mouth state, and then map the face in the closed mouth state.

[0038] The facial state feature can be regarded as a kind of data carrying the facial state, and the facial state can be obtained based on the facial state feature mapping. Different facial state features can be used to analyze different facial states. With the facial state feature, the preset face picture can also be adjusted by the facial state feature to obtain a simulated face picture. The simulated face picture is obtained by adjusting the facial state of the face in the preset face picture to the target facial state. For example, if the facial state feature indicates that the facial state includes an open mouth state, the mouth shape of the face in the preset face picture can be adjusted to an open mouth. Among them, the target facial state is obtained by facial state feature mapping. Among them, the preset face picture is recorded in the electronic device and is recorded as preset face picture 1.

[0039] The simulated face picture can be obtained by the face picture generation model 1 (after training). The electronic device can input the preset face picture 1 and the ultrasonic audio data 1 into the face picture generation model 1 (after training), and output the simulated face picture based on the preset face picture 1 and the ultrasonic audio data 1 through the face picture generation model 1 (after training). The ultrasonic audio data 1 is the ultrasonic audio data obtained by the electronic device through the microphone when obtaining facial information.

[0040] The face picture generation model 1 may include a facial state feature extraction network, a facial feature extraction network and a face generation network 1. The facial state feature extraction network is used to extract facial state features (referred to as facial state features 1) from the ultrasonic audio data 1. The facial feature extraction network is used to extract facial features (referred to as facial features 1) from the preset face picture 1. The face generation network 1 is used to adjust the facial features 1 through the facial state features 1, and map the adjusted facial features into a simulated face picture. In some possible cases, the face generation network 1 may include a convolutional network 11, a deconvolutional network 11 and a convolutional network 12. The functions of the convolutional network 11 and the deconvolutional network 11 include: adjusting the facial features 1 through the facial state features 1 to obtain the adjusted facial features. The convolutional network 12 is used to map the adjusted facial features into the simulated face picture.

[0041] The face picture generation model 1 is trained by using the ultrasonic audio data a and the preset face picture a as input data, the predicted face picture as output data and the sample face picture as reference data. The sample face picture matches the facial state features extracted based on the ultrasonic audio data a. For the relevant content of training the facial state analysis model 1, please refer to the following Figure 5A The description of is not repeated here.

[0042] The facial state can be obtained by the facial state analysis model 1 (after training). The electronic device can input the ultrasonic audio data 1 into the facial state analysis model 1 (after training), and the facial state analysis model 1 (after training) outputs the facial state based on the ultrasonic audio data 1.

[0043] The facial state analysis model 1 may include a facial state feature extraction network and a facial state analysis network 1. The facial state analysis network 1 is used to map the facial state feature 1 into a facial state. The facial state feature 1 is extracted based on the ultrasonic audio data 1 by the facial state feature extraction network.

[0044] The facial state analysis model 1 is trained by using the ultrasonic audio data a as input data, the predicted facial state as output data, and the sample facial state as reference data. The sample facial state matches the facial state features extracted from the ultrasonic audio data a. Fig. 7A The description of is not repeated here.

[0045] It should be noted here that when determining facial information through ultrasonic signals, changes in head posture and facial expression will affect the reflection of ultrasonic signals, and will be reflected in the ultrasonic audio data 1. When electronic devices analyze facial information through ultrasonic audio data 1, there is a problem of inaccuracy. For example, changes in head posture may be mistaken for changes in facial expression, resulting in incorrect facial information being generated.

[0046] In order to improve the accuracy of generating facial information, the data used to measure facial information may include, in addition to the ultrasonic audio data obtained through the ultrasonic signal, posture data for the head. The posture data may be obtained through a posture sensor in an electronic device. The posture data may be used to indicate the posture of the head and reflect the direction or movement of the face.

[0047] In the case of introducing posture data for the head, the aforementioned electronic device adjusts the preset face picture by facial state features to obtain a simulated face picture, which may include: the electronic device adjusts the preset face picture by facial state features and posture features to obtain a simulated face picture, and the simulated face picture has the posture indicated by the posture feature. Among them, the posture feature is extracted from the posture data. The posture feature can be used to determine the change in facial direction caused by the change in head posture, reduce the negative impact of head movement on facial expression recognition, and generate a more accurate simulated face picture.

[0048] At this time, the simulated face picture can be obtained by the face picture generation model a (after training). The electronic device can input the preset face picture 1, the ultrasonic audio data 1 and the posture data 1 into the face picture generation model a (after training), and output the simulated face picture based on the preset face picture 1, the ultrasonic audio data 1 and the posture data 1 through the face picture generation model a (after training).

[0049] The face picture generation model a may include a facial state feature extraction network, a facial feature extraction network, a posture feature extraction network and a face generation network a. The facial state feature extraction network is used to extract facial state features (referred to as facial state features 1) from the ultrasonic audio data 1. The facial feature extraction network is used to extract facial features (referred to as facial features 1) from the preset face picture 1. The posture feature extraction network is used to extract posture features (referred to as posture features 1) from the posture data 1. The face generation network a is used to adjust the facial features 1 through the facial state features 1 and the posture features 1, and map the adjusted facial features into a simulated face picture. In some possible cases, the face generation network a may include a convolution network 21, a deconvolution network 21 and a convolution network 22. The functions of the convolution network 21 and the deconvolution network 21 include: adjusting the facial features 1 through the facial state features 1 and the posture features 1 to obtain the adjusted facial features. The convolution network 22 is used to map the adjusted facial features into the simulated face picture.

[0050] The face image generation model a is trained by using the ultrasonic audio data a, the preset face image a and the posture data a as input data, the predicted face image as output data and the sample face image as reference data. For the relevant content of training the face image generation model a, please refer to the following Figure 4A as well as Figure 4B The description of is not repeated here.

[0051] In the case of introducing the posture data for the head, the aforementioned electronic device obtains the facial state based on the facial state feature mapping, which may include: the electronic device splices the facial state feature and the posture feature to obtain a spliced ​​feature, and maps the spliced ​​feature into the facial state.

[0052] At this time, the facial state can be obtained through the facial state analysis model a (after training). The electronic device can input the ultrasonic audio data 1 and the posture data 1 into the facial state analysis model a (after training), and the facial state analysis model a (after training) outputs the facial state based on the ultrasonic audio data 1 and the posture data 1.

[0053] The facial state analysis model a may include a facial state feature extraction network, a posture feature extraction network and a facial state analysis network a. The facial state analysis network a is used to map the spliced ​​features into facial states. The spliced ​​features include facial state feature 1 and posture feature 1. The facial state feature 1 is extracted by the facial state feature extraction network based on ultrasonic audio data 1. The posture feature 1 is extracted by the posture feature extraction network based on posture data 1.

[0054] The facial state analysis model a is trained using the ultrasonic audio data a and the posture data a as input data, the predicted facial state as output data, and the sample facial state as reference data. The sample facial state matches the facial state features extracted from the ultrasonic audio data a. Fig. 6A The description of is not repeated here.

[0055] It should be noted here that in the scheme where the posture data for the head is not introduced, the electronic device includes a wearable device that can be worn on the head. It may also include an electronic device that is not intended to be worn on the head. When implementing the facial information acquisition method, the wearer is prompted to align the electronic device worn on the non-head with the face. In the scheme where the posture data for the head is introduced, the electronic device includes a wearable device that can be worn on the head. Among them, the wearable device worn on the head may include: smart glasses, mixed reality headset (MR), AR glasses, VR glasses and other wearable devices. Electronic devices that are not intended to be worn on the head may include: smart watches, smart phones, etc. The embodiments of the present application are not limited to this.

[0056] It should also be noted that if the electronic device is a wearable device, a condition for executing the facial information acquisition method includes: the electronic device confirms that the user has worn the wearable device on the head. And in the process of executing the facial information acquisition method, it is also necessary to detect whether the wearable device is still worn on the head according to a preset period. If it is not worn on the head, the facial information acquisition method is stopped. Otherwise, it can continue to be executed.

[0057] The following content takes the electronic device as smart glasses as an example, and describes the training and application of the face picture generation model a and the facial state analysis model a when the head posture data is introduced. And the training and application of the face picture generation model 1 and the facial state analysis model 1 when the head posture data is not introduced.

[0058] Based on the above content, it can be known that in the case of introducing posture data for the head, a microphone and a speaker need to be configured in the smart glasses to obtain ultrasonic audio data. A posture sensor is also required to obtain posture data. Then, the ultrasonic audio data and posture data are transmitted to the face picture generation model a to generate a simulated face picture, or the ultrasonic audio data and posture data are transmitted to the facial state analysis model a to obtain the facial state.

[0059] In order to make the acquired ultrasonic audio data and posture data more accurately feedback facial information, microphones, speakers and posture sensors can be arranged on the smart glasses in a bilaterally symmetrical manner, including: at least one speaker, at least one microphone and at least one posture sensor are arranged on both sides of the central axis of the smart glasses.

[0060] like Figure 1 As shown, the smart glasses include a frame 1, a frame 2, a frame arm 1 connected to the frame 1, and a frame arm 2 connected to the frame 2. The smart glasses are configured with 4 microphones, 4 speakers, and 2 posture sensors, which are symmetrically arranged in the smart glasses. A speaker is arranged at position 1 of the frame 1 and the frame 2, and a microphone is arranged at position 2 of the frame 1 and the frame 2. A speaker is arranged at position 1 of the frame arm 1 and the frame arm 2, and a microphone is arranged at position 2 of the frame arm 1 and the frame arm 2. The 2 posture sensors can be symmetrically arranged in the frame 1 and the frame 2, or the 2 posture sensors can be symmetrically arranged in the frame arm 1 and the frame arm 2.

[0061] In some possible cases, the attitude sensor may be an inertial measurement unit (IMU). The frame carries the lenses, and the lenses of the smart glasses may serve as a display of the smart glasses to provide display content to the wearer.

[0062] It should be noted that the following description is based on an example in which four microphones, four speakers, and two gesture sensors are configured in the smart glasses. The description of other numbers of microphones, speakers, and gesture sensors can be referred to herein, and four microphones, four speakers, and two gesture sensors should not be used as limitations on the embodiments of the present application. If gesture data is not introduced, the gesture sensor may not be arranged in the smart glasses.

[0063] It should also be noted that Figure 1 The description in and the description of arranging the microphone, the speaker, and the gesture sensor in a bilaterally symmetrical manner are not limited to smart glasses, and other electronic devices may also be adopted. For example, other wearable devices other than smart glasses, such as VR glasses, etc.

[0064] The following is based on Figure 2 , Figure 3 , Figure 4A as well as Figure 4B Describe the relevant content of training a face image generation model a when introducing head posture data.

[0065] Training the facial image generation model a requires a large amount of training data. The training data includes: posture data required for training (the posture data a mentioned above), ultrasonic audio data required for training (the ultrasonic audio data a mentioned above), preset facial images required for training (the preset facial images a mentioned above) and sample facial images. Training the facial image generation model a once requires at least one frame of posture data a, one frame of ultrasonic audio data a, preset facial image a and one frame of sample facial image. The sample facial image frame used in each training is matched with the posture data a and the ultrasonic audio data a. It can also be understood that the posture data obtained when the face is in the posture indicated in the sample facial image frame is the posture data a. At the same time, the ultrasonic audio data obtained when the face is in the posture indicated in the sample facial image frame is the ultrasonic audio data a.

[0066] The process of obtaining training data can refer to the following Figure 2 as well as Figure 3 Description.

[0067] like Figure 2 As shown, at time Tc, the subject wears smart glasses to imitate the demonstrator displayed in the acquisition device to adjust the facial state. At time Tc, the acquisition device obtains the face image of the subject (the user wearing the smart glasses) through the camera connected to it, and obtains a frame of sample face image. At the same time, the smart glasses send ultrasonic signals through 4 speakers at time Tc, and obtain ultrasonic signals reflected by the face through 4 microphones to obtain a frame of ultrasonic audio data a. In addition, the smart glasses obtain a frame of posture data a through 2 posture sensors at time Tc, which is used to indicate the posture of the subject's head at time Tc. Among them, a frame of ultrasonic audio data a can include ultrasonic audio data obtained by 4 microphones at time Tc. A frame of posture data a can include posture data obtained by 2 posture sensors at time Tc. After multiple times Tc, more training data can be obtained.

[0068] It should be noted here that the acquisition device can display the presenter's screen in different facial states. When the subject follows the presenter to adjust the facial state, richer training data can be obtained. At the same time, for any presenter screen in the presenter screen in different facial states, the acquisition device can also display any presenter screen multiple times to obtain training data with reference. The differences in the presenter screen in different facial states may include any one or more of the following: Figure 3 As shown in (1), the differences in the presenter images under different facial states may include different facial expressions. Figure 3 As shown in (2), the differences in the presenter images under different facial states may include different head postures.

[0069] In some possible cases, the training data may be in the form of a data stream, which includes multiple frames of continuous ultrasonic audio data a, multiple frames of continuous posture data a, and multiple frames of continuous sample face picture frames. Any frame of ultrasonic audio data a in the multiple frames of continuous ultrasonic audio data a corresponds to a frame of posture data a and a frame of sample face picture with the same timestamp. The timestamp of a data indicates the time when the data is collected.

[0070] After obtaining the training data, the face image generation model a can be trained based on the training data. The description of this process can refer to the following Figure 4A .

[0071] like Figure 4A As shown, the face image generation model a in training may include a posture feature extraction network to be trained, a facial state feature extraction network, a face feature extraction network and a face generation network a. The face generation network a includes a convolution network 21 to be trained, a deconvolution network 21 and a convolution network 22. The role of each network can refer to the above-mentioned related content, which will not be repeated here.

[0072] The posture data a, ultrasonic audio data a and sample human face images are input into the human face image generation model a to be trained frame by frame, and the timestamps of the training data involved in the training of the human face image generation model a are the same. Figure 4A As shown, a frame of posture data a input into the face image generation model a can also be called a target posture data frame a, a frame of ultrasonic audio data a input into the face image generation model a can also be called an ultrasonic audio data frame a, and a frame of sample face image input into the face image generation model a can be called a sample face image frame.

[0073] In the process of training the face image generation model a, see Figure 4A In (1), the posture feature extraction network extracts features from the target posture data frame a and obtains posture feature a. The posture feature a can be represented as a 1*56*56 matrix. Figure 4A At (2), the facial state feature extraction network extracts features from the ultrasonic audio data frame a to obtain facial state feature a. The facial state feature a can be represented as a 128*56*56 matrix. Figure 4AIn (3), the facial feature extraction network can extract features from the preset facial image a to obtain facial features a. The facial features a can be represented as a 64*56*56 matrix. Then, the facial image generation model a performs feature splicing on the posture features a, facial state features a, and facial features a. The spliced ​​features are transmitted to the face generation network a. Here, feature splicing is to directly combine the posture features a, facial state features a, and facial features a to form a longer feature vector (i.e., spliced ​​features), without changing the values ​​of each element in each feature. The spliced ​​features can be represented as a 193 (1+128+64)*56*56 matrix.

[0074] Subsequently, the spliced ​​features are transmitted to the face generation network a, in which the face features a included in the spliced ​​features are adjusted by the convolution network 21 and the deconvolution network 21 using the facial state features a and the posture features a included in the spliced ​​features to obtain the adjusted face features. The convolution network 22 then maps the adjusted face features into a frame of predicted face image (predicted face image frame). The predicted face image frame can be represented as a 3*112*112 matrix. It should be noted that the predicted face image frame can be understood as a simulated face image output by the face image generation model a in training.

[0075] Subsequently, the difference between the predicted face frame and the sample face frame is calculated by the face image generation model a to obtain the loss function 1. The parameters in each network in the face image generation model a are adjusted based on the loss function 1. For example, when the loss function 1 is greater than the preset threshold, the parameters in each network in the face image generation model a are adjusted. When the loss function 1 is continuously less than the preset threshold, the parameters in each network in the face image generation model a are not adjusted in this training.

[0076] The conditions for the termination of the training of the face image generation model a include but are not limited to one or more of the following: Item 1, the loss function 1 is less than the preset threshold value N times in a row. Wherein N is an integer greater than or equal to 1. Item 2, the number of training times of the face image generation model a reaches the preset number of times 1. Item 3, the number of times the parameters in the face image generation model a are updated reaches the preset number of times 2. The preset number of times 1 and the preset number of times 2 may be equal or unequal. The preset number of times 1 and the preset number of times 2 are usually larger values, such as values ​​greater than 100, and this is not limited in the embodiments of the present application. The conditions for the termination of the training of other models in the embodiments of the present application also include but are not limited to at least one of the above three items.

[0077] The aforementioned Figure 4AIn the description, the posture data a, ultrasonic audio data a and sample face picture a input into the face picture generation model a to be trained are taken as one frame for explanation. The posture data a, ultrasonic audio data a and sample face picture a actually input into the face picture generation model a to be trained can also be multiple frames, and the face picture generation model a correspondingly outputs multiple frames of predicted face picture frames. The embodiment of the present application does not limit this, depending on the training method of the model. It can also be understood that the posture data a can include at least one frame of posture data. The ultrasonic audio data a can include at least one frame of ultrasonic audio data.

[0078] It should be noted here that the target posture data frame a can be understood as a posture data input into the face image generation model a to be trained as mentioned above. The target posture data frame a can be represented as a 12*100 matrix. Among them, 12 means that the target posture data frame a includes the absolute gravity acceleration on the three axes determined based on the initial posture data frame a1 and the linear force acceleration on the three axes, and also includes the absolute gravity acceleration on the three axes determined based on the initial posture data frame a2 and the linear force acceleration on the three axes. The absolute gravity acceleration on any axis and the linear force acceleration on any axis can be represented by 100 elements. The acquisition process of the target posture data frame a can refer to the following description.

[0079] The target attitude data frame a is obtained through the attitude sensor. Figure 4A As shown, the acquisition process includes: in the process of acquiring training data, the two posture sensors (posture sensor 1 and posture sensor 2) in the smart glasses continuously collect initial posture data. The initial posture data collected by posture sensor 1 can be called initial posture data a1, and the initial posture data collected by posture sensor 2 can be called initial posture data a2. Acquiring a target posture data frame a every time Tc includes: converting a frame of initial posture data a1 (for example, Figure 4A The initial posture data frame a1 in the figure) and the initial posture data frame a2 (for example Figure 4A The initial posture data frame a2 in the figure is processed by six-axis fusion to obtain the target posture data frame a. The initial posture data frame a1 and the initial posture data frame a2 may include the accelerometer data on three axes of the smart glasses within the time Tc, and the gyroscope data on three axes. The three axes include the X-axis, the Y-axis and the Z-axis. Among them, the X-axis generally represents an axis in the horizontal direction. The Y-axis is usually another axis in the horizontal direction, which is perpendicular to the X-axis. The Z-axis represents the vertical direction, which is perpendicular to the surface of the earth. Since the positions of the two posture sensors in the smart glasses are different, the accelerometer data on three axes and the gyroscope data on three axes included in the initial posture data frame a1 and the initial posture data frame a2 may be different, which can reflect more detailed postures.

[0080] Among them, the six-axis fusion processing includes fusing the accelerometer data on three axes and the gyroscope data on three axes in the initial posture data frame a1 to obtain more comprehensive target posture data. For example, the target posture data obtained after the six-axis fusion processing of the initial posture data frame a1 can be the absolute gravity acceleration on the three axes and the linear force acceleration on the three axes indicated in the initial posture data frame a1. In addition, the six-axis fusion processing also includes fusing the accelerometer data on three axes and the gyroscope data on three axes in the initial posture data frame a2 to obtain more comprehensive target posture data. For example, the target posture data obtained after the six-axis fusion processing of the initial posture data frame a2 can be the absolute gravity acceleration on the three axes and the linear force acceleration on the three axes indicated in the initial posture data frame a2.

[0081] In addition to the absolute gravity acceleration on three axes and the linear force acceleration on three axes, other data may be used to comprehensively represent the initial posture data frame a1, such as the pitch angle, yaw angle, and roll angle on three axes, etc. This embodiment of the application does not limit this.

[0082] The ultrasonic audio data frame a can be understood as the ultrasonic audio data input into the face image generation model a to be trained as mentioned above. The acquisition process of the ultrasonic audio data frame a can refer to the following description.

[0083] The ultrasonic audio data frame a is obtained through a microphone. Figure 4A As shown, the acquisition process includes: in the process of acquiring training data, the four microphones (including microphone 1 to microphone 4) in the smart glasses continuously acquire initial audio data. The initial audio data acquired by microphone 1, microphone 2, microphone 3 and microphone 4 are respectively referred to as initial audio data a1, initial audio data a2, initial audio data a3 and initial audio data a4. Acquiring an ultrasonic audio data frame a once every time Tc includes: converting a frame of initial audio data a1 (for example, Figure 4A An initial audio data frame a1 in the audio data, an initial audio data frame a2 (for example Figure 4A An initial audio data frame a2 in the audio data, an initial audio data frame a3 (for example Figure 4A The initial audio data frame a3 in the figure) and the initial audio data frame a4 (for example Figure 4A The initial audio data frame a4 in is recorded and Fourier transformed to obtain the ultrasonic audio data frame a.

[0084] The initial audio data collected by each microphone includes not only ultrasonic audio data indicating ultrasonic signals (reflected), but may also include audible audio data indicating audible sound signals. Therefore, it is necessary to filter the initial audio data a1-initial audio data a4 to remove the audible audio data therein. Only the ultrasonic audio data is retained. Then, the filtered initial audio data frame a1-filtered initial audio data frame a4 is Fourier transformed to obtain ultrasonic audio data frame a. It should be noted that the Fourier transform is to convert the filtered initial audio data frame a1-filtered initial audio data frame a4 from the time domain to the ultrasonic audio data frame a in the frequency domain that is easy to process.

[0085] The ultrasonic audio data frame a can be represented as a 4*2048*2048 matrix. Among them, 4 means that the ultrasonic audio data frame a includes ultrasonic audio data in the frequency domain determined based on 4 initial audio data frames (initial audio data frame a1-initial audio data frame a4). The ultrasonic audio data in the frequency domain determined on any initial audio data frame in the initial audio data frame a1-initial audio data frame a4 can be represented by a 2048*2048 matrix. One of the 2048 means that any initial audio data frame is divided into 2048 time points. Another 2048 means that the initial audio data frame can be analyzed into 2048 audio data components of different frequencies at one time point.

[0086] The preset face picture a and the sample face picture are both face pictures of the subject. The sample face picture is obtained by collecting the training data through the camera, and the preset face picture a can be preset. It can be obtained by image preprocessing the training picture a with a face. Figure 4B As shown, the training picture a with a face (from the subject) is image preprocessed, and after the face position is confirmed, the face image is segmented to obtain the subject's face. From the edge of the face, a rectangular area including the face is obtained to obtain a preset face picture a. The other content in the preset face picture a except the face can be set as the background, and the pixel values ​​of each pixel in the background are the same, such as gray or white. In order to make the obtained predicted face picture frame more realistic and reliable, the face in the training picture a with a face should be the front face of the subject after wearing the smart glasses.

[0087] It should be noted that the above content describes a training process once. In actual situations, the face image generation model a needs to be trained multiple times. Figure 4A As shown, subsequent initial posture data, initial audio data and sample face images are obtained from front to back according to time (t) to continue training the face image generation model a. The training process can refer to the aforementioned related content and will not be repeated here.

[0088] It should also be noted that in the embodiment of the present application, time Tc is a period of time, representing the time from the start to the end (which can be understood as the duration) of the audio corresponding to a frame of ultrasonic audio data (which can also be referred to as an ultrasonic audio data frame). Ultrasonic audio data with a duration of time Tc can also be referred to as an ultrasonic audio data frame.

[0089] When performing Fourier transform on the ultrasonic audio data frame, time Tc can be understood as the duration of the observation window in the Fourier transform. The length of time Tc determines the resolution of the observation window for the ultrasonic audio data. An observation window with a duration of Tc can distinguish audio data with a frequency interval exceeding 1 / Tc (Hz). For example, when the frequency interval between audio data 1 and audio data 2 in the ultrasonic audio data frame is greater than 1 / Tc (Hz), they can be distinguished as different audio by Fourier transform, otherwise they are the same audio.

[0090] According to the Doppler effect, at time Tc, after the speaker sends an ultrasonic signal, the ultrasonic signal is reflected by the face. In the process of collecting the reflected ultrasonic signal by the microphone to obtain the ultrasonic audio data frame, if the facial state changes, part or all of the position of the face will be relatively displaced relative to the speaker and the microphone. The relative displacement will cause the frequency of the reflected ultrasonic signal to change, and the change in frequency can be reflected in the ultrasonic audio data frame. Figure 4A As shown, the ultrasonic audio data frame can be considered to change linearly in a short period of time, and the frequency change slope (S) of the ultrasonic audio data frame can be expressed as: S indicates the frequency change of the ultrasonic signal emitted by the speaker that can be changed per unit time. Instructed The frequency interval of the ultrasonic audio data frame. If Fourier transform is required, the bandwidth (B) of the ultrasonic signal emitted by the speaker needs to satisfy: B = S × Tc. B determines the effective distance ( ). When the relative displacement of a facial position is greater than , the change of this position can be captured by the reflected ultrasonic signal, thus reflected in the ultrasonic audio data frame, and then converted into a simulated face image by the face image generation model. The determination process of is shown in the following formula (1) to formula (3).

[0091]

[0092] In formula (1), v represents the propagation speed of ultrasonic signals in air, which is about 340 m / s (meters per second). Other parameters can be referred to the above description and will not be repeated here. The equation in formula (2) is obtained by converting formula (1). Inequality Indicates: When the frequency interval of different audio data in the ultrasonic audio data frame When the frequency interval is greater than 1 / Tc, it can be distinguished by Fourier transform. Formula (3) can be obtained by converting formula (2). It indicates that when the audio data with a frequency interval greater than 1 / Tc in the ultrasonic audio data frame can be distinguished by Fourier transform, the effective distance and the bandwidth of the ultrasonic signal emitted by the speaker. This relationship can be expressed as For example, when the bandwidth of the ultrasonic signal emitted by the speaker is 80kHz, the effective distance It can be 2.125mm. It can be seen that the larger the bandwidth B of the ultrasonic signal emitted by the speaker, the greater the effective distance The smaller the distance, the easier it is to make a relative displacement of a position on the face larger than , the change in position can be more easily captured by the reflected ultrasonic signal.

[0093] The bandwidth of the ultrasonic signal refers to the frequency range of the ultrasonic signal, that is, the difference between the maximum frequency and the minimum frequency of the ultrasonic signal. For example, when the bandwidth of the ultrasonic signal is 80 kHz, the frequency range of the ultrasonic signal may be 20 kHz-100 kHz.

[0094] The above description of the observation window and the Doppler effect is not only applicable to Figure 4A The ultrasonic audio data involved in the invention are also applicable to other ultrasonic audio data.

[0095] Because the computing power of smart glasses is limited, the aforementioned process of training the face image generation model a can be performed on other electronic devices with better computing power. If the computing power of the electronic device implementing the facial information acquisition method is strong, the face image generation model a can also be trained in the electronic device.

[0096] After the training of the face image generation model a is completed, the trained face image generation model a can be placed in the smart glasses, so that the smart glasses can use the face image generation model a to obtain a simulated face image through the posture data 1, the ultrasonic audio data 1 and the preset face image 1. The process of the smart glasses using the face image generation model a can refer to the following Figure 4C Description.

[0097] like Figure 4CAs shown, the trained face image generation model a may include a trained posture feature extraction network, a facial state feature extraction network, a face feature extraction network, and a face generation network a. The face generation network a includes a trained convolution network 21, a deconvolution network 21, and a convolution network 22. The functions of each network can refer to the aforementioned related content and will not be repeated here.

[0098] In the scenario of using the trained face picture generation model a to generate a simulated face picture. The posture sensor 1 and the posture sensor 2 in the smart glasses can respectively continuously collect initial posture data 11 and initial posture data 12, and at the same time, the microphone 1-microphone 4 in the smart glasses can respectively continuously collect initial audio data 11-initial audio data 14. Obtain a frame of target posture data frame 1 and a frame of ultrasonic audio data frame 1 every time Tc. Transmit the target posture data frame 1, the ultrasonic audio data frame 1 and the preset face picture 1 to the trained face picture generation model a to obtain a frame of simulated face picture. This process is executed cyclically with time t, and multiple frames of simulated face pictures can be obtained.

[0099] In the scenario of generating multiple frames of simulated face images, the preset face image 1 input into the face image generation model a can remain unchanged, the posture data 1 and the ultrasonic audio data 1 are input into the trained face image generation model a frame by frame, and the timestamps of the frame of posture data 1 (also referred to as the target posture data frame 1) and the frame of ultrasonic audio data 1 (also referred to as the ultrasonic audio data frame 1) used when input into the face image generation model a to obtain a frame of simulated face image (also referred to as simulated face image frame) are the same. The process of generating a frame of simulated face image can refer to the following Figure 4C Middle (1) - Figure 4C The description in (3) above.

[0100] See Figure 4C In (1), the smart glasses extract features from the target posture data frame 1 through the posture feature extraction network to obtain posture feature 1. The posture feature 1 can be represented as a 1*56*56 matrix. Figure 4C At (2), the smart glasses extract features from the ultrasonic audio data frame 1 through the facial state feature extraction network to obtain facial state feature 1. The facial state feature 1 can be represented as a 128*56*56 matrix. Figure 4CIn (3), the smart glasses can extract features from the preset face image 1 through the face feature extraction network to obtain face feature 1. The face feature 1 can be represented as a 64*56*56 matrix. Then, the face image generation model a is used to perform feature splicing on the posture feature 1, the facial state feature 1 and the face feature 1. Here, feature splicing is to directly combine the posture feature 1, the facial state feature 1 and the face feature 1 to form a longer feature vector (i.e., splicing feature), without changing the value of each element in each feature. The splicing feature can be represented as a 193 (1+128+64)*56*56 matrix.

[0101] Subsequently, the smart glasses transmit the spliced ​​features to the face generation network a. In the face generation network a, the facial features 1 included in the spliced ​​features are adjusted by the convolution network 21 and the deconvolution network 21 using the facial state features 1 and the posture features 1 included in the spliced ​​features to obtain the adjusted facial features. The convolution network 22 then maps the adjusted facial features into a simulated face picture frame. The simulated face picture frame can be represented as a 3*112*112 matrix.

[0102] Among them, the target posture data frame 1 is obtained through the posture sensor. Figure 4C As shown, the acquisition process includes: a frame of initial posture data 11 (for example Figure 4C The initial posture data frame 11) and a frame of initial posture data 12 ( Figure 4C The initial posture data frame 12 in the image is processed by six-axis fusion to obtain the target posture data frame 1. Here, the process of six-axis fusion processing and obtaining the target posture data frame 1 is the same as the above Figure 4A The process of six-axis fusion processing and obtaining the target posture data frame a is the same, and you can refer to the description of the above-mentioned related content and make adaptive modifications. For example, the initial posture data a1 and the initial posture data a2 are replaced by the initial posture data 11 and the initial posture data 12 respectively, which will not be repeated here. The target posture data frame 1 and the target posture data frame a can both be represented as a 12*100 matrix.

[0103] Ultrasonic audio data frame 1 is obtained through a microphone. Figure 4C As shown, the acquisition process includes: a frame of initial audio data 11 (for example Figure 4C An initial audio data frame 11 in the audio data, from microphone 1), an initial audio data frame 12 ( Figure 4C An initial audio data frame 12 in the audio source, from microphone 2), an initial audio data frame 13 ( Figure 4C The initial audio data frame 13 in the image comes from microphone 3) and the initial audio data frame 14 ( Figure 4CThe initial audio data frame 14 in the image is from the microphone 4) and is filtered and Fourier transformed to obtain the ultrasonic audio data frame 1. Here, the filtering and Fourier transform process to obtain the ultrasonic audio data frame 1 is the same as the above process. Figure 4A The process of filtering and Fourier transforming and obtaining the ultrasonic audio data frame a is the same, and you can refer to the description of the above-mentioned related content and make adaptive modifications. For example, the initial audio data a1-initial audio data a4 are replaced with initial audio data 11-initial audio data 14 respectively, and no further description is given here. The ultrasonic audio data frame 1 and the ultrasonic audio data frame a can both be represented as a 4*2048*2048 matrix.

[0104] The preset face picture 1 is the face picture of the wearer. It can be obtained by performing image preprocessing on the preset face picture 1. In order to make the generated simulated face picture frame more realistic and reliable, the face in the preset face picture 1 should be the front face of the wearer after wearing the smart glasses. The process of obtaining the preset face picture 1 is the same as Figure 4A The process of obtaining the preset face picture a is the same as that of the training picture a of the face, and the adaptation modification can be performed. For example, the training picture a of the face is modified to the preset picture 1 of the face, and the embodiment of the present application will not be repeated. The preset face picture 1 and the preset face picture a can both be represented as a 3*112*112 matrix.

[0105] The aforementioned Figure 4C In the description, the posture data 1 and ultrasonic audio data 1 input into the trained face image generation model a are taken as an example for explanation. The posture data 1 and ultrasonic audio data 1 actually input into the trained face image generation model a can also be multiple frames, and the face image generation model a outputs multiple frames of simulated face image frames accordingly. It can also be understood that the posture data 1 can include at least one frame of posture data. The ultrasonic audio data 1 can include at least one frame of ultrasonic audio data.

[0106] The following is based on Figure 5A Describe the relevant content of training the face image generation model 1 without introducing the head posture data.

[0107] It should be noted that, compared with the face image generation model a, the training data used by the face image generation model 1 includes the ultrasonic audio data a, the preset face image a and the sample face image. However, it does not include the posture data a. The method for obtaining the training data required for training the face image generation model 1 can refer to the aforementioned Figure 2 as well as Figure 3 After obtaining the training data, the face image generation model a can be trained based on the training data. The description of this process can refer to the following Figure 5A .

[0108] like Figure 5A As shown, the face image generation model 1 in training may include a facial state feature extraction network to be trained, a face feature extraction network, and a face generation network 1. The face generation network 1 includes a convolutional network 11 to be trained, a deconvolutional network 11, and a convolutional network 12. The functions of each network can refer to the aforementioned related content, which will not be repeated here.

[0109] The ultrasonic audio data a and the sample human face images are input into the human face image generation model 1 to be trained frame by frame, and the timestamps of the training data involved in the training are the same when they are input into the human face image generation model 1 for one training. Figure 5A As shown, a frame of ultrasonic audio data a input into the facial image generation model 1 can also be called an ultrasonic audio data frame a, and a frame of sample facial image input into the facial image generation model 1 can be called a sample facial image frame.

[0110] In the process of training the face image generation model 1, see Figure 5A At (1), the facial state feature extraction network extracts features from the ultrasonic audio data frame a to obtain facial state feature a. The facial state feature a can be represented as a 128*56*56 matrix. Figure 5A At (2), the facial feature extraction network can extract features from the preset facial image a to obtain facial features a. The facial features a can be represented as a 64*56*56 matrix. Then, the facial image generation model 1 performs feature splicing on the posture features a, facial state features a, and facial features a. The spliced ​​features are transmitted to the face generation network 1. Here, feature splicing is to directly combine the posture features a, facial state features a, and facial features a to form a longer feature vector (i.e., spliced ​​features), without changing the values ​​of each element in each feature. The spliced ​​features can be represented as a 192 (128 + 64) * 56 * 56 matrix.

[0111] Subsequently, the spliced ​​features are transmitted to the face generation network 1. In the face generation network 1, the facial state features a included in the spliced ​​features are used by the convolution network 11 and the deconvolution network 11 to adjust the facial features a included in the spliced ​​features to obtain the adjusted facial features. The convolution network 12 then maps the adjusted facial features into a frame of predicted facial image (predicted facial image frame). The predicted facial image frame can be represented as a 3*112*112 matrix. It should be noted that the predicted facial image frame can be understood as a simulated facial image output by the facial image generation model 1 in training.

[0112] Subsequently, the difference between the predicted face frame and the sample face frame is calculated by the face frame generation model 1 to obtain the loss function 2. The parameters in each network in the face frame generation model 1 are adjusted based on the loss function 2. For example, when the loss function 2 is greater than a preset threshold, the parameters in each network in the face frame generation model 1 are adjusted. When the loss function 2 is continuously less than the preset threshold, the parameters in each network in the face frame generation model 1 are not adjusted in this training.

[0113] It should be noted that the above content describes a training process once. In actual situations, the face image generation model 1 needs to be trained multiple times. Figure 5A As shown, subsequent initial audio data and sample face images are obtained from the front to the back according to time (t) to continue training the face image generation model 1. The training process can refer to the aforementioned related content and will not be repeated here.

[0114] It should also be noted that Figure 5A and Figure 4A The acquisition process and definition of the data and methods with the same name can refer to the above Figure 4A The description of the relevant data in the above description will not be repeated here. Here, the data with the same name includes the sample face picture frame, the ultrasonic audio data frame a, and each initial data frame (including the initial audio data frame a1-initial audio data frame a4), etc. The methods with the same name include filtering, image preprocessing, etc.

[0115] After the training of the face image generation model 1 is completed, the trained face image generation model 1 can be placed in the smart glasses, so that the smart glasses can use the face image generation model 1 to obtain a simulated face image through the ultrasonic audio data 1 and the preset face image 1. The process of the smart glasses using the face image generation model 1 can refer to the following Figure 5B Description.

[0116] like Figure 5B As shown, the trained face image generation model 1 may include a trained posture feature extraction network, a facial state feature extraction network, a face feature extraction network, and a face generation network 1. The face generation network 1 includes a trained convolution network 11, a deconvolution network 11, and a convolution network 12. The functions of each network can refer to the aforementioned related content and will not be repeated here.

[0117] In a scenario where a trained face image generation model 1 is used to generate a simulated face image, microphones 1 to 4 in the smart glasses can respectively continuously collect initial audio data 11 to initial audio data 14. A frame of ultrasonic audio data frame 1 is acquired every time Tc. The ultrasonic audio data frame 1 and the preset face image 1 are transmitted to the trained face image generation model 1 to obtain a frame of simulated face image. This process is executed repeatedly with time t to obtain multiple frames of simulated face images.

[0118] In the scenario of generating multiple frames of simulated face images, the preset face image 1 input into the face image generation model 1 may remain unchanged, and the ultrasonic audio data 1 is input into the trained face image generation model 1 frame by frame. The process of generating a frame of simulated face image can refer to the following Figure 5B (1) and Figure 5B The description in (2) above.

[0119] See Figure 5B At (1), the smart glasses extract features from the ultrasonic audio data frame 1 through the facial state feature extraction network to obtain facial state feature 1. The facial state feature 1 can be represented as a 128*56*56 matrix. The facial feature extraction network can extract features from the preset face image 1 to obtain facial feature 1. The facial feature 1 can be represented as a 64*56*56 matrix. The facial image generation model 1 is then used to perform feature splicing on the facial state feature 1 and the facial feature 1. Here, feature splicing is to directly combine the facial state feature 1 and the facial feature 1 together to form a longer feature vector (i.e., spliced ​​feature), without changing the value of each element in each feature. The spliced ​​feature can be represented as a 192 (128+64)*56*56 matrix.

[0120] Subsequently, the smart glasses transmit the spliced ​​features to the face generation network 1. In the face generation network 1, the facial features 1 included in the spliced ​​features are adjusted by the convolution network 11 and the deconvolution network 11 using the facial state features 1 included in the spliced ​​features to obtain the adjusted facial features. The convolution network 12 then maps the adjusted facial features into a simulated face picture frame. The simulated face picture frame can be represented as a 3*112*112 matrix.

[0121] It should be noted here that Figure 5B and Figure 4C The acquisition process and definition of the data and methods with the same name can refer to the above Figure 4CThe description of the relevant data in the above description will not be repeated here. Here, the data with the same name includes ultrasonic audio data frame 1, preset screen 1 with human face, and each initial data frame (including initial audio data frame 11-initial audio data frame 14), etc. The methods with the same name include filtering, Fourier transform, etc.

[0122] The following is based on Fig. 6A Describe the relevant content of training the facial state analysis model a when introducing the posture data for the head.

[0123] Like the aforementioned training face image generation model a, the training facial state analysis model a also requires a large amount of training data. The training data includes: posture data required for training (the aforementioned posture data a), ultrasonic audio data required for training (the aforementioned ultrasonic audio data a), and sample facial states required for training. The sample facial states are obtained by performing facial state recognition on sample face image frames. A sample face image frame can also obtain a corresponding sample facial state.

[0124] Training a facial state analysis model a requires at least one frame of posture data a, one frame of ultrasonic audio data a, and a sample facial state corresponding to a frame of sample face picture. The sample facial state used in each training is matched with the posture data a and the ultrasonic audio data a. It can also be understood that the posture data obtained when the face is in the facial state indicated in the sample face picture frame is the posture data a. At the same time, the ultrasonic audio data obtained when the face is in the facial state indicated in the sample face picture frame is the ultrasonic audio data a.

[0125] The content involved in obtaining the posture data a, ultrasonic audio data a and sample face frame involved in the training data is the same as the above Figure 2 as well as Figure 3 The same can be referred to above. Figure 2 as well as Figure 3 The description is not repeated here.

[0126] After obtaining the training data, the facial state analysis model a can be trained based on the training data. The description of this process can refer to the following Fig. 6A .

[0127] like Fig. 6A As shown, the facial state analysis model a in training may include a posture feature extraction network to be trained, a facial state feature extraction network and a facial state analysis network a. The facial state analysis network a includes a convolution network a1 to be trained, a deconvolution network a1 and a convolution network a2. The role of each network can refer to the above-mentioned related content, which will not be repeated here.

[0128] The posture data a, ultrasonic audio data a, and the sample facial state corresponding to the sample face image are input into the facial state analysis model a to be trained frame by frame, and the timestamps of the training data involved in the input into the facial state analysis model a for one training are the same. Fig. 6A As shown, a frame of posture data a input into the facial state analysis model a can also be called target posture data frame a, a frame of ultrasonic audio data a input into the facial state analysis model a can also be called ultrasonic audio data frame a, and a sample facial state corresponding to a frame of sample face picture input into the facial state analysis model a can be simply referred to as sample facial state, and the sample facial state is represented by a 1*112*112 matrix.

[0129] In the process of training the facial state analysis model a, see Fig. 6A In (1), the posture feature extraction network extracts features from the target posture data frame a and obtains posture feature a. The posture feature a can be represented as a 1*56*56 matrix. Fig. 6A At (2), the facial state feature extraction network extracts features from the ultrasonic audio data frame a to obtain facial state feature a. The facial state feature a can be represented as a 128*56*56 matrix. The facial state analysis model a then performs feature splicing on the posture feature a and the facial state feature a. The spliced ​​features are transmitted to the facial state analysis network a. Here, feature splicing is to directly combine the posture feature a and the facial state feature a to form a longer feature vector (i.e., spliced ​​feature) without changing the value of each element in each feature. The spliced ​​feature can be represented as a 129 (1+128)*56*56 matrix.

[0130] Subsequently, the spliced ​​features (splicing features) are transmitted to the facial state analysis network a, where the splicing features are mapped into predicted facial states through the convolution network a1, the deconvolution network a1, and the convolution network a2. Among them, the convolution network a1 can be used to upsample the input splicing features, and the deconvolution network 1 is used to deconvolve the upsampled splicing features output by the convolution network a1, and then input the deconvolved splicing features into the convolution network a2. The convolution network a2 downsamples based on the deconvolution splicing features to obtain the final facial state.

[0131] It should be noted that the predicted facial state is represented by a 1*112*112 matrix. The predicted facial state can be understood as the facial state output by the facial state analysis model a in training.

[0132] Subsequently, the difference between the predicted facial state and the sample facial state is calculated by the facial state analysis model a to obtain the loss function a. The parameters in each network in the facial state analysis model a are adjusted based on the loss function a. For example, when the loss function a is greater than a preset threshold, the parameters in each network in the facial state analysis model a are adjusted. When the loss function a is continuously less than the preset threshold, the parameters in each network in the facial state analysis model a are not adjusted in this training.

[0133] It should be noted that the above content describes a training process once. In actual situations, the facial state analysis model a needs to be trained multiple times. Fig. 6A As shown, the subsequent initial audio data and sample face images are obtained from the front to the back according to the time (t) to continue training the facial state analysis model a. The training process can refer to the aforementioned related content and will not be repeated here.

[0134] It should also be noted that Fig. 6A and Figure 4A The acquisition process and definition of the data and methods with the same name can refer to the above Figure 4A The description of the relevant data in the above description will not be repeated here. Here, the data with the same name include the target posture data frame a, the ultrasonic audio data frame a, and each initial data frame (including the initial posture data frame a1, the initial posture data frame a2, and the initial audio data frame a1-initial audio data frame a4), the sample face picture frame, etc. The methods with the same name include six-axis fusion processing and filtering, etc.

[0135] After the facial state analysis model a is trained, the trained facial state analysis model a can be placed in the smart glasses, so that the smart glasses can use the facial state analysis model a to obtain the facial state through the posture data 1 and the ultrasonic audio data 1. The process of the smart glasses using the facial state analysis model a can refer to the following Figure 6B Description.

[0136] like Figure 6B As shown, the facial state analysis model a after training can include a posture feature extraction network after training, a facial state feature extraction network and a facial state analysis network a. The facial state analysis network a includes a convolution network a1, a deconvolution network a1 and a convolution network a2 after training. The role of each network can refer to the above-mentioned related content, which will not be repeated here.

[0137] In the scenario of using the trained facial state analysis model a to generate facial states. The posture sensor 1 and the posture sensor 2 in the smart glasses can respectively continuously collect initial posture data 11 and initial posture data 12, and at the same time, the microphones 1-microphones 4 in the smart glasses can respectively continuously collect initial audio data 11-initial audio data 14. Obtain a frame of target posture data frame 1 and a frame of ultrasonic audio data frame 1 at intervals of time Tc. Transmit the target posture data frame 1 and the ultrasonic audio data frame 1 to the trained facial state analysis model a to obtain a facial state. This process is executed cyclically with time t, and multiple facial states can be obtained.

[0138] In the scenario of generating multiple facial states, the posture data 1 and the ultrasonic audio data 1 are input into the trained facial state analysis model a frame by frame, and the timestamps of a frame of posture data 1 (also referred to as the target posture data frame 1) and a frame of ultrasonic audio data 1 (also referred to as the ultrasonic audio data frame 1) used when input into the facial state analysis model a to obtain a facial state are the same. The process of generating a facial state can refer to the following Figure 6B (1) and Figure 6B The description in (2) above.

[0139] See Figure 6B In (1), the smart glasses extract features from the target posture data frame 1 through the posture feature extraction network to obtain posture feature 1. The posture feature 1 can be represented as a 1*56*56 matrix. Figure 6B At (2), the smart glasses extract features from the ultrasonic audio data frame 1 through the facial state feature extraction network to obtain the facial state feature 1. The facial state feature 1 can be represented as a 128*56*56 matrix. Then, the facial state analysis model a is used to perform feature splicing on the posture feature 1 and the facial state feature 1. Here, feature splicing is to directly combine the posture feature 1 and the facial state feature 1 together to form a longer feature vector (i.e., spliced ​​feature), without changing the value of each element in each feature. The spliced ​​feature can be represented as a 129 (1+128)*56*56 matrix.

[0140] Subsequently, the smart glasses transmit the spliced ​​features (spliced ​​features) to the facial state analysis network a, where the facial state is obtained by using the spliced ​​feature mapping through the convolution network a1, the deconvolution network a1, and the convolution network a2. The facial state can be represented as a 1*112*112 matrix.

[0141] It should be noted here that Figure 6B and Figure 4C The acquisition process and definition of the data and methods with the same name can refer to the above Figure 4CThe description of the relevant data in the above description will not be repeated here. Here, the data with the same name include target posture data frame 1, ultrasonic audio data frame 1, and each initial data frame (including initial posture data frame 11, initial posture data frame 12, and initial audio data frame 11-initial audio data frame 14). The same name method includes six-axis fusion processing and filtering.

[0142] The following is based on Fig. 7A Describe the relevant content of training the facial state analysis model 1 without introducing the posture data for the head.

[0143] It should be noted that, compared with the facial state analysis model a, the training data used by the facial state analysis model 1 includes ultrasonic audio data a and sample facial states. However, it does not include posture data a. The method for obtaining the training data required for training the facial state analysis model 1 can refer to the description of obtaining the training data when training the facial state analysis model a, but there is no need to obtain posture data, which will not be repeated here. After obtaining the training data, the facial state analysis model 1 can be trained based on the training data. The description of this process can refer to the following Fig. 7A .

[0144] Training the facial state analysis model 1 once requires at least one frame of ultrasonic audio data a and a sample facial state corresponding to one frame of sample face picture. The sample facial state used in each training is matched with the ultrasonic audio data a. It can also be understood that the ultrasonic audio data obtained when the face is in the facial state indicated in the sample face picture frame is the ultrasonic audio data a.

[0145] like Fig. 6A As shown, the facial state analysis model 1 in training may include a posture feature extraction network to be trained, a facial state feature extraction network, and a facial state analysis network 1. The facial state analysis network 1 includes a convolutional network b1 to be trained, a deconvolutional network b1, and a convolutional network b2. The functions of each network can refer to the aforementioned related content and will not be repeated here.

[0146] The ultrasonic audio data a and the sample facial state corresponding to the sample face image are input into the facial state analysis model 1 to be trained frame by frame, and the timestamps of the training data involved in the training are the same when input into the facial state analysis model 1 for one training. Fig. 6A As shown, a frame of ultrasonic audio data a input into the facial state analysis model 1 can also be called ultrasonic audio data frame a, and the sample facial state corresponding to a frame of sample face picture input into the facial state analysis model 1 can be simply referred to as sample facial state, and the sample facial state is represented by a 1*112*112 matrix.

[0147] In the process of training the facial state analysis model 1, the facial state feature extraction network extracts features from the ultrasonic audio data frame a to obtain facial state features a. The facial state features a can be represented as a 128*56*56 matrix. The facial state features a are then transmitted to the facial state analysis network 1, where the facial state features a are mapped to predicted facial states through the convolution network b1, the deconvolution network b1, and the convolution network b2. Among them, the convolution network b1 can be used to upsample the input facial state features a, and the deconvolution network 1 is used to deconvolve the upsampled facial state features a output by the convolution network b1, and then the deconvolution facial state features a are input to the convolution network b2. The convolution network b2 downsamples based on the deconvolution facial state features a to obtain the final facial state.

[0148] It should be noted that the predicted facial state is represented by a 1*112*112 matrix. The predicted facial state can be understood as the facial state output by the facial state analysis model 1 in training.

[0149] Subsequently, the difference between the predicted facial state and the sample facial state is calculated by the facial state analysis model 1 to obtain the loss function b. The parameters in each network in the facial state analysis model 1 are adjusted based on the loss function b. For example, when the loss function b is greater than a preset threshold, the parameters in each network in the facial state analysis model 1 are adjusted. When the loss function b is continuously less than the preset threshold, the parameters in each network in the facial state analysis model 1 are not adjusted in this training.

[0150] It should be noted that the above content describes a training process once, and in actual situations, the facial state analysis model 1 needs to be trained multiple times. Fig. 6A As shown, subsequent initial audio data and sample face images are obtained from the front to the back according to time (t) to continue training the facial state analysis model 1. The training process can refer to the aforementioned related content and will not be repeated here.

[0151] It should also be noted that Fig. 7A With the aforementioned Fig. 6A The acquisition process and definition of the data and methods with the same name can refer to the above Fig. 6A The description of the relevant data in the above description will not be repeated here. Here, the data with the same name includes sample face picture frame, sample face state and each initial data frame (including initial audio data frame a1-initial audio data frame a4). The methods with the same name include filtering, Fourier transform and facial state recognition.

[0152] After the facial state analysis model 1 is trained, the trained facial state analysis model 1 can be placed in the smart glasses, so that the smart glasses can use the facial state analysis model 1 to obtain the facial state through the ultrasonic audio data 1. The process of the smart glasses using the facial state analysis model 1 can refer to the following Figure 7B Description.

[0153] like Figure 7B As shown, the trained facial state analysis model 1 may include a trained facial state feature extraction network and a facial state analysis network 1. The facial state analysis network 1 includes a trained convolutional network b1, a deconvolutional network b1, and a convolutional network b2. The functions of each network can refer to the aforementioned related content and will not be repeated here.

[0154] In a scenario where a trained facial state analysis model 1 is used to generate a facial state, microphones 1 to 4 in the smart glasses can respectively continuously collect initial audio data 11 to initial audio data 14. A frame of ultrasonic audio data frame 1 (also referred to as ultrasonic audio data frame 1) is acquired at intervals of time Tc. The ultrasonic audio data frame 1 is transmitted to the trained facial state analysis model 1 to obtain a facial state. This process is executed cyclically over time t to obtain multiple facial states.

[0155] See Figure 7B , the smart glasses extract features from the ultrasonic audio data frame 1 through the facial state feature extraction network to obtain facial state feature 1. The facial state feature 1 can be represented as a 128*56*56 matrix. The facial state feature 1 is then transmitted to the facial state analysis network 1. In the facial state analysis network 1, the facial state is mapped by the convolution network b1, the deconvolution network b1 and the convolution network b2 using the facial state feature 1. The facial state can be represented as a 1*112*112 matrix.

[0156] It should be noted here that Figure 7B With the aforementioned Figure 6B The acquisition process and definition of the data and methods with the same name can refer to the above Figure 6B The description of the relevant data in the above description will not be repeated here. Here, the data with the same name includes ultrasonic audio data frame 1 and each initial data frame (initial audio data frame 11-initial audio data frame 14), etc. The methods with the same name include filtering and Fourier transform, etc.

[0157] Figure 8 A schematic diagram of a system structure for implementing the facial information acquisition method in an embodiment of the present application is shown.

[0158] like Figure 8As shown, the system structure is divided into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the system structure is divided into 4 layers, from top to bottom, namely, the application layer, the application framework layer, the hardware abstraction layer (HAL), and the hardware layer.

[0159] The application layer may include a series of application packages, such as communication, music, and other applications.

[0160] The application framework layer provides application programming interface (API) and programming framework for applications in the application layer.

[0161] The application framework layer may include an audio framework layer and a facial information framework layer. The audio framework layer may be used to process audio data, including audio data received from other electronic devices and audio data sent to other electronic devices. The facial information framework layer is used to manage the generation of facial information. For example, the facial information framework layer may call a facial information generation module to generate facial information, and feed the facial information back to the application that calls the facial information framework layer.

[0162] The hardware abstraction layer is an interface layer located between the operating system kernel layer and other layers of the electronic device (such as the local service layer). Its purpose is to abstract the hardware and provide a virtual hardware platform for the operating system.

[0163] The hardware abstraction layer may include a facial information generation module.

[0164] The facial information generation module may include a sensor data processing module, a timestamp alignment module, an image preprocessing module, and a face image generation model. The face image generation model may include the face image generation model a or the face image generation model 1 mentioned above. When the face image generation model is the face image generation model 1, the hardware layer of the electronic device may include a posture sensor. For the contents related to the face image generation model a and the face image generation model 1, please refer to the aforementioned Figure 4C as well as Figure 5B The description of the relevant contents will not be repeated here.

[0165] The sensor data processing module is used to process the data uploaded by the microphone, speaker and gesture sensor to obtain data that can be input into the face image generation model (after training). The sensor data processing module can be used to implement the above Figure 4CThe six-axis fusion processing, filtering and Fourier transform involved in the sensor data processing module can obtain the target posture data 1 (at least one frame of target posture data) based on the initial posture data (such as the initial posture data 11 and the initial posture data 12), and obtain the ultrasonic audio data 1 (at least one frame of ultrasonic audio data) based on the initial audio data (such as the initial audio data 11-initial audio data 14).

[0166] The timestamp alignment module is used to control the timestamps of the target posture data 1 and the ultrasonic audio data 1 input into the face image generation model to be the same. For example, the timestamps of the target posture data frame 1 and the ultrasonic audio data frame 1 mentioned above are controlled to be the same.

[0167] The image preprocessing module is used to perform image preprocessing and other preprocessing on the aforementioned preset picture 1 with a human face to obtain the preset face picture 1. The description of this process can refer to the aforementioned Figure 4B In the process of performing image preprocessing and other preprocessing on the training picture a with a face to obtain a preset face picture a, the preset face picture a is modified to the preset face picture 1, and the training picture a with a face is modified to the preset picture 1 with a face, and the embodiments of the present application will not be repeated here.

[0168] The hardware layer may include at least one microphone (e.g., microphone 1-microphone m), at least one speaker (e.g., speaker 1-speaker h), a gesture sensor, and a display. Wherein, m is an integer greater than or equal to 1, such as 4. h is an integer greater than or equal to 1, such as 4. The number of data of the gesture sensor is greater than or equal to 1, such as 2. The gesture sensor is optional for the electronic device.

[0169] The display is used to display the generated simulated face image. During a video call, the display is also used to display the opposite end image sent by other electronic devices (also referred to as the opposite end electronic device).

[0170] It should be noted here that Figure 8 In the above, the facial information generation module includes a facial picture generation model as an example for explanation. In practical applications, the model included in the facial information generation module may not be a facial picture generation model but a facial state analysis model (including facial state analysis model 1 or facial state analysis model a). When it is a facial state analysis model, the facial information generation module is used to generate facial states, and the facial information generation module may not include a layer preprocessing module.

[0171] Each level cooperates with each other to complete the facial information acquisition method involved in the embodiment of the present application. The facial information acquisition method involved in the embodiment of the present application can be applied in a scenario (video call scenario) in which an electronic device makes a video call with another electronic device (also referred to as a peer electronic device). Fig. 9 A schematic diagram of interaction during a video call is shown.

[0172] like Fig. 9 As shown, during a video call between an electronic device (such as smart glasses) and an electronic device at the other end, the smart glasses can activate at least one microphone to collect initial audio data (such as the initial audio data 11 to the initial audio data 14 mentioned above). At the same time, at least one gesture sensor is activated to collect initial gesture data (such as the initial gesture data 11 and the initial gesture data 12 mentioned above).

[0173] The initial audio data collected by at least one microphone and the initial posture data collected by at least one posture sensor are transmitted to the timestamp alignment module for timestamp alignment. Then, the initial audio data and initial posture data after timestamp alignment are transmitted to the sensor data processing module for processing to obtain posture data 1 (also referred to as target posture data 1) and ultrasonic audio data 1 that can be input into the face generation model a (after training). For example, the target posture data frame 1 and ultrasonic audio data frame 1 mentioned above. The image preprocessing module performs image preprocessing on the preset picture 1 with a face to obtain a preset face picture 1.

[0174] Subsequently, the posture data 1, the ultrasonic audio data 1 and the preset face picture 1 are input into the face generation model a to obtain a simulated face picture. The relevant operations of generating the simulated face picture can be performed in the facial information generation module.

[0175] Then, the smart glasses send the simulated human face picture to the opposite electronic device, which can display the received simulated human face picture on a display (eg, screen) of the opposite electronic device.

[0176] The smart glasses can also receive the opposite end picture sent by the opposite end electronic device. Then, the opposite end picture is displayed on the display (such as lens) of the smart glasses. The opposite end picture can be collected by the opposite end electronic device through a camera.

[0177] It should be noted here that in the video call scenario, the microphone and the posture sensor collect data in real time, and can generate target posture data 1 and ultrasonic audio data 1 in real time. And the target posture data 1 and ultrasonic audio data 1 can be input into the face picture generation model a frame by frame, so that the face picture generation model can generate a simulated face picture in real time.

[0178] It should be noted here that Fig. 9 The face generation model a is used as an example for explanation. The model used in the actual video call process can also be the face screen generation model 1 instead of the face screen generation model a. For a description of the process, please refer to Fig. 9 The description of the face image generation model 1 does not require the use of a posture sensor to collect data.

[0179] Fig.10 An exemplary interactive process of implementing the facial information acquisition method in a video call scenario is shown in FIG. The description of the process can refer to the following description of steps S101 to S104, step S201a, and step S201b.

[0180] S101. An electronic device (such as smart glasses) establishes a video call connection with an electronic device at a peer end through a video call service.

[0181] After the video call connection is established, the electronic device may start to generate a simulated face image and transmit it to the other electronic device. For a description of this process, reference may be made to the description of steps S102a to S104 below.

[0182] S102a. The electronic device activates at least one speaker to continuously emit an ultrasonic signal, and activates at least one microphone to collect a sound signal, to obtain initial audio data carrying at least the ultrasonic signal.

[0183] In some possible cases, the relevant content involved in collecting the initial audio data can refer to the above Figure 4C The process of collecting initial audio data 11-initial audio data 14 is involved.

[0184] The ultrasonic signal carried in the initial audio data is the ultrasonic signal sent by the speaker after being reflected by the face. During the video call, the microphone needs to collect not only the ultrasonic signal but also the audible sound signal of the user (such as the wearer of smart glasses). The audible sound signal here refers to the sound signal that can be distinguished by the human ear, which is a non-ultrasonic signal. The frequency of the sound signal that can be distinguished by the human ear is usually in the range of 20 Hz to 20 kHz. The frequency of the ultrasonic signal is usually greater than 20 kHz.

[0185] The electronic device sends the initial audio data collected by the microphone to the facial information generation module to generate a simulated face image, and sends the initial audio data to the video call service so that the video call service obtains audible audio data from the initial audio data.

[0186] S102b. The electronic device starts at least one gesture sensor to continuously collect initial gesture data.

[0187] In some possible cases, the relevant content involved in collecting posture data can refer to the above Figure 4C The process of collecting initial posture data 11 and collecting initial posture data 12 is involved.

[0188] The electronic device transmits initial posture data collected by at least one posture sensor to the facial information generation module to generate a simulated human face picture.

[0189] S103a. The video call service obtains audible audio data from the initial audio data.

[0190] The electronic device performs low-pass filtering and Fourier transform on the initial audio data through the video call service, filters out the ultrasonic audio signal therein, and obtains audible audio data.

[0191] S103b. The facial information generation module processes the initial posture data and the initial audio data to obtain target posture data 1 and ultrasonic audio data 1.

[0192] The facial information generation module can perform timestamp alignment on the initial posture data and the initial audio data through the timestamp alignment module therein. Then, the initial audio data and the initial posture data after the timestamp alignment are processed through the sensor data processing module therein to obtain the target posture data 1 and the ultrasonic audio data 1 that can be input into the face generation model a (after training). For example, the target posture data frame 1 and the ultrasonic audio data frame 1 mentioned above.

[0193] The sensor data processing module filters and Fourier transforms the initial audio data after the timestamp alignment to obtain ultrasonic audio data 1. The process of generating a frame of ultrasonic audio data can refer to the above Figure 4C The sensor data processing module performs six-axis fusion processing on the initial posture data after timestamp alignment to obtain target posture data 1. The process of generating a frame of target posture data can refer to the above Figure 4C The description of the target posture data frame 1 is obtained in . It should be noted that the filtering process when obtaining the ultrasonic audio data is a high-pass filtering.

[0194] S104. The facial information generation module inputs the target posture data 1, the ultrasonic audio data 1 and the preset facial image 1 into the facial image generation model a to obtain a simulated facial image.

[0195] The process of obtaining a simulated face image based on the face image generation model a can be referred to in the aforementioned Figure 4C The relevant contents involved will not be repeated here.

[0196] Then, the electronic device transmits the simulated facial image and audible audio data to the other electronic device via the video call service.

[0197] After establishing the video call connection, the electronic device can also receive the opposite-end picture and opposite-end audio data sent by the opposite-end electronic device. For a description of this process, reference can be made to the description of the following step S201a and step S201b.

[0198] S201a. The electronic device processes the opposite-end picture sent by the opposite-end electronic device for implementing the video call through the video call service, and displays the opposite-end picture on the display.

[0199] S201b. The electronic device processes the peer audio data for implementing the video call sent by the peer electronic device through the video call service, and plays the peer audio through at least one speaker.

[0200] In step S201b, the speaker that plays the audio of the other end also needs to continuously send the ultrasonic signal, or the speaker that plays the audio of the other end and the speaker that continuously sends the ultrasonic signal may be different, depending on the number of speakers.

[0201] It should be noted here that the aforementioned Fig.10 In the embodiment, step S102a and step S102b can be executed simultaneously. Step S103a and step S103b can be executed simultaneously, or, since it takes longer time to generate the simulated face image from step S103a to step S103b, step S103a is allowed to be executed later than step S103b. However, the time when step S103a is executed should be controlled to be the same as the time when step S103b is executed. The aforementioned steps S201a and step S201b can be executed simultaneously.

[0202] In some possible cases, the aforementioned face image generation model a and facial state analysis model a can be integrated into the same model (facial information generation model), which can also be understood as constructing a branch of facial state analysis model a in the face image generation model a to obtain the facial information generation model. The training process of the facial information generation model can refer to the following Fig.11A .

[0203] like Fig.11A As shown, the facial information generation model in training may include a facial state analysis model a to be trained, and may also include a trained face picture generation model a. The process of training the facial information generation model is actually to train the facial state analysis model a and the face picture generation model a at the same time. The training data used can refer to the training data involved in the aforementioned training of the facial state analysis model a and the face picture generation model a. The training process can refer to the aforementioned Figure 4AIn the training of facial state analysis model a and Fig. 6A The process of training the face image generation model a in will not be repeated here. Fig.11A The splicing feature 1 in can be regarded as Figure 4A The spliced ​​features in the figure can be seen as Figure 4A The concatenated features in . Fig.11A The splicing feature 2 in can be regarded as Fig. 6A The spliced ​​features in the figure can be seen as Fig. 6A The concatenated features in .

[0204] It should be noted here that Fig.11A and Figure 4A or Fig. 6A The acquisition process and definition of the data and methods with the same name can refer to the above Figure 4A or Fig. 6A The description of the relevant data in the above description will not be repeated here. Here, the data with the same name includes ultrasonic audio data frame a and each initial data frame (initial posture data 11, initial posture data 12, initial audio data frame a1-initial audio data frame a4), etc. The methods with the same name include filtering and Fourier transform, etc.

[0205] The trained facial information generation model can be placed in smart glasses, so that the smart glasses can use the facial information generation model to generate facial states and / or simulated face images. The process of using the trained facial information generation model can refer to Fig. 11B .

[0206] like Fig. 11B As shown, the trained facial information generation model includes a trained face image generation model a and a trained face state generation model a. The trained facial information generation model can output facial states and simulated face images. The process of outputting simulated face images by the trained facial information generation model can refer to the aforementioned Figure 4C The content involved in the output of the simulated face image by the trained face image generation model a. The process of outputting the facial state by the trained facial information generation model can refer to the aforementioned Figure 6B The contents involved in the output of facial state by the trained facial state analysis model a are not described here. Fig. 11B The splicing feature 1 in can be regarded as Figure 4C The spliced ​​features in the figure can be seen as Figure 4C The concatenated features in . Fig. 11B The splicing feature 2 in can be regarded as Figure 6B The spliced ​​features in the figure can be seen as Figure 6B The concatenated features in .

[0207] Fig.11A as well as Fig. 11B The facial information generation model when the posture data is introduced is used as an example to illustrate. The facial information generation model when the posture data is not introduced can refer to Fig.11A as well as Fig. 11B The description of the gesture data can be deleted and adaptively modified. The embodiment of the present application is not limited to this.

[0208] It should be noted that the facial state features involved in the embodiments of the present application are extracted based on ultrasonic audio data, and the facial state features can also be referred to as ultrasonic audio features.

[0209] In the embodiment of the present application, the posture data is not introduced. In the scheme of generating the simulated human face picture based on the ultrasonic audio data, the obtained simulated human face picture can only adjust the facial expression based on the preset human face picture but not adjust the direction of the face. In this way, the focus can be placed on the expression of the facial expression, avoiding greater errors caused by introducing the direction of the face.

[0210] In some possible cases, the feature extraction networks (facial state feature extraction network, posture feature extraction network and face feature extraction network) and convolution networks (including convolution network 11, convolution network 12, convolution network a1, convolution network a2, etc.) involved in the above content can be convolutional neural networks (CNN) based on residual structures or Transfomer models. The deconvolution network (including deconvolution network 11, deconvolution network a1, etc.) can be a deep convolutional neural network (DCNN). The loss function (loss function 1, loss function 2, etc.) involved in the above content can be a Huber loss function.

[0211] In some possible cases, the posture features mentioned in the above content include posture features for the face, which may include facial direction information (looking up, looking down, facing left or right, etc.), etc. The direction information may include the deflection angle of the head around three axes, typically including pitch angle, yaw angle and roll angle.

[0212] Facial features generally refer to a set of features used to identify a face, including but not limited to the shape of the face, the position and size of the eyes, nose and mouth, and the texture of the skin. These features are relatively immutable and unique to each person. Because of this, they are often used for face recognition, such as determining whether a facial image matches a specific individual.

[0213] Compared with face features, facial state features tend to describe facial expressions, such as the opening and closing state of the eyes (wide open or closed), the direction of eye movement, the position of eyebrows (raised eyebrows or frowned eyebrows), the shape of the mouth (smiling face or frowning face), etc. These features can be used to understand and interpret facial states in a shorter time scale, and are often used for tasks such as expression recognition or fatigue detection.

[0214] Facial state features (facial state feature 1, facial state feature a, etc.) are more primitive and abstract data than facial states. By processing facial state features, facial states that are easy for users to understand can be obtained, such as closed eyes, open mouth, etc.

[0215] It should also be noted that the aforementioned Figures 4A-7B , Fig.11A as well as Fig. 11B The matrix size of each data involved in is a distance description, and can actually be other values. For example, the matrix size representing the preset face picture 1 and the preset face picture a is 3*112*112, where 112 is an example and can be other values, such as 114, 116, etc. The 3 indicates that the preset face picture 1 and the preset face picture a are color images, and a pixel in the picture can be described by 3 color channels, such as RGB color channels. The 3 in the matrix representing other pictures (such as simulated face pictures) also represents 3 color channels.

[0216] The following is an introduction to an exemplary electronic device provided in an embodiment of the present application.

[0217] Fig.12 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0218] It should be understood that electronic devices may have Fig.12 More or fewer components may be shown, two or more components may be combined, or there may be a different configuration of components. Fig.12 The various components shown in the EMBODIMENTS 2000 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0219] The electronic device may include: a processor 110 , a memory 120 , a universal serial bus (USB) interface 130 , a charging management module 140 , a power management module 141 , a battery 142 , a communication module 150 , a display 160 , an audio module 170 , and a gesture sensor 180 .

[0220] In an embodiment of the present application, the processor 110 may include one or more processing units, and the memory 120 may store computer instructions and models (such as a face image generation model) involved in implementing the facial information acquisition method, so that the electronic device executes the method in the embodiment of the present application.

[0221] The USB interface 130 can be used to connect a charger to charge the electronic device, and can also be used to transmit data between the electronic device and peripheral devices.

[0222] The charging management module 140 is used to receive charging input from the charger. The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to power the processor 110, the memory 120, the display 160, etc.

[0223] The communication module 150 is used to implement the communication function of the electronic device. For example, the aforementioned video call with other electronic devices can be completed based on the communication module 150.

[0224] The display 160 may be used to display content such as images or videos. The electronic device may include at least one display. For example, when the electronic device is a smart glasses, two displays may be included. In some possible scenarios, reference may be made to the aforementioned Figure 1 In the smart glasses shown in FIG. 1 , the display 160 includes a display in the frame 1 and a display in the frame 2. The display in the smart glasses is also a lens.

[0225] The audio module 170 may include a speaker 170A and a microphone 170B. The speaker 170A is used to play audio signals. Including ultrasonic signals and audible audio signals. The microphone 170B is used to collect sound signals to obtain ultrasonic audio data, or audible audio data. The electronic device includes at least one speaker and microphone. For example, when the electronic device is smart glasses, the speaker 170A may include 4 speakers and the microphone 170B may include 4 microphones. At this time, the layout of the speaker and the microphone in the electronic device can refer to the aforementioned Figure 1 Description in .

[0226] The posture sensor 180 can be used to collect initial posture data to obtain target posture data (referred to as posture data for short).

[0227] The present application also provides a chip system, which includes at least one processor for implementing the functions involved in the method executed by the electronic device in any of the above embodiments.

[0228] In one possible design, the chip system also includes a memory, which is used to store program instructions and data, and the memory is located inside or outside the processor.

[0229] The chip system may be composed of the chip, or may include the chip and other discrete devices.

[0230] Optionally, the processor in the chip system may be one or more. The processor may be implemented by hardware or by software. When implemented by hardware, the processor may be a logic circuit, an integrated circuit, etc. When implemented by software, the processor may be a general-purpose processor implemented by reading software code stored in a memory.

[0231] Optionally, the memory in the chip system may be one or more. The memory may be integrated with the processor or may be separated from the processor, which is not limited in the embodiment of the present application.

[0232] Exemplarily, the memory can be a non-transitory processor, such as a read-only memory ROM, which can be integrated with the processor on the same chip or can be set separately on different chips. The embodiments of the present application do not specifically limit the type of memory and the setting method of the memory and the processor.

[0233] The present application also provides a computer program product, which includes: a computer program (also referred to as code, or instruction), which enables a computer to execute the method executed by the electronic device in any of the above embodiments when the computer program is executed.

[0234] The present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program (also referred to as code or instruction). When the computer program is executed, the computer executes the method executed by the electronic device in any of the above embodiments.

[0235] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0236] As used in the above embodiments, the term "when..." may be interpreted to mean "if..." or "after..." or "in response to determining..." or "in response to detecting...", depending on the context. Similarly, the phrases "upon determining..." or "if (the stated condition or event) is detected" may be interpreted to mean "if determining..." or "in response to determining..." or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)", depending on the context.

[0237] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification and appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to and includes any or all possible combinations of one or more listed items.

[0238] The terms "first" and "second" are used for descriptive purposes only and should not be understood to imply or suggest relative importance or implicitly indicate the number of technical features indicated. Thus, features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0239] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.

[0240] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

Claims

1. A facial information acquisition method, characterized in that: Applied to a first electronic device, the first electronic device includes a speaker and a microphone, and the method includes: Acquire first ultrasonic audio data through the microphone, where the first ultrasonic audio data indicates an ultrasonic signal emitted by the speaker and reflected by the face; extracting facial state features of the face through the first ultrasonic audio data; The first preset face picture is adjusted based on the facial state feature to generate a simulated face picture.

2. The method according to claim 1, characterized in that: Adjusting the first preset face picture based on the facial state feature to generate a simulated face picture specifically includes: Extracting facial features in the first preset face picture through the first preset face; Adjusting the facial features according to the facial state features; The adjusted facial features are mapped into the simulated facial image.

3. The method according to claim 2, characterized in that The first electronic device also includes a gesture sensor, and the method further includes: Acquiring first posture data through the posture sensor, wherein the first posture data indicates the posture of the face; extracting a posture feature of the face through the first posture data; Adjusting the facial features according to the facial state features specifically includes: The facial features are adjusted through the facial state features and the posture features to generate a simulated facial image; the simulated facial image has the posture indicated by the posture features.

4. The method according to claim 2, characterized in that: The facial state feature is determined by a first feature extraction network in a first face picture generation model, and the face feature is determined by a second feature extraction network in the first face picture generation model; the face picture generation model also includes a first face generation network; the first face generation network includes a first convolution network, a first deconvolution network and a second convolution network; the first convolution network and the first deconvolution network have the following functions: adjusting the face feature based on the facial state feature to obtain the adjusted face feature; the second convolution network is used to map the adjusted face feature into the simulated face picture; Among them, the first facial image generation model is trained with the second ultrasonic audio data and the second preset facial image as input data, the predicted facial image as output data and the sample facial image as reference data; the sample facial image matches the facial state features extracted based on the second ultrasonic audio data.

5. The method according to claim 3, characterized in that: The facial state feature is determined by a first feature extraction network in a second face picture generation model, the face feature is determined by a second feature extraction network in the second face picture generation model, and the posture feature is determined by a third feature extraction network in the second face picture generation model; the face picture generation model also includes a second face generation network; the second face generation network includes a third convolution network, a second deconvolution network and a fourth convolution network; the functions of the third convolution network and the second deconvolution network include: adjusting the face feature based on the facial state feature and the posture feature to obtain the adjusted face feature; the fourth convolution network is used to map the adjusted face feature into the simulated face picture; Among them, the second facial image generation model is trained with the second ultrasonic audio data, the second posture data and the second preset facial image as input data, the predicted facial image as output data and the sample facial image as reference data; the sample facial image matches the facial state features extracted based on the second ultrasonic audio data.

6. The method according to any one of claims 1 to 5, characterized in that Before acquiring the first ultrasonic audio data through the microphone, the method further includes: Establishing a video call connection with the second electronic device; After generating the simulated human face picture, the method further comprises: The simulated face image is sent to the second electronic device via the video call connection.

7. The method according to claim 6, characterized in that Acquiring first ultrasonic audio data through the microphone specifically includes: Collecting sound signals through the microphone; Performing high-pass filtering on the sound signal to obtain the ultrasonic signal after being reflected by the face; The ultrasonic signal reflected by the face is processed to obtain the first ultrasonic audio data.

8. The method according to claim 7, characterized in that The method further comprises: Performing low-pass filtering on the sound signal to obtain an audible sound signal; the frequency of the audible sound signal is lower than the frequency of the ultrasonic signal; Processing the audible sound signal to obtain audible audio data; The audible audio data is transmitted to the second electronic device via the video call connection.

9. The method according to claim 3 or 5, characterized in that: The first electronic device is a wearable device worn on the head, and at least one speaker, at least one microphone and at least one gesture sensor are arranged on both sides of the central axis of the first electronic device.

10. The method according to claim 9, characterized in that The first electronic device is an eyeglass device, which includes a first frame, a second frame, a first frame arm connected to the first frame, and a second frame arm connected to the second frame; a speaker is arranged at a first position in the first frame and the second frame, a microphone is arranged at a second position in the first frame and the second frame, and a posture sensor is arranged at a third position in the first frame and the second frame; a speaker is arranged at a first position in the first frame arm and the second frame arm, and a microphone is arranged at a second position in the first frame arm and the second frame arm.

11. A facial information acquisition method, characterized in that: Applied to a first electronic device, the first electronic device includes a speaker and a microphone, and the method includes: Acquire first ultrasonic audio data through the microphone, where the first ultrasonic audio data indicates an ultrasonic signal emitted by the speaker and reflected by the face; extracting facial state features of the face through the first ultrasonic audio data; A facial state of the face is obtained based on the facial state feature map.

12. The method according to claim 11, characterized in that The first electronic device also includes a gesture sensor, and the method further includes: Acquiring first posture data through the posture sensor, wherein the first posture data indicates the posture of the face; extracting a posture feature of the face through the first posture data; Mapping the facial state of the face based on the facial state feature specifically includes: splicing the facial state feature and the posture feature to obtain a spliced ​​feature; The spliced ​​features are mapped into the facial states.

13. The method according to claim 11, characterized in that The facial state feature is determined by a first feature extraction network in a first facial state analysis model, wherein the first facial state analysis model further includes a first facial analysis network; the first facial analysis network is used to map the facial state feature into a first facial state of the face; Among them, the first facial state analysis model is trained with the second ultrasonic audio data as input data, the predicted facial state as output data and the sample facial state as reference data; the sample facial state matches the facial state features extracted from the second ultrasonic audio data.

14. The method according to claim 12, characterized in that The facial state feature is determined by a first feature extraction network in a second facial state analysis model, and the posture feature is determined by a second feature extraction network in the second facial state analysis model; the second facial state analysis model also includes a second facial state analysis network; the second facial state analysis network is used to map the splicing feature into the facial state of the face, and the splicing feature includes the posture feature and the facial state feature; Among them, the second facial state analysis model is trained with the second ultrasonic audio data and the second posture data as input data, the predicted facial state as output data and the sample facial state as reference data; the sample facial state matches the facial state features extracted from the second ultrasonic audio data.

15. A first electronic device, characterized in that: include: One or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the first electronic device to execute the method as described in any one of claims 1-12.

16. A computer-readable storage medium comprising computer instructions, characterized in that: When the computer instructions are executed on the first electronic device, the first electronic device is caused to execute the method as claimed in any one of claims 1 to 12.

17. A chip system, the chip system is applied to a first electronic device, characterized in that: The chip system includes one or more processors, and the processor is used to call computer instructions so that the first electronic device executes the method as described in any one of claims 1-12.

18. A computer program product comprising instructions, characterized in that When the computer program product runs on a first electronic device, the first electronic device is enabled to execute the method according to any one of claims 1 to 12.

Images