Voiceprint recognition and facial expression synchronization method and device

Abstract

The invention relates to the technical field of interaction, and particularly provides a voiceprint recognition and facial expression synchronization method and device, the device comprises a camera device and a head-mounted headset worn on the head of a user, and earmuff parts on the two sides of the head-mounted headset are connected with acquisition assemblies. The method comprises the following steps: acquiring voiceprint data and face data of a user synchronously collected in advance; voiceprint data and face data of a user are extracted to be preprocessed, and annotation data are obtained; training the deep learning model by using the labeled data to obtain a facial expression prediction model; synchronously obtaining user voiceprint data and cheek muscle data collected in real time, inputting the user voiceprint data and the cheek muscle data into the facial expression prediction model, outputting current expression parameters of the user by the facial expression prediction model based on the voiceprint data and the cheek muscle data, and mapping the expression parameters to the expression parameters of the virtual character; when the expression of the user is predicted, the current expression of the user can be speculated by acquiring real-time voiceprint data of the user and synchronously capturing corresponding cheek muscle activity data.

Description

Technical Field

[0001] The present invention relates to the field of interactive technology, and in particular to a method and device for synchronizing voiceprint recognition and facial expression. Background Art

[0002] In the fields of virtual reality (VR) and augmented reality (AR), interaction with users is achieved by constructing virtual images, and the construction of virtual images is generally achieved by capturing the user's voice and facial expressions to generate corresponding virtual image movements and expressions.

[0003] As for capturing the user's voice and facial expression, microphones, cameras / video devices and other devices are usually used for capture. However, when the user is in some outdoor scenes or environments where it is inconvenient to use cameras / video devices, the acquisition of the user's facial expression cannot achieve the desired effect, which has great limitations.

[0004] Based on this, the present invention provides a method and device for synchronizing voiceprint recognition and facial expression to solve the above problems. Summary of the invention

[0005] In order to overcome the deficiencies of the prior art, the present invention provides a method and device for synchronizing voiceprint recognition and facial expression to solve the problems in the prior art.

[0006] One embodiment of the present invention provides a method for synchronizing voiceprint recognition and facial expression, comprising a camera device and a headset worn on the head of a user, wherein the earmuffs on both sides of the headset are connected to a collection component, and the synchronization method comprises the following steps:

[0007] Acquire the voiceprint data and facial data of the user pre-synchronously collected by the microphone, the camera device and the collection component, wherein the facial data includes facial expression data and cheek muscle data, wherein the cheek muscle data is the activity of the cheek muscles corresponding to the user making a sound;

[0008] Extracting the user's voiceprint data and facial data for preprocessing to obtain annotated data applied to a deep learning model, wherein the annotated data includes expression samples, cheek muscle samples, and voiceprint samples;

[0009] Using the labeled data to train a deep learning model to obtain a trained facial expression prediction model;

[0010] The voiceprint data and cheek muscle data of the user collected in real time by the microphone and the collection component are synchronously obtained and input into the trained facial expression prediction model. The facial expression prediction model outputs the user's current expression parameters based on the voiceprint data and the cheek muscle data, and maps the output expression parameters to the expression parameters of the virtual character;

[0011] Receive user feedback information in real time, and determine whether the accuracy of the mapped facial expression is within a preset threshold range based on the received feedback information;

[0012] If not, a feedback training set is generated according to the received feedback information, and the facial expression prediction model is optimized by using the feedback training set.

[0013] By adopting the above scheme, the user's voiceprint data and cheek muscle data are synchronously obtained, and the facial expression prediction model is used to output the expression parameters and map them to the virtual character. The virtual character can be constructed without obtaining facial expression data, thereby improving the user's usage effect and reducing limitations. When a certain expression is shown and a sound is made, the muscles in the cheek part will follow the activity, that is, the three are synchronized. Therefore, the user's expression samples, cheek muscle samples and voiceprint samples are related. By using the user's expression samples, cheek muscle samples and voiceprint samples to train the facial expression prediction model, when predicting the user's expression, the user's voiceprint data can be obtained, and the corresponding cheek muscle activity data can be synchronously captured to infer the user's current expression. At the same time, by receiving user feedback information in real time, the facial expression prediction model is continuously optimized to improve the accuracy of the prediction.

[0014] In one embodiment, in the step of extracting the user facial data and voiceprint data for preprocessing to obtain the labeled data applied to the deep learning model, preprocessing the facial expression data in the user facial data specifically includes the following steps:

[0015] Extracting a plurality of facial expression images containing facial expression areas from a video acquired by a camera device, and performing scaling and normalization;

[0016] A facial landmark detection algorithm is used to mark the key points of facial expressions in each image, and the expressions are classified into predefined categories based on the location and distribution of the landmarks;

[0017] The classified facial expressions are encoded and integrated to obtain expression samples.

[0018] By adopting the above scheme, the video acquired by the camera device is processed to extract several facial expression images containing the user's facial expression area, and the facial expression images are used as the basis for constructing the user's expression. The facial landmark detection algorithm is used to mark the key points of the facial expression in each image, so as to classify the expression according to predefined categories, such as shock, smile, frown, etc., and then the classified facial expressions are encoded and integrated to obtain expression samples for training facial expression prediction models.

[0019] In one embodiment, the expression sample is a vocal expression, and the vocal expression is the sound emitted by the user while expressing the expression; in the step of extracting the user facial data and voiceprint data for preprocessing to obtain the labeled data applied to the deep learning model, the user voiceprint data is preprocessed, specifically comprising the following steps:

[0020] Preprocessing the voiceprint signal acquired by the microphone;

[0021] Extract the time point when the sound is emitted from the voiceprint signal, and obtain the time point when the sound is emitted from the voiced expression;

[0022] Use the time matching algorithm to calculate the same time point in two time points to determine the corresponding vocal expression in the voiceprint signal;

[0023] The determined voiceprint signal is marked into the voiced expression to obtain a voiceprint sample.

[0024] By adopting the above scheme, the expressions classified in the expression samples are all vocal expressions, which means that the user makes relevant sounds while expressing the expression; the voiceprint signal obtained by the microphone is processed to extract the time point when the sound is emitted in the voiceprint signal, and the time point of the sound emitted by the expression in the vocal expression is obtained, and the time matching algorithm is used to calculate the same time point between the two. The same time point between the two is obtained to determine the vocal expression corresponding to the voiceprint signal, and the determined voiceprint signal is marked to the corresponding vocal expression, that is, what is the voiceprint signal corresponding to the expression, so as to obtain the voiceprint sample for training the facial expression prediction model. This operation is used for subsequent, for example, when the user's real-time voiceprint data is obtained, the facial expression prediction model is used to determine the expression parameters corresponding to the current user's expression, and provide accurate prediction.

[0025] In one embodiment, the acquisition component is an electromyography sensor, and the cheek muscle data acquired by the acquisition component is an electrical signal of cheek muscle activity corresponding to when the user makes a sound; in the step of extracting the user facial data and voiceprint data for preprocessing to obtain the labeled data applied to the deep learning model, the preprocessing of the cheek muscle data in the user facial data specifically includes the following steps:

[0026] Preprocessing the electrical signals of cheek muscle activity collected by the electromyography sensor;

[0027] Extract the time point when the electrical signal of cheek muscle activity is generated, and extract the time point when the sound is emitted in the voiceprint signal corresponding to the vocal expression;

[0028] The time matching algorithm is used to calculate the same time point in two time points to determine the voiceprint signal corresponding to the electrical signal of cheek muscle activity;

[0029] The electrical signal of the determined cheek muscle activity is marked into the voiceprint signal to obtain a cheek muscle sample.

[0030] By adopting the above scheme, the acquisition component is an electromyography sensor, and the electromyography sensor is used to collect the electrical signal of the cheek muscle activity corresponding to the user's sound, that is, the cheek muscle data; the electrical signal of the cheek muscle activity collected by the electromyography sensor is processed to extract the time point when the electrical signal of the cheek muscle activity is generated, and the time point when the sound is emitted in the voiceprint signal corresponding to the voice expression is obtained, and the time matching algorithm is used to calculate the same time point between the two, and the same time point between the two is obtained to determine the voiceprint signal corresponding to the electrical signal of the cheek muscle activity, and the determined electrical signal is marked in the corresponding voiceprint signal, that is, what is the corresponding electrical signal in the voiceprint signal, so as to obtain the cheek muscle sample for training the facial expression prediction model. Through this operation, for example, when the electrical signal of the user's real-time cheek muscle activity is obtained, the facial expression prediction model is used to determine the expression parameters corresponding to the current user's expression, and the user's current corresponding expression parameters are determined by the electrical signal of the cheek muscle activity and the voiceprint signal, so as to provide accurate prediction.

[0031] In one embodiment, the step of synchronously acquiring the user's voiceprint data and cheek muscle data collected in real time by the microphone and the collection component and inputting them into a trained facial expression prediction model, the facial expression prediction model outputting the user's current expression parameters based on the voiceprint data and the cheek muscle data, and mapping the output expression parameters to the expression parameters of the virtual character specifically includes the following steps:

[0032] Acquire the user's voiceprint data collected by the microphone in real time, and synchronously acquire the cheek muscle data corresponding to the voiceprint data;

[0033] The acquired voiceprint data and cheek muscle data are pre-processed and then input into the facial expression prediction model;

[0034] Calculate the similarity between the voiceprint signal of the current voiceprint data and the voiceprint signal in the voiceprint sample, and the similarity between the electrical signal of the current cheek muscle data and the electrical signal in the cheek muscle sample according to the preset similarity matching algorithm;

[0035] Based on the calculation results, when the similarities all exceed a preset similarity threshold, the user's current expression parameters are determined;

[0036] The facial expression prediction model outputs the user's current expression parameters, and maps the output expression parameters to the expression parameters of the virtual character, wherein the output expression parameters are expression parameters corresponding to the classified predefined expressions.

[0037] By adopting the above scheme, since the voiceprint signal in the voiceprint sample corresponds to the corresponding facial expression, and the electrical signal of the cheek muscle activity in the cheek muscle sample corresponds to the corresponding voiceprint signal, when determining the current voiceprint signal and the electrical signal of the cheek muscle activity, the similarity between the current voiceprint signal, the cheek muscle electrical signal and the voiceprint sample and cheek muscle sample used to train the facial expression prediction model can be calculated by utilizing a preset similarity matching algorithm, so as to determine the user's current facial expression; through this, there is no need to obtain the user's current facial expression data, and the corresponding facial expression parameters can be calculated based on the current real-time voiceprint signal and cheek muscle electrical signal. If the current facial expression parameters are calculated solely by the voiceprint signal or the cheek muscle electrical signal, the accuracy will be greatly reduced.

[0038] The present application also relates to a voiceprint recognition and facial expression synchronization device, which is used to implement the steps of voiceprint recognition and facial expression synchronization as described in any one of the above, including a camera device and a headset worn on the user's head, wherein the earmuffs on both sides of the headset are connected to a collection component, and further includes:

[0039] A data acquisition module, used to acquire the voiceprint data and facial data of the user pre-synchronously acquired by the microphone, the camera device and the acquisition component, wherein the facial data includes facial expression data and cheek muscle data, wherein the cheek muscle data is the activity of the cheek muscles corresponding to the user making a sound;

[0040] A data preprocessing module, used to extract the user's voiceprint data and facial data for preprocessing to obtain annotated data applied to a deep learning model, wherein the annotated data includes expression samples, cheek muscle samples, and voiceprint samples;

[0041] A learning model training module, used to train the deep learning model using the labeled data to obtain a trained facial expression prediction model;

[0042] The prediction analysis module is used to synchronously obtain the user's voiceprint data and cheek muscle data collected in real time by the microphone and the collection component and input them into the trained facial expression prediction model. The facial expression prediction model outputs the user's current expression parameters based on the voiceprint data and cheek muscle data, and maps the output expression parameters to the expression parameters of the virtual character;

[0043] A feedback module is used to receive user feedback information in real time, and determine whether the accuracy of the mapped facial expression is within a preset threshold range based on the received feedback information;

[0044] The optimization module is used to generate a feedback training set according to the received feedback information, and optimize the facial expression prediction model with the feedback training set.

[0045] By adopting the above scheme, the user's voiceprint data and cheek muscle data are synchronously obtained, and the facial expression prediction model is used to output the expression parameters and map them to the virtual character. The virtual character can be constructed without obtaining facial expression data, thereby improving the user's usage effect and reducing limitations. When a certain expression is shown and a sound is made, the muscles in the cheek part will follow the activity, that is, the three are synchronized. Therefore, the user's expression samples, cheek muscle samples and voiceprint samples are related. By using the user's expression samples, cheek muscle samples and voiceprint samples to train the facial expression prediction model, when predicting the user's expression, the user's voiceprint data can be obtained, and the corresponding cheek muscle activity data can be synchronously captured to infer the user's current expression. At the same time, by receiving user feedback information in real time, the facial expression prediction model is continuously optimized to improve the accuracy of the prediction.

[0046] The present application also relates to a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned steps of synchronizing voiceprint recognition with facial expressions when executing the computer program.

[0047] The present application also relates to a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned method for synchronizing voiceprint recognition and facial expression are implemented.

[0048] The voiceprint recognition and facial expression synchronization method and device provided in the above embodiments have the following beneficial effects:

[0049] 1. By synchronously acquiring the user's voiceprint data and cheek muscle data, the facial expression prediction model is used to output expression parameters and map them to the virtual character. The virtual character can be constructed without acquiring facial expression data, thereby improving the user's usage effect and reducing limitations. When a certain expression is shown and a sound is made, the cheek muscles will follow the activity, that is, the three are synchronized. Therefore, the user's expression samples, cheek muscle samples and voiceprint samples are related. By using the user's expression samples, cheek muscle samples and voiceprint samples to train the facial expression prediction model, when predicting the user's expression, the user's voiceprint data can be acquired and the corresponding cheek muscle activity data can be synchronously captured to infer the user's current expression. At the same time, by receiving user feedback information in real time, the facial expression prediction model is continuously optimized to improve the accuracy of the prediction.

[0050] 2. In one embodiment, the electrical signal of cheek muscle activity collected by the electromyography sensor is processed to extract the time point when the electrical signal of cheek muscle activity is generated, and the time point when the sound is emitted in the voiceprint signal corresponding to the vocal expression is obtained, and the time matching algorithm is used to calculate the same time point between the two. The same time point between the two is obtained to determine the voiceprint signal corresponding to the electrical signal of cheek muscle activity, and the determined electrical signal is marked in the corresponding voiceprint signal, that is, what is the corresponding electrical signal in the voiceprint signal, so as to obtain the cheek muscle sample for training the facial expression prediction model. This operation is followed by, for example, when the electrical signal of the user's real-time cheek muscle activity is obtained, the facial expression prediction model is used to determine the expression parameters corresponding to the current user's expression, and the user's current expression parameters are determined by the electrical signal of the cheek muscle activity and the voiceprint signal, so as to provide accurate prediction.

[0051] 3. In one of the embodiments, a preset similarity matching algorithm is used to calculate the similarity between the current voiceprint signal, cheek muscle electrical signal and the voiceprint sample and cheek muscle sample used to train the facial expression prediction model, so as to determine the user's current facial expression. In this way, there is no need to obtain the user's current facial expression data. The corresponding facial expression parameters can be calculated based on the current real-time voiceprint signal and cheek muscle electrical signal. If the current facial expression parameters are calculated solely by voiceprint signal or cheek muscle electrical signal, the accuracy will be greatly reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0053] Figure 1 A flow chart of a method for synchronizing voiceprint recognition and facial expression provided by an embodiment of the present invention;

[0054] Figure 2 A schematic diagram of a headset and a collection component in a method for synchronizing voiceprint recognition and facial expression provided by an embodiment of the present invention;

[0055] Figure 3 A functional block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] It should be noted that if the embodiments of the present invention involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative position relationship, movement status, etc. between the components in a certain specific posture. If the specific posture changes, the directional indication will also change accordingly.

[0058] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features limited to "first" and "second" may explicitly or implicitly include at least one of the features. In addition, if "and / or" or "and / or" appears in the full text, its meaning includes three parallel solutions. Taking "A and / or B" as an example, it includes solution A, solution B, or solutions that satisfy both A and B. In addition, the technical solutions between the various embodiments can be combined with each other, but it must be based on the ability of ordinary technicians in this field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0059] Reference Figure 1 , Figure 2 One embodiment of the present invention provides a method for synchronizing voiceprint recognition and facial expression, comprising a camera device and a headset worn on the user's head, wherein the earmuffs on both sides of the headset are connected to a collection component, and the synchronization method comprises the following steps:

[0060] S100, obtaining the voiceprint data and facial data of the user pre-synchronously collected by the microphone, camera device and acquisition component, wherein the facial data includes facial expression data and cheek muscle data, wherein the cheek muscle data is the activity of the cheek muscles corresponding to when the user makes a sound.

[0061] In this embodiment, the voiceprint data and facial data of the user collected in advance are data collected through the camera device, microphone, and collection component when the user completes a specific instruction or when the user uses the camera device or head-mounted headset in daily life. The facial expression data is the expression shown by the user in the process of completing the instruction, the voiceprint data is the sound made by the user in the process of completing the instruction, and the cheek muscle data is the muscle activity of the cheek position when the user makes the sound.

[0062] S200, extracting the user's voiceprint data and facial data for preprocessing to obtain annotated data applied to a deep learning model, wherein the annotated data includes expression samples, cheek muscle samples, and voiceprint samples.

[0063] In this embodiment, the processing of voiceprint data such as denoising, feature extraction, normalization, etc., the processing of facial expression data such as cropping, scaling, normalization, labeling, etc., and the processing of cheek muscle data such as signal filtering, feature extraction, normalization, labeling, etc. are performed. By extracting these data for preprocessing, the data quality is ensured and the data is made suitable for the input requirements of the learning model, which helps to improve the performance and accuracy of the model. Through the preprocessed data, the corresponding expression samples, cheek muscle samples and voiceprint samples are obtained as labeled data to train the learning model.

[0064] Specifically, preprocessing the facial expression data in the user's facial data specifically includes the following steps:

[0065] S210A, extracting a plurality of facial expression images including facial expression areas from the video acquired by the camera device, and performing scaling and normalization processing;

[0066] S220A, using a facial landmark detection algorithm to mark key points of facial expressions in each image, and classifying the expressions into predefined categories based on the positions and distribution of the landmarks;

[0067] S230A: Encode and integrate the classified facial expressions to obtain expression samples.

[0068] By processing the video acquired by the camera device, several facial expression images containing the user's facial expression area are extracted, and the facial expression images are used as the basis for constructing the user's expression. By using facial landmark detection algorithms, such as Haar cascade, convolutional neural network, etc., the key points of facial expressions in each image are marked, so that the expressions are classified according to predefined categories, such as shock, smile, frown, etc., and then the classified facial expressions are encoded and integrated to obtain expression samples for training facial expression prediction models; for example, one-hot encoding, multi-label encoding, or continuous encoding is used to represent different expression categories. Through facial landmark detection and expression encoding, the user's facial expression can be converted into a data format that can be processed by the learning model, thereby realizing the synchronization of subsequent virtual character expressions.

[0069] Furthermore, the expression sample is a vocal expression, and the vocal expression is the sound emitted by the user while expressing the expression; preprocessing the user voiceprint data specifically includes the following steps:

[0070] S210B, preprocessing the voiceprint signal acquired by the microphone, such as denoising, feature extraction, normalization, etc.;

[0071] S220B, extracting the time point at which the sound is emitted from the voiceprint signal, and obtaining the time point at which the sound of the expression is emitted from the voiced expression;

[0072] S230B, using a time matching algorithm to calculate the same time point in the two time points, so as to determine the corresponding voice expression in the voiceprint signal;

[0073] S240B, marking the determined voiceprint signal into the voiced expression to obtain a voiceprint sample.

[0074] The expressions classified in the expression samples are all vocal expressions, which means that the user makes relevant sounds while expressing the expression; the voiceprint signal obtained by the microphone is processed to extract the time point when the sound is emitted in the voiceprint signal, and the time point of the sound emitted by the expression in the vocal expression is obtained, and the time matching algorithm, such as timestamp matching, dynamic time warping, time series synchronization, etc., is used to calculate the same time point between the two, and the same time point between the two is obtained to determine the corresponding vocal expression in the voiceprint signal, and the determined voiceprint signal is marked to the corresponding vocal expression, that is, what is the voiceprint signal corresponding to the expression, so as to obtain the voiceprint sample for training the facial expression prediction model. This operation is used for subsequent, for example, when the user's real-time voiceprint data is obtained, the facial expression prediction model is used to determine the expression parameters corresponding to the current user's expression, and provide accurate prediction.

[0075] Furthermore, the acquisition component is an electromyography sensor, and the cheek muscle data acquired by the acquisition component is an electrical signal of cheek muscle activity corresponding to when the user makes a sound; preprocessing the cheek muscle data in the user's facial data specifically includes the following steps:

[0076] S210C, preprocessing the electrical signal of cheek muscle activity collected by the electromyography sensor;

[0077] S220C, extracting the time point at which the electrical signal of cheek muscle activity is generated, and extracting the time point at which the sound is emitted in the voiceprint signal corresponding to the vocal expression;

[0078] S230C, using a time matching algorithm to calculate the same time point in the two time points, so as to determine the voiceprint signal corresponding to the electrical signal of cheek muscle activity;

[0079] S240C, marking the determined electrical signal of cheek muscle activity into the voiceprint signal to obtain a cheek muscle sample.

[0080] The acquisition component is an electromyogram sensor, which is used to collect the electrical signal of cheek muscle activity corresponding to the user's voice, that is, cheek muscle data; the electrical signal of cheek muscle activity collected by the electromyogram sensor is processed to extract the time point when the electrical signal of cheek muscle activity is generated, and the time point when the sound is emitted in the voiceprint signal corresponding to the voice expression is obtained, and the same time point between the two is calculated by using a time matching algorithm, such as timestamp matching, dynamic time warping, time series synchronization, etc., and the same time point between the two is obtained to determine the voiceprint signal corresponding to the electrical signal of cheek muscle activity, and the determined electrical signal is marked in the corresponding voiceprint signal, that is, what is the corresponding electrical signal in the voiceprint signal, so as to obtain cheek muscle samples for training facial expression prediction models. This operation is followed by, for example, when the electrical signal of the user's real-time cheek muscle activity is obtained, the facial expression prediction model is used to determine the expression parameters corresponding to the current user's expression, and the user's current corresponding expression parameters are determined by the electrical signal of the cheek muscle activity and the voiceprint signal, so as to provide accurate prediction.

[0081] S300: Using the labeled data to train a deep learning model to obtain a trained facial expression prediction model.

[0082] In this embodiment, the deep learning model is trained by using the labeled data (expression samples, cheek muscle samples, and voiceprint samples) obtained after preprocessing to obtain a facial expression prediction model, so that the user's facial expression can be predicted through the facial expression prediction model. The deep learning model adopts a multimodal deep learning model.

[0083] S400, synchronously obtain the user's voiceprint data and cheek muscle data collected in real time by the microphone and the collection component and input them into the trained facial expression prediction model, the facial expression prediction model outputs the user's current expression parameters based on the voiceprint data and cheek muscle data, and maps the output expression parameters to the expression parameters of the virtual character.

[0084] Specifically, the steps include:

[0085] S410, obtaining user voiceprint data collected by a microphone in real time, and synchronously obtaining cheek muscle data corresponding to the voiceprint data;

[0086] S420, pre-processing the acquired voiceprint data and cheek muscle data and inputting them into the facial expression prediction model;

[0087] S430, calculating the similarity between the voiceprint signal of the current voiceprint data and the voiceprint signal in the voiceprint sample, and the similarity between the electrical signal of the current cheek muscle data and the electrical signal in the cheek muscle sample respectively according to a preset similarity matching algorithm;

[0088] S440, based on the calculation results, when the similarities all exceed a preset similarity threshold, determining the user's current expression parameters;

[0089] S450, the facial expression prediction model outputs the user's current expression parameters, and maps the output expression parameters to the expression parameters of the virtual character, wherein the output expression parameters are expression parameters corresponding to the classified predefined expressions.

[0090] In this embodiment, since the voiceprint signal in the voiceprint sample corresponds to the corresponding facial expression, and the electrical signal of the cheek muscle activity in the cheek muscle sample corresponds to the corresponding voiceprint signal, when determining the current voiceprint signal and the electrical signal of the cheek muscle activity, a preset similarity matching algorithm, such as dynamic time warping, can be used to calculate the similarity between the current voiceprint signal, cheek muscle electrical signal and the voiceprint sample and cheek muscle sample used to train the facial expression prediction model, so as to determine the user's current facial expression; through this, there is no need to obtain the user's current facial expression data, and the corresponding facial expression parameters can be calculated based on the current real-time voiceprint signal and cheek muscle electrical signal. If the current facial expression parameters are calculated solely by the voiceprint signal or the cheek muscle electrical signal, the accuracy will be greatly reduced.

[0091] S500: Receive user feedback information in real time, and determine whether the accuracy of the mapped facial expression is within a preset threshold range according to the received feedback information.

[0092] S600: If not, generate a feedback training set according to the received feedback information, and optimize the facial expression prediction model with the feedback training set.

[0093] In this embodiment, feedback information can be sent to the user's mobile device after the expression parameters are mapped to the expression parameters of the virtual character. The information can be a notification on the mobile device. The user can reply to the construction of the facial expression parameters of the virtual character by clicking on the notification, and judge whether the accuracy is within the preset threshold range according to the content of the reply. For example, when the accuracy is lower than 85%, the user's expression, voiceprint, and muscle data can be collected again, and a feedback training set can be generated based on the collected data. The feedback training set can then be input into the facial expression prediction model for optimization training, so as to continuously correct the facial expression prediction model and provide users with accurate expression predictions.

[0094] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0095] In one embodiment, a voiceprint recognition and facial expression synchronization device is provided, and the voiceprint recognition and facial expression synchronization device corresponds to the voiceprint recognition and facial expression synchronization method in the above embodiment. The voiceprint recognition and facial expression synchronization device includes a camera device and a headset worn on the user's head, and the earmuffs on both sides of the headset are connected to a collection component, and further includes:

[0096] A data acquisition module, used to acquire the voiceprint data and facial data of the user pre-synchronously acquired by the microphone, the camera device and the acquisition component, wherein the facial data includes facial expression data and cheek muscle data, wherein the cheek muscle data is the activity of the cheek muscles corresponding to the user making a sound;

[0097] A data preprocessing module, used to extract the user's voiceprint data and facial data for preprocessing to obtain annotated data applied to a deep learning model, wherein the annotated data includes expression samples, cheek muscle samples, and voiceprint samples;

[0098] A learning model training module, used to train the deep learning model using the labeled data to obtain a trained facial expression prediction model;

[0099] The prediction analysis module is used to synchronously obtain the user's voiceprint data and cheek muscle data collected in real time by the microphone and the collection component and input them into the trained facial expression prediction model. The facial expression prediction model outputs the user's current expression parameters based on the voiceprint data and cheek muscle data, and maps the output expression parameters to the expression parameters of the virtual character;

[0100] A feedback module is used to receive user feedback information in real time, and determine whether the accuracy of the mapped facial expression is within a preset threshold range based on the received feedback information;

[0101] The optimization module is used to generate a feedback training set according to the received feedback information, and optimize the facial expression prediction model with the feedback training set.

[0102] For the specific definition of a voiceprint recognition and facial expression synchronization device, please refer to the definition of a voiceprint recognition and facial expression synchronization method above, which will not be repeated here. Each module in the above-mentioned voiceprint recognition and facial expression synchronization device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0103] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 3 The computer device includes a processor, a memory, a network interface and a database connected through a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store user data, data processing, predictive analysis, etc. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for synchronizing voiceprint recognition and facial expression is implemented.

[0104] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein a method for synchronizing voiceprint recognition and facial expression is implemented when the processor executes the computer program.

[0105] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, a method for synchronizing voiceprint recognition and facial expression is implemented.

[0106] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0107] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0108] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for synchronizing voiceprint recognition and facial expression, characterized in that: The invention comprises a camera device and a headset worn on the user's head, wherein the earmuffs on both sides of the headset are connected with a collection component, and the synchronization method comprises the following steps: Acquire the voiceprint data and facial data of the user pre-synchronously collected by the microphone, the camera device and the collection component, wherein the facial data includes facial expression data and cheek muscle data, wherein the cheek muscle data is the activity of the cheek muscles corresponding to the user making a sound; Extracting the user's voiceprint data and facial data for preprocessing to obtain annotated data applied to a deep learning model, wherein the annotated data includes expression samples, cheek muscle samples, and voiceprint samples; Using the labeled data to train a deep learning model to obtain a trained facial expression prediction model; The voiceprint data and cheek muscle data of the user collected in real time by the microphone and the collection component are synchronously obtained and input into the trained facial expression prediction model. The facial expression prediction model outputs the user's current expression parameters based on the voiceprint data and the cheek muscle data, and maps the output expression parameters to the expression parameters of the virtual character; Receive user feedback information in real time, and determine whether the accuracy of the mapped facial expression is within a preset threshold range based on the received feedback information; If not, a feedback training set is generated according to the received feedback information, and the facial expression prediction model is optimized by using the feedback training set.

2. The method for synchronizing voiceprint recognition and facial expression according to claim 1, characterized in that: In the step of extracting the user facial data and voiceprint data for preprocessing to obtain the labeled data applied to the deep learning model, preprocessing the facial expression data in the user facial data specifically includes the following steps: Extracting a plurality of facial expression images including facial expression areas from the video acquired by the camera device, and scaling and normalizing the images; A facial landmark detection algorithm is used to mark the key points of facial expressions in each image, and the expressions are classified into predefined categories based on the location and distribution of the landmarks; The classified facial expressions are encoded and integrated to obtain expression samples.

3. The method for synchronizing voiceprint recognition and facial expression as claimed in claim 2, characterized in that: The expression sample is a vocal expression, and the vocal expression is the sound emitted by the user while expressing the expression; in the step of extracting the user's facial data and voiceprint data for preprocessing to obtain the labeled data applied to the deep learning model, the user's voiceprint data is preprocessed, specifically including the following steps: Preprocessing the voiceprint signal acquired by the microphone; Extract the time point when the sound is emitted from the voiceprint signal, and obtain the time point when the sound is emitted from the voiced expression; Use the time matching algorithm to calculate the same time point in two time points to determine the corresponding vocal expression in the voiceprint signal; The determined voiceprint signal is marked into the voiced expression to obtain a voiceprint sample.

4. The method for synchronizing voiceprint recognition and facial expression as claimed in claim 3, characterized in that: The acquisition component is an electromyography sensor, and the cheek muscle data collected by the acquisition component is an electrical signal of cheek muscle activity corresponding to when the user makes a sound; in the step of extracting the user's facial data and voiceprint data for preprocessing to obtain the labeled data applied to the deep learning model, the cheek muscle data in the user's facial data is preprocessed specifically including the following steps: Preprocessing the electrical signals of cheek muscle activity collected by the electromyography sensor; Extract the time point when the electrical signal of cheek muscle activity is generated, and extract the time point when the sound is emitted in the voiceprint signal corresponding to the vocal expression; The time matching algorithm is used to calculate the same time point in two time points to determine the voiceprint signal corresponding to the electrical signal of cheek muscle activity; The electrical signal of the determined cheek muscle activity is marked into the voiceprint signal to obtain a cheek muscle sample.

5. The method for synchronizing voiceprint recognition and facial expression as claimed in claim 4, characterized in that: The step of synchronously acquiring the user's voiceprint data and cheek muscle data collected in real time by the microphone and the collection component and inputting them into the trained facial expression prediction model, the facial expression prediction model outputting the user's current expression parameters based on the voiceprint data and the cheek muscle data, and mapping the output expression parameters to the expression parameters of the virtual character specifically includes the following steps: Acquire the user's voiceprint data collected by the microphone in real time, and synchronously acquire the cheek muscle data corresponding to the voiceprint data; The acquired voiceprint data and cheek muscle data are pre-processed and then input into the facial expression prediction model; Calculate the similarity between the voiceprint signal of the current voiceprint data and the voiceprint signal in the voiceprint sample, and the similarity between the electrical signal of the current cheek muscle data and the electrical signal in the cheek muscle sample according to the preset similarity matching algorithm; Based on the calculation results, when the similarities all exceed a preset similarity threshold, the user's current expression parameters are determined; The facial expression prediction model outputs the user's current expression parameters, and maps the output expression parameters to the expression parameters of the virtual character, wherein the output expression parameters are expression parameters corresponding to the classified predefined expressions.

6. A voiceprint recognition and facial expression synchronization device, used to implement the steps of a voiceprint recognition and facial expression synchronization method as described in any one of claims 1 to 5, characterized in that: The device comprises a camera device and a headset worn on the user's head, wherein the earmuffs on both sides of the headset are connected to a collection component, and further comprises: A data acquisition module, used to acquire the voiceprint data and facial data of the user pre-synchronously acquired by the microphone, the camera device and the acquisition component, wherein the facial data includes facial expression data and cheek muscle data; A data preprocessing module, used to extract the user's voiceprint data and facial data for preprocessing to obtain annotated data applied to a deep learning model, wherein the annotated data includes expression samples, cheek muscle samples, and voiceprint samples; A learning model training module, used to train the deep learning model using the labeled data to obtain a trained facial expression prediction model; The prediction analysis module is used to synchronously obtain the user's voiceprint data and cheek muscle data collected in real time by the microphone and the collection component and input them into the trained facial expression prediction model. The facial expression prediction model outputs the user's current expression parameters based on the voiceprint data and cheek muscle data, and maps the output expression parameters to the expression parameters of the virtual character; A feedback module receives user feedback information in real time and determines whether the accuracy of the mapped facial expression is within a preset threshold range based on the received feedback information; If not, the judgment module generates a feedback training set according to the received feedback information, and optimizes the facial expression prediction model with the feedback training set.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method for synchronizing voiceprint recognition and facial expression as described in any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of a method for synchronizing voiceprint recognition and facial expression as described in any one of claims 1 to 5 are implemented.

Images