Audio data processing method, device, computer equipment and storage medium

Abstract

The embodiments of the present application disclose an audio data processing method, device, computer equipment and storage medium, which belong to the field of computer technology. The method includes: obtaining a spectrum image of target audio data, obtaining a target expression parameter set that matches the spectrum image of the target audio data based on the mapping relationship between the spectrum image and the expression parameter set, and adjusting the template face model according to the target expression parameter set to obtain a target face model with facial expressions. Based on the mapping relationship between the spectrum image and the expression parameter set, a target expression parameter set that matches the target audio data is obtained, and the target expression parameters in the target expression parameter set are used to simulate the facial expressions of the singer when the audio data is emitted, and the target face model with facial expressions is obtained according to the expression parameters in the expression parameter set. The target face model is then synchronously displayed when the audio data is played to simulate the effect of a real person emitting the audio data, thereby improving the playback effect.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to an audio data processing method, apparatus, computer equipment, and storage medium. Background Art

[0002] With the development of computer technology and the emergence of audio playback applications, playing audio data has become a common form of entertainment in people's daily lives. Usually, when playing the target audio data, the text content or other information in the target audio data is also displayed synchronously to facilitate user browsing. However, this audio playback form is relatively simple and has poor playback effect. Summary of the invention

[0003] The embodiments of the present application provide an audio data processing method, device, computer equipment and storage medium to improve the playback effect. The technical solution is as follows:

[0004] In one aspect, a method for processing audio data is provided, the method comprising:

[0005] Acquire a frequency spectrum image of the target audio data, wherein the frequency spectrum image is used to represent how the frequency of the sound in the target audio data changes over time;

[0006] Based on the mapping relationship between the spectrum image and the expression parameter set, a target expression parameter set matching the spectrum image of the target audio data is obtained, wherein the target expression parameters in the target expression parameter set are used to simulate the facial expression of the singer when the target audio data is emitted;

[0007] The template facial model is adjusted according to the target expression parameter set to obtain a target facial model having the facial expression.

[0008] In a possible implementation, the target audio data includes audio data in multiple time periods, and acquiring a frequency spectrum image of the target audio data includes:

[0009] Converting the audio data in the multiple time periods into matching frequency spectrum images respectively;

[0010] The step of acquiring a target expression parameter set matching the spectrum image of the target audio data based on a mapping relationship between the spectrum image and the expression parameter set includes:

[0011] Based on the mapping relationship, target expression parameter sets matching the multiple spectrum images obtained by conversion are respectively obtained.

[0012] In another possible implementation, the acquiring, based on the mapping relationship between the spectrum image and the expression parameter set, a target expression parameter set matching the spectrum image of the target audio data includes:

[0013] Based on the frequency spectrum image of the target audio data, query the mapping relationship to determine a reference frequency spectrum image in the mapping relationship that is similar to the frequency spectrum image of the target audio data;

[0014] The expression parameter set matched by the reference spectrum image in the mapping relationship is determined as the target expression parameter set.

[0015] In another possible implementation, the acquiring, based on the mapping relationship between the spectrum image and the expression parameter set, a target expression parameter set matching the spectrum image of the target audio data includes:

[0016] Based on a mapping model, the frequency spectrum image of the target audio data is mapped to obtain a target expression parameter set matching the frequency spectrum image of the target audio data, wherein the mapping model includes a mapping relationship between the frequency spectrum image and the expression parameter set.

[0017] In another possible implementation, mapping the frequency spectrum image of the target audio data based on the mapping model to obtain a target expression parameter set matching the frequency spectrum image of the target audio data includes:

[0018] Based on the mapping model, feature extraction is performed on the spectrum image of the target audio data to obtain image features of the spectrum image, where the image features are used to describe the relationship between the frequency and time of the sound in the spectrum image;

[0019] Based on the mapping model, feature transformation is performed on the image features to obtain the target expression parameter set.

[0020] In another possible implementation, extracting features from the spectrum image of the target audio data based on the mapping model to obtain image features of the spectrum image includes:

[0021] Based on the mapping model, feature extraction is performed on the frequency spectrum image of the target audio data to obtain image features corresponding to multiple time points, where the multiple time points belong to a time period corresponding to the target audio data;

[0022] The step of performing feature transformation on the image features based on the mapping model to obtain the target expression parameter set includes:

[0023] Based on the mapping model, feature transformation is performed on the image features corresponding to each time point to obtain the target expression parameters corresponding to each time point.

[0024] In another possible implementation, after extracting features from the spectrum image of the target audio data based on the mapping model to obtain image features corresponding to multiple time points, the method further includes:

[0025] Based on the mapping model, the product of the image feature corresponding to each time point and the corresponding weight is determined as the updated image feature corresponding to each time point, and the weights corresponding to the multiple time points are used to adjust the smoothness of the image features corresponding to the multiple time points.

[0026] In another possible implementation, the method further includes:

[0027] Acquire a sample spectrum image and a sample expression parameter set of the sample audio data, wherein the sample expression parameters in the sample expression parameter set are used to simulate the facial expression of the singer when the singer emits the sample audio data;

[0028] The mapping model is trained based on the sample spectrum image and the sample expression parameter set.

[0029] In another possible implementation, the target expression parameter set includes target expression parameters corresponding to a plurality of time points; and adjusting the template face model according to the target expression parameter set to obtain the target face model having the facial expression includes:

[0030] According to the target expression parameters corresponding to the multiple time points in the target expression parameter set, the template face model is adjusted respectively to obtain multiple target face models;

[0031] After adjusting the template face model according to the target expression parameters corresponding to the multiple time points in the target expression parameter set to obtain multiple target face models, the method further includes:

[0032] During the process of playing the target audio data, the expression animation formed by the multiple target facial models is played.

[0033] In another possible implementation, before adjusting the template face model according to the target expression parameters corresponding to the multiple time points in the target expression parameter set to obtain the multiple target face models, the method further includes:

[0034] The target expression parameters corresponding to the obtained multiple time points are smoothed.

[0035] In another possible implementation, the target expression parameters corresponding to any time point include expression parameters corresponding to multiple facial areas at the time point; the template face model is adjusted respectively according to the target expression parameters corresponding to multiple time points in the target expression parameter set to obtain multiple target face models, including:

[0036] According to the target expression parameters corresponding to the multiple facial regions at the multiple time points, the corresponding facial regions in the template facial model are adjusted respectively to obtain the multiple target facial models.

[0037] In another possible implementation, before respectively adjusting the corresponding facial regions in the template facial model according to the target expression parameters corresponding to the multiple facial regions at the multiple time points to obtain the multiple target facial models, the method further includes:

[0038] Smoothing is performed on the target expression parameters corresponding to the same facial region at the multiple time points.

[0039] In another possible implementation, after adjusting the template face model according to the target expression parameters corresponding to the multiple time points in the target expression parameter set to obtain multiple target face models, the method further includes:

[0040] storing the target audio data and the expression animations formed by the multiple target facial models in correspondence;

[0041] In the process of playing the target audio data, playing the expression animation formed by the multiple target face models includes:

[0042] In response to a play instruction for the target audio data, querying the expression animation stored corresponding to the target audio data;

[0043] The target audio data is played, and the expression animation is played synchronously.

[0044] In another possible implementation, the acquiring of the frequency spectrum image of the target audio data includes:

[0045] In response to a playback instruction for the target audio data, acquiring a frequency spectrum image of the target audio data;

[0046] In the process of playing the target audio data, playing the expression animation formed by the multiple target face models includes:

[0047] The target audio data is played, and the expression animation is played synchronously.

[0048] In another possible implementation, after acquiring the target expression parameter set matching the spectrum image of the target audio data based on the mapping relationship between the spectrum image and the expression parameter set, the method further includes:

[0049] The target audio data and the target expression parameter set are stored in correspondence;

[0050] The step of adjusting the template facial model according to the target expression parameter set to obtain a target facial model having the facial expression comprises:

[0051] In response to a play instruction for the target audio data, querying the target expression parameter set stored corresponding to the target audio data;

[0052] The template face model is adjusted according to the expression parameters in the queried target expression parameter set to obtain the target face model.

[0053] In another possible implementation, after adjusting the template facial model according to the target expression parameter set to obtain the target facial model having the facial expression, the method further includes:

[0054] In response to receiving a data playback request sent by a terminal, determining, based on an audio identifier carried in the data playback request, the target audio data indicated by the audio identifier and the target face model corresponding to the target audio data;

[0055] The target audio data and the target facial model are sent to the terminal, and the terminal plays the target audio data and synchronously plays the facial expression animation formed by the target facial model.

[0056] In another aspect, an audio data processing device is provided, the device comprising:

[0057] An acquisition module, used for acquiring a frequency spectrum image of target audio data, wherein the frequency spectrum image is used for representing how the frequency of the sound in the target audio data changes over time;

[0058] The acquisition module is further used to acquire a target expression parameter set that matches the spectrum image of the target audio data based on a mapping relationship between the spectrum image and the expression parameter set, wherein the target expression parameters in the target expression parameter set are used to simulate the facial expressions of the singer when the singer emits the target audio data;

[0059] The adjustment module is used to adjust the template face model according to the target expression parameter set to obtain the target face model with the facial expression.

[0060] In one possible implementation, the target audio data includes audio data within multiple time periods, and the acquisition module is used to convert the audio data within the multiple time periods into matching spectrum images respectively; based on the mapping relationship, target expression parameter sets matching the multiple spectrum images obtained by conversion are respectively acquired.

[0061] In another possible implementation, the acquisition module is also used to query the mapping relationship based on the spectrum image of the target audio data, determine a reference spectrum image in the mapping relationship that is similar to the spectrum image of the target audio data; and determine the expression parameter set that matches the reference spectrum image in the mapping relationship as the target expression parameter set.

[0062] In another possible implementation, the acquisition module is also used to map the spectrum image of the target audio data based on a mapping model to obtain a target expression parameter set that matches the spectrum image of the target audio data, and the mapping model includes a mapping relationship between the spectrum image and the expression parameter set.

[0063] In another possible implementation, the acquisition module includes:

[0064] A feature extraction unit, configured to extract features from the spectrum image of the target audio data based on the mapping model to obtain image features of the spectrum image, wherein the image features are used to describe the relationship between the frequency and time of the sound in the spectrum image;

[0065] A feature transformation unit is used to perform feature transformation on the image features based on the mapping model to obtain the target expression parameter set.

[0066] In another possible implementation, the feature extraction unit is used to extract features of the spectrum image of the target audio data based on the mapping model to obtain image features corresponding to multiple time points, where the multiple time points belong to a time period corresponding to the target audio data;

[0067] The feature transformation unit is used to perform feature transformation on the image features corresponding to each time point based on the mapping model to obtain the target expression parameters corresponding to each time point.

[0068] In another possible implementation, the device further includes:

[0069] A determination module is used to determine, based on the mapping model, the product of the image feature corresponding to each time point and the corresponding weight as the updated image feature corresponding to each time point, and the weights corresponding to the multiple time points are used to adjust the smoothness of the image features corresponding to the multiple time points.

[0070] In another possible implementation, the device further includes:

[0071] The acquisition module is further used to acquire a sample spectrum image and a sample expression parameter set of the sample audio data, wherein the sample expression parameters in the sample expression parameter set are used to simulate the facial expressions of the singer when the singer emits the sample audio data;

[0072] A training module is used to train the mapping model based on the sample spectrum image and the sample expression parameter set.

[0073] In another possible implementation, the target expression parameter set includes target expression parameters corresponding to multiple time points; the adjustment module is used to adjust the template face model according to the target expression parameters corresponding to the multiple time points in the target expression parameter set, so as to obtain multiple target face models;

[0074] The device also includes:

[0075] The playing module is used to play the expression animation composed of the multiple target face models during the process of playing the target audio data.

[0076] In another possible implementation, the device further includes:

[0077] The processing module is used to smooth the target expression parameters corresponding to the obtained multiple time points.

[0078] In another possible implementation, the target expression parameters corresponding to any time point include expression parameters corresponding to multiple facial areas at the time point; the adjustment module is used to adjust the corresponding facial areas in the template facial model according to the target expression parameters corresponding to the multiple facial areas at the multiple time points, so as to obtain the multiple target facial models.

[0079] In another possible implementation, the device further includes:

[0080] The processing module is used to perform smoothing processing on the target expression parameters corresponding to the same facial area at the multiple time points.

[0081] In another possible implementation, the device further includes:

[0082] A storage module, used for storing the target audio data and the expression animation formed by the multiple target face models in correspondence;

[0083] The playing module is used to query the expression animation stored corresponding to the target audio data in response to a playing instruction for the target audio data; play the target audio data, and synchronously play the expression animation.

[0084] In another possible implementation, the acquisition module is used to acquire a frequency spectrum image of the target audio data in response to a playback instruction for the target audio data;

[0085] The playing module is used to play the target audio data and synchronously play the expression animation.

[0086] In another possible implementation, the device further includes:

[0087] A storage module, used for storing the target audio data and the target expression parameter set in correspondence;

[0088] The adjustment module is used to query the target expression parameter set stored corresponding to the target audio data in response to a playback instruction for the target audio data; and adjust the template facial model according to the expression parameters in the queried target expression parameter set to obtain the target facial model.

[0089] In another possible implementation, the device further includes:

[0090] a determination module, configured to, in response to receiving a data playback request sent by a terminal, determine, based on an audio identifier carried in the data playback request, the target audio data indicated by the audio identifier and the target face model corresponding to the target audio data;

[0091] The sending module is used to send the target audio data and the target facial model to the terminal, and the terminal plays the target audio data and synchronously plays the facial expression animation composed of the target facial model.

[0092] On the other hand, a computer device is provided, which includes a processor and a memory, wherein the memory stores at least one program code, and the at least one program code is loaded and executed by the processor to implement the operations performed in the audio data processing method described in the above aspects.

[0093] On the other hand, a computer-readable storage medium is provided, in which at least one program code is stored. The at least one program code is loaded and executed by a processor to implement the operations performed in the audio data processing method described in the above aspects.

[0094] On the other hand, a computer program product or a computer program is provided, wherein the computer program product or the computer program comprises a computer program code, wherein the computer program code is stored in a computer-readable storage medium, and wherein the computer program code is loaded and executed by a processor to implement the operations performed in the audio data processing method as described in the above aspects.

[0095] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:

[0096] The method, apparatus, computer equipment and storage medium provided by the embodiments of the present application represent the frequency change of the sound in the audio data in the form of a spectrum image, and obtain a target expression parameter set matching the target audio data based on the mapping relationship between the spectrum image and the expression parameter set. The target expression parameters in the target expression parameter set are used to simulate the facial expression of the singer when the audio data is emitted. Therefore, a target facial model with facial expression is obtained according to the expression parameters in the expression parameter set, so that the facial expression of the target facial model matches the facial expression of the singer when the audio data is emitted, thereby ensuring the accuracy of the target facial model. Subsequently, the target facial model is displayed during the playback of the audio data, so that the audio data is synchronized with the facial expression of the target facial model to simulate the effect of a real person emitting the audio data, thereby improving the playback effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0097] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0098] Figure 1 It is a schematic diagram of an implementation environment provided by an embodiment of the present application;

[0099] Figure 2 is a flowchart of an audio data processing method provided by an embodiment of the present application;

[0100] Figure 3 is a flowchart of another audio data processing method provided by an embodiment of the present application;

[0101] Figure 4 is a schematic diagram of expression parameters after smoothing provided in an embodiment of the present application;

[0102] Figure 5 This is a flow chart of generating facial expression animation provided by an embodiment of the present application;

[0103] Figure 6is another flow chart for generating facial expression animation provided by an embodiment of the present application;

[0104] Figure 7 is a structural schematic diagram of an audio data processing device provided in an embodiment of the present application;

[0105] Figure 8 is a structural schematic diagram of another audio data processing device provided in an embodiment of the present application;

[0106] Fig. 9 is a schematic diagram of the structure of a terminal provided in an embodiment of the present application;

[0107] Fig.10 It is a structural diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0108] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0109] It can be understood that the terms "at least one", "multiple", "each", "any" and the like used in this application include one, two or more than two, multiple include two or more than two, each refers to each of the corresponding multiple, and any refers to any one of the multiple. For example, multiple time points include three time points, and each time point refers to each of the three time points, and any refers to any one of the three time points, which can be the first time point, the second time point, or the third time point.

[0110] The audio data processing method provided in the embodiment of the present application can be executed by a computer device. Optionally, the computer device is a terminal or a server. Optionally, the terminal is a computer, a mobile phone, a tablet computer or other terminals. Optionally, the server is a background server of the target application or a cloud server that provides services such as cloud computing and cloud storage.

[0111] Figure 1 is a schematic diagram of an implementation environment provided by an embodiment of the present application. Figure 1 , the implementation environment includes a terminal 101 and a server 102. The terminal 101 and the server 102 are connected via a wireless or wired network.

[0112] The target application provided by the server 102 is installed on the terminal 101, and the terminal 101 can realize functions such as data transmission, message interaction, audio playback, etc. through the target application. Optionally, the target application is a target application in the operating system of the terminal 101, or a target application provided by a third party. For example, the target application is an audio playback application, which has the function of audio playback.

[0113] In one possible implementation, the server 102 is used to store audio data, and the terminal 101 is used to log in to the target application based on the user identification. The target application can request audio data from the server 102, and the server 102 can send the audio data requested by the terminal 101 to the terminal 101. The terminal 101 obtains a target facial model with facial expressions based on the audio data, and displays the target facial model during the playback of the audio data.

[0114] In another possible implementation, the server 102 is used to store audio data, obtain a target facial model with facial expressions corresponding to the audio data based on the stored audio data, and store the audio data and the corresponding target facial model in correspondence. The terminal 101 is used to log in to the target application based on the user identifier, and can request the audio data and the corresponding target facial model from the server 102 through the target application. The server 102 can send the audio data and the corresponding target facial model requested by the terminal 101 to the terminal 101, and the terminal 101 plays the audio data and synchronously displays the target facial model.

[0115] In another possible implementation, the server 102 is used to store audio data, and based on the stored audio data, obtain an expression parameter set corresponding to the facial expression matching the audio data, and store the audio data and the corresponding expression parameter set in correspondence. The terminal 101 is used to log in to the target application based on the user identification, and can request the audio data and the corresponding expression parameter set from the server 102 through the target application. The server 102 can send the audio data and the corresponding expression parameter set requested by the terminal 101 to the terminal 101, and the terminal 101 adjusts the template face model according to the expression parameters in the expression parameter set to obtain a target face model with facial expressions, plays the audio data, and synchronously plays the expression animation composed of the target face model.

[0116] The method provided in the embodiment of the present application can be applied to the audio data playback scenario.

[0117] For example, the audio data is music, a music playing application is installed in the terminal, a server provides services for the music playing application, a plurality of music is stored in the server, and the method provided in the embodiment of the present application is adopted to obtain a target facial model with facial expressions corresponding to each music, and each music is stored in correspondence with the corresponding target facial model with facial expressions. The terminal logs in to the music playing application based on the user identification, and sends a music acquisition request to the server through the music playing application, and the music acquisition request carries the target music identification. The server receives the music acquisition request, queries the target music corresponding to the target music identification and the plurality of target facial models with facial expressions corresponding to the target music, and returns the queried target music and the plurality of target facial models to the terminal. After receiving the target music and the plurality of target facial models, the terminal plays the target music and synchronously plays the expression animation composed of the plurality of target facial models to simulate the facial expression of the singer singing the target music, thereby improving the playback effect.

[0118] Figure 2 1 is a flowchart of an audio data processing method provided by an embodiment of the present application. The execution subject of the embodiment of the present application is a computer device. Figure 2 , the method comprises the following steps:

[0119] 201. A computer device obtains a frequency spectrum image of target audio data.

[0120] The spectrum image is used to represent how the frequency of the sound in the target audio data changes over time.

[0121] 202. The computer device obtains a target expression parameter set that matches the spectrum image of the target audio data based on a mapping relationship between the spectrum image and the expression parameter set.

[0122] The target expression parameters in the target expression parameter set are used to simulate the facial expression of the singer when the target audio data is emitted. The mapping relationship can reflect the relationship between the spectrum image and the expression parameter set, and the expression parameter set corresponding to any spectrum image can be determined through the mapping relationship. The expression parameters in the determined expression parameter set can simulate the facial expression of the singer when the audio data corresponding to the spectrum image is emitted.

[0123] 203. The computer device adjusts the template facial model according to the target expression parameter set to obtain a target facial model with facial expressions.

[0124] The template face model is a model for presenting facial expressions, and the template face model is an arbitrary face model. The facial expression of the target face model matches the facial expression of the singer when the target audio data is emitted. The target face model is used to be displayed during the process of playing the target audio data, that is, while playing the target audio data, the target face model is also displayed so that the presented facial expression matches the target audio data, thereby improving the playback effect.

[0125] The method provided in the embodiment of the present application uses a spectrum image to represent the frequency change of the sound in the audio data, and obtains a target expression parameter set that matches the target audio data based on the mapping relationship between the spectrum image and the expression parameter set. The target expression parameters in the target expression parameter set are used to simulate the facial expression of the singer when the audio data is emitted. Therefore, a target facial model with facial expressions is obtained according to the expression parameters in the expression parameter set, so that the facial expression of the target facial model matches the facial expression of the singer when the audio data is emitted, thereby ensuring the accuracy of the target facial model. Subsequently, the target facial model is displayed during the playback of the audio data, so that the audio data is synchronized with the facial expression of the target facial model to simulate the effect of a real person emitting the audio data, thereby improving the playback effect.

[0126] Figure 3 1 is a flowchart of an audio data processing method provided by an embodiment of the present application. The execution subject of the embodiment of the present application is a computer device. Figure 3 , the method comprises the following steps:

[0127] 301. A computer device obtains a frequency spectrum image of target audio data.

[0128] The target audio data is any type of audio data, for example, the target audio data is music, news broadcast data, storytelling audio data, etc. The spectrum image is used to represent the frequency of the sound in the matching audio data changing over time, that is, any spectrum image is used to represent the frequency of the sound in the matching audio data, and changes over time in the time period corresponding to the matching audio data. Optionally, the spectrum image includes the frequency size of the sound at multiple time points in the corresponding time period. Optionally, the spectrum diagram includes a frequency curve and a coordinate axis, the horizontal axis of the coordinate axis is used to represent time, and the vertical axis is used to represent the frequency of the sound. The frequency curve can reflect the frequency of the sound changing over time.

[0129] In a possible implementation, step 301 includes: converting the target audio data into a matching frequency spectrum image.

[0130] By converting the target audio data into a matching spectrum image, the converted spectrum image can describe how the frequency of the sound in the target audio data changes over time, so that a set of expression parameters matching the target audio data can be subsequently obtained based on the spectrum image.

[0131] Optionally, the process of converting the audio data into a matching frequency spectrum image includes: the computer device converts the target audio data into a frequency spectrum image matching the target audio data in response to a play instruction for the target audio data.

[0132] The play instruction is used to indicate that the target audio data needs to be played. If a play instruction for the target audio data is received, it means that the target audio data needs to be played, so the spectrum image of the target audio data is obtained, so as to subsequently obtain a target face model with facial expressions matching the target audio data, so as to display it synchronously when the target audio data is played.

[0133] In a possible implementation, step 301 includes: converting audio data in multiple time periods included in the target audio data into matching frequency spectrum images respectively.

[0134] The target audio data includes audio data in multiple time periods, and optionally, the duration after splicing the multiple time periods is equal to the duration of the target audio data. Optionally, the durations of any two time periods in the multiple time periods are equal. For example, the total duration of the target audio data is 20 seconds, and the duration of each time period is 5 seconds. Then the target audio data includes audio data in 4 time periods, the first time period is 0 seconds to 5 seconds, the second time period is 5 seconds to 10 seconds, the third time period is 10 seconds to 15 seconds, and the fourth time period is 15 seconds to 20 seconds.

[0135] Since the target audio data includes audio data in multiple time periods, the audio data in each time period is converted into a matching spectrum image, thereby obtaining spectrum images matching the audio data in multiple time periods, that is, obtaining multiple spectrum images.

[0136] Optionally, step 301 includes: in response to a play instruction for the target audio data, the computer device converts audio data in multiple time periods included in the target audio data into matching frequency spectrum images respectively.

[0137] If a play instruction for the target audio data is received, it means that the target audio data needs to be played, and therefore, the target audio data is converted into a plurality of matching frequency spectrum images.

[0138] 302. The computer device extracts features of the frequency spectrum image of the target audio data based on the mapping model to obtain image features corresponding to multiple time points.

[0139] In an embodiment of the present application, the mapping model includes a mapping relationship between a spectrum image and an expression parameter set. The input of the mapping model is the spectrum image, and the output is the expression parameter set. That is, based on the mapping model, the expression parameter set of the spectrum image can be obtained by utilizing the mapping relationship between the spectrum image and the expression parameter set in the mapping model.

[0140] The multiple time points belong to the time period corresponding to the target audio data, and the multiple time points are any multiple time points within the time period. Optionally, among the multiple time points, the duration between any two adjacent time points is equal. For example, the time period corresponding to the target audio data is divided into multiple sub-time periods, each sub-time period has a start time point and an end time point, and the end time period of each sub-time period is used as a time point within the time period, thereby obtaining multiple time points within the time period.

[0141] Image features are used to describe the characteristics of spectrum images. Image features are used to describe the relationship between the frequency and time of the sound in the spectrum image. The image features corresponding to each time point can describe the relationship between the frequency and time of the sound corresponding to the corresponding time point in the spectrum image. By extracting features from the spectrum image based on the mapping model, multiple image features corresponding to the spectrum image can be obtained, and each image feature corresponds to a time point.

[0142] In a possible implementation, the mapping model includes a feature extraction sub-model, and 302 includes: based on the feature extraction sub-model, performing feature extraction on the spectrum image to obtain image features corresponding to the multiple time points.

[0143] The feature extraction sub-model is used to extract the image features of the spectrum image. Optionally, the feature extraction sub-model is an arbitrary network model, for example, the feature extraction sub-model is a CNN (Convolutional Neural Networks).

[0144] Optionally, the feature extraction sub-model includes multiple channels, each channel is used to output an image feature corresponding to a time point. When the feature extraction sub-model extracts features from any spectrum image, the image features output by the multiple channels can form a three-dimensional image feature.

[0145] For example, based on the feature extraction sub-model, feature extraction is performed on the spectrum image to obtain a three-dimensional image feature whose size is C×W×H, wherein C is used to represent the number of channels included in the feature extraction sub-model, and W×H is used to represent the image features output by each channel, and each channel corresponds to a time point within the time period corresponding to the spectrum image, that is, image features of multiple time points within the time period corresponding to the spectrum image are obtained.

[0146] 303. Based on the mapping model, the computer device determines the product of the image feature corresponding to each time point and the corresponding weight as the updated image feature corresponding to each time point.

[0147] Among them, each time point corresponds to a weight, and the weights corresponding to multiple time points are used to adjust the smoothness of the image features corresponding to the multiple time points. By integrating the corresponding weight into the image features corresponding to each time point, the accuracy of the updated image features corresponding to each time point can be guaranteed, and the smoothness between the expression parameters subsequently obtained based on multiple image features can also be guaranteed, thereby ensuring the continuity between facial expressions and making the presented facial expressions natural, that is, ensuring the subsequent display effect.

[0148] In one possible implementation, the mapping model includes a feature extraction sub-model, and step 303 includes: based on the feature extraction sub-model, the computer device multiplies the image feature corresponding to each time point by the corresponding weight to determine the updated image feature corresponding to each time point.

[0149] In the embodiment of the present application, the image features corresponding to the multiple time points corresponding to the spectrum image are obtained by the feature extraction submodel performing feature extraction on the spectrum image, and the image features corresponding to the multiple time points are output by multiple channels in the feature extraction submodel, and each channel in the feature extraction submodel has a corresponding weight. After the multiple channels of the feature extraction submodel output the image features corresponding to the multiple time points, based on the feature extraction submodel, the product of the image features output by each channel and the corresponding weight is determined as the updated image features corresponding to each channel, that is, the updated image features corresponding to each time point.

[0150] 304. The computer device performs feature transformation on the image features corresponding to each time point based on the mapping model to obtain the target expression parameters corresponding to each time point.

[0151] The target expression parameter is used to simulate the facial expression of the singer when the target audio data is emitted. Optionally, the expression parameter includes a mouth area parameter, an eye area parameter, a smile parameter, an anger parameter or a head posture parameter. For example, under any facial expression, when the facial expression is presented by describing the expression parameter, the mouth area parameter is used to describe the size of the mouth opening, the eye area parameter is used to describe the size of the eye opening, etc., the smile parameter, the anger parameter, etc. are used to describe whether the facial expression is a smile, anger, etc., and the head posture parameter is used to describe the head raising angle, tilt angle, etc.

[0152] In the embodiment of the present application, if the image features corresponding to any time point can reflect the facial expression that matches the target audio data at that time point, the image features corresponding to that time point are subjected to feature transformation to obtain the target expression parameters corresponding to that time point. In step 303, the image features corresponding to each time point have been updated, and the target expression parameters corresponding to each time point are obtained by performing feature transformation on the updated image features corresponding to each time point. That is, the updated image features corresponding to each time point are subjected to feature transformation to obtain the target expression parameters corresponding to each time point, and the target expression parameters corresponding to the obtained multiple time points constitute a target expression parameter set that matches the spectrum image.

[0153] In a possible implementation, the mapping model includes a feature conversion sub-model. The step 304 includes: based on the feature conversion sub-model, performing feature transformation on the image features corresponding to each time point respectively to obtain the expression parameters corresponding to each time point.

[0154] The feature conversion submodel is used to obtain expression parameters corresponding to image features, and the feature conversion submodel is an arbitrary network model. For example, the feature conversion submodel is RNN (Recurrent Neural Networks) or DNN (Deep Neural Networks).

[0155] It should be noted that the embodiment of the present application takes obtaining the image features of multiple time points corresponding to the spectrum image as an example to obtain the target expression parameters corresponding to multiple time points. In another embodiment, there is no need to execute steps 302-304, and multiple time points are no longer divided. Based on the mapping model, the target expression parameter set is obtained by taking the spectrum image as a whole.

[0156] It should be noted that the embodiment of the present application first obtains the image features corresponding to each spectrum image at multiple time points, and then obtains the target expression parameters corresponding to each time point. In another embodiment, there is no need to execute steps 302-304, and other methods can be adopted. The process of feature extraction and feature conversion is no longer performed. Based on the mapping model, a target expression parameter set that matches the spectrum image of the target audio data is directly mapped.

[0157] It should be noted that the embodiment of the present application is based on a mapping model to first obtain the image features corresponding to each spectrum image at multiple time points, and then obtain the expression parameters corresponding to each time point. In another embodiment, there is no need to execute steps 302-304, and it is no longer based on a mapping model. Other mapping methods can be adopted to obtain a target expression parameter set that matches the spectrum image of the target audio data based on the mapping relationship between the spectrum image and the expression parameter set.

[0158] In one possible implementation, the process of obtaining a target expression parameter set corresponding to a spectrum image includes: a computer device queries a mapping relationship between a spectrum image and an expression parameter set based on the spectrum image of target audio data, determines a reference spectrum image in the mapping relationship that is similar to the spectrum image of the target audio data, and determines the expression parameter set that matches the reference spectrum image in the mapping relationship as the target expression parameter set.

[0159] In an embodiment of the present application, a mapping relationship between a spectrum image and an expression parameter set is stored in a computer device, and the mapping relationship includes multiple spectrum images and an expression parameter set corresponding to each spectrum image. After acquiring the spectrum image of the target audio data, the spectrum image of the target audio data is compared with each spectrum image in the mapping relationship to determine a reference spectrum image similar to the spectrum image of the target audio data, that is, the frequency variation of the sound described by the spectrum image of the target audio data over time is similar to the frequency variation of the sound described by the reference spectrum image over time, and therefore, the expression parameter set matched by the reference spectrum image is used as the expression parameter set matched by the spectrum image of the target audio data.

[0160] 305. The computer device performs smoothing processing on the target expression parameters corresponding to the obtained multiple time points.

[0161] Since the multiple expression parameters obtained are discontinuous, that is, the multiple expression parameters are discrete, smoothing is performed on the expression parameters corresponding to the multiple time points obtained to ensure continuity between the multiple expression parameters after smoothing and reduce the difference between the expression parameters corresponding to any two adjacent time points.

[0162] In a possible implementation, polynomial interpolation is used to smooth the expression parameters corresponding to the multiple time points obtained.

[0163] In a possible implementation, the effect of smoothing the obtained expression parameters is represented in the form of a curve. Figure 4 As shown, in Figure 4 The expression parameters 401 corresponding to the multiple time points are multiple discrete points. By adopting smoothing processing, the smoothed expression parameters present a curve that changes with time, so as to ensure that the difference between the smoothed expression parameters at any two adjacent time points is small, so as to ensure the continuity between the facial expressions at multiple time points when the template face model is subsequently adjusted based on the expression parameters, so that the presented facial expressions are natural and the subsequent display effect is guaranteed.

[0164] In a possible implementation, the target expression parameter corresponding to each time point includes the expression parameter corresponding to at least one facial region at each time point. For example, the target expression parameter corresponding to each time point includes the expression parameter corresponding to the mouth region at each time point, or the target expression parameter corresponding to each time point includes the expression parameter corresponding to the mouth region and the eye region at each time point.

[0165] In a possible implementation, the target expression parameters corresponding to any time point include expression parameters corresponding to multiple facial regions at the time point; then step 305 includes: smoothing the target expression parameters corresponding to the same facial region at the multiple obtained time points.

[0166] For example, the multiple facial regions are eye regions, mouth regions, forehead regions, etc. For the expression parameters corresponding to the multiple time points obtained, it is equivalent to obtaining the expression parameters corresponding to the multiple facial regions at the multiple time points, and smoothing is performed according to the expression parameters corresponding to each facial region at the multiple time points, so that the difference between any two adjacent time points in the expression parameters corresponding to each facial region after smoothing is small, so as to ensure that when the facial regions in the template facial model are subsequently adjusted based on the expression parameters, the expressions of the obtained facial regions are natural, that is, the facial expressions of the obtained target facial model are natural, and the continuity between the facial expressions at the multiple time points is obtained, so that the presented facial expressions are natural, and the subsequent display effect is guaranteed.

[0167] 306. The computer device adjusts the template facial model according to the target expression parameters corresponding to the multiple time points in the target expression parameter set to obtain multiple target facial models.

[0168] The template face model is a model for presenting facial expressions, and the template face model is any face model, for example, the template face model is a human face model, a cartoon character face model, an animal face model, etc. For example, the template face model is a 3D face model that supports Morph (deformation).

[0169] The target facial model is used to display during the playback of audio data that the target facial model has a certain facial expression, and the facial expression of any target facial model is obtained through the corresponding expression parameters. After obtaining the expression parameters corresponding to multiple time points, the template facial model is adjusted according to the expression parameters corresponding to each time point to obtain the target facial template corresponding to each time point, that is, to obtain multiple target facial models. Since the expression parameters corresponding to the obtained multiple time points are smoothed in the above step 305, the template facial model is adjusted respectively according to the multiple expression parameters after smoothing to obtain multiple target facial models. In the embodiment of the present application, the expression parameters corresponding to different time points may be different, and the facial expressions in the obtained multiple target facial models may be different.

[0170] In one possible implementation, step 306 includes: mapping the expression parameters corresponding to the multiple time points obtained to the target interval, obtaining the adjustment coefficients corresponding to the multiple time points, and adjusting the template face model according to the obtained multiple adjustment coefficients to obtain multiple target face models.

[0171] In the embodiment of the present application, the target interval is the adjustment range of the face area in the template face model, and the adjustment coefficient is any value within the target interval. For example, for the opening size of the mouth area in the template face model, the target interval is the adjustment interval of the mouth area. If the target interval is (0, 1), the expression parameter can describe the opening size of the mouth area. For example, the opening size of the mouth area is divided into 10 levels. For any expression parameter, the expression parameter indicates that the level to which the opening size of the mouth area belongs is level 5. The expression parameter is mapped to the target area, and the obtained adjustment coefficient is 0.5. The template face model is subsequently adjusted according to the adjustment coefficient to ensure the accuracy of the adjusted target face model.

[0172] In one possible implementation, the target expression parameters corresponding to any time point include expression parameters corresponding to multiple facial regions at the time point; step 306 includes: adjusting the corresponding facial regions in the template facial model according to the target expression parameters corresponding to the multiple facial regions at the multiple time points, to obtain multiple target facial models.

[0173] For any time point, the expression parameters corresponding to the multiple facial regions at the time point are used to adjust the corresponding facial regions in the template facial model to obtain the target facial model corresponding to the time point. Based on the above method, according to the expression parameters corresponding to the multiple facial regions at the multiple time points, the corresponding facial regions in the template facial model are adjusted respectively, and the target facial model corresponding to each time point can be obtained, that is, multiple target facial models can be obtained.

[0174] It should be noted that the embodiment of the present application first smoothes the expression parameters corresponding to multiple time points, and then obtains the target facial models corresponding to the multiple time points. In another embodiment, there is no need to execute steps 305-306, and other methods can be adopted to adjust the template facial model according to the target expression parameter set to obtain the target facial model with facial expressions.

[0175] 307. The computer device plays the facial expression animation composed of multiple target facial models during the process of playing the target audio data.

[0176] Since each target face model has a facial expression, the multiple target face models can form an expression animation, in which the effect of facial expression changing over time can be displayed. In the process of playing the target audio data, the expression animation is played, and the facial expression in the expression animation matches the target audio data to present the effect of a real person speaking the sound data, thereby improving the playback effect.

[0177] For example, the target audio data is target music. When the target music is played, the expression animation is played synchronously, and the facial expressions in the expression animation match the target music, thereby presenting the effect of a real person singing the target music.

[0178] In a possible implementation, the method further includes: forming an expression animation from a plurality of target facial models in a sequence of a plurality of time points.

[0179] Each time point is used to represent the interval between the target audio data and the starting time point, for example, the multiple time points are 1 second, 2 seconds, 3 seconds, etc. Each target facial model has a facial expression, and the multiple target facial models are switched over time in the order of the multiple time points corresponding to the multiple target facial models, thereby obtaining an expression animation in which the facial expression changes over time, that is, obtaining an expression animation composed of multiple target facial models. When the target audio data and the expression animation are subsequently played synchronously, the multiple target facial models are switched and displayed in sequence with the playback time of the target audio data and in the order of the multiple time points corresponding to the multiple target facial models, thereby presenting an expression animation in which the facial expression changes over time, that is, achieving the effect of synchronous playback of the target audio data and the expression animation.

[0180] In a possible implementation, step 301 includes: in response to a play instruction for target audio data, obtaining a frequency spectrum image of the target audio data, and step 307 includes: playing the target audio data and synchronously playing an expression animation composed of multiple target facial models.

[0181] If a play instruction for the target audio data is received, it means that the target audio data needs to be played. According to the above steps 301-306, multiple target facial models are obtained in real time. After obtaining the multiple target facial models, the target audio data is played, and the expression animation is played synchronously to present the effect that the facial expression changes with the playback of the target audio data, so as to present the effect of a real person speaking the sound data.

[0182] It should be noted that, in the embodiment of the present application, after obtaining multiple target facial models, the target audio data is directly played and the facial expression animation is synchronously played. In another embodiment, after executing step 306, there is no need to directly execute step 307, and the target audio data and facial expression animation can be played in other situations.

[0183] In one possible implementation, after step 306, the method further includes: the computer device stores the target audio data corresponding to the facial expression animation composed of multiple target facial models, and in response to a play instruction for the target audio data, queries the facial expression animation stored corresponding to the target audio data; plays the target audio data, and synchronously plays the facial expression animation.

[0184] After obtaining multiple target facial models, the multiple target facial models are combined into an expression animation, and the target audio data and the expression animation are stored correspondingly. If a playback instruction for the target audio data is received later, the expression animation corresponding to the target audio data is queried, and the target audio data and the expression animation are played synchronously. That is, there is no need to generate the expression animation again, and the expression animation can be reused, thereby reducing the consumption of device resources and ensuring playback efficiency.

[0185] Optionally, when the target audio data is stored in correspondence with the facial expression animation composed of multiple target facial models, the multiple target facial models and the corresponding time points are stored in correspondence with the target audio data.

[0186] Among them, each time point is used to represent the interval between the starting time point of the target audio data, so that when the target audio data is played subsequently, according to the playback duration of the target audio data and in the order of the time points corresponding to the multiple target facial models, the multiple target facial models are switched and displayed in sequence to present the effect of facial expressions changing over time.

[0187] Optionally, facial expression animation is generated based on multiple target facial models, and when the target audio data is stored in correspondence with the facial expression animation formed by the multiple target facial models, the facial expression animation and the target audio data are stored in correspondence with each other.

[0188] The duration of the expression animation is equal to the duration of the target audio data.

[0189] Optionally, the process of generating the facial expression animation includes: generating the facial expression animation in a sequence of multiple time points based on target facial models corresponding to the multiple time points.

[0190] Each time point is used to represent the interval between the start time point of the target audio data, for example, the multiple time points are 1 second, 2 seconds, 3 seconds, etc. According to the order of the multiple time points and the time intervals between the multiple time points, an expression animation is generated based on the multiple target facial models, so that the duration of the expression animation is equal to the duration of the target audio data, and starting from the start time point of the expression animation, the corresponding target facial model is displayed each time a time point is reached. That is, when the expression animation is played later, the facial expressions of the multiple target facial models can be displayed in sequence as time changes, and the time point for displaying each facial expression is the same as the time point corresponding to the corresponding target facial model.

[0191] For example, the multiple time points are 1 second, 2 seconds, 3 seconds, 4 seconds, 5 seconds, etc. When playing the facial expression animation, when the playing time reaches 1 second, the displayed facial expression corresponds to the target facial model corresponding to 1 second, when the playing time reaches 2 seconds, the displayed facial expression corresponds to the target facial model corresponding to 2 seconds, and so on, presenting an animation of facial expressions changing over time.

[0192] After the facial expression animation is generated, the facial expression animation and the target audio data are stored correspondingly, so that the facial expression animation can be played synchronously when the target audio data is played later.

[0193] Optionally, among the multiple time points, the interval between any two adjacent time points is of any length, for example, 1 second, or 0.04 seconds.

[0194] In order to ensure the playback effect of the facial expression animation, in the above embodiment, the facial expression parameters corresponding to the multiple time points are smoothed to obtain the facial expression parameters corresponding to the multiple time points after the smoothing process, thereby obtaining the target facial model corresponding to the multiple time points after the smoothing process. By adopting the smoothing process, the interval between any two adjacent time points is reduced. For example, among the multiple time points after the smoothing process, the interval between any two adjacent time points is 0.04 seconds. Then, based on the target facial model corresponding to the multiple time points after the smoothing process, the generated facial expression animation can reflect the natural switching of facial expressions, thereby ensuring the playback effect of the facial expression animation.

[0195] Optionally, the computer device includes a terminal and a server, and the above steps 301-306 are executed by the server, and the server stores the target audio data corresponding to the facial expression animation, and the above step 307 is executed by the terminal. In response to the play instruction for the target audio data, the terminal sends a data acquisition request to the server, and the data acquisition request carries the data identifier of the target audio data. The server receives the data acquisition request, queries the target audio data corresponding to the data identifier, and queries the facial expression animation stored corresponding to the target audio data, and sends the queried target audio data and facial expression animation to the terminal. After receiving the target audio data and facial expression animation, the terminal synchronously plays the target audio data and facial expression animation.

[0196] It should be noted that, in the embodiment of the present application, after obtaining multiple target facial models, the target audio data is directly played and the facial expression animation is synchronously played. In another embodiment, after executing step 305, there is no need to directly execute step 306, and other methods can be adopted to obtain multiple target facial models.

[0197] In one possible implementation, after step 305, the method further includes: storing the target audio data and the target expression parameter set in correspondence; in response to a playback instruction for the target audio data, querying the target expression parameter set stored in correspondence with the target audio data; and adjusting the template facial model according to the expression parameters in the queried target expression parameter set to obtain the target facial model.

[0198] By storing the smoothed expression parameters in correspondence with the target audio data, and storing the target audio data in correspondence with the expression parameters, if a playback instruction for the target audio data is received later, after querying the expression parameters corresponding to the target audio data, the template facial model is adjusted according to the queried expression parameters to obtain the target facial model.

[0199] Optionally, after obtaining the target facial model, the target audio data is played, and the expression animation composed of multiple target facial models is played synchronously.

[0200] If a playback instruction for target audio data is received and the expression parameters corresponding to the target audio data are queried, the template facial model is adjusted according to the queried expression parameters to obtain the target facial model, and the target audio data and the expression animation composed of multiple target facial models are played synchronously, thereby achieving real-time driving of the facial model expression when playing the target audio data, and real-time simulation of the expression changes during real-person singing, enriching the expressiveness of the music and realizing the perceptibility of the music.

[0201] Optionally, the computer device includes a terminal and a server, and the above steps 301-305 are executed by the server, and the server stores the target audio data and the target expression parameter set in correspondence; in response to receiving a data playback request sent by the terminal, based on the audio identifier carried by the data playback request, the target audio data indicated by the audio identifier and the target expression parameter set corresponding to the target audio data are determined, and the target audio data and the target expression parameter set are sent to the terminal. The terminal receives the target audio data and the target expression parameter set, adjusts the template facial model according to the expression parameters in the queried target expression parameter set, obtains the target facial model, plays the target audio data, and synchronously plays the expression animation composed of the target facial model.

[0202] In response to the play instruction for the target audio data, the terminal sends a data acquisition request to the server, and the data acquisition request carries the data identifier of the target audio data. The server receives the data acquisition request sent by the terminal, queries the target audio data corresponding to the data identifier, and queries the expression parameter set stored corresponding to the target audio data, and sends the queried target audio data and expression parameter set to the terminal. After the terminal receives the target audio data and the expression parameter set, it obtains multiple target facial models according to the above step 306, obtains the expression animation composed of the multiple target facial models, and synchronously plays the target audio data and the expression animation.

[0203] Optionally, the computer device includes a terminal and a server, and the above steps 301-305 are executed by the server, and the server stores the target audio data and the target facial model in correspondence; the server responds to receiving a data playback request sent by the terminal, determines the target audio data indicated by the audio identifier and the target facial model corresponding to the target audio data based on the audio identifier carried in the data playback request, sends the target audio data and the target facial model to the terminal, and the terminal receives the target audio data and the target facial model, plays the target audio data, and synchronously plays the expression animation composed of the target facial model.

[0204] It should be noted that in the embodiment of the present application, the facial expression animation is composed of multiple target facial models after obtaining multiple target facial models. In another embodiment, when playing audio data, the template facial model is adjusted in real time based on the expression parameters corresponding to the multiple time points obtained, and the template facial model adjusted in real time is displayed to present the effect that the facial expression in the template facial model gradually changes over time.

[0205] It should be noted that all the above-mentioned optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described one by one here.

[0206] The method provided in the embodiment of the present application uses a spectrum image to represent the frequency change of the sound in the audio data, and obtains a target expression parameter set that matches the target audio data based on the mapping relationship between the spectrum image and the expression parameter set. The target expression parameters in the target expression parameter set are used to simulate the facial expression of the singer when the audio data is emitted. Therefore, a target facial model with facial expressions is obtained according to the expression parameters in the expression parameter set, so that the facial expression of the target facial model matches the facial expression of the singer when the audio data is emitted, thereby ensuring the accuracy of the target facial model. Subsequently, the target facial model is displayed during the playback of the audio data, so that the audio data is synchronized with the facial expression of the target facial model to simulate the effect of a real person emitting the audio data, thereby improving the playback effect.

[0207] In addition, target facial models at multiple time points are obtained, each target facial model has a facial expression that matches the audio data. When the audio data is played, the expression animation composed of multiple target facial models is played synchronously to present the effect of a real person speaking the sound data, thereby realizing a new form of audio data playback, enriching the expressiveness of the audio data, and thus improving the playback effect.

[0208] Furthermore, based on the mapping model, the spectrum image matching the target audio data is processed to obtain expression parameters, so as to ensure the accuracy of the acquired expression parameters, thereby ensuring the accuracy of the subsequent expression animation.

[0209] exist Figure 3 Based on the embodiment shown, before obtaining a target expression parameter set matching the spectrum image based on the mapping model, it is necessary to train the mapping model. The process of training the mapping model includes the following steps 308-310:

[0210] 308. Obtain a sample spectrum image and a sample expression parameter set that match the sample audio data, where the sample expression parameters in the sample expression parameter set are used to simulate the facial expressions of the singer when the sample audio data is emitted.

[0211] The sample audio data is any type of audio data, for example, the sample audio data is music, news broadcast data, storytelling audio data, etc. The sample spectrum image is used to represent the frequency of the sound in the sample audio data changing over time. The sample expression parameters are used to describe the facial expression matching the sample audio data, and are used to simulate the facial expression of the singer when the sample audio data is emitted. The sample expression parameters are obtained by manual annotation. For example, sample video data matching the sample audio data is obtained, and the facial expression in the sample video data is annotated to obtain the sample expression parameters. For example, the sample video data is a standard sample video data selected by manual, and the standard sample video data is used to describe the scene of the singer emitting audio data in a formal occasion, such as the standard sample video data is used to describe the scene of the singer singing a song at a large party. The selected standard sample video data is manually annotated to ensure that the sample expression parameters obtained are consistent with the facial expression of most people when emitting the sample audio data, so as to ensure the accuracy of the mapping model trained subsequently.

[0212] In a possible implementation, the sample expression parameters include sample expression parameters corresponding to multiple facial regions at multiple time points. For example, at every 0.04 second interval, the facial expressions in the video frames in the sample video data are annotated to obtain a sample expression parameter for multiple facial regions, and the annotation is repeated multiple times to obtain sample expression parameters corresponding to multiple facial regions at multiple time points.

[0213] In a possible implementation, step 308 includes: obtaining sample audio data and sample expression parameters, and converting the sample audio data in multiple time periods included in the sample audio data into matching sample spectrum images. The process of converting the sample audio data is the same as the conversion process in step 301 above, and will not be repeated here.

[0214] 309. Based on the mapping model, the sample spectrum image is processed to obtain a predicted expression parameter set corresponding to the facial expression matching the sample audio data.

[0215] This step is similar to steps 302-304 in the above embodiment and will not be described again.

[0216] 310. Based on the difference between the predicted expression parameter set and the sample expression parameter set, the mapping model is adjusted.

[0217] Among them, the predicted expression parameters are obtained by the mapping model, and the sample expression parameter set is obtained by manual annotation. The difference between the predicted expression parameter set and the sample expression parameter set can reflect the accuracy of the mapping model. Based on the difference between the predicted expression parameter set and the sample expression parameter set, the mapping model is adjusted to improve the accuracy of the mapping model, thereby realizing the training of the mapping model.

[0218] Optionally, 301 includes: training the mapping model based on the difference between the predicted expression parameters in the predicted expression parameter set and the sample expression parameters in the sample expression parameter set.

[0219] Optionally, the mapping model is iteratively trained in the above manner.

[0220] It should be noted that the embodiment of the present application adjusts the mapping model based on the difference between the predicted expression parameters and the sample expression parameters. In another embodiment, there is no need to execute steps 309-310, and other methods can be adopted to train the mapping model based on the sample spectrum image and the sample expression parameter set.

[0221] The method provided in the embodiment of the present application trains the mapping model through sample spectrum images and sample expression parameter sets, so that the trained mapping model can learn the relationship between the spectrum image and the sample expression parameter set to improve the accuracy of the mapping model, so as to ensure that the trained mapping model can map the spectrum image into a matching expression parameter set.

[0222] It should be noted that the above Figure 3 The embodiment shown is only described with one spectrum image of the target audio data. In another embodiment, the target audio data includes audio data in multiple time periods. The audio data in the multiple time periods are respectively converted into matching spectrum images, that is, multiple spectrum images are obtained. Figure 3 In the embodiment shown, steps 302-304, or the above-mentioned query mapping relationship, are used to obtain the expression parameter set matched by each spectrum image, and any expression parameter set matched by any spectrum image includes the target expression parameters corresponding to multiple time points in the time period corresponding to the spectrum image. Then, steps 305-307 are performed to realize the solution of synchronously playing the expression animation when playing the target audio data.

[0223] Since the target audio data in the embodiment of the present application corresponds to multiple spectrum images, and each spectrum image corresponds to expression parameters corresponding to multiple time points, when executing step 305, the expression parameters corresponding to all time points corresponding to the multiple spectrum images are smoothed. For example, if the target audio data corresponds to 3 spectrum images, and each spectrum image corresponds to 4 time points, expression parameters corresponding to 12 time points are obtained, and then, when executing step 305, the expression parameters corresponding to the 12 time points are smoothed.

[0224] Figure 5 is a flowchart of generating an expression animation provided by an embodiment of the present application, such as Figure 5 As shown, the process includes:

[0225] The audio data in multiple time periods included in the audio data are converted into matching spectrum images, and the obtained multiple spectrum images constitute a spectrum map sequence.

[0226] Based on the mapping model, each spectrum image is processed respectively to obtain expression parameters corresponding to multiple time points. According to the expression parameters corresponding to the multiple time points, the template face model is driven to obtain multiple target face models, and the expression animation is composed of the obtained multiple target face models.

[0227] Figure 6 is another flowchart of generating facial expression animation provided by an embodiment of the present application, such as Figure 6 As shown, the process includes:

[0228] The audio data in multiple time periods included in the audio data are converted into matching spectrum images, and the obtained multiple spectrum images constitute a spectrum map sequence.

[0229] For each spectrum image, feature extraction is performed on the spectrum image based on the feature extraction sub-model in the mapping model to obtain image features corresponding to multiple time points corresponding to the spectrum image, and the product of the image features corresponding to each time point and the corresponding weight is determined as the updated image feature, that is, the updated image features corresponding to multiple time points corresponding to the spectrum image are obtained.

[0230] Based on the feature conversion sub-model in the mapping model, feature conversion is performed on the updated image features corresponding to each spectrum image to obtain expression parameters corresponding to multiple time points corresponding to each spectrum image.

[0231] The expression parameters corresponding to the obtained multiple time points are smoothed to obtain updated expression parameters, and the template face model is driven according to the updated expression parameters corresponding to the multiple time points to obtain multiple target face models, and the expression animation is composed of the obtained multiple target face models.

[0232] Figure 7 is a structural diagram of an audio data processing device provided in an embodiment of the present application. Figure 7 , the device comprises:

[0233] An acquisition module 701 is used to acquire a spectrum image of target audio data, where the spectrum image is used to represent how the frequency of the sound in the target audio data changes over time;

[0234] The acquisition module 701 is further used to acquire a target expression parameter set that matches the spectrum image of the target audio data based on a mapping relationship between the spectrum image and the expression parameter set, wherein the target expression parameters in the target expression parameter set are used to simulate the facial expression of the singer when the singer emits the target audio data;

[0235] The adjustment module 702 is used to adjust the template face model according to the target expression parameter set to obtain a target face model with facial expressions.

[0236] In one possible implementation, the target audio data includes audio data within multiple time periods, and the acquisition module 701 is used to convert the audio data within the multiple time periods into matching spectrum images respectively; based on the mapping relationship, respectively obtain target expression parameter sets that match the multiple spectrum images obtained by conversion.

[0237] In another possible implementation, the acquisition module 701 is also used to query the mapping relationship based on the spectrum image of the target audio data, determine a reference spectrum image in the mapping relationship that is similar to the spectrum image of the target audio data; and determine the expression parameter set that matches the reference spectrum image in the mapping relationship as the target expression parameter set.

[0238] In another possible implementation, the acquisition module 701 is also used to map the spectrum image of the target audio data based on a mapping model to obtain a target expression parameter set that matches the spectrum image of the target audio data, and the mapping model includes a mapping relationship between the spectrum image and the expression parameter set.

[0239] In another possible implementation, Figure 8 As shown, the acquisition module 701 includes:

[0240] A feature extraction unit 7101 is used to extract features from the spectrum image of the target audio data based on the mapping model to obtain image features of the spectrum image, where the image features are used to describe the relationship between the frequency and time of the sound in the spectrum image;

[0241] The feature transformation unit 7102 is used to perform feature transformation on the image features based on the mapping model to obtain a target expression parameter set.

[0242] In another possible implementation, the feature extraction unit 7101 is used to extract features from the spectrum image of the target audio data based on the mapping model to obtain image features corresponding to multiple time points, where the multiple time points belong to a time period corresponding to the target audio data;

[0243] The feature transformation unit 7102 is used to perform feature transformation on the image features corresponding to each time point based on the mapping model to obtain the target expression parameters corresponding to each time point.

[0244] In another possible implementation, Figure 8 As shown, the device also includes:

[0245] Determination module 703 is used to determine the updated image features corresponding to each time point by multiplying the image features corresponding to each time point by the corresponding weight based on the mapping model, and the weights corresponding to multiple time points are used to adjust the smoothness of the image features corresponding to multiple time points.

[0246] In another possible implementation, Figure 8 As shown, the device also includes:

[0247] The acquisition module 701 is also used to acquire a sample spectrum image and a sample expression parameter set of the sample audio data, wherein the sample expression parameters in the sample expression parameter set are used to simulate the facial expression of the singer when the singer emits the sample audio data;

[0248] The training module 704 is used to train the mapping model based on the sample spectrum image and the sample expression parameter set.

[0249] In another possible implementation, the target expression parameter set includes target expression parameters corresponding to multiple time points; the adjustment module 702 is used to adjust the template face model according to the target expression parameters corresponding to the multiple time points in the target expression parameter set, so as to obtain multiple target face models;

[0250] like Figure 8 As shown, the device also includes:

[0251] The playing module 705 is used to play the expression animation composed of multiple target face models during the process of playing the target audio data.

[0252] In another possible implementation, Figure 8 As shown, the device also includes:

[0253] The processing module 706 is used to perform smoothing processing on the target expression parameters corresponding to the obtained multiple time points.

[0254] In another possible implementation, the target expression parameters corresponding to any time point include expression parameters corresponding to multiple facial areas at the time point; the adjustment module 702 is used to adjust the corresponding facial areas in the template facial model according to the target expression parameters corresponding to the multiple facial areas at multiple time points, so as to obtain multiple target facial models.

[0255] In another possible implementation, Figure 8 As shown, the device also includes:

[0256] The processing module 706 is used to perform smoothing processing on the target expression parameters corresponding to the same facial region at multiple time points.

[0257] In another possible implementation, Figure 8 As shown, the device also includes:

[0258] The storage module 707 is used to store the target audio data and the expression animation composed of multiple target face models in correspondence;

[0259] The playing module 705 is used to query the expression animation stored corresponding to the target audio data in response to the playing instruction of the target audio data; play the target audio data, and synchronously play the expression animation.

[0260] In another possible implementation, the acquisition module 701 is configured to acquire a frequency spectrum image of the target audio data in response to a playback instruction for the target audio data;

[0261] The playing module 705 is used to play the target audio data and synchronously play the expression animation.

[0262] In another possible implementation, Figure 8 As shown, the device also includes:

[0263] Storage module 707, used for storing target audio data and target expression parameter set in correspondence;

[0264] The adjustment module 702 is used to query the target expression parameter set stored corresponding to the target audio data in response to the playback instruction of the target audio data; adjust the template face model according to the expression parameters in the queried target expression parameter set to obtain the target face model.

[0265] In another possible implementation, Figure 8 As shown, the device also includes:

[0266] The determination module 703 is used to determine the target audio data indicated by the audio identifier and the target face model corresponding to the target audio data based on the audio identifier carried in the data playback request in response to receiving the data playback request sent by the terminal;

[0267] The sending module 708 is used to send the target audio data and the target facial model to the terminal, and the terminal plays the target audio data and synchronously plays the facial expression animation composed of the target facial model.

[0268] It should be noted that: the audio data processing device provided in the above embodiment only uses the division of the above functional modules as an example when displaying the screen. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device is divided into different functional modules to complete all or part of the functions described above. In addition, the audio data processing device provided in the above embodiment and the audio data processing method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0269] An embodiment of the present application also provides a computer device, which includes a processor and a memory, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor to implement the operations performed in the audio data processing method of the above embodiment.

[0270] Optionally, the computer device is provided as a terminal. Fig. 9 1 is a schematic diagram of the structure of a terminal 900 provided in an embodiment of the present application. The terminal 900 may be a portable mobile terminal, such as a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer or a desktop computer. The terminal 900 may also be referred to as a user device, a portable terminal, a laptop terminal, a desktop terminal or other names.

[0271] The terminal 900 includes a processor 901 and a memory 902 .

[0272] The processor 901 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 901 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 901 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 901 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 901 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0273] The memory 902 may include one or more computer-readable storage media, which may be non-transitory. The memory 902 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 902 is used to store at least one computer program, which is used to be executed by the processor 901 to implement the audio data processing method provided in the method embodiment of the present application.

[0274] In some embodiments, the terminal 900 may further optionally include: a peripheral device interface 903 and at least one peripheral device. The processor 901, the memory 902 and the peripheral device interface 903 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 903 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 904, a display screen 905, a camera assembly 906, an audio circuit 907, a positioning assembly 908 and a power supply 909.

[0275] The peripheral device interface 903 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 901 and the memory 902. In some embodiments, the processor 901, the memory 902, and the peripheral device interface 903 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 901, the memory 902, and the peripheral device interface 903 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0276] The radio frequency circuit 904 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 904 communicates with communication networks and other communication devices through electromagnetic signals. The radio frequency circuit 904 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 904 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 904 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 904 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0277] The display screen 905 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 905 is a touch display screen, the display screen 905 also has the ability to collect touch signals on the surface or above the surface of the display screen 905. The touch signal can be input to the processor 901 as a control signal for processing. At this time, the display screen 905 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 905 can be one, set on the front panel of the terminal 900; in other embodiments, the display screen 905 can be at least two, respectively set on different surfaces of the terminal 900 or in a folding design; in other embodiments, the display screen 905 can be a flexible display screen, set on the curved surface or folding surface of the terminal 900. Even, the display screen 905 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 905 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode, organic light-emitting diode).

[0278] The camera assembly 906 is used to capture images or videos. Optionally, the camera assembly 906 includes a front camera and a rear camera. The front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 906 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0279] The audio circuit 907 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 901 for processing, or input them into the radio frequency circuit 904 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal 900. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 901 or the radio frequency circuit 904 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 907 may also include a headphone jack.

[0280] The positioning component 908 is used to locate the current geographical location of the terminal 900 to implement navigation or LBS (Location Based Service). The positioning component 908 can be a positioning component based on the US GPS (Global Positioning System), China's Beidou system, Russia's Grenas positioning system, or the European Union's Galileo positioning system.

[0281] The power supply 909 is used to power various components in the terminal 900. The power supply 909 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 909 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery charged through a wired line, and a wireless rechargeable battery is a battery charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0282] In some embodiments, the terminal 900 further includes one or more sensors 910 , including but not limited to: an acceleration sensor 911 , a gyroscope sensor 912 , a pressure sensor 913 , a fingerprint sensor 914 , an optical sensor 915 , and a proximity sensor 916 .

[0283] The acceleration sensor 911 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal 900. For example, the acceleration sensor 911 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 901 can control the display screen 905 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 911. The acceleration sensor 911 can also be used to collect motion data of games or users.

[0284] The gyro sensor 912 can detect the body direction and rotation angle of the terminal 900, and the gyro sensor 912 can cooperate with the acceleration sensor 911 to collect the user's 3D actions on the terminal 900. The processor 901 can implement the following functions based on the data collected by the gyro sensor 912: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0285] The pressure sensor 913 can be set on the side frame of the terminal 900 and / or the lower layer of the display screen 905. When the pressure sensor 913 is set on the side frame of the terminal 900, it can detect the user's holding signal of the terminal 900, and the processor 901 performs left and right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor 913. When the pressure sensor 913 is set on the lower layer of the display screen 905, the processor 901 controls the operability controls on the UI interface according to the user's pressure operation on the display screen 905. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0286] The fingerprint sensor 914 is used to collect the user's fingerprint, and the processor 901 identifies the user's identity based on the fingerprint collected by the fingerprint sensor 914, or the fingerprint sensor 914 identifies the user's identity based on the collected fingerprint. When the user's identity is identified as a trusted identity, the processor 901 authorizes the user to perform relevant sensitive operations, which include unlocking the screen, viewing encrypted information, downloading software, paying, and changing settings. The fingerprint sensor 914 can be set on the front, back, or side of the terminal 900. When a physical button or a manufacturer logo is set on the terminal 900, the fingerprint sensor 914 can be integrated with the physical button or the manufacturer logo.

[0287] The optical sensor 915 is used to collect the ambient light intensity. In one embodiment, the processor 901 can control the display brightness of the display screen 905 according to the ambient light intensity collected by the optical sensor 915. Specifically, when the ambient light intensity is high, the display brightness of the display screen 905 is increased; when the ambient light intensity is low, the display brightness of the display screen 905 is reduced. In another embodiment, the processor 901 can also dynamically adjust the shooting parameters of the camera component 906 according to the ambient light intensity collected by the optical sensor 915.

[0288] The proximity sensor 916, also called a distance sensor, is disposed on the front panel of the terminal 900. The proximity sensor 916 is used to collect the distance between the user and the front of the terminal 900. In one embodiment, when the proximity sensor 916 detects that the distance between the user and the front of the terminal 900 is gradually decreasing, the processor 901 controls the display screen 905 to switch from the screen-on state to the screen-off state; when the proximity sensor 916 detects that the distance between the user and the front of the terminal 900 is gradually increasing, the processor 901 controls the display screen 905 to switch from the screen-off state to the screen-on state.

[0289] Those skilled in the art will understand that Fig. 9 The structure shown in the figure does not constitute a limitation on the terminal 900, and the terminal 900 may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0290] Optionally, the computer device is provided as a server. Fig.10 This is a schematic diagram of the structure of a server provided in an embodiment of the present application. The server 1000 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 1001 and one or more memories 1002, wherein the memory 1002 stores at least one computer program, and the at least one computer program is loaded and executed by the processor 1001 to implement the methods provided in the above-mentioned various method embodiments. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output, and the server may also include other components for implementing device functions, which will not be described in detail here.

[0291] An embodiment of the present application also provides a computer device, which includes a processor and a memory, wherein the memory stores at least one program code, and the at least one program code is loaded and executed by the processor to implement the operations performed in the audio data processing method of the above embodiment.

[0292] An embodiment of the present application further provides a computer-readable storage medium, in which at least one program code is stored. The at least one program code is loaded and executed by a processor to implement the operations performed in the audio data processing method of the above embodiment.

[0293] The present application also provides a computer program product or a computer program, which includes a computer program code stored in a computer-readable storage medium. The computer program code is loaded and executed by a processor to implement the operations performed in the audio data processing method of the above embodiment.

[0294] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0295] The above are only optional embodiments of the embodiments of the present application and are not intended to limit the embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the protection scope of the present application.

Claims

1. An audio data processing method, characterized in that: The method comprises: Acquire a frequency spectrum image of the target audio data, wherein the frequency spectrum image is used to represent how the frequency of the sound in the target audio data changes over time; Based on the feature extraction sub-model in the mapping model, feature extraction is performed on the spectrum image of the target audio data, and each channel in the feature extraction sub-model outputs an image feature corresponding to a time point, the feature extraction sub-model includes multiple channels, and the time points corresponding to the multiple channels belong to the time period corresponding to the target audio data, and the image feature is used to describe the relationship between the frequency and time of the sound in the spectrum image; Based on the feature extraction submodel, the product of the image feature output by each channel and the corresponding weight is determined as the updated image feature corresponding to each time point, each channel in the feature extraction submodel has a corresponding weight, and the weights corresponding to the multiple channels are used to adjust the smoothness of the image features corresponding to the multiple time points; Based on the mapping model, feature transformation is performed on the updated image features corresponding to each time point to obtain the target expression parameters corresponding to each time point, the target expression parameters corresponding to the multiple time points constitute a target expression parameter set that matches the frequency spectrum image of the target audio data, the target expression parameters in the target expression parameter set are used to simulate the facial expressions of the singer when the target audio data is emitted, and the mapping model includes a mapping relationship between the frequency spectrum image and the expression parameter set; Performing smoothing processing on the target expression parameters corresponding to the multiple time points in the target expression parameter set; Mapping the target expression parameters corresponding to the multiple smoothed time points to the target interval to obtain the adjustment coefficients corresponding to the multiple time points, wherein the target interval is the adjustment range of the facial area in the template facial model; According to the obtained multiple adjustment coefficients, the template face model is adjusted respectively to obtain multiple target face models, and the multiple target face models are used to form an expression animation, and the expression animation is used to show the effect of the facial expression changing over time.

2. The method according to claim 1, characterized in that: The target audio data includes audio data in multiple time periods, and the step of obtaining a frequency spectrum image of the target audio data includes: Converting the audio data in the multiple time periods into matching frequency spectrum images respectively; The method further comprises: Based on the mapping relationship, target expression parameter sets matching the multiple spectrum images obtained by conversion are respectively obtained.

3. The method according to claim 1, characterized in that The method further comprises: Based on the frequency spectrum image of the target audio data, query the mapping relationship to determine a reference frequency spectrum image in the mapping relationship that is similar to the frequency spectrum image of the target audio data; The expression parameter set matched by the reference spectrum image in the mapping relationship is determined as the target expression parameter set.

4. The method according to claim 1, characterized in that: The method further comprises: Acquire a sample spectrum image and a sample expression parameter set of the sample audio data, wherein the sample expression parameters in the sample expression parameter set are used to simulate the facial expression of the singer when the singer emits the sample audio data; The mapping model is trained based on the sample spectrum image and the sample expression parameter set.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: According to the target expression parameters corresponding to the multiple time points in the target expression parameter set, the template face model is adjusted respectively to obtain multiple target face models; After adjusting the template face model according to the target expression parameters corresponding to the multiple time points in the target expression parameter set to obtain multiple target face models, the method further includes: During the process of playing the target audio data, the expression animation formed by the multiple target facial models is played.

6. The method according to claim 5, characterized in that The target expression parameters corresponding to any time point include expression parameters corresponding to multiple facial areas at the time point; the template face model is adjusted respectively according to the target expression parameters corresponding to multiple time points in the target expression parameter set to obtain multiple target face models, including: According to the target expression parameters corresponding to the multiple facial regions at the multiple time points, the corresponding facial regions in the template facial model are adjusted respectively to obtain the multiple target facial models.

7. The method according to claim 6, characterized in that Before respectively adjusting the corresponding facial regions in the template facial model according to the target expression parameters corresponding to the multiple facial regions at the multiple time points to obtain the multiple target facial models, the method further includes: Smoothing is performed on the target expression parameters corresponding to the same facial region at the multiple time points.

8. The method according to claim 5, characterized in that After adjusting the template face model according to the target expression parameters corresponding to the multiple time points in the target expression parameter set to obtain multiple target face models, the method further includes: storing the target audio data and the expression animations formed by the multiple target facial models in correspondence; In the process of playing the target audio data, playing the expression animation formed by the multiple target face models includes: In response to a play instruction for the target audio data, querying the expression animation stored corresponding to the target audio data; The target audio data is played, and the expression animation is played synchronously.

9. The method according to claim 5, characterized in that The step of obtaining a frequency spectrum image of target audio data comprises: In response to a playback instruction for the target audio data, acquiring a frequency spectrum image of the target audio data; In the process of playing the target audio data, playing the expression animation formed by the multiple target face models includes: The target audio data is played, and the expression animation is played synchronously.

10. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: The target audio data and the target expression parameter set are stored in correspondence; The method further comprises: In response to a play instruction for the target audio data, querying the target expression parameter set stored corresponding to the target audio data; The template face model is adjusted according to the expression parameters in the queried target expression parameter set to obtain the target face model.

11. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: In response to receiving a data playback request sent by a terminal, determining, based on an audio identifier carried in the data playback request, the target audio data indicated by the audio identifier and the target face model corresponding to the target audio data; The target audio data and the target facial model are sent to the terminal, and the terminal plays the target audio data and synchronously plays the facial expression animation formed by the target facial model.

12. An audio data processing device, characterized in that: The device comprises: An acquisition module, used for acquiring a frequency spectrum image of target audio data, wherein the frequency spectrum image is used for representing how the frequency of the sound in the target audio data changes over time; The acquisition module includes a feature extraction unit and a feature transformation unit; The feature extraction unit is used to extract features from the spectrum image of the target audio data based on the feature extraction sub-model in the mapping model, and each channel in the feature extraction sub-model outputs an image feature corresponding to a time point, the feature extraction sub-model includes multiple channels, and the time points corresponding to the multiple channels belong to the time period corresponding to the target audio data, and the image feature is used to describe the relationship between the frequency and time of the sound in the spectrum image; A determination module, configured to determine, based on the feature extraction submodel, the product of the image feature output by each channel and the corresponding weight as the updated image feature corresponding to each time point, wherein each channel in the feature extraction submodel has a corresponding weight, and the weights corresponding to the multiple channels are used to adjust the smoothness of the image features corresponding to the multiple time points; The feature transformation unit is used to perform feature transformation on the updated image features corresponding to each time point based on the mapping model to obtain the target expression parameters corresponding to each time point, the target expression parameters corresponding to the multiple time points constitute a target expression parameter set that matches the spectrum image of the target audio data, the target expression parameters in the target expression parameter set are used to simulate the facial expressions of the singer when the target audio data is emitted, and the mapping model includes a mapping relationship between the spectrum image and the expression parameter set; A processing module, used for smoothing the target expression parameters corresponding to the multiple time points in the target expression parameter set; The adjustment module is used to map the target expression parameters corresponding to the smoothed multiple time points to the target interval to obtain the adjustment coefficients corresponding to the multiple time points, and the target interval is the adjustment range of the facial area in the template facial model; according to the obtained multiple adjustment coefficients, the template facial model is adjusted respectively to obtain multiple target facial models, and the multiple target facial models are used to form an expression animation, and the expression animation is used to show the effect of the facial expression changing over time.

13. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one program code, and the at least one program code is loaded and executed by the processor to implement the operations performed in the audio data processing method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one program code, and the at least one program code is loaded and executed by the processor to implement the operations performed in the audio data processing method according to any one of claims 1 to 11.

Images