Virtual image expression control method, device and equipment fusing singing emotions

Abstract

The invention provides a virtual image expression control method, device and equipment fusing singing emotions, and relates to the technical field of computers. In the application, firstly, a user face image and a singing song audio of a target singing user are acquired; secondly, determining initial expression control data of the target virtual image based on the user face image; then, semantic mining is carried out on the singing song audio, and a song audio semantic vector is formed; further, based on the song audio semantic vector, optimizing the initial expression control data to form target expression control data; and finally, based on the target expression control data, controlling the expression of the target virtual image. On the basis of the content, the problem that in the prior art, the accuracy of virtual image expression control in a song singing application scene is relatively low can be solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a virtual image expression control method and device fusing song emotion and equipment. BACKGROUND

[0002] Virtual image, namely virtual digital person, generally refers to a virtual character with digitalized appearance, which generally depends on display device. With the development of computer graphics (CG) technology, digital person can be realized through three-dimensional modeling technology. In some application scenarios, in order to improve the stickiness of users to the platform, a corresponding virtual image is generated in the process of users singing a song to improve the interest. However, in the prior art, the control of the virtual image is mainly the control of the overall action of the virtual image, and the expression with high precision is not the focus of research. The inventors found that in other application scenarios, the expression control of the virtual image is generally directly based on the user's face image, and in the singing scenario, the overall expression is generally ignored. That is, in the prior art, the expression control of the virtual image has the problem of relatively low precision. SUMMARY

[0003] Therefore, the purpose of the present application is to provide a virtual image expression control method, device and equipment fusing song emotion to improve the problem of relatively low precision of virtual image expression control in the singing application scenario in the prior art.

[0004] To achieve the above purpose, the present application adopts the following technical solutions: A virtual image expression control method fusing song emotion, comprising: obtaining a user face image of a target singing user and a singing song audio, wherein the user face image is an image formed by collecting the face of the target singing user in the process of singing a song, and the singing song audio is an audio formed by the target singing user singing a song; determining initial expression control data of a target virtual image based on the user face image, wherein the initial expression control data at least includes coordinates of each facial feature point of the target virtual image; performing semantic mining on the singing song audio to form a song audio semantic vector, wherein the song audio semantic vector is at least used to represent the semantic information related to emotion of the target singing user in the singing process; optimizing the initial expression control data based on the song audio semantic vector to form target expression control data, wherein the target expression control data at least includes coordinates of each facial feature point of the target virtual image; controlling the expression of the target virtual image based on the target expression control data.

[0005] In a preferred selection of the present application, in the virtual image expression control method of fusing song emotion, the step of performing semantic mining on the singing song audio to form a song audio semantic vector includes: extracting local features of the singing song audio to obtain first local feature information, and strengthening semantic information of a time domain of the singing song audio based on the first local feature information to form a song time domain vector; performing frequency domain conversion on the singing song audio to obtain a singing song spectrogram; extracting local features of the singing song spectrogram to obtain second local feature information, and strengthening semantic information of a frequency domain of the singing song spectrogram based on the second local feature information to form a song frequency domain vector; fusing the song time domain vector and the song frequency domain vector across data domains to form a song audio semantic vector corresponding to the singing song audio.

[0006] In a preferred selection of the present application, in the virtual image expression control method of fusing song emotion, the step of extracting local features of the singing song audio to obtain first local feature information, and strengthening semantic information of a time domain of the singing song audio based on the first local feature information to form a song time domain vector includes: performing zero-crossing point analysis on the singing song audio to obtain first local feature information including a zero-crossing rate, wherein the zero-crossing point refers to a position where a signal in the singing song audio changes from a positive value to a negative value or from a negative value to a positive value; mapping the singing song audio and the first local feature information into vector spaces respectively to form a song audio mapping vector of a time domain and a zero-crossing mapping vector; performing attention coding on the song audio mapping vector based on the zero-crossing mapping vector to form a zero-crossing attention vector; performing residual connection on the zero-crossing attention vector and the song audio mapping vector to form a song time domain vector.

[0007] In a preferred selection of the present application, in the virtual image expression control method for fusing song emotion, the step of extracting local features of the singing song spectrogram to obtain second local feature information, and strengthening semantic information of a frequency domain of the singing song spectrogram based on the second local feature information to form a song frequency domain vector, comprises: calculating mean, variance, peak value, skewness and kurtosis of the singing song spectrogram to obtain second local feature information including mean, variance, peak value, skewness and kurtosis; deeply mining the singing song spectrogram to form a singing song depth vector; vector space mapping the second local feature information to form a local feature mapping vector; attention coding the singing song depth vector based on the local feature mapping vector to form a local feature attention vector; residual connection of the local feature attention vector and the singing song depth vector to form a song frequency domain vector.

[0008] In a preferred selection of the present application, in the virtual image expression control method for fusing song emotion, the step of deeply mining the singing song spectrogram to form a singing song depth vector comprises: calculating the difference of adjacent data points of the singing song spectrogram along the frequency axis direction to form a frequency axis change spectrogram corresponding to the singing song spectrogram; deep convolution of the singing song spectrogram and the frequency axis change spectrogram respectively to form a song spectrogram vector and a change spectrogram vector; adjusting the change spectrogram vector based on a target weight parameter distribution to form a change adjustment vector, wherein the target weight parameter distribution is formed by learning user face image samples, singing song audio samples and expression control data labels; nonlinear activation of the change adjustment vector to form a gating parameter distribution; adjusting the song spectrogram vector based on the gating parameter distribution to form a singing song depth vector.

[0009] In a preferred selection of the present application, in the virtual image expression control method for fusing song emotion, the step of fusing the song time domain vector and the song frequency domain vector across data domains to form a song audio semantic vector corresponding to the singing song audio comprises: a plurality of convolution mappings or a plurality of pooling mappings are performed on the song time domain vector to form a plurality of song mapping vectors corresponding to the song time domain vector; performing attention score calculation based on each of the song mapping vectors and the song frequency domain vector, to obtain a plurality of attention scores; fusing the plurality of attention scores to form a fused attention score; performing weighted sum calculation on the song frequency domain vector based on the fused attention score, to form a song audio semantic vector corresponding to the sung song audio.

[0010] In a preferred selection of the present application, in the virtual image expression control method fusing song emotion as described above, the step of performing optimization on the initial expression control data based on the song audio semantic vector to form target expression control data, comprises: mapping the initial expression control data into a vector space to form an expression control mapping vector; performing semantic optimization on the expression control mapping vector based on the song audio semantic vector to form an expression control optimized vector; performing semantic restoration based on the expression control optimized vector to output target expression control data.

[0011] In a preferred selection of the present application, in the virtual image expression control method fusing song emotion as described above, the step of performing semantic optimization on the expression control mapping vector based on the song audio semantic vector to form an expression control optimized vector, comprises: In the encoding stage, respectively in a plurality of semantic depths, performing attention encoding on the song audio semantic vector based on the expression control mapping vector, so that the semantic information in the expression control mapping vector is fused into the song audio semantic vector to form a plurality of encoding fusion vectors, wherein, starting from the second semantic depth, the objects of attention encoding include the encoding fusion vector corresponding to the previous semantic depth and the semantic vector of the expression control mapping vector at the corresponding semantic depth; In the decoding stage, respectively in a plurality of semantic depths, performing attention decoding on the expression control mapping vector based on the plurality of encoding fusion vectors, so that the semantic information in the encoding fusion vector is fused into the expression control mapping vector to form a plurality of decoding fusion vectors, wherein, starting from the second semantic depth, the objects of attention decoding include the decoding fusion vector corresponding to the previous semantic depth, the encoding fusion vector of the corresponding semantic depth, and the semantic vector of the expression control mapping vector at the corresponding semantic depth; determining the expression control optimized vector based on the last decoding fusion vector.

[0012] The present application also provides a virtual image expression control device fusing song emotion.

[0013] On the basis of the above, the application further provides an electronic device, comprising: a memory for storing a computer program; a processor connected with the memory, for executing the computer program stored in the memory to realize the virtual image expression control method fusing song emotion as described above.

[0014] The virtual image expression control method, device and equipment fusing song emotion provided by the application firstly acquire a user facial image of a target singing user and a singing song audio; secondly, determine initial expression control data of a target virtual image based on the user facial image; then, perform semantic mining on the singing song audio to form a song audio semantic vector; further, optimize the initial expression control data based on the song audio semantic vector to form target expression control data; finally, control the expression of the target virtual image based on the target expression control data. Based on the above, on the one hand, since the initial expression control data of the target virtual image is directly determined based on the user facial image, the global and overall expression control of the target virtual image can be performed, and on the other hand, since the initial expression control data is optimized based on the semantic information related to emotion in the singing song audio, the target expression control data formed can further increase more expression detail control on the basis of the global and overall expression control, thereby improving the control precision and further improving the problem of relatively low precision of virtual image expression control in the song singing application scenario in the prior art. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to make the above-mentioned purposes, features and advantages of the application more obvious and easy to understand, the following preferred embodiments are specifically described below with reference to the accompanying drawings.

[0016] Figure 1 The structural block diagram of the electronic device provided by the embodiments of the application is shown.

[0017] Figure 2 The flowchart of the virtual image expression control method fusing song emotion provided by the embodiments of the application is shown.

[0018] Figure 3 The schematic diagram of controlling the expression of the target virtual image provided by the embodiments of the application is shown.

[0019] Figure 4 The schematic diagram of deep mining of the singing song spectrum graph provided by the embodiments of the application is shown.

[0020] Figure 5 The schematic diagram of optimizing the expression control mapping vector provided by the embodiments of the application is shown.

[0021] Figure 6A block diagram of a virtual image expression control device fusing a singing emotion is provided for embodiments of the present application. DETAILED DESCRIPTION

[0022] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in connection with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application but not all embodiments of the present application. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations.

[0023] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work are within the scope of protection of the present application.

[0024] As shown in Figure 1 The embodiments of the present application provide an electronic device. The electronic device can include a memory, a processor and a virtual image expression control device fusing a singing emotion.

[0025] In detail, the memory and the processor are directly or indirectly electrically connected to realize data transmission or interaction. For example, the memory and the processor can be electrically connected through one or more communication buses or signal lines. The virtual image expression control device fusing a singing emotion includes at least one software function module stored in the memory in the form of software or firmware. The processor is configured to execute the executable computer programs stored in the memory, for example, the software function modules and computer programs included in the virtual image expression control device fusing a singing emotion, to realize the virtual image expression control method fusing a singing emotion provided by the embodiments of the present application.

[0026] Optionally, the memory can be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electric erasable programmable read-only memory (EEPROM), and the like. Moreover, the processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), a system on chip (SoC), and the like; or can be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0027] It can be understood that, Figure 1 The structure shown is only schematic, and the electronic device can further include more or fewer components than those shown in Figure 1 or have a different configuration from that shown in Figure 1 For example, it can further include a communication unit for information interaction with other devices (such as an image acquisition device, an audio acquisition device, etc., and a display device for displaying a virtual image, etc.).

[0028] In combination with Figure 2 The embodiments of the present application also provide a virtual image expression control method for fusing song emotion, which can be applied to the above-mentioned electronic device. The method steps defined by the flow of the virtual image expression control method for fusing song emotion can be implemented by the electronic device. The specific flow shown in Figure 2 will be described in detail below.

[0029] In step S110, a user face image of a target singing user and a singing song audio are acquired.

[0030] In the embodiment of the present application, the electronic device can obtain a user face image of a target singing user and a singing song audio. The user face image is an image formed by collecting the face of the target singing user during song singing. The singing song audio is an audio formed by the target singing user singing a song. In addition, the user face image can be collected based on an image collection device, the singing song audio can be collected based on an audio collection device, and the user face image and the singing song audio can have time dimension synchronization.

[0031] In step S120, initial expression control data of a target virtual image is determined based on the user face image.

[0032] In the embodiment of the present application, after obtaining the user face image, the electronic device can determine initial expression control data of a target virtual image based on the user face image. The initial expression control data at least includes coordinates of each facial feature point of the target virtual image. It should be noted that the specific method of determining the initial expression control data can adopt any existing technology. For example, a neural network model based on a training process can be used to directly generate the coordinates of each facial feature point of the target virtual image based on the user face image, and the coordinates are used as the initial expression control data. For another example, the user face image can be subjected to facial feature point recognition to obtain the coordinates of each facial feature point, and then the coordinates of each facial feature point are mapped (a mapping relationship between the facial feature points of the user and the facial feature points of the target virtual image can be determined in advance) to the coordinates of the facial feature points of the target virtual image, thereby forming the initial expression control data.

[0033] In step S130, the singing song audio is subjected to semantic mining to form a song audio semantic vector.

[0034] In the embodiment of the present application, after obtaining the singing song audio, the electronic device can perform semantic mining on the singing song audio to form a song audio semantic vector. The song audio semantic vector is at least used to represent semantic information related to emotion of the target singing user during singing. That is, the semantic information related to emotion in the singing song audio can be mainly mined (other related semantic information can also be mined synchronously), and represented in the form of a vector, so that the song audio semantic vector can be formed.

[0035] In step S140, the initial expression control data is optimized based on the song audio semantic vector to form target expression control data.

[0036] In the embodiments of the present application, after the song audio semantic vector and the initial expression control data are formed, the electronic device can optimize the initial expression control data based on the song audio semantic vector to form target expression control data. The target expression control data at least includes coordinates of each facial feature point of the target virtual image. That is, the emotional-related semantic information in the song audio semantic vector can be fused into the initial expression control data, so as to adjust and update the coordinates of each facial feature point, and then obtain the coordinates of each facial feature point with more expression details, that is, the target expression control data.

[0037] In step S150, the expression of the target virtual image is controlled based on the target expression control data.

[0038] In the embodiments of the present application, after the target expression control data is formed, the electronic device can control the expression of the target virtual image based on the target expression control data. That is, the coordinates of each facial feature point of the target virtual image are updated from the current coordinates to the coordinates in the target expression control data, so as to realize dynamic control of the expression.

[0039] Based on the above, on the one hand, since the initial expression control data of the target virtual image is directly determined based on the user's face image, the global and overall expression control of the target virtual image can be realized, and on the other hand, since the initial expression control data is optimized based on the emotional-related semantic information in the song audio, the target expression control data formed can further increase the expression detail control on the basis of the global and overall expression control, thereby improving the control accuracy, and further improving the problem that the expression control accuracy of the virtual image in the song singing application scenario in the prior art is relatively low.

[0040] In the first aspect, it needs to be explained that the specific way of performing semantic mining on the song audio is not limited, and can be selected according to actual needs.

[0041] For example, in an alternative implementation, a trained neural network model can be used to perform semantic mining on the song audio to form a corresponding song audio semantic vector. The neural network model is trained based on song audio samples and emotion labels. In this way, through training, the neural network model can mine emotional-related semantic information.

[0042] For example, in another alternative implementation, in order to fully excavate the semantic information related to emotion in the sung song audio, so that the formed song audio semantic vector can fully and reliably represent the emotion, the step S130 can further include steps S131, S132, S133 and S134, and the specific contents of each step are as follows.

[0043] Step S131, extracting local features of the sung song audio to obtain first local feature information, and strengthening semantic information in the time domain of the sung song audio based on the first local feature information to form a song time domain vector.

[0044] In the embodiments of the present application, local features of the sung song audio can be extracted to obtain first local feature information, and semantic information in the time domain of the sung song audio can be strengthened based on the first local feature information to form a song time domain vector. That is, the local feature information related to emotion can be extracted from the sung song audio first, and then the global semantic information of the sung song audio in the time domain can be strengthened based on the local feature information related to emotion, so that the formed song time domain vector can represent the global information of the song and highlight the semantic information related to emotion.

[0045] Step S132, performing frequency domain conversion on the sung song audio to obtain a sung song spectrogram.

[0046] In the embodiments of the present application, frequency domain conversion can be performed on the sung song audio to obtain a sung song spectrogram, for example, Fourier conversion can be performed on the sung song audio to form a corresponding spectrogram, i.e., the sung song spectrogram.

[0047] Step S133, extracting local features of the sung song spectrogram to obtain second local feature information, and strengthening semantic information in the frequency domain of the sung song spectrogram based on the second local feature information to form a song frequency domain vector.

[0048] In the embodiment of the present application, after the singing song spectrogram is formed, local features of the singing song spectrogram can be extracted to obtain second local feature information, and semantic information of the frequency domain of the singing song spectrogram can be strengthened based on the second local feature information to form a song frequency domain vector. That is, the local feature information related to emotion representation can be extracted from the singing song spectrogram first, and then the global semantic information of the singing song spectrogram in the frequency domain can be strengthened based on the local feature information related to emotion, so that the song frequency domain vector formed can represent the global information of the song and highlight the semantic information related to emotion.

[0049] In step S134, the song time domain vector and the song frequency domain vector are fused across data domains to form a song audio semantic vector corresponding to the singing song audio.

[0050] In the embodiment of the present application, after the song time domain vector and the song frequency domain vector are formed, the song time domain vector and the song frequency domain vector can be fused across data domains to form a song audio semantic vector corresponding to the singing song audio. That is, the semantic information in the time domain and the semantic information in the frequency domain can be fused to obtain a song audio semantic vector with richer semantic information, that is, to realize multi-angle representation of emotional semantic information.

[0051] It can be understood that in the above step S131, the specific way of forming the song time domain vector is not limited. For example, in an alternative implementation, it is found through research that the zero-crossing rate can be used to represent the degree of change of the signal waveform, which is related to the emotional state of the audio signal. Specifically, a high zero-crossing rate is usually related to intense emotional fluctuations (such as anger), and a low zero-crossing rate is related to mild emotions (such as sadness). Based on this, the above step S131 can further include steps S131a, S131b, S131c and S131d, and the specific contents are as follows.

[0052] In step S131a, the singing song audio is analyzed for zero-crossing points to obtain first local feature information including a zero-crossing rate.

[0053] In the embodiment of the present application, the singing song audio can be analyzed for zero-crossing points to obtain first local feature information including a zero-crossing rate. The zero-crossing point refers to a position in the singing song audio where the signal changes from a positive value to a negative value or from a negative value to a positive value, that is, the signs of two adjacent signals are different (one is positive and the other is negative), and it is considered that zero-crossing occurs. The zero-crossing rate can refer to the frequency of occurrence of zero-crossing points, that is, the ratio between the number of occurrences and the total number of signals.

[0054] Step S131b, mapping the singing song audio and the first local feature information into vector spaces respectively to form a song audio mapping vector and a zero-crossing mapping vector in a time domain.

[0055] In the embodiments of the present application, after obtaining the first local feature information, the singing song audio and the first local feature information can be mapped into vector spaces respectively to form a song audio mapping vector and a zero-crossing mapping vector in a time domain. In detail, the singing song audio can be mapped into a vector space by a 1D convolutional neural network (1D CNN), a convolution kernel slides on an input signal (i.e. the singing song audio) and calculates a convolution operation to generate a new feature map (feature map), i.e. the song audio mapping vector. The first local feature information can be mapped into a vector space by a word embedding model.

[0056] Step S131c, performing attention encoding on the song audio mapping vector based on the zero-crossing mapping vector to form a zero-crossing attention vector.

[0057] In the embodiments of the present application, after forming the zero-crossing mapping vector and the song audio mapping vector, the song audio mapping vector can be attention encoded based on the zero-crossing mapping vector to form a zero-crossing attention vector. Exemplarily, the song audio mapping vector can be cross-attention processed based on the zero-crossing mapping vector to form a zero-crossing attention vector.

[0058] Step S131d, performing residual connection on the zero-crossing attention vector and the song audio mapping vector to form a song time domain vector.

[0059] In the embodiments of the present application, after forming the zero-crossing attention vector, the zero-crossing attention vector and the song audio mapping vector can be residual connected to form a song time domain vector. Exemplarily, the zero-crossing attention vector and the song audio mapping vector can be added, and then the result of the addition operation can be normalized, nonlinearly activated, etc. In this way, a song time domain vector can be formed.

[0060] It can be understood that in the step S133, the specific way of forming the song frequency domain vector is not limited, for example, in an alternative embodiment, it is found through research that the melody, rhythm and other elements in the audio have an impact on the representation of emotion in the frequency domain, therefore, the local feature information related to the melody, rhythm and other elements can be mined, and based on this, the step S133 can further include steps S133a, S133b, S133c, S133d and S133e, and the specific contents of each step are as follows.

[0061] In step S133a, the mean, variance, peak, skewness and kurtosis of the singing song spectrogram are calculated to obtain second local feature information including the mean, variance, peak, skewness and kurtosis.

[0062] In the embodiments of the present application, the mean, variance, peak, skewness and kurtosis of the singing song spectrogram can be calculated to obtain second local feature information including the mean, variance, peak, skewness and kurtosis. The mean can be used to reflect the average energy intensity in the singing song spectrogram. The variance can be used to reflect the degree of change in signal energy in the singing song spectrogram, a higher variance indicates a larger change in signal frequency or time, and a lower variance indicates a relatively stable signal. The peak can be used to reflect the maximum energy value in the singing song spectrogram. The skewness can be used to reflect the asymmetry of the energy distribution in the singing song spectrogram, if the energy of the spectrogram is mostly concentrated in the low frequency part, the skewness value will be negative, otherwise, if the energy is concentrated in the high frequency part, the skewness value will be positive. The kurtosis can be used to reflect the energy concentration degree of the singing song spectrogram, a higher kurtosis indicates that the energy distribution has a clear concentration peak, and a lower kurtosis indicates that the energy distribution is relatively smooth. Specifically, the calculation formula of the skewness is as follows: ; Wherein, is the skewness, T is the length of the time axis, |X(t,f)| is the amplitude of the point (t,f), is the mean, is the variance.

[0063] In addition, the calculation formula of the kurtosis is as follows: ; Wherein, is the kurtosis, T is the length of the time axis, |X(t,f)| is the amplitude of the point (t,f), is the mean, is the variance.

[0064] In step S133b, the singing song spectrogram is deeply mined to form a singing song depth vector.

[0065] In the embodiment of the present application, the singing song spectrogram can also be deeply mined to form a singing song deep vector, that is, the global semantic information in the singing song spectrogram is mined.

[0066] In step S133c, the second local feature information is mapped in a vector space to form a local feature mapping vector.

[0067] In the embodiment of the present application, after the second local feature information is formed, the second local feature information can be mapped in a vector space to form a local feature mapping vector. For example, the second local feature information can be embedded by a word embedding model to realize corresponding vector space mapping, so as to obtain a corresponding local feature mapping vector.

[0068] In step S133d, the singing song deep vector is encoded based on the local feature mapping vector to form a local feature attention vector.

[0069] In the embodiment of the present application, after the local feature mapping vector and the singing song deep vector are formed, the singing song deep vector can be encoded based on the local feature mapping vector (that is, cross-attention processing) to form a local feature attention vector. In this way, the semantic information related to emotion represented by the local feature mapping vector can be fused into the global semantic information in the frequency domain represented by the singing song deep vector.

[0070] In step S133e, the local feature attention vector and the singing song deep vector are residual connected to form a song frequency domain vector.

[0071] In the embodiment of the present application, after the local feature attention vector is formed, the local feature attention vector and the singing song deep vector can be residual connected to form a song frequency domain vector. For example, the local feature attention vector and the singing song deep vector can be added, and then the result of the addition operation can be normalized, nonlinearly activated, or the like. In this way, a song frequency domain vector can be formed.

[0072] It can be understood that in the step S133b, the specific way of deep mining of the singing song spectrogram is not limited, for example, in an alternative embodiment, it is found through research that emotions tend to change with the rapid changes of voice or music, especially in terms of speed, tone, intensity and timbre, etc., therefore, the frequency change operation in the spectrogram helps to capture these rapid changes and provides key features in emotion recognition, based on which, the above step S133b can further include steps b1, b2, b3, b4 and b5, the specific contents of each step are as follows, and can be understood in combination with the contents shown in the above. Figure 4

[0073] Step b1, difference calculation of adjacent data points of the singing song spectrogram is performed along the frequency axis direction, forming a frequency axis change spectrogram corresponding to the singing song spectrogram.

[0074] In the embodiment of the present application, the difference calculation of adjacent data points of the singing song spectrogram can be performed along the frequency axis direction (such as the column direction in Figure 4 , for example, for each data point, the difference between the data point and the previous data point (or the ratio of the difference to the current data point as the output of the calculation, i.e. the change rate) can be calculated, to obtain the data point at the corresponding position in the frequency axis change spectrogram. It should be noted that the previous data point of the first data point is 0. Based on this, the frequency change can be effectively characterized through the frequency axis change spectrogram.

[0075] Step b2, deep convolution is performed on the singing song spectrogram and the frequency axis change spectrogram respectively, forming a song spectrogram vector and a change spectrogram vector.

[0076] In the embodiment of the present application, deep convolution can be performed on the singing song spectrogram and the frequency axis change spectrogram respectively, forming a song spectrogram vector and a change spectrogram vector. The deep convolution can be realized through a neural network model including multiple convolution layers, for example, including convolution layer 1, pooling layer 1, convolution layer 2, pooling layer 2, convolution layer 3, pooling layer 3 and full connection layer. It should be noted that the singing song spectrogram and the frequency axis change spectrogram need to pay attention to different semantic information, therefore, different neural network models can be used, specifically, the architectures of the two neural network models can be the same (as described above), but the specific model parameters can be different, and the specific parameters can be formed during learning according to the corresponding samples and labels.

[0077] Step b3, based on the target weight parameter distribution, adjusting the change spectrogram vector to form a change adjustment vector.​

[0078] In the embodiment of the present application, after the change spectrum map vector is formed, the change spectrum map vector can be adjusted based on a target weight parameter distribution to form a change adjustment vector. The target weight parameter distribution is formed by learning user facial image samples, singing song audio samples and expression control data labels. In this way, through learning, it can be learned from the samples which position semantic information is more important, that is, represented by the target weight parameter distribution. Therefore, the target weight parameter distribution and the change spectrum map vector can be multiplied bit by bit to form the change adjustment vector, so that important semantic information is represented.

[0079] Step b4, performing nonlinear activation on the change adjustment vector to form a gating parameter distribution.

[0080] In the embodiment of the present application, after the change adjustment vector is formed, the change adjustment vector can be nonlinearly activated to form a gating parameter distribution. For example, the change adjustment vector can be processed by a sigmiod function, so that the parameters are mapped to 0-1.

[0081] Step b5, adjusting the song spectrum map vector based on the gating parameter distribution to form a singing song depth vector.

[0082] In the embodiment of the present application, after the gating parameter distribution is formed, the song spectrum map vector can be adjusted based on the gating parameter distribution to form a singing song depth vector. That is, the gating parameter distribution and the song spectrum map vector can be multiplied bit by bit, so that the semantic information corresponding to the important positions represented by the gating parameter distribution is highlighted in the singing song depth vector.

[0083] It can be understood that the specific manner of fusing the song time domain vector and the song frequency domain vector across the data domain in the above step S134 is not limited, for example, in an alternative embodiment, in order to be able to reduce the computing overhead and improve the computing efficiency in the process of fusing in multiple ways, the above step S134 can further include steps S134a, S134b, S134c and S134d, and the specific contents of each step are as follows.

[0084] Step S134a, performing multiple convolution mapping or multiple pooling mapping on the song time domain vector to form multiple song mapping vectors corresponding to the song time domain vector.

[0085] In the embodiments of the present application, the song time domain vector can be subjected to various convolution mapping or various pooling mapping to form a plurality of song mapping vectors corresponding to the song time domain vector. In this way, various high-level semantic information can be captured through various convolution mapping or various pooling mapping. For example, the song time domain vector can be subjected to maximum value pooling to obtain a first song mapping vector, and subjected to mean value pooling to obtain a second song mapping vector.

[0086] In step S134b, attention score calculation is performed based on each of the song mapping vectors and the song frequency domain vector to obtain a plurality of attention scores.

[0087] In the embodiments of the present application, after the plurality of song mapping vectors are formed, attention score calculation can be performed based on each of the song mapping vectors and the song frequency domain vector to obtain a plurality of attention scores. For example, the first song mapping vector and the transpose vector of the song frequency domain vector can be multiplied to form a first attention score, and the second song mapping vector and the transpose vector of the song frequency domain vector can be multiplied to form a second attention score.

[0088] In step S134c, the plurality of attention scores are fused to form a fused attention score.

[0089] In the embodiments of the present application, after the plurality of attention scores are formed, the plurality of attention scores can be fused to form a fused attention score. For example, mean value calculation can be performed on the plurality of attention scores to obtain the fused attention score.

[0090] In step S134d, based on the fused attention score, weighted sum calculation is performed on the song frequency domain vector to form a song audio semantic vector corresponding to the sung song audio.

[0091] In the embodiments of the present application, after the fused attention score is formed, based on the fused attention score, weighted sum calculation can be performed on the song frequency domain vector to form a song audio semantic vector corresponding to the sung song audio. Based on this, both capturing various high-level semantic information for semantic information fusion and performing weighted sum calculation only once can be realized, so that the calculation amount is reduced and the calculation efficiency is ensured.

[0092] In the second aspect, it is necessary to explain that the specific manner of optimizing the initial expression control data is not limited, and can be selected according to actual needs.

[0093] For example, in an alternative implementation, the song audio semantic vector can be decoded and output to form other expression control data, wherein the other expression control data includes coordinates of each facial feature point of the target virtual image. Then, the other expression control data and the initial expression control data can be fused, for example, the coordinates of the same facial feature point are weighted and summed to obtain the coordinates of the corresponding facial feature point in the target expression control data. In addition, the weighting coefficient (such as 0.1) corresponding to the other expression control data can be smaller than the weighting coefficient (such as 0.9) corresponding to the initial expression control data when performing the weighted sum calculation. In this way, fine tuning of the initial expression control data can be achieved.

[0094] For example, in an alternative implementation, in order to enable sufficient fusion of the semantic information represented by the song audio semantic vector and the initial expression control data, so as to enable high-precision optimization of the initial expression control data, the above step S140 can further include steps S141, S142 and S143, and the specific contents of each step are as follows.

[0095] Step S141, mapping the initial expression control data into a vector space to form an expression control mapping vector.

[0096] In the embodiments of the present application, the initial expression control data can be mapped into a vector space to form an expression control mapping vector. For example, the coordinates of each facial feature point in the initial expression control data can be embedded by a word embedding model to map into a vector space, thereby forming the expression control mapping.

[0097] Step S142, based on the song audio semantic vector, performing semantic optimization on the expression control mapping vector to form an expression control optimization vector.

[0098] In the embodiments of the present application, after forming the expression control mapping vector, the expression control mapping vector can be optimized based on the song audio semantic vector to form an expression control optimization vector. That is, the optimization of the initial expression control data can be transferred to the optimization of the corresponding semantic vector, so that the optimization granularity is smaller and the precision is higher, or the potential semantic information can be fully utilized to make the optimization more reliable.

[0099] Step S143, based on the expression control optimization vector, performing semantic restoration to output target expression control data.

[0100] In the embodiment of the present application, after the expression control optimization vector is formed, semantic restoration can be performed based on the expression control optimization vector, and target expression control data is output. That is, the expression control optimization vector can be decoded and output, which can be implemented by a corresponding decoding neural network. For example, the expression control optimization vector can be fully connected to obtain a fully connected vector with a size of 1*3, and then the fully connected vector can be identity mapped or linearly mapped to obtain a 1*3 parameter distribution (x, y, z), each parameter serving as a coordinate of a dimension in a three-dimensional space. In this way, the coordinates of the first facial feature point can be obtained, and then the coordinates of the first facial feature point can be mapped to a vector space to obtain a first coordinate mapping vector. Then, based on the attention mechanism, the coordinate mapping vector and the expression control optimization vector can be fused to form a fusion output vector, and then the fusion output vector is fully connected, identity mapped or linearly mapped in the manner described above to obtain the coordinates of the second facial feature point. Further, the coordinates of the second facial feature point can be mapped to a vector space to obtain a second coordinate mapping vector, and then based on the attention mechanism, the coordinate mapping vector and the previous fusion output vector can be fused to form the current fusion output vector, and then the fusion output vector is fully connected, identity mapped or linearly mapped in the manner described above to obtain the coordinates of the third facial feature point. In this way, starting from the second facial feature point, the generation of the coordinates of each facial feature point is constrained by the coordinates of the previous facial feature point, so that the accuracy can be higher.

[0101] It can be understood that the specific manner of performing semantic optimization on the expression control mapping vector in step S142 is not limited. For example, in an alternative embodiment, in order to ensure the accuracy of semantic optimization, that is, to sufficiently fuse the semantic information in the song audio semantic vector into the expression control mapping vector, step S142 can further include steps S142a, S142b and S142c, and the specific contents of each step are as follows.

[0102] Step S142a, in the encoding stage, based on the expression control mapping vector, the song audio semantic vector is attention encoded in multiple semantic depths, so that the semantic information in the expression control mapping vector is fused into the song audio semantic vector to form multiple encoding fusion vectors.

[0103] In the embodiments of the present application, in the encoding stage, in multiple semantic depths respectively, the song audio semantic vector is encoded based on the expression control mapping vector (i.e. cross-attention processing), so that the semantic information in the expression control mapping vector is fused into the song audio semantic vector, forming multiple encoding fusion vectors. Wherein, combining Figure 5 In the first semantic depth, the song audio semantic vector can be cross-attention processed based on the expression control mapping vector to obtain a first encoding fusion vector. Starting from the second semantic depth, the objects of attention encoding include the encoding fusion vector corresponding to the previous semantic depth and the semantic vector of the expression control mapping vector at the corresponding semantic depth. For example, for the second semantic depth, the encoding fusion vector corresponding to the first semantic depth and the expression control mapping vector can be respectively pooled and compressed to increase the semantic depth, and then the pooled and compressed encoding fusion vector can be cross-attention processed based on the pooled and compressed expression control mapping vector to obtain a second encoding fusion vector. For example, for the third semantic depth, the encoding fusion vector corresponding to the second semantic depth and the previous pooled and compressed expression control mapping vector can be respectively pooled and compressed, and then the pooled and compressed encoding fusion vector can be cross-attention processed based on the pooled and compressed expression control mapping vector to obtain a third encoding fusion vector.

[0104] Step S142b, in the decoding stage, in multiple semantic depths respectively, the expression control mapping vector is decoded based on the multiple encoding fusion vectors, so that the semantic information in the encoding fusion vector is fused into the expression control mapping vector, forming multiple decoding fusion vectors.

[0105] In the embodiments of the present application, after the semantic information is fused in the multiple semantic depths by encoding, the multiple semantic depths can be sequentially decoded and output. Specifically, in the decoding stage, in the multiple semantic depths, the expression control mapping vector is decoded based on the multiple encoding fusion vectors, so that the semantic information in the encoding fusion vectors is fused into the expression control mapping vector, forming multiple decoding fusion vectors. For the first semantic depth, the expression control mapping vector of the last semantic depth (pooling compression) can be cross-attention processed based on the last encoding fusion vector to form the first decoding fusion vector. Starting from the second semantic depth, the objects of attention decoding include the decoding fusion vector corresponding to the previous semantic depth, the encoding fusion vector of the corresponding semantic depth, and the semantic vector of the expression control mapping vector at the corresponding semantic depth. For example, for the second last semantic depth, the first decoding fusion vector can be de-pooling expanded, then the de-pooling expanded decoding fusion vector and the encoding fusion vector of the second last semantic depth can be averaged to obtain a mean fusion vector, and the expression control mapping vector of the second last semantic depth is cross-attention processed based on the mean fusion vector to form the second decoding fusion vector. Based on the same logic, the third decoding fusion vector can also be formed.

[0106] In step S142c, the expression control optimization vector is determined based on the last decoding fusion vector.

[0107] In the embodiments of the present application, after the last decoding fusion vector is obtained, the expression control optimization vector can be determined based on the last decoding fusion vector. For example, the last decoding fusion vector can be directly determined as the expression control optimization vector.

[0108] In combination Figure 6 The embodiments of the present application also provide a virtual image expression control device for fusing song emotion, which can be applied to the electronic device described above. The virtual image expression control device for fusing song emotion can include a data acquisition module, a control data determination module, a semantic mining module, a control data optimization module, and a virtual image control module.

[0109] The data acquisition module is configured to acquire a user face image of a target singing user and a singing song audio. The user face image is an image formed by collecting the face of the target singing user during singing of a song. The singing song audio is an audio formed by the target singing user singing a song. In the embodiments of the present application, the data acquisition module can be configured to perform the steps S110 shown in the figure. Figure 2 The related content of the data acquisition module can be referred to the description of step S110 above.

[0110] The control data determination module is configured to determine initial expression control data of the target virtual image based on the facial image of the user, wherein the initial expression control data at least includes coordinates of each facial feature point of the target virtual image. Figure 2 The step S120 is shown, and the related content of the control data determination module can be referred to the description of the step S120.

[0111] The semantic mining module is configured to perform semantic mining on the singing song audio to form a song audio semantic vector, wherein the song audio semantic vector is used to represent at least semantic information related to emotion of the target singing user in the singing process. Figure 2 The step S130 is shown, and the related content of the semantic mining module can be referred to the description of the step S130.

[0112] The control data optimization module is configured to optimize the initial expression control data based on the song audio semantic vector to form target expression control data, wherein the target expression control data at least includes coordinates of each facial feature point of the target virtual image. Figure 2 The step S140 is shown, and the related content of the control data optimization module can be referred to the description of the step S140.

[0113] The virtual image control module is configured to control the expression of the target virtual image based on the target expression control data. Figure 2 The step S150 is shown, and the related content of the virtual image control module can be referred to the description of the step S150.

[0114] In the embodiments of the present application, corresponding to the virtual image expression control method for fusing song voice emotion applied to the electronic device, a computer readable storage medium is also provided, and the computer readable storage medium stores a computer program.

[0115] The steps executed by the computer program when running are not described here again, and can be referred to the foregoing description of the virtual image expression control method for fusing song voice emotion.

[0116] In summary, the virtual image expression control method, device and equipment fusing song emotion provided by the application firstly acquire the user face image and the singing song audio of the target singing user; secondly, determine the initial expression control data of the target virtual image based on the user face image; then, perform semantic mining on the singing song audio to form a song audio semantic vector; further, optimize the initial expression control data based on the song audio semantic vector to form target expression control data; finally, control the expression of the target virtual image based on the target expression control data. Based on the above, on the one hand, since the initial expression control data of the target virtual image is directly determined based on the user face image, the global and overall expression control of the target virtual image can be performed, and on the other hand, since the initial expression control data is optimized based on the semantic information related to emotion in the singing song audio, the target expression control data formed can further increase the expression detail control on the basis of the global and overall expression control, thereby improving the control precision, and further improving the problem of relatively low precision of virtual image expression control in the song singing application scenario in the prior art.

[0117] In several embodiments provided by the embodiments of the present application, it should be understood that the disclosed apparatus and method can also be implemented by other manners. The apparatus and method embodiments described above are only illustrative, for example, the flowchart and block diagram in the drawings show the possible implementation architecture, function and operation of the apparatus, method and computer program product according to the embodiments of the present application. In this regard, each block in the flowchart or block diagram can represent a module, program segment or a part of code containing one or more executable instructions for implementing the specified logic function. It should also be noted that in some alternative implementation manners, the functions noted in the blocks can also occur in different order from that noted in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and sometimes they can be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for executing the specified function or action, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0118] In addition, the function modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0119] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product in essence or in the form of a part of the prior art or part of the technical solutions of the present application. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, an electronic device, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media. It should be noted that in this paper, the terms "include", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0120] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for controlling the expression of a virtual image integrating singing emotions, characterized in that: include: Obtaining a user facial image and singing song audio of a target singing user, wherein the user facial image is an image formed by capturing the face of the target singing user during the singing process, and the singing song audio is audio formed by the target singing user singing the song; Determining initial expression control data of a target virtual image based on the user facial image, wherein the initial expression control data at least includes coordinates of facial feature points of the target virtual image; Performing semantic mining on the singing song audio to form a song audio semantic vector, wherein the song audio semantic vector is at least used to represent semantic information related to emotions of the target singing user during the singing process; Based on the song audio semantic vector, the initial expression control data is optimized to form target expression control data, wherein the target expression control data at least includes the coordinates of each facial feature point of the target avatar; The expression of the target avatar is controlled based on the target expression control data.

2. The method for controlling the expression of a virtual image integrating singing emotions according to claim 1, wherein: The step of performing semantic mining on the sung song audio to form a song audio semantic vector includes: Extracting local features of the sung song audio to obtain first local feature information, and based on the first local feature information, enhancing the semantic information of the time domain of the sung song audio to form a song time domain vector; Performing frequency domain conversion on the audio of the sung song to obtain a sung song spectrum graph; Extracting local features from the sung song spectrogram to obtain second local feature information, and based on the second local feature information, enhancing the semantic information of the frequency domain of the sung song spectrogram to form a song frequency domain vector; The song time domain vector and the song frequency domain vector are fused across data domains to form a song audio semantic vector corresponding to the sung song audio.

3. The method for controlling the expression of a virtual image integrating singing emotions according to claim 2, wherein: The step of extracting local features from the sung song audio to obtain first local feature information, and enhancing the time domain semantic information of the sung song audio based on the first local feature information to form a song time domain vector includes: Performing a zero-crossing point analysis on the singing song audio to obtain first local feature information including a zero-crossing rate, wherein the zero-crossing point refers to a position in the singing song audio where a signal changes from a positive value to a negative value or from a negative value to a positive value; Mapping the singing song audio and the first local feature information into vector space respectively to form a song audio mapping vector and a zero-crossing mapping vector in the time domain; Based on the zero-crossing mapping vector, performing attention encoding on the song audio mapping vector to form a zero-crossing attention vector; Perform a residual connection on the zero-crossing attention vector and the song audio mapping vector to form a song time domain vector.

4. The method for controlling the expression of a virtual image integrating singing emotions according to claim 2, wherein: The step of extracting local features from the sung song spectrogram to obtain second local feature information, and based on the second local feature information, enhancing the semantic information of the frequency domain of the sung song spectrogram to form a song frequency domain vector includes: Calculating the mean, variance, peak value, skewness and kurtosis of the singing song spectrum graph to obtain second local feature information including the mean, variance, peak value, skewness and kurtosis; Deeply mining the sung song spectrum graph to form a sung song depth vector; Performing vector space mapping on the second local feature information to form a local feature mapping vector; Based on the local feature mapping vector, the singing song depth vector is subjected to attention encoding to form a local feature attention vector; Perform residual connection on the local feature attention vector and the singing song depth vector to form a song frequency domain vector.

5. The method for controlling the expression of a virtual image integrating singing emotions according to claim 4, characterized in that: The step of performing deep mining on the sung song spectrum graph to form a sung song depth vector comprises: Calculating the difference of adjacent data points on the singing song spectrum graph along the frequency axis to form a frequency axis change spectrum graph corresponding to the singing song spectrum graph; Performing deep convolution on the singing song spectrum graph and the frequency axis change spectrum graph respectively to form a song spectrum graph vector and a change spectrum graph vector; Adjusting the change spectrum graph vector based on a target weight parameter distribution to form a change adjustment vector, wherein the target weight parameter distribution is formed by learning user facial image samples, singing song audio samples, and expression control data labels; Performing nonlinear activation on the change adjustment vector to form a gating parameter distribution; Based on the gate parameter distribution, the song spectrum graph vector is adjusted to form a singing song depth vector.

6. The method for controlling the expression of a virtual image integrating singing emotions according to claim 2, characterized in that: The step of fusing the song time domain vector and the song frequency domain vector across data domains to form a song audio semantic vector corresponding to the sung song audio includes: Performing multiple convolution mappings or multiple pooling mappings on the song time domain vector to form multiple song mapping vectors corresponding to the song time domain vector; Calculating an attention score based on each of the song mapping vectors and the song frequency domain vector to obtain a plurality of attention scores; fusing the multiple attention scores to form a fused attention score; Based on the fused attention score, a weighted summation calculation is performed on the song frequency domain vector to form a song audio semantic vector corresponding to the sung song audio.

7. The method for controlling the expression of a virtual image integrating singing emotions according to any one of claims 1 to 6, characterized in that: The step of optimizing the initial expression control data based on the song audio semantic vector to form target expression control data includes: Mapping the initial expression control data into a vector space to form an expression control mapping vector; Based on the song audio semantic vector, semantically optimizing the expression control mapping vector to form an expression control optimization vector; Semantic restoration is performed based on the expression control optimization vector, and target expression control data is output.

8. The method for controlling the expression of a virtual image integrating singing emotions according to claim 7, characterized in that: The step of semantically optimizing the expression control mapping vector based on the song audio semantic vector to form an expression control optimization vector includes: In the encoding stage, the song audio semantic vector is subjected to attention encoding based on the expression control mapping vector at multiple semantic depths, so that the semantic information in the expression control mapping vector is fused into the song audio semantic vector to form multiple encoding fusion vectors, wherein, starting from the second semantic depth, the objects of attention encoding include the encoding fusion vector corresponding to the previous semantic depth and the semantic vector of the expression control mapping vector at the corresponding semantic depth; In the decoding stage, the expression control mapping vector is subjected to attention decoding based on the multiple encoding fusion vectors at multiple semantic depths, so that the semantic information in the encoding fusion vector is fused into the expression control mapping vector to form multiple decoding fusion vectors, wherein, starting from the second semantic depth, the objects of attention decoding include the decoding fusion vector corresponding to the previous semantic depth, the encoding fusion vector of the corresponding semantic depth, and the semantic vector of the expression control mapping vector at the corresponding semantic depth; Based on the last decoded fusion vector, an expression control optimization vector is determined.

9. A virtual image expression control device integrating singing emotions, characterized in that: include: A data acquisition module is used to acquire a user facial image and singing audio of a target singing user, wherein the user facial image is an image formed by capturing the face of the target singing user during the singing process, and the singing audio is audio formed by the target singing user singing the song; a control data determination module, configured to determine initial expression control data of a target virtual image based on the user's facial image, wherein the initial expression control data at least includes coordinates of facial feature points of the target virtual image; A semantic mining module is used to perform semantic mining on the singing song audio to form a song audio semantic vector, wherein the song audio semantic vector is used to at least characterize semantic information related to emotions of the target singing user during the singing process; a control data optimization module, configured to optimize the initial expression control data based on the song audio semantic vector to form target expression control data, wherein the target expression control data at least includes the coordinates of each facial feature point of the target avatar; The virtual image control module is used to control the expression of the target virtual image based on the target expression control data.

10. An electronic device, characterized in that: include: Memory for storing computer programs; A processor connected to the memory is used to execute the computer program stored in the memory to implement the virtual image expression control method integrating singing emotions as described in any one of claims 1-8.

Images