Video animation generation method, device, equipment and storage medium

Abstract

The present invention discloses a method, apparatus, device, and storage medium for generating video animation. The method comprises: converting a target user's voice after denoising into pure voice; determining a specific original voice corresponding to the target user's voice based on the voice features of the target user's voice; and fusing the voice features of the pure voice, the voice features of the specific original voice, and the image features of a target image to obtain a video animation. Because the present invention generates the video animation by converting the target user's voice after denoising into pure voice, determining the corresponding specific original voice based on the voice features of the target user's voice, and then fusing the voice features of the pure voice, the voice features of the specific original voice, and the image features of the target image, it is possible to remove noise from the input voice and generate a video animation based on the pure voice, the specific original voice, and the target image, thereby ensuring smooth and fluent video animation and improving the quality of the video animation.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a video animation generation method, device, equipment and storage medium. Background Art

[0002] At present, when generating video animation based on speech and images, facial feature parameters are generally extracted from the image, and then the video animation is generated by inputting the speech and facial feature parameters. Since there is a lot of noise in the input speech, and the video animation is generated directly based on the input speech and the extracted facial feature parameters, the quality of the generated video animation is low.

[0003] The above content is only used to assist in understanding the technical solution of the present invention and does not constitute an admission that the above content is related technology. Summary of the Invention

[0004] The main purpose of the present invention is to provide a video animation generation method, device, equipment and storage medium, aiming to solve the technical problem of low quality of video animation generated directly based on input voice and facial feature parameters in the prior art.

[0005] To achieve the above object, the present invention provides a method for generating a video animation, the method comprising the following steps:

[0006] Convert the denoised target user's voice into pure voice;

[0007] Determining a specific original voice corresponding to the target user's voice based on the voice features of the target user's voice;

[0008] The speech features of the pure speech, the speech features of the specific original speech and the image features of the target image are fused to obtain a video animation.

[0009] Optionally, determining the specific original voice corresponding to the target user voice according to the voice feature of the target user voice includes:

[0010] Determining whether the target user's voice has fluctuations based on the voice characteristics of the target user's voice;

[0011] When the target user's voice fluctuates, determining factors influencing the fluctuation of the target user's voice based on ambient sound;

[0012] When the influencing factor is an environmental factor, the specific original voice corresponding to the target user voice is determined according to the voice feature difference between the environmental sound and the target user voice.

[0013] Optionally, when the target user's voice fluctuates, after determining the factors influencing the fluctuation of the target user's voice according to the ambient sound, the method further includes:

[0014] When the influencing factor is a non-environmental factor, determining the duration and position of the fluctuating speech segment in the target user's speech according to the speech features of the target user's speech;

[0015] When the duration is less than a preset duration, determining a neighboring speech segment of the fluctuating speech segment according to the position of the speech segment;

[0016] The fluctuating voice segment is processed according to the neighborhood voice segment to obtain a specific original voice corresponding to the target user voice.

[0017] Optionally, fusing the voice features of the clean voice, the voice features of the specific original voice, and the image features of the target image to obtain a video animation includes:

[0018] Performing speech feature extraction on the clean speech and the specific original speech to obtain corresponding clean speech feature coding vectors and specific speech feature coding vectors;

[0019] Perform image feature extraction on the target image to obtain lip feature coding vectors and facial feature coding vectors;

[0020] Fusing the lip feature coding vector with the clean speech feature coding vector to obtain a lip fusion feature vector;

[0021] Fusing the facial feature coding vector with the specific speech feature coding vector to obtain a facial fusion feature vector;

[0022] The lip fusion feature vector and the facial fusion feature vector are spliced together, and the spliced vector is decoded to obtain a video animation.

[0023] Optionally, extracting image features from the target image to obtain a lip feature coding vector and a facial feature coding vector includes:

[0024] Performing image feature extraction on the lip region of the target image through a first image feature extraction network to obtain a lip feature encoding vector;

[0025] Image features are extracted from the facial area of the target image through a second image feature extraction network to obtain a facial feature encoding vector, wherein the number of convolutional layers of the second image feature extraction network is greater than the number of convolutional layers of the first image feature extraction network.

[0026] Optionally, fusing the facial feature coding vector with the specific speech feature coding vector to obtain a facial fusion feature vector includes:

[0027] Dividing the clean speech feature coding vector and the specific speech feature coding vector into blocks to obtain a plurality of feature blocks;

[0028] Comparing the similarity between the feature block of the clean speech feature coding vector and the feature block corresponding to the specific speech feature coding vector to obtain a similarity value set;

[0029] Performing block feature fusion on the clean speech feature coding vector and the specific speech feature coding vector according to the similarity set to obtain a specific fused coding vector;

[0030] The facial feature coding vector is fused with the specific fusion coding vector to obtain a facial fusion feature vector.

[0031] Optionally, converting the denoised target user voice into pure voice includes:

[0032] Performing noise reduction processing on the input speech to obtain noise-reduced speech;

[0033] Performing voice separation on the noise-reduced voice, and determining the target user voice based on voice information of each voice after voice separation;

[0034] Converting the target user's speech into text to obtain speech-to-text information;

[0035] The speech and text information is converted into pure speech through a speech synthesis system.

[0036] In addition, to achieve the above-mentioned purpose, the present invention further provides a video animation generating device, the device comprising:

[0037] A conversion module, used to convert the denoised target user voice into pure voice;

[0038] A determination module, configured to determine a specific original voice corresponding to the target user's voice based on the voice features of the target user's voice;

[0039] The fusion module is used to fuse the voice features of the pure voice, the voice features of the specific original voice and the image features of the target image to obtain a video animation.

[0040] In addition, to achieve the above-mentioned purpose, the present invention also proposes a video animation generation device, which includes: a memory, a processor, and a video animation generation program stored on the memory and runnable on the processor, and the video animation generation program is configured to implement the steps of the video animation generation method described above.

[0041] In addition, to achieve the above-mentioned purpose, the present invention also proposes a storage medium, on which a video animation generation program is stored. When the video animation generation program is executed by a processor, the steps of the video animation generation method described above are implemented.

[0042] The present invention converts the denoised target user voice into pure voice; determines the specific original voice corresponding to the target user voice based on the voice features of the target user voice; and fuses the voice features of the pure voice, the voice features of the specific original voice, and the image features of the target image to obtain a video animation. Because the present invention generates a video animation by converting the denoised target user voice into pure voice, determining the corresponding specific original voice based on the voice features of the target user voice, and then fusing the voice features of the pure voice, the voice features of the specific original voice, and the image features of the target image, it is able to remove noise from the input voice and generate a video animation based on the pure voice, the specific original voice, and the target image, ensuring smooth and fluent video animation and improving the quality of the video animation. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a structural diagram of a video animation generating device in a hardware operating environment involved in an embodiment of the present invention;

[0044] Figure 2 This is a flow chart of a first embodiment of a method for generating video animation according to the present invention;

[0045] Figure 3 This is a flow chart of a second embodiment of the method for generating video animation according to the present invention;

[0046] Figure 4 This is a flow chart of a third embodiment of a method for generating video animation according to the present invention;

[0047] Figure 5 A schematic diagram of determining a specific fusion coding vector in the third embodiment of the video animation generation method of the present invention;

[0048] Figure 6 This is a structural block diagram of the first embodiment of the video animation generation device of the present invention.

[0049] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0050] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0051] Reference Figure 1 , Figure 1This is a schematic diagram of the structure of a video animation generation device in the hardware operating environment involved in an embodiment of the present invention.

[0052] like Figure 1 As shown, the video animation generation device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the user interface 1003 may optionally include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (Wireless-Fidelity, WI-FI) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM) or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk storage. The memory 1005 may optionally be a storage device independent of the aforementioned processor 1001.

[0053] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the video animation generating device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0054] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a video animation generation program.

[0055] exist Figure 1 In the video animation generating device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the video animation generating device of the present invention can be set in the video animation generating device, and the video animation generating device calls the video animation generation program stored in the memory 1005 through the processor 1001, and executes the video animation generation method provided by the embodiment of the present invention.

[0056] The embodiment of the present invention provides a method for generating video animation, referring to Figure 2 , Figure 2 2 is a flow chart of the first embodiment of the video animation generation method of the present invention.

[0057] In this embodiment, the video animation generation method includes the following steps:

[0058] Step S10: Convert the denoised target user voice into pure voice.

[0059] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, mobile phone, etc., or an electronic device capable of implementing the above functions, a video animation generation device, etc. The following uses a video animation generation device as an example to illustrate this embodiment and the following embodiments.

[0060] It can be understood that the target user voice can be the voice of the target user used to generate the video animation; the pure voice can be the voice with a specific timbre from which noise has been removed; converting the denoised target user voice into the pure voice can be performing noise reduction processing on the input voice to obtain the noise-reduced voice, selecting the target user voice from the noise-reduced voice, and converting the target user voice into the pure voice.

[0061] Step S20: determining the specific original speech corresponding to the target user's speech according to the speech features of the target user's speech.

[0062] It can be understood that the specific original voice can be obtained by processing the fluctuating voice segments in the target user's voice to obtain the voice corresponding to the target user's voice; determining the specific original voice corresponding to the target user according to the voice features of the target user's voice can be determining the fluctuating voice segments in the target user's voice according to the voice features of the target user's voice, processing each fluctuating voice segment, and obtaining the specific original voice corresponding to the target user's voice.

[0063] Step S30: fusing the speech features of the clean speech, the speech features of the specific original speech, and the image features of the target image to obtain a video animation.

[0064] It can be understood that the target image can be an image used to generate a video animation; fusing the voice features of the pure speech, the voice features of the specific original speech and the image features of the target image to obtain the video animation can be fusing the voice features of the pure speech with the lip area features in the image features, fusing the voice features of the specific original speech with the facial area features in the image features, and splicing the above-mentioned fused features to obtain splicing features, and decoding the splicing features to obtain the video animation.

[0065] In a specific implementation, the video animation generation device performs noise reduction processing on the input voice to obtain noise-reduced voice, separates the target user voice from the noise-reduced voice, and converts the target user voice into pure voice, determines the fluctuating voice segments existing in the target user voice based on the voice features of the target user voice, processes each fluctuating voice segment to obtain the specific original voice corresponding to the target user voice, fuses the voice features of the pure voice with the lip area features in the image features, fuses the voice features of the specific original voice with the facial area features in the image features, splices the above-mentioned fused features to obtain splicing features, decodes the splicing features, and obtains video animation.

[0066] Furthermore, since there may be multiple people talking or multiple people making noise in real scenes, and there may also be noise in the environment, in order to obtain more accurate pure speech, thereby improving the accuracy of lip area movement and improving the quality of video animation, step S10 includes: performing noise reduction processing on the input speech to obtain noise-reduced speech; performing speech separation on the noise-reduced speech, and determining the target user speech based on the speech information of each speech after speech separation; performing text conversion on the target user speech to obtain speech-to-text information; and converting the speech-to-text information into pure speech through a speech synthesis system.

[0067] It is understandable that the input speech may be composed of the target user's speech and other speech, which can be expressed by the following formula: S sound =S speech +S env , where S sound Indicates the input speech, S speech represents the target user’s voice, S env represents other voices, including environmental sounds and voices of non-target users; performing noise reduction on the input voice to obtain the noise-reduced voice can be performing front-end noise reduction on the input voice to remove noise in the input voice to obtain the noise-reduced voice, in which only the human voice is retained.

[0068] It should be understood that voice information includes information such as voice volume and voice duration; performing voice separation on the noise reduction voice and determining the target user voice based on the voice information of each voice after voice separation can be performing voice separation on the noise reduction voice to obtain the voice corresponding to each sound source, sorting the voice corresponding to each sound source according to the voice duration of each voice, obtaining a voice duration sequence, selecting the voice with the longest voice duration from the voice sequence as the target user voice, if the difference in voice duration between the first several voices in the voice sequence is less than a preset threshold, re-sorting the several voices according to the voice volume of the several voices to obtain a voice volume sequence, and selecting the voice with the largest volume from the voice volume sequence as the target user voice.

[0069] In a specific implementation, the video animation generation device obtains input voice, performs front-end noise reduction on the input voice to eliminate the noise in the input voice, obtains noise-reduced voice, performs voice separation on the noise-reduced voice, obtains the voices corresponding to each sound source, sorts them according to voice duration, selects the voice with the longest voice duration as the target user voice, and discards the remaining voices. If the difference in voice duration between several voices is less than a preset threshold, the several voices are sorted according to voice volume, and the voice with the largest voice volume is selected as the target voice.

[0070] It is understandable that converting the target user's voice into text to obtain the voice-to-text information may be performed by converting the target user's voice into voice-to-text information through Automatic Speech Recognition (ASR) technology.

[0071] It should be understood that the speech synthesis system can convert text information into pure speech output with a specific timbre; pure speech is obtained through speech separation, speech-to-text and text-to-speech. Speech-to-text only extracts speech and text information from the target user's speech, thereby discarding and eliminating the voice differences between different users, and converting speech and text information into pure speech, which can effectively remove noise in the input speech, thereby improving the quality of video animation.

[0072] This embodiment converts the denoised target user voice into pure voice; determines the specific original voice corresponding to the target user voice based on the voice features of the target user voice; and fuses the voice features of the pure voice, the voice features of the specific original voice, and the image features of the target image to obtain a video animation. Because this embodiment generates a video animation by converting the denoised target user voice into pure voice, determining the corresponding specific original voice based on the voice features of the target user voice, and then fusing the voice features of the pure voice, the voice features of the specific original voice, and the image features of the target image, it can remove noise from the input voice and generate a video animation based on the pure voice, the specific original voice, and the target image. This ensures that the video animation is smooth and fluent, thereby improving the quality of the video animation.

[0073] refer to Figure 3 , Figure 3 2 is a flow chart of a second embodiment of a method for generating video animation according to the present invention.

[0074] Based on the above first embodiment, in this embodiment, step S20 includes:

[0075] Step S201: determining whether the target user's voice has fluctuations according to the voice features of the target user's voice.

[0076] It can be understood that the target user's voice can be feature extracted by a voice extractor to obtain a mel-spectrogram corresponding to the target user's voice, and the voice feature value at each moment can be determined based on the mel-spectrogram; judging whether the target user's voice fluctuates based on the voice features of the target user's voice can be by performing feature extraction on the target user's voice through a voice feature extractor to obtain a corresponding mel-spectrogram, determining the voice feature value at each moment based on the mel-spectrogram, calculating the feature difference between the voice feature values at adjacent moments, and judging whether the target user's voice fluctuates based on the feature difference.

[0077] It should be understood that when the feature difference is greater than the preset threshold, it is determined that there is no fluctuation in the target user's voice, and the target user's voice is regarded as the specific original voice; when the feature difference is greater than or equal to the preset threshold, it is determined that there is fluctuation in the target user's voice; the preset threshold can be set according to the specific scenario, and this embodiment does not limit this.

[0078] Step S202: When the target user's voice fluctuates, factors influencing the fluctuation of the target user's voice are determined according to ambient sound.

[0079] It can be understood that the environmental sound can be other voices in the input voice; when the target user voice fluctuates, determining the influencing factors of the target user voice fluctuation based on the environmental sound can be that when the target user voice fluctuates, the fluctuation moment of the target user voice is obtained, the moment corresponding to the fluctuation moment in the environmental sound is determined, and whether the environmental sound fluctuates within a preset time period before and after the moment. If so, it is determined that the influencing factors of the target user voice fluctuation are environmental factors; if not, it is determined that the influencing factors of the target user voice fluctuation are non-environmental factors.

[0080] Step S203: When the influencing factor is an environmental factor, the specific original voice corresponding to the target user's voice is determined according to the voice feature difference between the environmental sound and the target user's voice.

[0081] It can be understood that when the influencing factor is an environmental factor, the voice feature difference between the voice features of the target user's voice and the voice features of the corresponding environmental sound is obtained, a new voice is generated based on the voice feature difference, and the new voice is used as the specific original voice corresponding to the target user's voice.

[0082] In a specific implementation, the video animation generation device extracts features of the target user's voice through a voice extractor, obtains the Mel-atlas corresponding to the target user's voice, determines the voice feature value at each moment based on the Mel-atlas, and calculates the feature difference between the voice feature values at adjacent moments. When the feature difference is less than a preset threshold, it is determined that there is no fluctuation in the target user's voice, and the target user's voice is used as the specific original voice; when the feature difference is greater than or equal to the preset threshold, it is determined that there is fluctuation in the target user's voice, obtains the fluctuation moment when the fluctuation occurs in the target user's voice, and determines the moment in the ambient sound corresponding to the fluctuation moment, and judges whether the ambient sound fluctuates within a preset time period before and after the moment. If so, it is determined that the influencing factor causing the fluctuation of the target user's voice is an environmental factor; if not, it is a non-environmental factor; when the influencing factor is an environmental factor, the feature difference between the voice feature of the target user's voice and the voice feature of the ambient sound is obtained, and a new voice is generated based on the feature difference, and the new voice is used as the specific original voice.

[0083] Furthermore, when the user inputs voice, he or she may actively increase the pitch or volume to express emotions within a certain period of time, thereby causing the target user's voice to fluctuate. It is also possible that the user encounters an emergency at a certain moment and passively increases the pitch or volume, thereby causing the target user's voice to fluctuate. It is necessary to distinguish different situations to remove abnormal fluctuations in the target user's voice, thereby improving the quality of the video animation. After step S202, it also includes: when the influencing factor is a non-environmental factor, determining the duration and voice segment position of the fluctuating voice segment in the target user's voice according to the voice features of the target user's voice; when the duration is less than a preset duration, determining the neighboring voice segment of the fluctuating voice segment according to the voice segment position; processing the fluctuating voice segment according to the neighboring voice segments to obtain the specific original voice corresponding to the target user's voice.

[0084] It can be understood that the fluctuating voice segment can be the voice segment where fluctuation occurs in the target user's voice; the voice segment position can be the moment when the fluctuating voice segment is in the target user's voice; and the neighborhood voice segment can be the voice segment adjacent to the fluctuating voice segment.

[0085] It should be understood that when the duration is greater than or equal to the preset duration, it can be determined that the fluctuations in the target user's voice are actively caused by the user, and the target user's voice is determined to be a specific original voice; when the duration is less than the preset duration, it can be determined that the fluctuations in the target user's voice are passively caused, and the neighboring voice segments adjacent to the fluctuating voice segment are determined according to the moment when the fluctuating voice segment is in the target user's voice.

[0086] It can be understood that processing the fluctuating voice segment according to the neighborhood voice segment to obtain the specific original voice corresponding to the target user's voice can be done by deleting the fluctuating voice segment and filling the voice features at the fluctuating voice segment with the average of the voice feature values of the neighborhood voice segments to obtain the specific original voice.

[0087] This embodiment determines whether the target user's voice fluctuates based on the voice features of the target user's voice; when the target user's voice fluctuates, the influencing factor of the target user's voice fluctuation is determined based on the ambient sound; when the influencing factor is an environmental factor, the specific original voice corresponding to the target user's voice is determined based on the difference in voice features between the ambient sound and the target user's voice. This embodiment determines whether the target user's voice fluctuates based on the voice features of the target user. When there is a fluctuation, since the voice features of the target user's voice and the voice features of the ambient sound are consistent with the influence of the external environment, it is possible to determine whether the influencing factor of the target user's voice fluctuation is an environmental factor based on the voice features of the ambient sound at the corresponding moment. If so, the specific original voice is determined based on the difference in features between the ambient sound and the target user's voice. This can remove the influence of the external environment, improve the accuracy of the specific original voice, and improve the quality of the video animation.

[0088] refer to Figure 4 , Figure 4 2 is a flow chart of a third embodiment of the video animation generation method of the present invention.

[0089] Based on the above embodiments, in this embodiment, step S30 includes:

[0090] Step S301: performing speech feature extraction on the clean speech and the specific original speech to obtain corresponding clean speech feature coding vectors and specific speech feature coding vectors.

[0091] It is understandable that the clean speech and the specific original speech can be extracted by the speech feature extraction network to obtain the clean speech feature coding vector F_S trans and specific speech feature encoding vector F_S ori ; The speech feature extraction network can be a residual network, MFCC or other networks with the same or similar functions, and this embodiment does not limit this.

[0092] Step S302: extract image features from the target image to obtain lip feature coding vectors and facial feature coding vectors.

[0093] It is understandable that the image feature extraction network can be used to extract the image features of the target image and obtain the lip encoding feature encoding vector F_img mouthAnd facial feature encoding vector F_img face2 ; The image feature extraction network can be a VGG network or other networks with the same or similar functions, and this embodiment does not limit this.

[0094] In a specific implementation, the image feature extraction network can be used to extract features from the lip area matrix corresponding to the target image to obtain a lip feature coding vector, and the image feature extraction network can be used to extract features from the facial area matrix corresponding to the target image to obtain a facial feature coding vector.

[0095] Step S303: Fusing the lip feature coding vector with the clean speech feature coding vector to obtain a lip fusion feature vector.

[0096] It is understandable that the lip feature coding vector and the pure speech feature coding vector can be fused through concat or add.

[0097] In the specific implementation, the lip feature encoding vector F_img is concat mouth With pure speech feature coding to F_S trans The lip fusion feature vector F is obtained by fusion merge1 , that is, F merge1 =F_img mouth concatF_S trans .

[0098] Step S304: Fusing the facial feature coding vector with the specific speech feature coding vector to obtain a facial fusion feature vector.

[0099] In the specific implementation, the facial feature encoding vector F_img is concat face2 and F_S ori Fusion, obtains the facial fusion feature vector.

[0100] Step S305: splicing the lip fusion feature vector and the facial fusion feature vector, and decoding the spliced vector to obtain a video animation.

[0101] In the specific implementation, the lip fusion feature vector and the facial fusion feature vector are spliced together, and the spliced vector is input into the multimodal information fusion decoder to obtain the facial video animation in motion state.

[0102] Furthermore, since the feature dimensions corresponding to the lip area and the facial area are different, the amount of information contained is quite different. In order to make the feature size of the lip feature vector consistent with that of the facial feature vector, step S302 includes: performing image feature extraction on the lip area of the target image through a first image feature extraction network to obtain a lip feature coding vector; performing image feature extraction on the facial area of the target image through a second image feature extraction network to obtain a facial feature coding vector, and the number of convolutional layers of the second image feature extraction network is greater than the number of convolutional layers of the first image feature extraction network.

[0103] In a specific implementation, image features of the lip area are extracted through a first image feature extraction network to obtain a lip feature coding vector, and image features of the facial area are extracted through a second image feature extraction network to obtain a facial feature coding vector. The number of convolutional layers of the second image feature network is greater than that of the first image feature extraction network, so that the feature size of the lip feature coding vector is consistent with that of the facial feature coding vector.

[0104] Furthermore, in order to obtain smooth and fluent facial changes to improve the quality of video animation, the step S304 includes: dividing the clean speech feature coding vector and the specific speech feature coding vector into blocks to obtain a number of feature blocks; performing similarity comparison between the feature blocks of the clean speech feature coding vector and the feature blocks corresponding to the specific speech feature coding vector to obtain a similarity set; performing block feature fusion on the clean speech feature coding vector and the specific speech feature coding vector according to the similarity set to obtain a specific fusion coding vector; and fusing the facial feature coding vector with the specific fusion coding vector to obtain a facial fusion feature vector.

[0105] In a specific implementation, the pure speech feature coding vector and the specific speech feature coding vector can be equally divided into blocks to obtain several feature blocks, and the corresponding feature blocks can be compared by cosine similarity to obtain several similarity values corresponding to each feature block. The several similarity values constitute a similarity set, and the similarity data is masked to obtain a mask set. According to the similarity set and the mask set, the pure speech feature coding vector and the specific speech feature coding vector are block-feature fused to obtain a specific fused coding vector, and the facial feature coding vector is fused with the specific fused coding vector to obtain a facial fused feature vector.

[0106] For example, refer to Figure 5 , Figure 5To determine the schematic diagram of a specific fusion coding vector, the specific original speech is input into a shallow feature extraction network with an attention mechanism for speech feature extraction to obtain a specific speech feature coding vector; the clean speech is input into a shallow feature extraction network for speech feature extraction to obtain a clean speech feature coding vector; the specific speech feature coding vector and the clean speech feature coding vector are equally divided into blocks, and the corresponding blocks are compared by cosine similarity to obtain a similarity set List = [0.1, 0.8, 0.9, ...], then its mask is List2 = [0.9, 0.2, 0.1, ...], and the block feature fusion is performed on the specific speech feature coding vector and the clean speech feature coding vector: F_S trans3 =[0.9*F+0.1*F2,0.2*F+0.8*F2,0.2*F+0.9*F2,...], where F_S trans3 Represents a specific fusion coding vector, F represents the feature block corresponding to the specific speech feature coding vector, and F2 represents the feature block corresponding to the pure speech feature coding vector; the specific fusion coding vector is fused with the facial feature coding vector to obtain a facial fusion feature vector.

[0107] This embodiment performs speech feature extraction on the clean speech and the specific original speech to obtain corresponding clean speech feature coding vectors and specific speech feature coding vectors; performs image feature extraction on the target image to obtain a lip feature coding vector and a facial feature coding vector; fuses the lip feature coding vector with the clean speech feature coding vector to obtain a lip fusion feature vector; fuses the facial feature coding vector with the specific speech feature coding vector to obtain a facial fusion feature vector; concatenates the lip fusion feature vector and the facial fusion feature vector, and decodes the concatenated vector to obtain a video animation. Because this embodiment fuses the lip feature coding vector with the clean speech feature coding vector, the facial feature coding vector with the specific speech feature coding vector, and the lip fusion feature vector with the facial fusion feature vector, and decodes the concatenated vector to obtain a video animation, it is possible to deeply fuse the clean speech with the lip region and the specific original speech with the facial region, achieving refined processing of lip changes based on the clean speech while obtaining smooth and fluent overall facial changes, thereby improving the smoothness of the video animation and thereby enhancing the quality of the video animation.

[0108] In addition, an embodiment of the present invention further provides a storage medium on which a video animation generation program is stored. When the video animation generation program is executed by a processor, the steps of the video animation generation method described above are implemented.

[0109] Reference Figure 6 , Figure 6 This is a structural block diagram of the first embodiment of the video animation generation device of the present invention.

[0110] like Figure 6 As shown, the video animation generating device proposed in the embodiment of the present invention includes: a conversion module 10, a determination module 20 and a fusion module 30.

[0111] The conversion module 10 is used to convert the denoised target user voice into pure voice;

[0112] The determining module 20 is configured to determine the specific original speech corresponding to the target user's speech according to the speech features of the target user's speech;

[0113] The fusion module 30 is used to fuse the voice features of the pure voice, the voice features of the specific original voice, and the image features of the target image to obtain a video animation.

[0114] This embodiment converts the denoised target user voice into pure voice; determines the specific original voice corresponding to the target user voice based on the voice features of the target user voice; and fuses the voice features of the pure voice, the voice features of the specific original voice, and the image features of the target image to obtain a video animation. Because this embodiment generates a video animation by converting the denoised target user voice into pure voice, determining the corresponding specific original voice based on the voice features of the target user voice, and then fusing the voice features of the pure voice, the voice features of the specific original voice, and the image features of the target image, it can remove noise from the input voice and generate a video animation based on the pure voice, the specific original voice, and the target image. This ensures that the video animation is smooth and fluent, thereby improving the quality of the video animation.

[0115] Based on the above-mentioned first embodiment of the video animation generating device of the present invention, a second embodiment of the video animation generating device of the present invention is proposed.

[0116] In this embodiment, the determination module 20 is also used to determine whether the target user voice fluctuates based on the voice features of the target user voice; when the target user voice fluctuates, determine the influencing factors of the fluctuation of the target user voice based on the ambient sound; when the influencing factors are environmental factors, determine the specific original voice corresponding to the target user voice based on the difference in voice features between the ambient sound and the target user voice.

[0117] The determination module 20 is further configured to determine, when the influencing factor is a non-environmental factor, the duration and position of the fluctuating voice segment in the target user's voice according to the voice features of the target user's voice; determine, when the duration is less than a preset duration, a neighboring voice segment of the fluctuating voice segment according to the voice segment position; and process the fluctuating voice segment according to the neighboring voice segments to obtain a specific original voice corresponding to the target user's voice.

[0118] The fusion module 30 is further used to perform speech feature extraction on the clean speech and the specific original speech to obtain corresponding clean speech feature coding vectors and specific speech feature coding vectors; perform image feature extraction on the target image to obtain lip feature coding vectors and facial feature coding vectors; fuse the lip feature coding vector with the clean speech feature coding vector to obtain a lip fusion feature vector; fuse the facial feature coding vector with the specific speech feature coding vector to obtain a facial fusion feature vector; splice the lip fusion feature vector and the facial fusion feature vector, and decode the spliced vectors to obtain a video animation.

[0119] The fusion module 30 is further used to perform image feature extraction on the lip area of the target image through a first image feature extraction network to obtain a lip feature coding vector; and perform image feature extraction on the facial area of the target image through a second image feature extraction network to obtain a facial feature coding vector, wherein the number of convolutional layers of the second image feature extraction network is greater than the number of convolutional layers of the first image feature extraction network.

[0120] The fusion module 30 is further configured to divide the clean speech feature coding vector and the specific speech feature coding vector into blocks to obtain a plurality of feature blocks; perform similarity comparison between the feature blocks of the clean speech feature coding vector and the feature blocks corresponding to the specific speech feature coding vector to obtain a similarity set; perform block feature fusion on the clean speech feature coding vector and the specific speech feature coding vector according to the similarity set to obtain a specific fused coding vector; and fuse the facial feature coding vector with the specific fused coding vector to obtain a facial fused feature vector.

[0121] The conversion module 10 is further configured to perform noise reduction processing on the input speech to obtain noise-reduced speech; perform speech separation on the noise-reduced speech and determine the target user speech based on the speech information of each speech after speech separation; perform text conversion on the target user speech to obtain speech-to-text information; and convert the speech-to-text information into pure speech through a speech synthesis system.

[0122] Other embodiments or specific implementations of the video animation generating device of the present invention can refer to the above-mentioned method embodiments and will not be repeated here.

[0123] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0124] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0125] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as read-only memory / random access memory, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0126] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A video animation generation method, characterized in that: The method comprises: Converting the denoised target user voice into a clean voice, wherein the clean voice is a voice with a specific timbre from which noise has been removed; Determining a specific original voice corresponding to the target user's voice based on the voice features of the target user's voice; fusing the voice features of the clean voice, the voice features of the specific original voice, and the image features of the target image to obtain a video animation; The determining of the specific original speech corresponding to the target user's speech according to the speech feature of the target user's speech includes: Determining whether the target user's voice has fluctuations based on the voice characteristics of the target user's voice; When the target user's voice fluctuates, determining factors influencing the fluctuation of the target user's voice based on ambient sound; When the influencing factor is an environmental factor, the specific original voice corresponding to the target user voice is determined according to the voice feature difference between the environmental sound and the target user voice, wherein the specific original voice is a new voice generated according to the voice feature difference.

2. The method according to claim 1, wherein When the target user's voice fluctuates, after determining the factors affecting the fluctuation of the target user's voice according to the ambient sound, the method further includes: When the influencing factor is a non-environmental factor, determining the duration and position of the fluctuating speech segment in the target user's speech according to the speech features of the target user's speech; When the duration is less than a preset duration, determining a neighboring speech segment of the fluctuating speech segment according to the position of the speech segment; The fluctuating voice segment is processed according to the neighborhood voice segment to obtain a specific original voice corresponding to the target user voice.

3. The method according to claim 1 or 2, wherein: The step of fusing the voice features of the pure voice, the voice features of the specific original voice, and the image features of the target image to obtain a video animation includes: Performing speech feature extraction on the clean speech and the specific original speech to obtain corresponding clean speech feature coding vectors and specific speech feature coding vectors; Perform image feature extraction on the target image to obtain lip feature coding vectors and facial feature coding vectors; Fusing the lip feature coding vector with the clean speech feature coding vector to obtain a lip fusion feature vector; Fusing the facial feature coding vector with the specific speech feature coding vector to obtain a facial fusion feature vector; The lip fusion feature vector and the facial fusion feature vector are spliced together, and the spliced vector is decoded to obtain a video animation.

4. The method according to claim 3, wherein The step of extracting image features from the target image to obtain a lip feature coding vector and a facial feature coding vector includes: Performing image feature extraction on the lip region of the target image through a first image feature extraction network to obtain a lip feature encoding vector; Image features are extracted from the facial area of the target image through a second image feature extraction network to obtain a facial feature encoding vector, wherein the number of convolutional layers of the second image feature extraction network is greater than the number of convolutional layers of the first image feature extraction network.

5. The method according to claim 3, wherein The step of fusing the facial feature coding vector with the specific speech feature coding vector to obtain a facial fusion feature vector includes: Dividing the clean speech feature coding vector and the specific speech feature coding vector into blocks to obtain a plurality of feature blocks; Comparing the similarity between the feature block of the clean speech feature coding vector and the feature block corresponding to the specific speech feature coding vector to obtain a similarity value set; Performing block feature fusion on the clean speech feature coding vector and the specific speech feature coding vector according to the similarity set to obtain a specific fused coding vector; The facial feature coding vector is fused with the specific fusion coding vector to obtain a facial fusion feature vector.

6. The method according to claim 1 or 2, wherein: Converting the denoised target user voice into pure voice includes: Performing noise reduction processing on the input speech to obtain noise-reduced speech; Performing voice separation on the noise-reduced voice, and determining the target user voice based on voice information of each voice after voice separation; Converting the target user's speech into text to obtain speech-to-text information; The speech and text information is converted into pure speech through a speech synthesis system.

7. A video animation generating device, characterized in that: The device comprises: a conversion module, configured to convert the denoised target user speech into a clean speech, wherein the clean speech is a speech with a specific timbre from which noise has been removed; A determination module, configured to determine a specific original voice corresponding to the target user's voice based on the voice features of the target user's voice; A fusion module, configured to fuse the speech features of the clean speech, the speech features of the specific original speech, and the image features of the target image to obtain a video animation; Among them, the determination module is also used to determine whether the target user voice fluctuates based on the voice features of the target user voice; when the target user voice fluctuates, determine the influencing factors of the fluctuation of the target user voice based on the ambient sound; when the influencing factors are environmental factors, determine the specific original voice corresponding to the target user voice based on the voice feature difference between the ambient sound and the target user voice, wherein the specific original voice is a new voice generated based on the voice feature difference.

8. A video animation generating device, characterized in that: The device includes: a memory, a processor, and a video animation generation program stored in the memory and executable on the processor, wherein the video animation generation program is configured to implement the steps of the video animation generation method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium stores a video animation generation program, which, when executed by a processor, implements the steps of the video animation generation method according to any one of claims 1 to 6.

Images