Virtual object driving method and device, electronic equipment and storage medium

Abstract

The application provides a virtual object driving method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence. The method comprises the following steps: obtaining a song dry audio of a target object and a first dry audio segment, the first dry audio segment comprising at least two first audio sub-segments, and the song dry audio comprising at least two second dry audio segments; determining whether to amplify the volume of each second dry audio segment based on the volume of each first audio sub-segment; and driving a virtual object based on the volume of the target second dry audio segment after amplification and the volume of other second dry audio segments except the target second dry audio segment when it is determined to amplify the volume of the target second dry audio segment. The application drives a virtual object based on the volume of the song dry audio of a target object, thereby improving the synchronization of the driving of the virtual object and the song dry audio played.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a virtual object driving method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the continuous development of social technology and the virtual avatar industry, more and more people are entering the virtual live streaming industry. Companies that incubate virtual avatars do not want to bind the person driving the avatar too closely to the avatar, to prevent the person's exposure or misconduct from affecting the avatar's value. Therefore, the demand for artificial intelligence (AI) virtual avatars independent of the person driving the avatar is extremely strong.

[0003] In related technologies, the system typically plays songs in the live streaming room, and during the song playback, it drives the lips and other parts of the virtual character in a preset manner.

[0004] However, in the aforementioned technologies, the virtual avatar's driver is not linked to the song being played, resulting in the virtual avatar's driver failing to synchronize with the song being played. Summary of the Invention

[0005] This invention provides a virtual object driving method, apparatus, electronic device, and storage medium to solve the defect in the prior art that the driving of virtual images cannot be synchronized with the played song, so as to realize that as long as the dry audio of the song has volume, the virtual object can be driven, thereby improving the synchronization between the driving of the virtual object and the dry audio of the played song.

[0006] This invention provides a virtual object driving method, comprising:

[0007] The target object's dry audio of a song and a first dry audio segment are obtained. The first dry audio segment is the audio within a preset time period in the dry audio of the song. The first dry audio segment includes at least two first audio sub-segments, and the dry audio of the song includes at least two second dry audio segments.

[0008] Based on the volume of each of the first audio sub-segments, determine whether to amplify the volume of each of the second dry audio segments;

[0009] If it is determined that the volume of the target second dry audio segment is amplified, the virtual object is driven based on the amplified volume of the target second dry audio segment and the volumes of other second dry audio segments besides the target second dry audio segment.

[0010] According to a virtual object driving method provided by the present invention, each second dry audio segment includes at least two second audio sub-segments;

[0011] The step of determining whether to amplify the volume of each of the second dry audio segments based on the volume of each of the first audio sub-segments includes:

[0012] Based on the volume of each of the first audio sub-segments, the volume of each of the second audio sub-segments is normalized;

[0013] For each of the second dry audio segments, the first median volume of the second dry audio segment is determined based on the normalized volume of each second audio sub-segment in the second dry audio segment;

[0014] If the first median volume is less than a preset value, it is determined that the normalized volume of each of the second audio sub-segments in the second dry audio segment will be amplified.

[0015] According to a virtual object driving method provided by the present invention, the normalization of the volume of each second audio sub-segment based on the volume of each first audio sub-segment includes:

[0016] Among the volumes of each of the first audio sub-segments, determine the maximum volume and the second median volume;

[0017] Determine the volume difference between the maximum volume and the second median volume;

[0018] The volume of each of the second audio sub-segments is normalized based on the second median volume and the volume difference.

[0019] According to a virtual object driving method provided by the present invention, the step of obtaining the dry audio of a song for a target object includes:

[0020] The acoustic features in the original dry audio of the song are encoded to obtain first encoded information, which includes first timbre information in the original dry audio of the song.

[0021] Based on the original dry audio of the song, the timbre features of the source object are extracted. Based on the timbre features of the source object, the first timbre information in the original dry audio of the song is removed from the first encoding information to obtain the second encoding information.

[0022] The second encoded information and the timbre features of the target object are fused to obtain the third encoded information;

[0023] Based on the third encoding information, the dry audio of the song of the target object is determined.

[0024] According to a virtual object driving method provided by the present invention, the method further includes:

[0025] Upon receiving target information, response information corresponding to the target information is obtained asynchronously based on the target information.

[0026] If the response information is text, it is converted into initial audio asynchronously.

[0027] The virtual object is driven based on the volume corresponding to the initial audio.

[0028] According to a virtual object driving method provided by the present invention, driving the virtual object based on the volume corresponding to the initial audio includes:

[0029] The acoustic features of the initial audio are encoded to obtain fourth encoded information, which includes the second timbre information in the initial audio.

[0030] Based on the initial audio, extract the timbre features of the initial object; based on the timbre features of the initial object, remove the second timbre information of the initial audio from the fourth encoding information to obtain the fifth encoding information;

[0031] The fifth encoded information and the timbre features of the target object are fused to obtain the sixth encoded information;

[0032] The target audio is determined based on the sixth encoded information;

[0033] The virtual object is driven based on the volume corresponding to the target audio.

[0034] According to a virtual object-driven method provided by the present invention, the asynchronous conversion of the text information into initial audio includes:

[0035] If the response information includes sensitive information, the response information is desensitized; the desensitization process includes replacing the sensitive information in the response information with preset information, or deleting the sensitive information in the response information.

[0036] The anonymized response information is converted into the initial audio using an asynchronous method.

[0037] A virtual object driving device provided by the present invention includes:

[0038] The first acquisition unit is used to acquire the dry audio of the song and a first dry audio segment of the target object. The first dry audio segment is the audio within a preset time period in the dry audio of the song. The first dry audio segment includes at least two first audio sub-segments, and the dry audio of the song includes at least two second dry audio segments.

[0039] The determining unit is configured to determine, based on the volume of each of the first audio sub-segments, whether to amplify the volume of each of the second dry audio segments;

[0040] The first driving unit is configured to drive the virtual object based on the amplified volume of the target second dry audio segment and the volumes of other second dry audio segments besides the target second dry audio segment, when it is determined that the volume of the target second dry audio segment is amplified.

[0041] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the virtual object driving method as described above.

[0042] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the virtual object driving method as described above.

[0043] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the virtual object driving method as described above.

[0044] The present invention provides a virtual object driving method, apparatus, electronic device, and storage medium. It acquires the dry audio of a song for a target object and a first dry audio segment within a preset time period from the dry audio. The first dry audio segment is divided into at least two first audio sub-segments, and the dry audio of the song is divided into at least two second dry audio segments. When it is determined, based on the volume of each first audio sub-segment, that the volume of the target second dry audio segment needs to be processed, the virtual object is driven based on the amplified volume of the target second dry audio segment and the volumes of other second dry audio segments besides the target second dry audio segment. It is understood that the present invention drives the virtual object based on the volume of the dry audio of the song for the target object, and amplifies the volume of the target second dry audio segment that requires volume amplification. This avoids the inability to drive the virtual object when the volume of the target second dry audio segment is low, achieving the ability to drive the virtual object as long as the dry audio of the song has volume, thereby improving the synchronization between the driving of the virtual object and the played dry audio of the song. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0046] Figure 1 This is one of the flowcharts of the virtual object driving method provided by the present invention;

[0047] Figure 2 This is the second flowchart of the virtual object driving method provided by the present invention;

[0048] Figure 3 This is the third flowchart of the virtual object driving method provided by the present invention;

[0049] Figure 4 This is a schematic diagram of the multi-queue asynchronous communication mechanism provided in an embodiment of the present invention;

[0050] Figure 5 This is a schematic diagram of the response information generation process provided in an embodiment of the present invention;

[0051] Figure 6 This is a schematic diagram of the structure of the virtual object driving device provided by the present invention;

[0052] Figure 7 This is a schematic diagram of the physical structure of the electronic device provided by the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0054] In live streaming, virtual objects based on Live2D models or 3D motion capture equipment are mostly driven by a virtual person. Whether speaking, singing, or moving, the virtual object is driven by this virtual person. Recently, with the emergence of cognitive models like ChatGPT, live streams featuring fully AI-driven virtual objects that respond to user questions and communicate with users have also appeared. However, in existing live streams, especially in fully AI-driven scenarios, when a complete song is playing, the virtual object cannot be synchronized with the music to move its lips or other body parts. This significantly reduces user engagement and playability.

[0055] Based on this, the present invention provides a virtual object driving method, which drives the virtual object based on the volume of the target object's dry audio of the song, and amplifies the volume of the target second dry audio segment that requires volume amplification processing, so as to avoid the virtual object being unable to be driven when the volume of the target second dry audio segment is low. This realizes that as long as the dry audio of the song has volume, the virtual object can be driven to move, thereby improving the synchronization between the driving of the virtual object and the played dry audio of the song, and greatly improving the user's fun and playability.

[0056] The following is combined with Figures 1-5 The virtual object driving method of the present invention is described.

[0057] Figure 1 This is one of the flowcharts illustrating the virtual object driving method provided in this embodiment of the invention. The executing entity of this method can be an electronic device such as a terminal device or a computer, or a virtual object driving device installed in that electronic device. This virtual object driving device can be implemented through software, hardware, or a combination of both. This method can be applied in application scenarios such as live streaming using virtual objects, or hosting using virtual objects. Figure 1 As shown, the virtual object driving method includes the following steps:

[0058] Step 101: Obtain the dry audio of the song and the first dry audio segment of the target object. The first dry audio segment is the audio within the first preset time period of the dry audio of the song. The first dry audio segment includes at least two first audio sub-segments, and the dry audio of the song includes at least two second dry audio segments.

[0059] The target object can be a specific singer, whose song's dry audio is the audio corresponding to the song played using the singer's timbre in the application scenario. Alternatively, the target object can be a virtual singer, whose timbre can be a preset sweet timbre, a female timbre, etc., and whose song's dry audio can be the dry audio of an entire song. Since the volume of the dry audio of each song may differ, to ensure that the movements of the virtual object's lips and other parts are not affected by the default volume differences in the dry audio itself, it is necessary to obtain the first dry audio segment within the first preset time period of the song's dry audio and calculate the baseline volume based on this first dry audio segment. Typically, vocals appear 30 seconds (s) before the song begins, so to ensure sampling accuracy, the preset time period can be set to 60 seconds.

[0060] For example, since the same song changes according to the singer's mood and rhythm, when the first dry audio segment within the first preset time period is obtained from the dry audio of the song, the first dry audio segment can be further divided according to the first preset time interval to obtain at least two first audio sub-segments. The first preset time interval can be set according to the requirements, for example, the first preset time interval can be 100 milliseconds (ms); In addition, in order to achieve the effect of the virtual object's lips and other parts producing a fluctuation effect whenever there is a human voice, it is also necessary to divide the dry audio of the target object's song according to the second preset time interval to obtain at least two second dry audio segments; the second preset time interval can be set according to the requirements, for example, the second preset time interval can be 1 second.

[0061] It should be noted that this invention can manage the dry audio of each song in the application scenario, that is, store the dry audio of each song in an audio list.

[0062] Step 102: Based on the volume of each of the first audio sub-segments, determine whether to amplify the volume of each of the second dry audio segments.

[0063] For example, when each first audio sub-segment and each second dry audio segment are obtained, the volume of each first audio sub-segment and the volume of each second dry audio segment are calculated. Then, for each second dry audio segment, based on the volume of each first audio sub-segment, it is determined whether the volume of the second dry audio segment is less than a preset value. The second dry audio segment with a volume less than the preset value is identified as the target second dry audio segment, and the target second dry audio segment is identified as the segment that needs to be amplified. The second dry audio segments with a volume greater than or equal to the preset value are identified as other second dry audio segments, and the other second dry audio segments are identified as the segments that do not need to be amplified.

[0064] The specific method for calculating volume is as follows:

[0065] 1) For each first audio sub-segment, convert the first audio sub-segment into a byte array byte[] buffer, and set the length of the byte array to r.

[0066] 2) Calculate the sum of squares of the byte array based on the following formula (1):

[0067]

[0068] Where buffer[i] represents the audio data corresponding to the array element in the byte array, i is the index of the array element, and v represents the volume corresponding to the first audio sub-segment.

[0069] 3) Calculate the mean of the volume corresponding to the first audio sub-segment based on the following formula (2).

[0070]

[0071] 4) When the mean value is determined to be greater than a preset threshold, the volume of the first audio segment is set to 10 * log0. 10 (mean); When the average value mean is less than or equal to a preset threshold, the volume of the first audio segment is determined to be zero, and the volume is in decibels; The preset threshold can be set according to the requirements, for example, the preset threshold can be 0.1.

[0072] It should be noted that the volume calculation method for the second dry audio segment is the same as that for the first audio sub-segment, and will not be repeated here.

[0073] Step 103: If it is determined that the volume of the target second dry audio segment is amplified, the virtual object is driven based on the amplified volume of the target second dry audio segment and the volumes of other second dry audio segments besides the target second dry audio segment.

[0074] For example, when the volume of the target second dry audio segment is determined to be less than a preset value, it indicates that the volume of the target second dry audio segment is too low. In this case, the volume of the target second dry audio segment needs to be amplified, specifically by a preset multiplier, such as 1x, to obtain the amplified volume of the target second dry audio segment. Other second dry audio segments with volumes greater than or equal to the preset value do not require amplification because their volumes are already high. Finally, the virtual object is driven based on the amplified volumes of each target second dry audio segment and the volumes of other second dry audio segments besides the target second dry audio segments. Specifically, this is done by directing the image capture software (VTube). Studio (VTS) sends a first custom parameter message, which carries the amplified volume of each target second dry audio segment and the volume of other second dry audio segments besides the target second dry audio segments, so that VTS controls the virtual object's action based on the amplified volume of each target second dry audio segment and the volume of other second dry audio segments besides the target second dry audio segments.

[0075] It should be noted that driving a virtual object can include controlling the opening and closing of the virtual object's lips, as well as controlling the virtual object's facial expressions or limbs, etc., and this invention does not limit this.

[0076] It should be noted that the volume of the accompaniment audio corresponding to the dry audio of the song can also be carried in the first custom parameter message, so that VTS can also drive the virtual object based on the volume of the accompaniment audio. This invention does not limit this.

[0077] It should be noted that if the audio format of the obtained dry audio of the target object is not WAV, it is necessary to convert the audio format of the dry audio of the song to WAV first. Of course, if the electronic device is compatible with audio formats other than WAV, it is not necessary to convert the audio format of the dry audio of the song. This invention does not limit this.

[0078] It should be noted that when sending the first custom parameter message to the VTS, carrying the amplified volume of each target second dry audio segment and the volume of other second dry audio segments besides the target second dry audio segments, it is also necessary to simultaneously send the target object's song dry audio to the speaker. The speaker then plays the target object's song dry audio in the application scenario, allowing the virtual object's lips and other parts to move synchronously during the playback. Alternatively, both the target object's song dry audio and the song's accompaniment audio can be sent to the speaker, and both can be played through the speaker in the application scenario.

[0079] The virtual object driving method provided by this invention obtains the dry audio of a song for a target object and a first dry audio segment within a preset time period of the dry audio. The first dry audio segment is divided into at least two first audio sub-segments, and the dry audio of the song is divided into at least two second dry audio segments. When it is determined, based on the volume of each first audio sub-segment, that the volume of the target second dry audio segment needs to be processed, the virtual object is driven based on the amplified volume of the target second dry audio segment and the volumes of other second dry audio segments besides the target second dry audio segment. It can be seen that this invention drives the virtual object based on the volume of the dry audio of the song for the target object, and amplifies the volume of the target second dry audio segment that requires volume amplification. This avoids the inability to drive the virtual object when the volume of the target second dry audio segment is low, achieving the ability to drive the virtual object as long as the dry audio of the song has volume, thereby improving the synchronization between the driving of the virtual object and the played dry audio of the song.

[0080] In one embodiment, each of the second dry audio segments includes at least two second audio sub-segments. Figure 2 This is a second flowchart illustrating the virtual object driving method provided in this embodiment of the invention, as shown below. Figure 2 As shown, step 102 above can be implemented in the following ways:

[0081] Step 1021: Normalize the volume of each second audio sub-segment based on the volume of each first audio sub-segment.

[0082] To achieve audio-visual synchronization between the player and VTS, each second dry audio segment can be divided based on a preset time sub-interval, resulting in at least two second audio sub-segments corresponding to each second dry audio segment. The preset time sub-interval can be set according to requirements; for example, the preset time sub-interval can be 100ms.

[0083] For example, when calculating the volume of each first audio sub-segment, a volume time series is formed based on the volume of each first audio sub-segment. A reference parameter is determined from the volume time series, and the reference parameter is determined as global volume data. The volume of each second audio sub-segment is normalized based on the reference parameter, and the normalized volume is determined as interval volume data. Since the format, sampling rate, bit rate, and channels of the dry audio file of the song are not fixed, and the volume distribution of the same song will fluctuate at different time periods according to the changes in the singing mood and rhythm, this invention compares and corrects based on global volume data and interval volume data to make the volume fluctuation more obvious, achieving the effect of the opening and closing of the lips of the virtual object. The normalization calculation is to map the volume value of each second audio sub-segment to the interval range of 0 to 1.

[0084] It should be noted that the volume calculation method for the second audio sub-segment is the same as that for the first audio sub-segment, and will not be repeated here.

[0085] Step 1022: For each of the second dry audio segments, determine the first median volume of the second dry audio segment based on the normalized volume of each second audio sub-segment in the second dry audio segment.

[0086] For example, traverse each second dry audio segment, sort the normalized volumes of each second audio sub-segment in the currently traversed second dry audio segment in ascending order to obtain a sorted volume sequence, obtain the median volume from the sorted volume sequence, and determine the median volume obtained from the sorted volume sequence as the first median volume of the currently traversed second dry audio segment. The first median volume of each second dry audio segment can be obtained in this way.

[0087] Step 1023: When the first median volume is less than a preset value, determine to amplify the normalized volume of each of the second audio sub-segments in the second dry audio segment.

[0088] For example, when determining the first median volume of a second dry audio segment, the first median volume is compared with a preset value. If the first median volume is less than the preset value, it indicates that the volume of the corresponding second dry audio segment is low. In this case, the second dry audio segment is identified as the target second dry audio segment, and the normalized volume of the target second dry audio segment needs to be amplified to avoid the inability to drive the movements of the virtual object's lips and other parts when the volume is low. If the first median volume is greater than or equal to the preset value, it indicates that the volume of the corresponding second dry audio segment is high. In this case, the second dry audio segment is identified as another second dry audio segment, and it is not necessary to amplify the normalized volume of other second dry audio segments.

[0089] In this embodiment, the data corresponding to each sampled second audio sub-segment is relatively small. When traversing the entire dry audio of the song, a slider is scrolled with a time window of 100ms sampling of the second audio sub-segment. The data of the current second audio sub-segment is first sent to the system's player, and the target volume corresponding to the current second audio sub-segment is sent to the VTS. Because the sampled data is small, the player and VTS respond quickly, thus achieving audio-visual synchronization. Here, "audio" in audio-visual refers to the sound of the player playing the current second audio sub-segment, and "visual" refers to the movements of the virtual object's lips and other parts. In addition, when the normalized volume of the current second audio sub-segment is amplified, the target volume is the amplified volume of the current second audio sub-segment; when the normalized audio of the current second audio sub-segment is not amplified, the target volume is the normalized volume of the current second audio sub-segment.

[0090] In this embodiment, when the first median volume of the second dry audio segment is less than a preset value, the normalized volume of each second audio sub-segment in the second dry audio segment is amplified to avoid the inability to drive the movement of the virtual object's lips and other parts when the volume is low. This realizes that as long as the dry audio of the song has volume, the virtual object can be driven to move, thus improving the synchronization between the driving of the virtual object and the played dry audio of the song.

[0091] In one embodiment, step 1021 described above can be implemented in the following way:

[0092] In each of the first audio sub-segments, determine the maximum volume and the second median volume; determine the volume difference between the maximum volume and the second median volume; and normalize the volume of each of the second audio sub-segments based on the second median volume and the volume difference.

[0093] For example, in the volume time series composed of the volume of each first audio sub-segment, the volume of each first audio sub-segment is sorted in ascending order to obtain a new volume time series. The maximum volume max and the second median volume mid are obtained in the new volume time series. The volume difference between the maximum volume max and the second median volume mid is calculated, i.e., the volume difference diff = max - mid. The volume difference diff and the second median volume mid are determined as the reference parameters. Then, the volume of each second audio sub-segment is normalized based on the following formula (3).

[0094] a j =(b j -mid) / diff (3)

[0095] Where a represents the normalized volume of the j-th second audio sub-segment, b j This represents the volume of the j-th second audio segment, where j ranges from 1 to n, and n represents the total number of second audio segments.

[0096] In this embodiment, the volume of each second audio sub-segment is normalized based on the volume difference and the second median volume, so that the normalized volume of each second audio sub-segment is more accurate.

[0097] In one embodiment, obtaining the dry audio of the song of the target object in step 101 above can be achieved in the following way:

[0098] The acoustic features in the original dry audio of the song are encoded to obtain first encoded information, which includes first timbre information in the original dry audio of the song. Based on the original dry audio of the song, the timbre features of the source object are extracted. Based on the timbre features of the source object, the first timbre information in the original dry audio of the song is removed from the first encoded information to obtain second encoded information. The second encoded information and the timbre features of the target object are fused to obtain third encoded information. Based on the third encoded information, the dry audio of the song of the target object is determined.

[0099] The source object can be the source singer, that is, the person who sings the song corresponding to the original dry audio of the song.

[0100] Specifically, the acoustic features of the original song dry audio are encoded using a pre-trained song conversion model to obtain first encoded information. Based on the timbre features of the source object in the original song dry audio, the first timbre information in the original song dry audio is removed from the first encoded information to obtain second encoded information. The second encoded information is then fused with the timbre features of the target object to obtain third encoded information. The target object's song dry audio is generated by decoding the fundamental frequency information of the original song dry audio and the third encoded information.

[0101] The song conversion model includes: an encoding network, a timbre encoding network, a reversible probability distribution model, and a decoding network.

[0102] For example, in the application of the song conversion model, the raw audio of the original song is input into a timbre encoding network. The timbre encoding network encodes the acoustic features of the raw audio to obtain the timbre features of the source object in the raw audio. These acoustic features can include speech waveform features, linear amplitude spectrum features, Mel spectrum features, etc. The acoustic features of the raw audio are then input into an encoding network, which encodes them to obtain first encoded information. The first encoded information and the timbre features of the source object are then input into a reversible probability distribution model. A forward transform is performed using the reversible probability distribution model to obtain second encoded information independent of timbre. The second encoded information and the timbre features of the target object are then input into the reversible probability distribution model again. An inverse transform is performed using the reversible probability distribution model to obtain third encoded information containing the timbre features of the target object. Finally, this third encoded information is transmitted to a decoding network. The decoding network decodes the fundamental frequency information and the third encoded information of the raw audio to obtain the raw audio of the target object.

[0103] It should be noted that during the decoding process of the third encoded information, issues such as fundamental frequency jitter and discontinuity may occur, affecting the pitch accuracy of the generated dry audio file. Pitch accuracy is one of the fundamental factors affecting the overall audio quality during dry audio production, and the fundamental frequency information of the dry audio file is the data reflecting pitch accuracy. In dry audio production, the pitch accuracy of the dry audio file can be guaranteed based on the fundamental frequency information. Therefore, to ensure stable pitch accuracy in the generated dry audio file, this embodiment requires fundamental frequency extraction from the original dry audio file to obtain its fundamental frequency information. While decoding the third encoded information, the pitch accuracy is controlled using the fundamental frequency information of the original dry audio file, ensuring that the generated dry audio file has a stable and continuous fundamental frequency, conforming to the pitch accuracy corresponding to the fundamental frequency information of the original dry audio file, thereby improving the pitch accuracy of the dry audio file.

[0104] For the extraction of fundamental frequency from the dry audio of the original song, this embodiment can use existing speech signal processing tools, such as STRAIGHT, Praat, WORLD, etc., or a pre-trained neural network-based fundamental frequency extraction model. This fundamental frequency extraction model is an existing model, and will not be described in detail in this embodiment.

[0105] Furthermore, when extracting the fundamental frequency from the original song's dry audio using the aforementioned fundamental frequency extraction method, inaccurate fundamental frequency extraction may occur. For example, errors may occur in voiced / unvoiced sounds, half / second harmonic frequencies, or inaccurate extraction of guttural sounds, thus affecting the final quality of the determined dry audio. Therefore, after extracting the fundamental frequency information from the original song's dry audio, this embodiment requires checking and correcting this information to ensure it matches the fundamental frequency of the original song's dry audio, thereby achieving the desired release-grade dry audio for the target audience.

[0106] In this embodiment, the encoding network and decoding network are the same as the posterior encoding network and decoding network in the variational autoencoder. The timbre encoding network can be an existing network for timbre extraction of speech. The reversible probability distribution model can be the Glow model or the NICE stream model, etc., which will not be described further in this embodiment.

[0107] It should be noted that, to ensure the accuracy of the acoustic features of the original song's dry audio, the source singer needs to record in a noise-free recording studio when performing the original song's dry audio. This ensures that the obtained original song's dry audio is a high-fidelity performance. Because a live singer's performance reveals singing details, such as emotion, rhythm, and tone, the original song's dry audio contains not only pronunciation content and timbre information but also singing details from the source singer. Therefore, the first encoded information after encoding the acoustic features of the original song's dry audio also includes pronunciation content information, the source singer's timbre information, and singing details information. These singing details information include at least one of emotion, rhythm, and tone information. Therefore, the final target song's dry audio contains both the target object's timbre information and the source object's singing details.

[0108] It should be noted that when training the song conversion model, some dry audio samples of the target song can be collected. The initial song conversion model can be trained based on multiple dry audio samples to obtain the song conversion model. The dry audio samples need to be recorded in a high-fidelity recording environment, and the data format can be 48kHz / 16bit. The length of the dry audio samples can be a preset length, for example, fifteen seconds.

[0109] In this embodiment, the timbre characteristics of the original song's dry audio are determined. Then, based on these timbre characteristics, the first timbre information of the source object is removed from the first encoding information corresponding to the original song's dry audio, resulting in second encoding information that does not contain the source object's timbre information (i.e., is unrelated to timbre). This second encoding information merely removes the first timbre information of the source object; therefore, it still contains the singing details of the source object. The second encoding information is then fused with the timbre characteristics of the target object to obtain third encoding information that includes the timbre characteristics of the target object. This third encoding information includes both the timbre characteristics of the target object and the singing details of the source object. In other words, only the timbre in the original song's dry audio corresponding to the source song sung by a real person is replaced with the timbre of the target object, while preserving details such as emotion, rhythm, or tone in the source song. This achieves the conversion from original song dry audio containing the source object's timbre to song dry audio containing the target object's timbre.

[0110] To ensure timely responses to user-sent target information, this invention employs a multi-queue, asynchronous processing design, dividing the entire process into a message preprocessing queue, a dialogue model processing queue, a sound synthesis processing queue, and a playback queue. Each queue's corresponding thread operates independently without interference or blocking. Each queue can join the next process after completing its current stage of processing. This ensures parallel processing of multiple target messages. When multiple target messages exist, the playback interval of the playback queue can be adaptively adjusted based on demand.

[0111] In one embodiment, Figure 3 This is the third flowchart of the virtual object driving method provided in this embodiment of the invention, as shown below. Figure 3 As shown, the virtual object driving method also includes the following steps:

[0112] Step 104: Upon receiving the target information, obtain the response information corresponding to the target information asynchronously.

[0113] The target information can be user-sent bullet comments or pre-stored information to be played in the application scenario. For example, in a live streaming scenario, the target information could be "Welcome to the live stream." The response information is the reply information in response to the target information.

[0114] For example, upon receiving target information, it can be preprocessed using a message preprocessing queue. The preprocessed information is then sent to the dialogue model's processing queue. The dialogue model's processing queue calls the question-answering model, inputting the preprocessed target information to obtain the response information output by the model. This response information can be text or audio. If the response information is text, it is sent to the voice synthesis processing queue. The response model can be a cognitive big model, which takes the preprocessed target information as a question and submits it along with a pre-defined prompt as a message to the cognitive big model. For example, the cognitive big model could be ChatGPT (Chat Generative Pre-trained Transformer) or IFlytek Spark, and returns the response information. Furthermore, to obtain the expected answer, the prompt needs to be customized based on the specific circumstances of the virtual object, including basic information such as age and name; personality traits such as personality and hobbies; interpersonal relationships such as friends and idols; and social positioning.

[0115] Specifically, preprocessing the target information includes at least one of the following: preprocessing the target information based on filtering rules, preprocessing the target information based on the priority of the information, or preprocessing the target information based on the maximum length of the message processing pool.

[0116] The filtering rules can be as follows: target information with more than a preset number of characters is considered valid information to reduce low-quality information; that is, when the number of characters in the target information is less than the preset number of characters, the target information is identified as low-quality information and discarded; when the number of characters in the target information is greater than or equal to the preset number of characters, the target information is identified as valid information and sent to the dialogue model's waiting queue.

[0117] Filtering rules can also be used to filter target information that includes sensitive information.

[0118] The filtering rules can also identify target information containing specific characters as valid information. These specific characters can be "?", etc. For example, if the target information is "**Has the product been loaded yet?", then the target information is identified as valid information and will not be filtered. If the target information is "I bought this product last time!", then the target information is identified as invalid information and needs to be filtered out.

[0119] When the target information is bullet screen information, the filtering rule can also be to obtain valid information according to the generation time of the bullet screen information, that is, after the current target information is processed, the latest generated bullet screen information is obtained as valid information.

[0120] The priority of preset settings can be, for example, that platform gift information has a higher priority than bullet screen information. That is, higher priority target information is processed first. For example, if the received target information is platform gift information, then the target information is determined to be valid information and processed first.

[0121] Preprocessing target information based on the maximum length of the message processing pool can be done as follows: when the message processing pool is idle, send the target information to the dialogue model's waiting queue; when the message processing pool is not idle, discard the target information. Additionally, when the message processing pool is idle, low-priority target information can also be sent to the dialogue model's waiting queue. For example, if the target information is a bullet screen message, it can be sent to the dialogue model's waiting queue.

[0122] Step 105: If the response information is text information, convert the response information into initial audio in an asynchronous manner.

[0123] For example, the voice synthesis queue receives response information sent by the dialogue model queue. In order to achieve a fully AI application scenario, such as a fully AI live broadcast, when the voice synthesis queue receives the response information, it can call the text-to-speech (TTS) model to convert the response information into initial audio based on the TTS model. The initial audio includes the timbre information of the initial object, which can be a preset user or an automatically generated audio with timbre information.

[0124] It should be noted that when the response information is audio information, the virtual object is driven directly based on the volume corresponding to the response information.

[0125] Step 106: Drive the virtual object based on the volume corresponding to the initial audio.

[0126] For example, when the sound synthesis processing queue receives the initial audio, it sends the initial audio to the playback queue, calculates the volume corresponding to the initial audio through the playback queue, and sends the volume corresponding to the initial audio to the VTS, so that the VTS controls the virtual object's actions based on the volume corresponding to the initial audio.

[0127] It should be noted that all communication between the queues and between the models within the queues uses an asynchronous communication mechanism.

[0128] Figure 4 This is a schematic diagram of the multi-queue asynchronous communication mechanism provided in an embodiment of the present invention, as shown below. Figure 4As shown, it includes a message preprocessing queue, a dialogue model waiting queue, a sound synthesis waiting queue, and a playback queue. The message preprocessing queue preprocesses each target information, and then sends the preprocessed target information to the dialogue model waiting queue. The dialogue model waiting queue calls the question-answering model, inputs the preprocessed target information into the question-answering model, and obtains the response information output by the question-answering model.

[0129] If the response information is text, it is sent to the voice synthesis processing queue. When the voice synthesis processing queue receives the response information, it can call the TTS model to convert the response information into initial audio based on the TTS model, and then convert the initial audio into target audio including the timbre features of the target object based on the speech conversion model. The target audio including the timbre features of the target object is sent to the playback queue. The volume of the target audio is calculated by the playback queue, and the virtual object is driven based on the volume of the target audio.

[0130] In this embodiment, the response information of the received target information is determined by a multi-queue asynchronous processing method, and the response information is converted into initial audio. The virtual object is driven based on the volume corresponding to the initial audio, and the initial audio is played by a player. This improves the timeliness of the initial audio playback and the timeliness of driving the virtual object based on the volume of the initial audio.

[0131] In one embodiment, the virtual object is driven based on the volume corresponding to the initial audio, which can be implemented in the following way:

[0132] The acoustic features of the initial audio are encoded to obtain fourth encoded information, which includes second timbre information from the initial audio. Based on the initial audio, the timbre features of the initial object are extracted. Based on the timbre features of the initial object, the second timbre information of the initial audio is removed from the fourth encoded information to obtain fifth encoded information. The fifth encoded information and the timbre features of the target object are fused to obtain sixth encoded information. Based on the sixth encoded information, the target audio is determined. Based on the volume corresponding to the target audio, the virtual object is driven.

[0133] Specifically, the acoustic features of the initial audio are encoded using a pre-trained speech conversion model to obtain fourth encoded information. Based on the timbre features of the initial object in the initial audio, the second timbre information in the initial audio is removed from the fourth encoded information to obtain fifth encoded information. The fifth encoded information is then fused with the timbre features of the target object to obtain sixth encoded information. The target audio is generated by decoding the fundamental frequency information of the initial audio and the sixth encoded information, and the virtual object is driven based on the volume of the target audio.

[0134] The speech conversion model has the same network structure as the song conversion model mentioned above. For details, please refer to the relevant description of the song conversion model. This invention will not repeat it here.

[0135] For example, when the initial audio is obtained, the speech synthesis queue calls the speech conversion model again, inputs the initial audio into the speech conversion model, and replaces the timbre features of the initial object in the initial audio with the timbre features of the target object through the speech conversion model, and obtains the target audio output by the speech conversion model. The target audio is the speech that needs to be played in the application scenario. The target audio includes the timbre features of the target object and the acoustic details of the initial object, such as audio content, the emotion, rhythm and tone of the initial object.

[0136] Optionally, the virtual object can be driven based on the volume corresponding to the target audio, which can be implemented in the following ways:

[0137] The process involves: acquiring the target audio segment within a pre-defined time period preceding the target audio segment; dividing the target audio segment into at least two third audio sub-segments; determining the volume of each third audio sub-segment; sorting the volumes of each third audio sub-segment from smallest to largest to obtain a target volume time series; acquiring the target maximum volume and target median volume from the target volume time series; determining the target volume difference between the target maximum volume and the target median volume; normalizing the volume of each third audio sub-segment based on the target volume difference and the target median volume; and driving the virtual object based on the normalized volume of each third audio sub-segment.

[0138] In the context of speaking scenarios, i.e. voice scenarios, the sound is synthesized by the TTS module and the speech conversion model. The volume of the target audio is relatively fixed, so the target audio segment before the target preset time period can be collected to calculate the volume benchmark. The target preset time period can be set according to the requirements, for example, the target preset time period can be 10 seconds.

[0139] For example, when the sound synthesis processing queue receives the target audio, it sends the target audio to the playback queue. The playback queue determines the normalized volume of each third audio sub-segment corresponding to the target audio and drives the virtual object based on the normalized volume of each third audio sub-segment. The process of normalizing the volume of each third audio sub-segment based on the target volume difference and the target median volume can refer to the above formula (3), which will not be repeated here.

[0140] Specifically, driving the virtual object based on the normalized volume of each third audio sub-segment involves sending a second custom parameter message to the VTS, which carries the normalized volume of each third audio sub-segment, so that the VTS controls the virtual object's actions based on the normalized volume of each third audio sub-segment.

[0141] It should be noted that driving the virtual object can include controlling the opening and closing of the virtual object's lips, that is, controlling the virtual object to speak in the application scenario, and can also control the virtual object's facial expressions or body parts, etc., which are not limited in this invention.

[0142] It should be noted that when sending the second custom parameter message carrying the normalized volume of each third audio sub-segment to the VTS, the target audio also needs to be sent to the speaker at the same time. The target audio is then played through the speaker in the application scenario so that the lips and other parts of the virtual object can move synchronously during the playback of the target audio in the application scenario.

[0143] It should be noted that since the acoustic characteristics of the same person speaking and singing are different, it is necessary to collect some dry audio samples of the target object's speech when training the speech conversion model. That is, dry audio samples of speech. The initial speech conversion model is trained based on multiple dry audio samples to obtain the final speech conversion model. These dry audio samples need to be recorded in a high-fidelity recording environment, with a data format of 48kHz / 16bit. The length of the dry audio sample can be the same as the length of the dry audio sample of the song mentioned above; for example, the length of the dry audio sample can also be a preset length, such as fifteen seconds.

[0144] In this embodiment, the timbre characteristics of the initial audio are determined. Then, based on the timbre characteristics of the initial audio, the second timbre information of the initial object in the fourth encoding information corresponding to the initial audio is removed, resulting in fifth encoding information that does not contain the timbre information of the initial object (i.e., is unrelated to timbre). This fifth encoding information only removes the second timbre information of the initial object, so it still contains the speech detail information of the initial object. The fifth encoding information is then fused with the timbre characteristics of the target object to obtain sixth encoding information that includes the timbre characteristics of the target object. This sixth encoding information includes both the timbre characteristics of the target object and the speech detail information of the initial object. That is, only the timbre in the initial audio is replaced with the timbre of the target object, while retaining details such as emotion, rhythm, or tone in the initial audio. This achieves the conversion from initial audio containing the timbre of the initial object to target audio containing the timbre of the target object. Since both the target audio and the dry audio of the song contain the timbre of the target object, it ensures that the timbre of the spoken voice and the timbre of the played song are consistent in the application scenario.

[0145] In one embodiment, the above-mentioned asynchronous conversion of the response information into initial audio can be achieved in the following way:

[0146] If the response information includes sensitive information, the response information is desensitized; the desensitization process includes replacing the sensitive information in the response information with preset information, or deleting the sensitive information in the response information; the desensitized response information is converted into the initial audio in an asynchronous manner.

[0147] For example, when the response information is obtained, it is determined whether the response information contains sensitive information from the sensitive word database. If the response information contains sensitive information from the sensitive word database, it is determined that the response information is unsafe. The sensitive information in the response information can be replaced with preset information to obtain the desensitized response information. For example, the preset information can be "someone", "cross", etc. The specific preset information can be set according to the requirements. Alternatively, the sensitive information in the response information can be deleted to obtain the desensitized response information. Then, the desensitized response information is converted into the initial audio in an asynchronous manner so that the initial audio played by the final player does not contain sensitive information.

[0148] If it is determined that the response information does not contain sensitive information from the sensitive word database, the response information is considered safe, and the response information can be converted into the initial audio asynchronously.

[0149] Figure 5 This is a schematic diagram of the response information generation process provided in an embodiment of the present invention, such as... Figure 5 As shown, when target information is received, the target information is input into the cognitive big model. The cognitive big model outputs the response information corresponding to the target information and reviews the response information. If the response information is qualified, the response information is synthesized into speech to obtain the initial audio. The initial audio is converted based on the speech conversion model to obtain the target audio including the timbre features of the target object. The virtual object is driven based on the volume of the target audio.

[0150] In this embodiment, each response is reviewed upon receipt, and if the response contains sensitive information, it is desensitized. The target audio is then determined based on the desensitized response, thereby improving the security of playing the target audio in the application scenario.

[0151] The virtual object driving device provided by the present invention will be described below. The virtual object driving device described below can be referred to in correspondence with the virtual object driving method described above.

[0152] Figure 6This is a schematic diagram of the structure of the virtual object driving device provided in an embodiment of the present invention, as shown below. Figure 6 As shown, the virtual object driving device 600 includes a first acquisition unit 601, a determination unit 602, and a first driving unit 603; wherein:

[0153] The first acquisition unit 601 is used to acquire the dry audio of the song and the first dry audio segment of the target object. The first dry audio segment is the audio within the first preset time period in the dry audio of the song. The first dry audio segment includes at least two first audio sub-segments, and the dry audio of the song includes at least two second dry audio segments.

[0154] The determining unit 602 is used to determine whether to amplify the volume of each of the second dry audio segments based on the volume of each of the first audio sub-segments.

[0155] The first driving unit 603 is used to drive the virtual object based on the amplified volume of the target second dry audio segment and the volumes of other second dry audio segments besides the target second dry audio segment when it is determined that the volume of the target second dry audio segment is amplified.

[0156] The virtual object driving device provided by this invention acquires the dry audio of a song for a target object and a first dry audio segment within a preset time period of the dry audio. It divides the first dry audio segment into at least two first audio sub-segments and the dry audio of the song into at least two second dry audio segments. When it is determined, based on the volume of each first audio sub-segment, that the volume of the target second dry audio segment needs to be processed, the virtual object is driven based on the amplified volume of the target second dry audio segment and the volumes of other second dry audio segments besides the target second dry audio segment. It can be seen that this invention drives the virtual object based on the volume of the dry audio of the song for the target object, and amplifies the volume of the target second dry audio segment that requires volume amplification. This avoids the inability to drive the virtual object when the volume of the target second dry audio segment is low, achieving the ability to drive the virtual object as long as the dry audio of the song has volume, thereby improving the synchronization between the driving of the virtual object and the played dry audio of the song.

[0157] Based on any of the above embodiments, each second dry audio segment includes at least two second audio sub-segments; the determining unit 602 is specifically used for:

[0158] Based on the volume of each of the first audio sub-segments, the volume of each of the second audio sub-segments is normalized;

[0159] For each of the second dry audio segments, the first median volume of the second dry audio segment is determined based on the normalized volume of each second audio sub-segment in the second dry audio segment;

[0160] If the first median volume is less than a preset value, it is determined that the normalized volume of each of the second audio sub-segments in the second dry audio segment will be amplified.

[0161] Based on any of the above embodiments, the determining unit 602 is further specifically used for:

[0162] Among the volumes of each of the first audio sub-segments, determine the maximum volume and the second median volume;

[0163] Determine the volume difference between the maximum volume and the second median volume;

[0164] The volume of each of the second audio sub-segments is normalized based on the second median volume and the volume difference.

[0165] Based on any of the above embodiments, the first acquisition unit 601 is specifically used for:

[0166] The acoustic features in the original dry audio of the song are encoded to obtain first encoded information, which includes first timbre information in the original dry audio of the song.

[0167] Based on the original dry audio of the song, the timbre features of the source object are extracted. Based on the timbre features of the source object, the first timbre information in the original dry audio of the song is removed from the first encoding information to obtain the second encoding information.

[0168] The second encoded information and the timbre features of the target object are fused to obtain the third encoded information;

[0169] Based on the third encoding information, the dry audio of the song of the target object is determined.

[0170] Based on any of the above embodiments, the virtual object driving device 600 further includes:

[0171] The second acquisition unit is used to acquire, in an asynchronous manner, response information corresponding to the target information based on the target information upon receiving the target information.

[0172] The conversion unit is used to convert the response information into initial audio asynchronously when the response information is text information;

[0173] The second driving unit is used to drive the virtual object based on the volume corresponding to the initial audio.

[0174] Based on any of the above embodiments, the second driving unit is specifically used for:

[0175] The acoustic features of the initial audio are encoded to obtain fourth encoded information, which includes the second timbre information in the initial audio.

[0176] Based on the initial audio, extract the timbre features of the initial object; based on the timbre features of the initial object, remove the second timbre information of the initial audio from the fourth encoding information to obtain the fifth encoding information;

[0177] The fifth encoded information and the timbre features of the target object are fused to obtain the sixth encoded information;

[0178] The target audio is determined based on the sixth encoded information;

[0179] The virtual object is driven based on the volume corresponding to the target audio.

[0180] Based on any of the above embodiments, the conversion unit is specifically used for:

[0181] If the response information includes sensitive information, the response information is desensitized; the desensitization process includes replacing the sensitive information in the response information with preset information, or deleting the sensitive information in the response information.

[0182] The anonymized response information is converted into the initial audio using an asynchronous method.

[0183] Figure 7 This is a schematic diagram of the physical structure of the electronic device provided in the embodiments of the present invention, such as... Figure 7 As shown, the electronic device may include: a processor 710, a communications interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communications interface 720, and the memory 730 communicate with each other through the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a virtual object driving method, which includes: acquiring the dry audio of a song and a first dry audio segment of a target object, wherein the first dry audio segment is audio within a preset time period of the dry audio of the song; the first dry audio segment includes at least two first audio sub-segments, and the dry audio of the song includes at least two second dry audio segments;

[0184] Based on the volume of each of the first audio sub-segments, determine whether to amplify the volume of each of the second dry audio segments;

[0185] If it is determined that the volume of the target second dry audio segment is amplified, the virtual object is driven based on the amplified volume of the target second dry audio segment and the volumes of other second dry audio segments besides the target second dry audio segment.

[0186] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0187] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program, the computer program being able to be stored on a non-transitory computer-readable storage medium, the computer program being executed by a processor, the computer being able to execute the virtual object driving method provided by the above methods, the method including: acquiring the dry audio of a song and a first dry audio segment of a target object, the first dry audio segment being the audio within a preset time period in the dry audio of the song; the first dry audio segment including at least two first audio sub-segments, the dry audio of the song including at least two second dry audio segments;

[0188] Based on the volume of each of the first audio sub-segments, determine whether to amplify the volume of each of the second dry audio segments;

[0189] If it is determined that the volume of the target second dry audio segment is amplified, the virtual object is driven based on the amplified volume of the target second dry audio segment and the volumes of other second dry audio segments besides the target second dry audio segment.

[0190] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the virtual object driving method provided by the above methods, the method comprising: acquiring the dry audio of a song and a first dry audio segment of a target object, wherein the first dry audio segment is audio within a preset time period in the dry audio of the song; the first dry audio segment includes at least two first audio sub-segments, and the dry audio of the song includes at least two second dry audio segments;

[0191] Based on the volume of each of the first audio sub-segments, determine whether to amplify the volume of each of the second dry audio segments;

[0192] If it is determined that the volume of the target second dry audio segment is amplified, the virtual object is driven based on the amplified volume of the target second dry audio segment and the volumes of other second dry audio segments besides the target second dry audio segment.

[0193] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0194] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0195] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A virtual object-driven method, characterized in that, include: The target object's dry audio of a song and a first dry audio segment are obtained. The first dry audio segment is the audio within a preset time period in the dry audio of the song. The first dry audio segment includes at least two first audio sub-segments, and the dry audio of the song includes at least two second dry audio segments. Based on the volume of each of the first audio sub-segments, determine whether to amplify the volume of each of the second dry audio segments; If it is determined that the volume of the target second dry audio segment is amplified, the virtual object is driven based on the amplified volume of the target second dry audio segment and the volume of other second dry audio segments besides the target second dry audio segment. Each of the second dry audio segments includes at least two second audio sub-segments; The step of determining whether to amplify the volume of each of the second dry audio segments based on the volume of each of the first audio sub-segments includes: Based on the volume of each of the first audio sub-segments, the volume of each of the second audio sub-segments is normalized; For each of the second dry audio segments, the first median volume of the second dry audio segment is determined based on the normalized volume of each second audio sub-segment in the second dry audio segment; If the first median volume is less than a preset value, it is determined that the normalized volume of each of the second audio sub-segments in the second dry audio segment will be amplified.

2. The virtual object driving method according to claim 1, characterized in that, The normalization of the volume of each second audio sub-segment based on the volume of each first audio sub-segment includes: Among the volumes of each of the first audio sub-segments, determine the maximum volume and the second median volume; Determine the volume difference between the maximum volume and the second median volume; The volume of each of the second audio sub-segments is normalized based on the second median volume and the volume difference.

3. The virtual object driving method according to claim 1 or 2, characterized in that, The acquisition of the dry audio of the song for the target object includes: The acoustic features in the original dry audio of the song are encoded to obtain first encoded information, which includes first timbre information in the original dry audio of the song. Based on the original dry audio of the song, the timbre features of the source object are extracted. Based on the timbre features of the source object, the first timbre information in the original dry audio of the song is removed from the first encoding information to obtain the second encoding information. The second encoded information and the timbre features of the target object are fused to obtain the third encoded information; Based on the third encoding information, the dry audio of the song of the target object is determined.

4. The virtual object driving method according to claim 1 or 2, characterized in that, The method further includes: Upon receiving target information, response information corresponding to the target information is obtained asynchronously based on the target information. If the response information is text, it is converted into initial audio asynchronously. The virtual object is driven based on the volume corresponding to the initial audio.

5. The virtual object driving method according to claim 4, characterized in that, The step of driving the virtual object based on the volume corresponding to the initial audio includes: The acoustic features of the initial audio are encoded to obtain fourth encoded information, which includes the second timbre information in the initial audio. Based on the initial audio, extract the timbre features of the initial object; based on the timbre features of the initial object, remove the second timbre information of the initial audio from the fourth encoding information to obtain the fifth encoding information; The fifth encoded information and the timbre features of the target object are fused to obtain the sixth encoded information; Based on the sixth encoded information, the target audio is determined; The virtual object is driven based on the volume corresponding to the target audio.

6. The virtual object driving method according to claim 4, characterized in that, The asynchronous conversion of the text information into initial audio includes: If the response information includes sensitive information, the response information is desensitized; the desensitization process includes replacing the sensitive information in the response information with preset information, or deleting the sensitive information in the response information. The anonymized response information is converted into the initial audio using an asynchronous method.

7. A virtual object driving device, characterized in that, include: The first acquisition unit is used to acquire the dry audio of the song and a first dry audio segment of the target object. The first dry audio segment is the audio within a preset time period in the dry audio of the song. The first dry audio segment includes at least two first audio sub-segments, and the dry audio of the song includes at least two second dry audio segments. The determining unit is configured to determine, based on the volume of each of the first audio sub-segments, whether to amplify the volume of each of the second dry audio segments; The first driving unit is configured to drive the virtual object based on the amplified volume of the target second dry audio segment and the volumes of other second dry audio segments besides the target second dry audio segment when it is determined that the volume of the target second dry audio segment is amplified. Each of the second dry audio segments includes at least two second audio sub-segments; The determining unit is specifically used for: Based on the volume of each of the first audio sub-segments, the volume of each of the second audio sub-segments is normalized; For each of the second dry audio segments, the first median volume of the second dry audio segment is determined based on the normalized volume of each second audio sub-segment in the second dry audio segment; If the first median volume is less than a preset value, it is determined that the normalized volume of each of the second audio sub-segments in the second dry audio segment will be amplified.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the virtual object driving method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the virtual object driving method as described in any one of claims 1 to 6.

Images