A real-time voice changing method and device based on sliding window

Abstract

The application relates to the technical field of voice changing processing, and discloses a real-time voice changing method and device based on a sliding window, which has the technical scheme as follows: S1, acquiring a real-time voice stream and segmenting the real-time voice stream to obtain n voice segments; S2, sequentially performing voice changing smoothing processing on the n voice segments through the sliding window, performing voice changing conversion on an i-th voice segment to obtain an i-th voice changing voice segment, and performing head-tail segmentation on the i-th voice changing voice segment to obtain an i-th voice head segment and an i-th voice tail segment; performing voice changing conversion on an i+1-th voice segment to obtain an i+1-th voice changing voice segment, and performing head-tail segmentation on the i+1-th voice changing voice segment to obtain an i+1-th voice head segment and an i+1-th voice tail segment; performing smoothing transition processing on the i+1-th voice head segment through the i-th voice tail segment to obtain an i+1-th smoothly voice changing voice segment; and S3, outputting in real time after obtaining the first voice changing voice segment or the i+1-th smoothly voice changing voice segment.

Description

Technical Field

[0001] This invention relates to the field of voice changing technology, and more specifically, to a real-time voice changing method and apparatus based on a sliding window. Background Technology

[0002] Voice changing, as a long-standing technology, mainly achieved its purpose by manipulating the frequency domain of sound before the advent of AI voice changing. However, this traditional technology produces a very mechanical and unnatural voice changing effect, making it easy to tell that the voice is changed by software, which greatly limits the scope of application of this technology.

[0003] AI voice changing, as an upgraded product of traditional voice changing technology, is developed from AI speech synthesis technology. Since 2016, the academic community has been conducting more and more research on deep learning voice changing technology. After years of research, AI voice changing technology has made great progress, and the voice changing effect in experimental scenarios has achieved good results.

[0004] Most existing AI voice changing technologies are based on offline complete audio files for voice conversion. When used in real-time voice changing scenarios, to solve the real-time problem, they can only be based on offline voice changing of small window audio files, and then the audio segments are spliced ​​together to form a changed audio stream. Although this method solves the real-time voice changing problem, segmented voice changing can lead to unnatural and abrupt transitions between changed audio segments, affecting the actual user experience. Summary of the Invention

[0005] The purpose of this invention is to provide a real-time voice changing method and device based on a sliding window, which realizes real-time AI voice changing of audio streams, resulting in a smoother and more natural voice changing effect, and greatly improving the user's actual voice changing experience.

[0006] The above-mentioned technical objective of the present invention is achieved through the following technical solution: a real-time voice changing method based on a sliding window, comprising the following steps: S1. Acquire the real-time audio stream and segment the real-time audio stream to obtain n consecutive audio segments, where n > 1; S2. Using a sliding window, perform voice smoothing processing on n speech segments in sequence according to the segmentation order; The process of voice smoothing is as follows: The voice of the i-th speech segment is converted into a voice, i=1…n-1, to obtain the i-th voice-modified speech segment. The i-th voice-modified speech segment is then split into the first and last segments to obtain the i-th speech beginning segment and the i-th speech ending segment. Slide the window to the (i+1)th speech segment, perform voice conversion, and obtain the (i+1)th voice-modified speech segment; then split the (i+1)th voice-modified speech segment into its beginning and end segments to obtain the (i+1)th speech beginning segment and the (i+1)th speech end segment. By using the i-th speech tail segment, the i+1-th speech head segment is smoothed to obtain the i+1-th smooth voice-changing speech segment with the smoothed speech head segment. S3. After obtaining the first voice-changing speech segment or the (i+1)th smooth voice-changing speech segment, output it in real time.

[0007] As a preferred technical solution of the present invention, in S1, the real-time voice stream is segmented according to a preset fixed duration.

[0008] As a preferred technical solution of the present invention, in S2, after each time the i-th voice-changing speech segment is split into its beginning and end, the i-th speech end segment is cached, and when the (i+1)-th speech beginning segment is processed smoothly, the cached i-th speech end segment is used for execution.

[0009] As a preferred technical solution of the present invention, when dividing the voice-changing speech segment into its beginning and end, the segment with a preset beginning segment length starting from the beginning of the segment is recorded as the speech beginning segment, and the remainder is the speech end segment, wherein the speech beginning segment is less than or equal to the speech end segment.

[0010] As a preferred technical solution of the present invention, when the length of the nth voice-changing speech segment is less than the preset head segment length, the entire nth voice-changing speech segment is taken as the nth voice head segment; the (n-1)th voice tail segment is truncated from back to front to obtain the (n-1)th voice tail segment length, which is the same as the length of the nth voice head segment, and the truncated (n-1)th voice tail segment and the nth voice head segment are subjected to smooth transition processing to obtain the nth smooth voice-changing speech segment.

[0011] As a preferred embodiment of the present invention, the i-th speech tail segment includes an i-th body segment and an i-th tail segment located at the beginning, wherein the length of the i-th body segment is greater than or equal to 0, and the length of the i-th tail segment is the same as the length of the i-th speech head segment; When the length of the nth speech segment is greater than or equal to the preset head segment length, when the i+1th speech head segment is smoothed by the i-th speech tail segment, the i-th body segment is removed from the i-th speech tail segment, the i-th tail segment is retained and used, and the i+1th speech head segment is smoothed.

[0012] As a preferred embodiment of the present invention, in S2, the smooth transition processing process is as follows: Determine the target speech beginning segment b and the target speech ending segment a, and perform weighted smoothing fitting on them, including: assuming the length of the target speech beginning segment b is m sample points, align the target speech beginning segment b and the target speech ending segment a along the time axis, and perform point-by-point weighted calculation to obtain the smoothed speech beginning segment c, whose expression is: c[j] = a[j] * [1 - r] + b[j] * r; r = exp(-k * (m - j)); Where a[j] is the discrete sample value of the tail segment a of the target voice, b[j] is the discrete sample value of the head segment b of the target voice; c[j] is the discrete sample value of the voice segment c after smooth fitting, j is the sample index, and j < m; r is the weight coefficient, m is the length of the voice sample; k is the curvature control constant.

[0013] A real-time voice conversion device based on a sliding window, comprising: A voice input module for collecting and obtaining a real-time voice stream; A voice stream segmentation module for segmenting the real-time voice stream to obtain n consecutive voice segments, where n > 1; A sliding window processing module for sequentially calling a voice conversion module and a smoothing processing module to process n voice segments in the segmentation order through a sliding window; performing voice conversion and smoothing processing to obtain the first voice-converted segment and the (i + 1)-th smoothed voice-converted segment; A voice conversion module for performing voice conversion on n voice segments; A smoothing processing module for performing head and tail segmentation on n voice-converted segments to obtain corresponding voice head segments and voice tail segments, and performing smooth transition processing on the (i + 1)-th voice head segment through the i-th voice tail segment to obtain the (i + 1)-th smoothed voice-converted segment with a smoothed voice head segment; A voice output module for outputting according to the smoothed voice-converted segment.

[0014] As a preferred technical solution of the present invention, the real-time voice conversion device further includes: A voice conversion selection module for selecting whether to perform voice conversion on the real-time voice stream. If so, it calls the voice stream segmentation module; if not, it directly calls the voice output module to perform voice stream transparent transmission output; A voice decoding module for decoding the voice stream; A voice encoding module for encoding the smoothed voice-converted segment.

[0015] In summary, the present invention has the following beneficial effects: In real-time voice changing scenarios, after receiving a real-time audio stream, it can perform segmented voice changing. At the same time, it performs beginning and end segmentation on the audio segments after segmented voice changing, and smoothly transitions the beginning of the current audio segment through the end of the previous audio segment. This not only solves the problem of real-time voice changing, but also solves the problem of unnatural and abrupt transitions between voice-changing audio segments. Combined with offline AI voice changing methods, it realizes real-time audio stream AI voice changing, resulting in a smoother and more natural voice changing effect, greatly improving the user's actual voice changing experience. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the real-time voice streaming process of the present invention; Figure 2 This is a schematic diagram of the real-time voice streaming processing of the present invention; Figure 3 This is a schematic diagram of the smooth transition processing of speech segments in this invention; Figure 4 This is a schematic diagram of the real-time voice changing device of the present invention. Detailed Implementation

[0017] The present invention will be further described in detail below with reference to the accompanying drawings.

[0018] This invention provides a real-time voice changing method based on a sliding window, comprising the following steps: S1. Acquire the real-time audio stream and segment the real-time audio stream into n consecutive audio segments according to a preset fixed duration, where n > 1; S2. Using a sliding window, perform voice smoothing processing on n speech segments in sequence according to the segmentation order to obtain the first voice-changing speech segment and the (i+1)th smoothed voice-changing speech segment. like Figure 2 As shown, the voice smoothing process is as follows: The voice of the i-th speech segment is converted into a voice, i=1…n-1, to obtain the i-th voice-modified speech segment. The i-th voice-modified speech segment is then split into the first and last segments to obtain the i-th speech beginning segment and the i-th speech ending segment. Slide the window to the (i+1)th speech segment, perform voice conversion, and obtain the (i+1)th voice-modified speech segment; then split the (i+1)th voice-modified speech segment into its beginning and end segments to obtain the (i+1)th speech beginning segment and the (i+1)th speech end segment. By using the i-th speech tail segment, the i+1-th speech head segment is smoothed to obtain the i+1-th smooth voice-changing speech segment with the smoothed speech head segment. S3. After obtaining the first voice-changing speech segment or the (i+1)th smooth voice-changing speech segment, output it in real time.

[0019] In S2, when converting voice segments, any AI voice conversion method that matches the device of the present invention can be used, as long as it can support voice conversion of voice segments.

[0020] In one embodiment of the present invention, in S2, after each time the i-th voice-changing speech segment is split into its beginning and end, the i-th speech end segment is cached, and when the (i+1)-th speech beginning segment is processed smoothly, the cached i-th speech end segment is used.

[0021] Specifically, when segmenting a voice-changing speech segment into its beginning and end, the segment with a preset beginning segment length starting from the beginning of the segment is recorded as the speech beginning segment, and the remainder is the speech end segment, with the speech beginning segment being less than or equal to the speech end segment.

[0022] Furthermore, when the length of the nth voice-changing speech segment is less than the preset head segment length, the entire nth voice-changing speech segment is taken as the nth voice head segment; the (n-1)th voice tail segment is truncated from back to front to obtain the (n-1)th voice tail segment length, which is the same as the length of the nth voice head segment, and the truncated (n-1)th voice tail segment and the nth voice head segment are subjected to smooth transition processing to obtain the nth smooth voice-changing speech segment.

[0023] The i-th speech tail segment includes the i-th body segment and the i-th tail segment located in the beginning segment, wherein the length of the i-th body segment is greater than or equal to 0, and the length of the i-th tail segment is the same as the length of the i-th speech head segment; When the length of the nth speech segment is greater than or equal to the preset head segment length, when the i+1th speech head segment is smoothed by the i-th speech tail segment, the i-th body segment is removed from the i-th speech tail segment, the i-th tail segment is retained and used, and the i+1th speech head segment is smoothed.

[0024] As an embodiment of the present invention, the smooth transition processing process is as follows: like Figure 3 As shown, the target speech beginning segment b and the target speech ending segment a are determined, and a weighted smoothing fitting process is performed on them, including: assuming the length of the target speech beginning segment b is m sample points, aligning the target speech beginning segment b and the target speech ending segment a along the time axis, and performing point-by-point weighted calculation to obtain the smoothed speech beginning segment c, whose expression is: c[j] = a[j] * [1-r] + b[j] * r; r = exp(-k * (m - j)); Where, a[j] is the discrete sample value of the target voice tail segment a, b[j] is the discrete sample value of the target voice head segment b; c[j] is the discrete sample value of the voice segment c after smooth fitting, j is the sample index, and j < m; r is the weight coefficient, m is the length of the voice sample; k is the curvature control constant.

[0025] As an embodiment of the method of the present invention, as Figure 1 shown, it includes the following steps: S101. Cut the real-time voice stream into continuous voice segments every 0.2 seconds; S102. Slide the window, process the current voice segment, starting from the first voice segment of the real-time voice; S103. In the current window, use a voice conversion method for the voice segment to convert it into a specified object voice segment, and implement the voice conversion of the current voice segment; S104. Cut the voice segment into a 0.08-second voice head and a 0.12-second voice tail; S105. Smoothly transition the voice head segment with the voice tail segment of the previous voice segment in the system cache to form a new voice head segment. If it is the first voice segment, skip it without processing.

[0026] S106. In the current window, store the voice tail in the system cache; S107. Merge the new voice head and the voice tail to form a voice-converted voice segment, S108. Output the merged voice-converted voice segment; S109. Slide the window to the next voice segment, and repeat step S102 until all voice segments are voice-converted and smoothly processed.

[0027] Corresponding to the above method, the present invention also provides a real-time voice conversion device based on a sliding window, including: A voice input module for collecting and obtaining a real-time voice stream; A voice conversion selection module for selecting whether to perform voice conversion processing on the real-time voice stream. If so, call the voice stream segmentation module; if not, directly call the voice output module to perform voice stream pass-through output; A voice decoding module for decoding the voice stream; A voice stream segmentation module for segmenting the real-time voice stream to obtain n continuous voice segments, where n > 1; A sliding window processing module for sequentially calling the voice conversion module and the smooth processing module to process the n voice segments through the sliding window according to the segmentation order; perform voice conversion and smoothing processing to obtain the first voice-converted voice segment and the (i + 1)-th smoothly voice-converted voice segment A voice conversion module for performing voice conversion on the n voice segments; The smoothing module is used to split the n voice-changing speech segments into their beginning and end segments to obtain the corresponding beginning and end segments. Then, the i-th end segment is used to perform a smooth transition on the (i+1)-th beginning segment to obtain the (i+1)-th smooth voice-changing speech segment with the smooth transitioned beginning segment. The speech encoding module is used to encode smoothly altered speech segments.

[0028] The voice output module is used to smoothly change the voice stream output.

[0029] The device of the present invention provides an open API interface, supports multiple voice changing methods, and provides a voice changing switch through a voice changing selection module. When turned on, it collects input voice, uses the method in claim 3 to realize real-time voice changing of the voice stream, and outputs it to the voice output interface.

[0030] As an embodiment of the device of the present invention, such as Figure 4 As shown, it includes the following steps: S401, Voice changer activated; S402. Check the voice changing settings. If voice changing is not enabled, proceed to step S403; if voice changing is enabled, proceed to step S404. S403, Voice changer not enabled, voice stream pass-through, outputting original voice; S404. Detect the designated speaker's voice changer and load the designated speaker's voice changer model. S405, Voice changer collects input voice stream; S406: Input voice stream is decoded into 16kPCM voice stream, supporting high-definition voice decoding such as G.722, opus, and speex. S407. The decoded speech stream is subjected to real-time speech stream voice changing and smoothing processing using the method of the present invention. S408: The voice-changing audio stream is re-encoded into the input audio encoding format, supporting high-definition audio such as G.722, opus, and speex. S409, Voice stream output.

[0031] The advantages of the real-time voice changing method and device in this invention are as follows: In real-time voice changing scenarios, after receiving a real-time audio stream, it can perform segmented voice changing. At the same time, it performs head-to-tail segmentation on the audio segments after segmented voice changing, and smoothly transitions the beginning of the current audio segment through the end of the previous audio segment. This not only solves the problem of real-time voice changing, but also solves the problem of unnatural and abrupt transitions between voice segments. Combined with offline AI voice changing methods, it realizes real-time audio stream AI voice changing, resulting in a smoother and more natural voice changing effect, greatly improving the user's actual voice changing experience.

[0032] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A real-time voice changing method based on a sliding window, characterized by: It includes the following steps: S1. Obtain a real-time voice stream, segment the real-time voice stream to obtain n consecutive voice segments, where n > 1; S2. Through a sliding window, perform voice conversion and smoothing processing on the n voice segments in sequence according to the segmentation order; The process of voice conversion and smoothing processing is as follows: Perform voice conversion on the i-th voice segment, i = 1…n - 1, to obtain the i-th voice-converted voice segment, and perform head and tail segmentation on the i-th voice-converted voice segment to obtain the i-th voice head segment and the i-th voice tail segment; Slide the window to the (i + 1)-th voice segment, perform voice conversion to obtain the (i + 1)-th voice-converted voice segment; and perform head and tail segmentation on the (i + 1)-th voice-converted voice segment to obtain the (i + 1)-th voice head segment and the (i + 1)-th voice tail segment; when performing head and tail segmentation on the voice-converted voice segment, the segment with a preset head segment length starting from the segment head is recorded as the voice head segment, and the rest is the voice tail segment, and the voice head segment is less than or equal to the voice tail segment; Perform smooth transition processing on the (i + 1)-th voice head segment through the i-th voice tail segment to obtain the (i + 1)-th smoothly voice-converted voice segment with the smoothly processed voice head segment; when the length of the n-th voice-converted voice segment is less than the preset head segment length, the entire n-th voice-converted voice segment is used as the n-th voice head segment; intercept the n - 1 voice tail segment with the same length as the n-th voice head segment from the n - 1 voice tail segment from back to front, and perform smooth transition processing through the intercepted n - 1 voice tail segment and the n-th voice head segment to obtain the n-th smoothly voice-converted voice segment; the i-th voice tail segment includes the i-th body segment and the i-th tail segment located in the front segment, where the length of the i-th body segment is greater than or equal to 0, and the length of the i-th tail segment is the same as the length of the i-th voice head segment; when the length of the n-th voice segment is greater than or equal to the preset head segment length, when performing smooth transition processing on the (i + 1)-th voice head segment through the i-th voice tail segment, for the i-th voice tail segment, remove the i-th body segment, retain and use the i-th tail segment, and perform smooth transition processing on the (i + 1)-th voice head segment; the process of smooth transition processing is: determine the target voice head segment b and the target voice tail segment a, and perform weighted smooth fitting processing on the two, including: assuming that the length of the target voice head segment b is m, align the target voice head segment b and the target voice tail segment a along the time axis, and calculate point by point with weights to obtain the smoothly processed voice head segment c, and its expression is: c[j] = a[j] * [1 - r] + b[j] * r; r = exp(-k * (m - j)); where, a[j] is the discrete sample value of the target voice tail segment a, b[j] is the discrete sample value of the target voice head segment b; c[j] is the discrete sample value of the smoothly fitted voice head segment c, j is the sample index, and j < m; r is the weight coefficient; k is the curvature control constant; S3. After obtaining the (i + 1)-th smoothly voice-converted voice segment, output it in real time.

2. The real-time voice changing method based on a sliding window according to claim 1, characterized in that: In S1, segment the real-time voice stream according to a preset fixed duration.

3. The real-time voice changing method based on a sliding window according to claim 2, characterized in that: S2 In it, each time after the head and tail of the i-th voice conversion voice segment are segmented, the obtained i-th voice tail segment is cached, and when the i+1-th voice head segment is subjected to a smooth transition process, it is executed through the cached i-th voice tail segment.

4. A real-time voice changing device based on a sliding window, characterized in that: It includes: A voice input module for collecting and obtaining a real-time voice stream; A voice stream segmentation module for segmenting the real-time voice stream to obtain n consecutive voice segments, where n>1; A sliding window processing module for sequentially calling a voice conversion module and a smoothing processing module to process n voice segments in the segmentation order through a sliding window; performing voice conversion and smoothing processing to obtain the first voice conversion voice segment and the i+1-th smoothed voice conversion voice segment; A voice conversion module for performing voice conversion on n voice segments; A smoothing processing module for performing head and tail segmentation on n voice conversion voice segments to obtain corresponding voice head segments and voice tail segments, and performing a smooth transition process on the i+1-th voice head segment through the i-th voice tail segment to obtain the i+1-th smoothed voice conversion voice segment with a smoothly transitioned voice head segment; when performing head and tail segmentation on the voice conversion voice segment, a segment with a preset head segment length starting from the segment head is recorded as the voice head segment, and the remainder is the voice tail segment, and the voice head segment is less than or equal to the voice tail segment; when the length of the n-th voice conversion voice segment is less than the preset head segment length, the entire n-th voice conversion voice segment is used as the n-th voice head segment; the n-1-th voice tail segment is intercepted from back to front to obtain the n-1-th voice tail segment length that is the same as the length of the n-th voice head segment, and a smooth transition process is performed through the intercepted n-1-th voice tail segment and the n-th voice head segment to obtain the n-th smoothed voice conversion voice segment; the i-th voice tail segment includes the i-th body segment and the i-th tail segment located in the front segment, where the length of the i-th body segment is greater than or equal to 0, and the length of the i-th tail segment is the same as the length of the i-th voice head segment; when the length of the n-th voice segment is greater than or equal to the preset head segment length, when performing a smooth transition process on the i+1-th voice head segment through the i-th voice tail segment, for the i-th voice tail segment, the i-th body segment is removed, and the i-th tail segment is retained and used to perform a smooth transition process on the i+1-th voice head segment; the process of the smooth transition process is: determining a target voice head segment b and a target voice tail segment a, and performing a weighted smooth fitting process on the two, including: setting the length of the target voice head segment b as m, aligning the target voice head segment b and the target voice tail segment a along the time axis, and performing weighted calculation point by point to obtain the smoothly transitioned voice head segment c, and its expression is: c[j]=a[j]*[1-r]+b[j]*r; r = exp(-k*(m-j)); where, a[j] is the discrete sample value of the target voice tail segment a, b[j] is the discrete sample value of the target voice head segment b; c[j] is the discrete sample value of the smoothly fitted voice head segment c, j is the sample index, and j < m; r is the weight coefficient; k is the curvature control constant; The voice output module is used to output the (i+1)th smoothly morphed voice segment.

Images