Real-time voice changing method and device based on sliding window

Abstract

The invention relates to the technical field of voice changing processing, and discloses a real-time voice changing method and device based on a sliding window, and the key point of the technical scheme is that the method comprises the following steps: S1, obtaining a real-time voice stream, and segmenting the real-time voice stream to obtain n voice segments; s2, through a sliding window, sequentially carrying out voice change smoothing processing on the n voice segments, carrying out voice change conversion on the i-th voice segment to obtain an i-th voice change voice segment, and carrying out head and tail segmentation to obtain an i-th voice head segment and an i-th voice tail segment; performing voice change conversion on the (i + 1) th voice segment to obtain an (i + 1) th voice change voice segment, and performing head and tail segmentation to obtain an (i + 1) th voice head segment and an (i + 1) th voice tail segment; performing smooth transition processing on the (i + 1) th voice head segment through the ith voice tail segment to obtain an (i + 1) th smooth voice change voice segment; and S3, after the first voice changing voice segment or the (i + 1) th smooth voice changing voice segment is obtained, real-time output is carried out.

Description

Technical Field

[0001] The present invention relates to the technical field of voice change processing, and more particularly to a real-time voice change method and device based on a sliding window. Background Art

[0002] Voice changing is a long-standing technology. Before the emergence of AI voice changing, it was mainly achieved through frequency domain operations on the sound. However, the voice changing effect of this traditional technology is very mechanical and unnatural. It is easy to hear that the voice is changed by software, which greatly limits the scope of application of this technology.

[0003] AI voice changing is an upgraded product of traditional voice changing technology. It is developed from AI speech synthesis technology. Since 2016, the academic community has conducted more and more research on deep learning voice changing technology. After years of research, AI voice changing technology has made great progress, and the voice changing effect in experimental scenarios has achieved good results.

[0004] Existing AI voice-changing systems typically use offline, complete audio files for conversion. To address real-time voice-changing requirements, these systems must only be used offline to convert audio files in a small window, then splice the audio segments together to create a voice-changed stream. While this approach solves the problem of real-time voice-changing, the segmented voice-changing process can create unnatural and abrupt transitions between segments, impacting the user experience. Summary of the Invention

[0005] The purpose of the present invention is to provide a real-time voice changing method and device based on a sliding window, which realizes real-time AI voice changing of voice streams, makes the voice changing effect smoother and more natural, and greatly improves the user's actual voice changing experience.

[0006] The above technical objectives of the present invention are achieved through the following technical solutions: A real-time voice changing method based on a sliding window, comprising the following steps:

[0007] S1. Acquire a real-time voice stream and segment the real-time voice stream to obtain n continuous voice segments, where n>1;

[0008] S2. Perform voice change and smoothing processing on n speech segments in sequence through a sliding window according to the segmentation order;

[0009] The process of voice changing and smoothing is as follows:

[0010] Perform voice change conversion on the i-th voice segment, where i=1…n-1, to obtain the i-th voice-changed voice segment, and perform head-to-tail segmentation on the i-th voice-changed voice segment to obtain the i-th voice head segment and the i-th voice tail segment;

[0011] Slide the window to the i+1th voice segment and perform voice change conversion to obtain the i+1th voice-changed voice segment; and split the i+1th voice-changed voice segment into the beginning and the end to obtain the i+1th voice head segment and the i+1th voice tail segment;

[0012] Performing smooth transition processing on the (i+1)th speech head segment through the (i)th speech tail segment to obtain the (i+1)th smoothly voice-changed speech segment with the speech head segment after smooth transition processing;

[0013] S3. After obtaining the first voice-changed speech segment or the (i+1)th smoothly voice-changed speech segment, output it in real time.

[0014] As a preferred technical solution of the present invention, in S1, the real-time voice stream is segmented according to a preset fixed duration.

[0015] As a preferred technical solution of the present invention, in S2, each time the i-th voice-changed speech segment is split into the beginning and the end, the obtained i-th voice tail segment is cached, and when the i+1-th voice head segment is smoothly transitioned, it is executed through the cached i-th voice tail segment.

[0016] As an optimal technical solution of the present invention, when the voice-changed speech segment is divided into the head and tail, the segment of the preset head segment length starting from the segment head is recorded as the speech head segment, and the rest is the speech tail segment, and the speech head segment is less than or equal to the speech tail segment.

[0017] As a preferred technical solution of the present invention, when the length of the nth voice-changed speech segment is less than the preset head segment length, the entire nth voice-changed speech segment is used as the nth voice head segment; the n-1th voice tail segment is cut from back to front to obtain the n-1th voice tail length that is the same as the length of the nth voice head segment, and a smooth transition processing is performed on the cut n-1th voice tail segment and the nth voice head segment to obtain the nth smooth voice-changed speech segment.

[0018] As a preferred technical solution of the present invention, the i-th voice tail segment includes the i-th body segment located in the front segment and the i-th tail segment, wherein the length of the i-th body segment is greater than or equal to 0, and the length of the i-th tail segment is the same as the length of the i-th voice head segment;

[0019] When the length of the nth voice segment is greater than or equal to the preset head segment length, when the i+1th voice head segment is smoothly transitioned through the i-th voice tail segment, the i-th body segment is removed from the i-th voice tail segment, the i-th tail segment is retained and used, and the i+1th voice head segment is smoothly transitioned.

[0020] As a preferred technical solution of the present invention, in S2, the process of smooth transition processing is:

[0021] Determine the target voice head segment b and the target voice tail segment a, and perform weighted smoothing fitting on both of them, including: assuming the length of the target voice head b is m sample points, align the target voice head segment b and the target voice tail segment a along the time axis, and calculate point-by-point weighting to obtain the voice head segment c after smooth transition processing, and its expression is:

[0022] c[j] = a[j] * [1 - r] + b[j] * r;

[0023] r = exp(-k * (n - j));

[0024] Where, a[j] is the discrete sample value of the target voice tail segment a, b[j] is the discrete sample value of the target voice head segment b; c[j] is the discrete sample value of the voice segment c after smooth fitting, j is the sample index, and j < m; r is the weight coefficient, m is the length of the voice sample; k is the curvature control constant.

[0025] A real-time voice conversion device based on a sliding window, including:

[0026] A voice input module, used to collect and obtain a real-time voice stream;

[0027] A voice stream segmentation module, used to segment the real-time voice stream to obtain n consecutive voice segments, where n > 1;

[0028] A sliding window processing module, used to sequentially call a voice conversion module and a smoothing processing module to process n voice segments in the segmentation order through a sliding window; perform voice conversion and smoothing processing to obtain the first voice conversion segment and the (i + 1)-th smoothed voice conversion segment

[0029] A voice conversion module, used to perform voice conversion on n voice segments;

[0030] A smoothing processing module, used to perform head and tail segmentation on n voice conversion segments to obtain the corresponding voice head segment and voice tail segment, and perform smooth transition processing on the (i + 1)-th voice head segment through the i-th voice tail segment to obtain the (i + 1)-th smoothed voice conversion segment with a smoothed voice head segment;

[0031] A voice output module, used to output according to the smoothed voice conversion segment.

[0032] As a preferred technical solution of the present invention, the real-time voice conversion device further includes:

[0033] A voice conversion selection module, used to select whether to perform voice conversion on the real-time voice stream. If so, call the voice stream segmentation module; if not, directly call the voice output module to perform voice stream transparent transmission output;

[0034] A voice decoding module, used for decoding the voice stream;

[0035] The speech coding module is used to encode the smoothly changed voice segments.

[0036] In summary, the present invention has the following beneficial effects: in the real-time voice changing scenario, after receiving the real-time voice stream, it can perform segmented voice changing, and at the same time, the voice segments after the segmented voice changing are divided into the beginning and the end, and the segment head of the current segment of the voice segment is smoothly transitioned through the segment tail of the previous segment of the voice stage. It not only solves the problem of real-time voice changing, but also solves the problem that the connection between the voice-changed voice segments is not natural and abrupt. Combined with the offline AI voice changing method, real-time voice stream AI voice changing is realized, and the voice changing effect is smoother and more natural, which greatly improves the user's actual voice changing experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a schematic diagram of the real-time speech flow process of the present invention

[0038] Figure 2 This is a schematic diagram of the real-time speech rheological processing of the present invention;

[0039] Figure 3 This is a schematic diagram of the smooth transition processing of speech segments according to the present invention;

[0040] Figure 4 The figure is a flow chart of the real-time voice changing device of the present invention. DETAILED DESCRIPTION

[0041] The present invention will be further described in detail below with reference to the accompanying drawings.

[0042] The present invention provides a real-time voice changing method based on a sliding window, comprising the following steps:

[0043] S1. Obtain a real-time voice stream and segment the real-time voice stream according to a preset fixed time length to obtain n continuous voice segments, where n>1;

[0044] S2. Perform voice-changing and smoothing processing on n speech segments in sequence through a sliding window to obtain the first voice-changing speech segment and the (i+1)th smooth voice-changing speech segment;

[0045] like Figure 2 As shown in the figure, the process of voice change smoothing is as follows:

[0046] Perform voice change conversion on the i-th voice segment, where i=1…n-1, to obtain the i-th voice-changed voice segment, and perform head-to-tail segmentation on the i-th voice-changed voice segment to obtain the i-th voice head segment and the i-th voice tail segment;

[0047] Slide the window to the i+1th voice segment and perform voice change conversion to obtain the i+1th voice-changed voice segment; and split the i+1th voice-changed voice segment into the beginning and the end to obtain the i+1th voice head segment and the i+1th voice tail segment;

[0048] Performing smooth transition processing on the (i+1)th speech head segment through the (i)th speech tail segment to obtain the (i+1)th smoothly voice-changed speech segment with the speech head segment after smooth transition processing;

[0049] S3. After obtaining the first voice-changed speech segment or the (i+1)th smoothly voice-changed speech segment, output it in real time.

[0050] In S2, when performing voice change conversion on the speech segment, any AI voice change processing method that matches the device of the present invention can be used, as long as it can support the voice change conversion of the speech segment.

[0051] As an embodiment of the present invention, in S2, each time the i-th voice-changed speech segment is split into the beginning and the end, the obtained i-th voice tail segment is cached, and when the i+1-th voice head segment is smoothly transitioned, it is executed using the cached i-th voice tail segment.

[0052] Specifically, when the voice-changed speech segment is divided into the beginning and the end, the segment of the preset head segment length starting from the segment head is recorded as the speech head segment, and the rest is the speech tail segment, and the speech head segment is less than or equal to the speech tail segment.

[0053] Furthermore, when the length of the nth voice-changing speech segment is less than the preset head segment length, the entire nth voice-changing speech segment is used as the nth speech head segment; the n-1th speech tail segment is cut out from back to front to obtain the n-1th speech tail length that is the same as the length of the nth speech head segment, and the cut-out n-1th speech tail segment and the nth speech head segment are smoothly transitioned to obtain the nth smooth voice-changing speech segment.

[0054] The i-th voice tail segment includes the i-th body segment and the i-th tail segment located in the front segment, wherein the length of the i-th body segment is greater than or equal to 0, and the length of the i-th tail segment is the same as the length of the i-th voice head segment;

[0055] When the length of the nth voice segment is greater than or equal to the preset head segment length, when the i+1th voice head segment is smoothly transitioned through the i-th voice tail segment, the i-th body segment is removed from the i-th voice tail segment, the i-th tail segment is retained and used, and the i+1th voice head segment is smoothly transitioned.

[0056] As an embodiment of the present invention, the process of smooth transition processing is as follows:

[0057] like Figure 3As shown, the target speech head segment b and the target speech tail segment a are determined, and weighted smoothing fitting processing is performed on both, including: setting the length of the target speech head b as m sample points, aligning the target speech head segment b and the target speech tail segment a along the time axis, and performing point-by-point weighted calculation to obtain the speech head segment c after smooth transition processing, and its expression is:

[0058] c[j] = a[j] * [1 - r] + b[j] * r;

[0059] r = exp(-k * (n - j));

[0060] Where, a[j] is the discrete sample value of the target speech tail segment a, b[j] is the discrete sample value of the target speech head segment b; c[j] is the discrete sample value of the speech segment c after smooth fitting, j is the sample index, and j < m; r is the weight coefficient, m is the speech sample length; k is the curvature control constant.

[0061] As an embodiment of the method of the present invention, as Figure 1 shown, the following steps are included:

[0062] S101. Cut the real-time speech stream into continuous speech segments every 0.2 seconds;

[0063] S102. Slide the window, process the current speech segment, starting from the first speech segment of the real-time speech;

[0064] S103. In the current window, use a voice conversion method for the speech segment to convert it into a specified object speech segment to achieve voice conversion of the current speech segment;

[0065] S104. Cut the speech segment into a 0.08-second speech head and a 0.12-second speech tail;

[0066] S105. Perform smooth transition processing on the speech head segment and the speech tail segment of the previous speech segment in the system cache to form a new speech head segment. If it is the first speech segment, skip it without processing.

[0067] S106. In the current window, store the speech tail in the system cache;

[0068] S107. Merge the new speech head and the speech tail to form a voice-converted speech segment,

[0069] S108. Output the merged voice-converted speech segment;

[0070] S109. Slide the window to the next speech segment, and repeat step S102 until all speech segments are voice-converted and smooth processed.

[0071] Corresponding to the above method, the present invention also provides a real-time voice changing device based on a sliding window, comprising:

[0072] Voice input module, used to collect and obtain real-time voice stream;

[0073] The voice change selection module is used to select whether to perform voice change processing on the real-time voice stream. If so, the voice stream segmentation module is called; if not, the voice output module is directly called to transparently transmit the voice stream out;

[0074] A voice decoding module, used for decoding the voice stream;

[0075] The voice stream segmentation module is used to segment the real-time voice stream into n continuous voice segments, where n>1;

[0076] The sliding window processing module is used to call the voice change conversion module and the smoothing processing module to process n voice segments in sequence through the sliding window; perform voice change and smoothing processing to obtain the first voice change voice segment and the (i+1) smooth voice change voice segment.

[0077] A voice change conversion module is used to change the voice of n speech segments;

[0078] A smoothing processing module is used to split the n voice-changed speech segments into head and tail segments to obtain corresponding speech head segments and speech tail segments, and to perform smooth transition processing on the (i+1)th speech head segment through the (i)th speech tail segment to obtain the (i+1)th smooth voice-changed speech segment with the smooth transition processed speech head segment;

[0079] The speech coding module is used to encode the smoothly changed voice segments.

[0080] The voice output module is used to output a smooth voice-changed voice stream.

[0081] The device of the present invention provides an open API interface, supports multiple voice changing methods, provides a voice changing switch through a voice changing selection module, collects input voice after turning it on, uses the method in claim 3 to realize real-time voice stream voice changing, and outputs it to the voice output interface.

[0082] As an embodiment of the device of the present invention, Figure 4 As shown, the following steps are included:

[0083] S401, voice changing device turned on;

[0084] S402, check the voice change setting, if the voice change is not enabled, execute step S403; if the voice change is enabled, execute step S404;

[0085] S403: Voice changing is not enabled, the voice stream is transparently transmitted, and the original voice is output;

[0086] S404, detecting a designated speaker with a changed voice, and loading a voice change model for the designated speaker;

[0087] S405, the voice changing device collects the input voice stream;

[0088] S406, decode the input voice stream into 16kPCM voice stream, support G.722, opus, speex and other high-definition voice decoding;

[0089] S407: The decoded voice stream is subjected to real-time voice stream de-sounding and smoothing processing using the method of the present invention;

[0090] S408, the voice stream after voice change is re-encoded into the input voice coding format, supporting high-definition voice such as G.722, opus, speex, etc.

[0091] S409: Output the voice stream.

[0092] The advantages of the real-time voice changing method and device in the present invention are: in the real-time voice changing scenario, after receiving the real-time voice stream, it can perform segmented voice changing, and at the same time, the voice segments after the segmented voice changing are divided into the beginning and the end, and the segment head of the current segment of the voice segment is smoothly transitioned through the segment tail of the previous segment of the voice stage. This not only solves the problem of real-time voice changing, but also solves the problem that the connection between the voice-changed voice segments is not natural and abrupt. Combined with the offline AI voice changing method, real-time voice stream AI voice changing is realized, and the voice changing effect is smoother and more natural, which greatly improves the user's actual voice changing experience.

[0093] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiment. All technical solutions based on the concept of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A real-time voice changing method based on a sliding window, characterized by: The steps include: S1. Acquire a real-time voice stream and segment the real-time voice stream to obtain n continuous voice segments, where n>1; S2. Perform voice change and smoothing processing on n speech segments in sequence through a sliding window according to the segmentation order; The process of voice changing and smoothing is as follows: Perform voice change conversion on the i-th voice segment, where i=1…n-1, to obtain the i-th voice-changed voice segment, and perform head-to-tail segmentation on the i-th voice-changed voice segment to obtain the i-th voice head segment and the i-th voice tail segment; Slide the window to the i+1th voice segment and perform voice change conversion to obtain the i+1th voice-changed voice segment; and split the i+1th voice-changed voice segment into the beginning and the end to obtain the i+1th voice head segment and the i+1th voice tail segment; Performing smooth transition processing on the (i+1)th speech head segment through the (i)th speech tail segment to obtain the (i+1)th smoothly voice-changed speech segment with the speech head segment after smooth transition processing; S3. After obtaining the first voice-changed speech segment or the (i+1)th smoothly voice-changed speech segment, output it in real time.

2. The real-time voice changing method based on a sliding window according to claim 1, wherein: In S1, the real-time voice stream is segmented according to a preset fixed duration.

3. The real-time voice changing method based on sliding window according to claim 2, characterized in that: S2 In the embodiment, each time the i-th voice-changed speech segment is divided into the beginning and the end, the obtained i-th speech tail segment is cached, and when the i+1-th speech head segment is smoothly transitioned, the cached i-th speech tail segment is used for execution.

4. A real-time voice changing method based on sliding window according to claim 3, characterized in that: When the voice-changed speech segment is divided into the head and tail, the segment of the preset head segment length starting from the segment head is recorded as the speech head segment, and the rest is the speech tail segment, and the speech head segment is less than or equal to the speech tail segment.

5. The real-time voice changing method based on sliding window according to claim 4, characterized in that: When the length of the nth voice-changed speech segment is less than the preset head segment length, the entire nth voice-changed speech segment is used as the nth voice head segment; the n-1th voice tail segment is cut from back to front to obtain the n-1th voice tail length that is the same as the length of the nth voice head segment, and a smooth transition process is performed between the cut n-1th voice tail segment and the nth voice head segment to obtain the nth smooth voice-changed speech segment.

6. The real-time voice changing method based on sliding window according to claim 5, characterized in that: The i-th voice tail segment includes the i-th body segment and the i-th tail segment located in the front segment, wherein the length of the i-th body segment is greater than or equal to 0, and the length of the i-th tail segment is the same as the length of the i-th voice head segment; When the length of the nth voice segment is greater than or equal to the preset head segment length, when the i+1th voice head segment is smoothly transitioned through the i-th voice tail segment, the i-th body segment is removed from the i-th voice tail segment, the i-th tail segment is retained and used, and the i+1th voice head segment is smoothly transitioned.

7. The real-time voice changing method based on sliding window according to claim 6, wherein: in S2, the process of smooth transition processing is: Determine the target speech head segment b and the target speech tail segment a, and perform weighted smooth fitting processing on the two, including: Assume that the length of the target speech header b is m sample points. Align the target speech header segment b and the target speech tail segment a along the time axis and perform weighted calculation point by point to obtain the speech header segment c after smooth transition processing. Its expression is: c[j]=a[j]*[1-r]+b[j]*r; r = exp(-k*(nj)); Where, a[j] is the discrete sample value of the target voice tail segment a, and b[j] is the discrete sample value of the target voice head segment b; c[j] is the discrete sample value of the voice segment c after smooth fitting, j is the sample index, and j < m; r is the weight coefficient, m is the length of the voice sample; k is the curvature control constant.

8. A real-time voice changing device based on a sliding window, characterized by: It includes: A voice input module for collecting and obtaining a real-time voice stream; A voice stream segmentation module for segmenting the real-time voice stream to obtain n consecutive voice segments, where n > 1; A sliding window processing module for sequentially calling a voice conversion module and a smoothing processing module to process the n voice segments in the segmentation order through a sliding window; performing voice conversion and smoothing processing to obtain the first voice-converted segment and the (i + 1)-th smoothed voice-converted segment A voice conversion module for performing voice conversion on the n voice segments; A smoothing processing module for performing head and tail segmentation on the n voice-converted segments to obtain the corresponding voice head segment and voice tail segment, and performing smooth transition processing on the (i + 1)-th voice head segment through the i-th voice tail segment to obtain the (i + 1)-th smoothed voice-converted segment with a smoothed voice head segment; A voice output module for outputting the smoothed voice-converted segment according to the output.

9. The real-time voice conversion device according to claim 1, wherein: The real-time voice conversion device further includes: A voice conversion selection module for selecting whether to perform voice conversion on the real-time voice stream. If so, it calls the voice stream segmentation module; if not, it directly calls the voice output module to perform voice stream pass-through output; A voice decoding module for decoding the voice stream; A voice encoding module for encoding the smoothed voice-converted segment.

Images