Time delay determination method based on voice alignment and related device

Abstract

The invention discloses a time delay determination method based on voice alignment and a related device, and the method comprises the steps: respectively segmenting a standard voice signal of a standard channel and a participation voice signal of a participation channel into a plurality of standard signal segments and a plurality of participation signal segments in a voice alignment scene; and carrying out pitch period calculation on each frame of signal to obtain a plurality of standard pitch period strings corresponding to the plurality of standard signal segments and a plurality of participating pitch period strings corresponding to the plurality of participating signal segments. And for a to-be-matched pitch period string in the plurality of standard pitch period strings and a plurality of candidate pitch period strings in the plurality of participating pitch period strings, performing dynamic time warping to obtain a plurality of distances between the to-be-matched pitch period string and the plurality of candidate pitch period strings. And when the minimum distance in the plurality of distances meets a preset similarity condition, determining more accurate time delay of the participating channel relative to the standard channel through the first alignment point time of the pitch period string to be matched and the second alignment point time of the candidate pitch period string corresponding to the minimum distance.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method for determining a delay based on speech alignment and a related device. Background Art

[0002] With the rapid development of computer technology and Internet technology, multiple users can connect to the Internet through multiple terminals for voice interaction. In the voice alignment scenario, multiple voice data corresponding to multiple terminals need to be transmitted to the same device for voice alignment. There are large delay differences between multiple voice data transmitted by multiple terminals, which affects the voice alignment effect.

[0003] In the related art, the voice data transmitted by each terminal carries a timestamp, which is obtained by correcting the local clock of the terminal through a reference clock; based on this, voice alignment is performed on multiple voice data according to multiple timestamps carried by multiple voice data transmitted by multiple terminals.

[0004] However, in the above method, the reference clock is obtained through network transmission, and there are delay differences in the reference clocks obtained by different terminals, and the reference clock cannot correct the processing delay of the terminal and the pronunciation delay of the voice, etc., resulting in the alignment of multiple voice data through multiple timestamps carried by multiple voice data is not accurate enough, resulting in poor voice alignment effect. Summary of the invention

[0005] In order to solve the above technical problems, the present application provides a delay determination method based on speech alignment and related devices, which can determine a more accurate delay of a participating channel relative to a standard channel; based on this, the participating speech signal can be more accurately aligned with the standard speech signal through the delay, thereby improving the speech alignment effect.

[0006] The embodiments of the present application disclose the following technical solutions:

[0007] On the one hand, an embodiment of the present application provides a method for determining a delay based on speech alignment, the method comprising:

[0008] In the speech alignment scenario, the standard speech signal of the standard channel and the participating speech signal of the participating channel are segmented to obtain multiple standard signal segments and multiple participating signal segments;

[0009] For the plurality of standard signal segments and the plurality of participating signal segments, the pitch period calculation is performed on each frame signal to obtain a plurality of standard pitch period strings corresponding to the plurality of standard signal segments and a plurality of participating pitch period strings corresponding to the plurality of participating signal segments;

[0010] Dynamic time warping is performed on a pitch cycle string to be matched among the plurality of standard pitch cycle strings and a plurality of candidate pitch cycle strings within a preset time period from a preset moment among the plurality of participating pitch cycle strings, to obtain a plurality of distances between the pitch cycle string to be matched and the plurality of candidate pitch cycle strings; the preset moment is determined according to a plurality of moments corresponding to the pitch cycle string to be matched;

[0011] If the minimum distance among the multiple distances meets the preset similarity condition, the delay of the participating channel relative to the standard channel is determined according to the first alignment point time of the fundamental pitch cycle string to be matched and the second alignment point time of the candidate fundamental pitch cycle string corresponding to the minimum distance.

[0012] On the other hand, an embodiment of the present application provides a delay determination device based on speech alignment, the device comprising: a segmentation unit, a calculation unit, a regularization unit and a determination unit;

[0013] The segmentation unit is used to perform signal segmentation on the standard speech signal of the standard channel and the participating speech signal of the participating channel in the speech alignment scenario to obtain multiple standard signal segments and multiple participating signal segments;

[0014] The calculation unit is used to calculate the pitch period of each frame signal for the multiple standard signal segments and the multiple participating signal segments, and obtain a plurality of standard pitch period strings corresponding to the multiple standard signal segments and a plurality of participating pitch period strings corresponding to the multiple participating signal segments;

[0015] The regularization unit is used to perform dynamic time regularization on the pitch cycle string to be matched in the multiple standard pitch cycle strings and the multiple candidate pitch cycle strings within a preset time period from the preset moment in the multiple participating pitch cycle strings, so as to obtain multiple distances between the pitch cycle string to be matched and the multiple candidate pitch cycle strings; the preset moment is determined according to the multiple moments corresponding to the pitch cycle string to be matched;

[0016] The determination unit is used to determine the delay of the participating channel relative to the standard channel according to the first alignment point time of the fundamental pitch cycle string to be matched and the second alignment point time of the candidate fundamental pitch cycle string corresponding to the minimum distance if the minimum distance among the multiple distances meets the preset similarity condition.

[0017] On the other hand, an embodiment of the present application provides a computer device, the computer device comprising a processor and a memory:

[0018] The memory is used to store a computer program and transmit the computer program to the processor;

[0019] The processor is configured to execute the method described in any one of the preceding aspects according to instructions in the computer program.

[0020] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which is used to store a computer program. When the computer program is executed on a computer device, the computer device executes the method described in any one of the aforementioned aspects.

[0021] On the other hand, an embodiment of the present application provides a computer program product, including a computer program, which, when executed on a computer device, enables the computer device to execute the method described in any one of the aforementioned aspects.

[0022] It can be seen from the above technical scheme that in the speech alignment scenario, first, the standard speech signal of the standard channel is divided into multiple standard signal segments, and the participating speech signal of the participating channel is divided into multiple participating signal segments; secondly, the pitch period is calculated for each frame signal in the multiple standard signal segments to obtain multiple standard pitch period strings corresponding to the multiple standard signal segments, and the pitch period is calculated for each frame signal in the multiple participating signal segments to obtain multiple participating pitch period strings corresponding to the multiple participating signal segments; the pitch period calculation method obtains the speech features of each signal segment based on the pronunciation features of the valid speech segment. Then, the pitch period string to be matched is selected from the multiple standard pitch period strings, the preset time is determined by the multiple moments corresponding to the pitch period string to be matched, and multiple candidate pitch period strings within the preset time period from the multiple participating pitch period strings are selected; the pitch period string to be matched and the multiple candidate pitch period strings are dynamically time-warped to obtain multiple distances between the pitch period string to be matched and the multiple candidate pitch period strings; based on the speech features of each signal segment, the dynamic time warping method provides a data basis for determining the signal segment of the participating channel that is most similar in speech to the standard channel. Finally, when the minimum distance among multiple distances meets the preset similarity condition, the signal segment of the participating channel that is most similar in voice to the standard channel is determined. Through the first alignment point time of the fundamental pitch cycle string to be matched and the second alignment point time of the candidate fundamental pitch cycle string corresponding to the minimum distance, not only the transmission delay of the terminal but also the processing delay of the terminal and the pronunciation delay of the voice are considered to determine a more accurate delay of the participating channel relative to the standard channel; based on this, the participating voice signal can be more accurately aligned with the standard voice signal through this delay, thereby improving the voice alignment effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technical members in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0024] Figure 1 A schematic diagram of a speech alignment process provided for related technologies;

[0025] Figure 2 A schematic diagram of the system architecture of a method for determining a delay based on speech alignment provided in an embodiment of the present application;

[0026] Figure 3 A flowchart of a method for determining a delay based on speech alignment provided in an embodiment of the present application;

[0027] Figure 4 A schematic diagram of the distance between a pitch cycle string to be matched and a candidate pitch cycle string provided in an embodiment of the present application;

[0028] Figure 5 A schematic diagram of an overall speech alignment process provided in an embodiment of the present application;

[0029] Figure 6 A schematic diagram of a speech alignment scenario provided in an embodiment of the present application;

[0030] Figure 7 A structural diagram of a delay determination device based on speech alignment provided in an embodiment of the present application;

[0031] Figure 8 A structural diagram of a server provided in an embodiment of the present application;

[0032] Fig. 9 A structural diagram of a terminal provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] The embodiments of the present application are described below in conjunction with the accompanying drawings.

[0034] At present, in the voice alignment scenario, multiple voice data corresponding to multiple terminals need to be transmitted to the same device for voice alignment. Figure 1 , Figure 1 A schematic diagram of a voice alignment process provided for related technologies; wherein, in order to avoid the large delay differences between multiple voice data transmitted by multiple terminals affecting the voice alignment effect, the local device can obtain a timestamp when starting recording, and calculate the deviation of the recording start time relative to the reference clock to correct the timestamp. The local device can also obtain the corrected timestamp carried by the voice data sent by the remote device; based on this, the voice data of the local device and the voice data sent by the remote device are aligned by aligning the corrected timestamp.

[0035] However, after research, it was found that the reference clock is obtained through network transmission, and there is a delay difference between the reference clocks obtained by the local device and the remote device, and the reference clock cannot correct the processing delay of the device and the pronunciation delay of the voice, etc., resulting in the alignment of the corrected timestamps to align the voice data of the local device and the voice data sent by the remote device is not accurate enough, resulting in poor voice alignment effect.

[0036] The embodiment of the present application provides a method for determining a time delay based on speech alignment. In a speech alignment scenario, a standard speech signal of a standard channel is divided into a plurality of standard signal segments, and a participating speech signal of a participating channel is divided into a plurality of participating signal segments; a pitch period calculation is performed on each frame signal in the plurality of standard signal segments to obtain a plurality of standard pitch period strings corresponding to the plurality of standard signal segments, and a pitch period calculation is performed on each frame signal in the plurality of participating signal segments to obtain a plurality of participating pitch period strings corresponding to the plurality of participating signal segments; the pitch period calculation method obtains the speech features of each signal segment based on the pronunciation features of the valid speech segment. A pitch period string to be matched is selected from the plurality of standard pitch period strings, a preset moment is determined from a plurality of moments corresponding to the pitch period string to be matched, and a plurality of candidate pitch period strings within a preset time period from the plurality of participating pitch period strings are selected; dynamic time warping is performed on the pitch period string to be matched and a plurality of candidate pitch period strings to obtain a plurality of distances between the pitch period string to be matched and the plurality of candidate pitch period strings; based on the speech features of each signal segment, the dynamic time warping method provides a data basis for determining the signal segment of the participating channel that is most similar in speech to the standard channel. When the minimum distance among multiple distances meets the preset similarity condition, the signal segment of the participating channel that is most similar in voice to the standard channel is determined. Through the first alignment point time of the fundamental pitch cycle string to be matched and the second alignment point time of the candidate fundamental pitch cycle string corresponding to the minimum distance, not only the transmission delay of the terminal but also the processing delay of the terminal and the pronunciation delay of the voice are considered to determine a more accurate delay of the participating channel relative to the standard channel; based on this, the participating voice signal can be more accurately aligned with the standard voice signal through this delay, thereby improving the voice alignment effect.

[0037] Next, the system architecture of the delay determination method based on speech alignment will be introduced. Figure 2 , Figure 2 A system architecture diagram of a delay determination method based on voice alignment provided in an embodiment of the present application, wherein the system architecture includes a first terminal 201, a second terminal 202 and a server 203, wherein the first terminal 201 transmits standard voice data to the server 203, and the server 203 receives the standard voice data through a standard channel, and the second terminal 202 transmits participating voice data to the server 203, and the server 203 receives the participating voice data through a participating channel.

[0038] Among them, the first terminal 201 or the second terminal 202 can be a smart phone, a tablet computer, a laptop computer, a desktop computer, an intelligent voice interaction device, a vehicle terminal or an aircraft, etc., but is not limited to this. The server 203 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services, but is not limited to this. The first terminal 201 and the server 203, the second terminal 202 and the server 203 can be directly or indirectly connected via wired or wireless communication, and this application does not limit this. For example, the terminal and the server can be connected via a network, and the network can be a wired or wireless network.

[0039] The server 203 decodes the standard voice data received by the standard channel into a standard voice signal, and decodes the participating voice data received by the participating channel into a participating voice signal; in the voice alignment scenario, the server 203 performs signal segmentation on the standard voice signal of the standard channel and the participating voice signal of the participating channel to obtain multiple standard signal segments and multiple participating signal segments. As an example, the voice alignment scenario is an online chorus scenario, the standard voice data is standard singing data, and the participating voice data is participating singing data. The server 203 can decode the standard singing data into a standard voice signal, i.e., Sig_std, and decode the participating singing data into a participating voice signal, i.e., Sig_join; on this basis, the server 203 divides Sig_std into multiple standard signal segments, i.e., multiple Seg_std, and divides Sig_join into multiple participating signal segments, i.e., multiple Seg_join.

[0040] The server 203 calculates the pitch period for each frame signal for multiple standard signal segments and multiple participating signal segments, and obtains multiple standard pitch period strings corresponding to the multiple standard signal segments and multiple participating pitch period strings corresponding to the multiple participating signal segments. As an example, based on the above example, the pitch period is calculated for each frame signal in multiple Seg_std to obtain multiple standard pitch period strings corresponding to multiple Seg_std, that is, multiple T_std; the pitch period is calculated for each frame signal in multiple Seg_join to obtain multiple participating pitch period strings corresponding to multiple Seg_join, that is, multiple T_join.

[0041] The server 203 performs dynamic time regularization on the pitch cycle string to be matched in the multiple standard pitch cycle strings and the multiple candidate pitch cycle strings in the multiple participating pitch cycle strings within a preset time period from a preset moment, and obtains multiple distances between the pitch cycle string to be matched and the multiple candidate pitch cycle strings; the preset moment is determined based on the multiple moments corresponding to the pitch cycle string to be matched. As an example, the preset moment is the current moment i, and the preset time period is t; based on the above example, a fundamental frequency period string to be matched is selected from multiple T_stds as T_std(i), and multiple candidate fundamental frequency period strings within a distance of t from i are selected from multiple participating fundamental frequency period strings, including T_join(i), T_join(i-1),..., T_join(iN), where N is a positive integer, and N+1 is the number of T_joins within a distance of t from i in multiple T_joins; dynamic time warping is performed on T_std(i) and T_join(ij), and the distance between T_std(i) and T_join(ij) is obtained as Dist(i, ij), where j is a natural number, j=0, 1, 2,..., N-1.

[0042] If the minimum distance among the multiple distances meets the preset similarity condition, the server 203 determines the delay of the participating channel relative to the standard channel according to the first alignment point time of the pitch cycle string to be matched and the second alignment point time of the candidate pitch cycle string corresponding to the minimum distance. As an example, the preset similarity condition is that the minimum distance is less than or equal to the preset distance Dist. On the basis of the above example, when the minimum distance Dist(i, iX) in Dist(i, ij) is less than Dist, the more accurate delay of the participating channel relative to the standard channel is determined by the first alignment point time in T_std(i) and the second alignment point time in the candidate pitch cycle string T_join(iX) corresponding to the minimum distance, where X is a natural number, and X is a value among 0, 1, 2, ..., N-1.

[0043] That is to say, in the speech alignment scenario, the standard speech signal of the standard channel is divided into multiple standard signal segments, and the participating speech signal of the participating channel is divided into multiple participating signal segments; the pitch period is calculated for each frame signal in the multiple standard signal segments to obtain multiple standard pitch period strings corresponding to the multiple standard signal segments, and the pitch period is calculated for each frame signal in the multiple participating signal segments to obtain multiple participating pitch period strings corresponding to the multiple participating signal segments; the pitch period calculation method obtains the speech features of each signal segment based on the pronunciation features of the valid speech segment. The pitch period string to be matched is selected from the multiple standard pitch period strings, the preset time is determined by the multiple time points corresponding to the pitch period string to be matched, and multiple candidate pitch period strings within the preset time period from the multiple participating pitch period strings are selected; the pitch period string to be matched and the multiple candidate pitch period strings are dynamically time-warped to obtain multiple distances between the pitch period string to be matched and the multiple candidate pitch period strings; based on the speech features of each signal segment, the dynamic time-warping method provides a data basis for determining the signal segment of the participating channel that is most similar in speech to the standard channel. When the minimum distance among multiple distances meets the preset similarity condition, the signal segment of the participating channel that is most similar in voice to the standard channel is determined. Through the first alignment point time of the fundamental pitch cycle string to be matched and the second alignment point time of the candidate fundamental pitch cycle string corresponding to the minimum distance, not only the transmission delay of the terminal but also the processing delay of the terminal and the pronunciation delay of the voice are considered to determine a more accurate delay of the participating channel relative to the standard channel; based on this, the participating voice signal can be more accurately aligned with the standard voice signal through this delay, thereby improving the voice alignment effect.

[0044] It should be noted that in the embodiments of the present application, the computer device may be a server or a terminal, and the method provided in the embodiments of the present application may be executed by the terminal or the server alone, or by the terminal and the server in cooperation. Figure 2 The corresponding embodiments are mainly introduced by taking the server executing the method provided in the embodiments of the present application as an example.

[0045] In addition, when the method provided in the embodiment of the present application is executed by a terminal alone, its execution method is the same as Figure 2 The corresponding embodiments are similar, mainly replacing the server with a terminal. In addition, when the terminal and the server cooperate to execute the method provided in the embodiment of the present application, the steps that need to be reflected on the front-end interface can be executed by the terminal, and some steps that require background calculation and do not need to be reflected on the front-end interface can be executed by the server.

[0046] It should also be noted that in the specific implementation of the present application, the data carried by the standard voice signal and the participating voice signal may involve user information and other related data. When the above embodiments of the present application are applied to specific products or technologies, the user's separate consent is required, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0047] In addition, the embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, audio and video, assisted driving, etc.

[0048] Next, the method for determining the delay based on voice alignment provided in the embodiment of the present application will be described in detail by taking the server executing the method provided in the embodiment of the present application as an example and combining the accompanying drawings. Figure 3 , Figure 3 A flowchart of a method for determining a delay based on speech alignment provided in an embodiment of the present application, the method comprising:

[0049] S301: In a speech alignment scenario, signal segmentation is performed on a standard speech signal of a standard channel and a participating speech signal of a participating channel to obtain a plurality of standard signal segments and a plurality of participating signal segments.

[0050] Because in the related art, when multiple voice data corresponding to multiple terminals need to be transmitted to the same device for voice alignment, the voice data transmitted by each terminal carries a timestamp, which is obtained by correcting the local clock of the terminal through a reference clock; the multiple timestamps carried by the multiple voice data transmitted by the multiple terminals are aligned to align the multiple voice data. However, the reference clock is obtained through network transmission, and there are delay differences between the reference clocks obtained by different terminals, and the reference clock cannot correct the processing delay of the terminal and the pronunciation delay of the voice, etc., resulting in the inaccuracy of aligning the multiple voice data through the multiple timestamps carried by the multiple voice data, resulting in poor voice alignment effect.

[0051] Therefore, in an embodiment of the present application, in order to solve the above problems, in the voice alignment scenario, it is considered to divide the multiple voice data transmitted by multiple terminals into standard voice data and participating voice data according to the voice category; for standard voice data and participating voice data, the participating voice data needs to be aligned with the standard voice data, and the standard voice data is received through the standard channel and decoded into a standard voice signal, and the participating voice data is received through the participating channel to be decoded into a participating voice signal. On the basis of the pitch period calculation method and the dynamic time warping method (dynamic time bending method), the two signal segments that are most similar in voice can be determined from the standard voice signal of the standard channel and the participating voice signal of the participating channel. Not only the transmission delay of the terminal is considered, but also the processing delay of the terminal and the pronunciation delay of the voice are considered to determine a more accurate delay of the participating channel relative to the standard channel, so that the participating voice signal can be more accurately aligned with the standard voice signal through the delay later, thereby improving the voice alignment effect. Based on this, first, the standard voice signal of the standard channel is divided into multiple standard signal segments, and the participating voice signal of the participating channel is divided into multiple participating signal segments, so that the two signal segments that are most similar in voice can be determined later for voice alignment.

[0052] In the specific implementation of S301, the standard voice signal of the standard channel is divided into multiple standard signal segments, which is actually determined by voice activity detection (VAD) to determine the voice activity value of each frame signal in the standard voice signal, so that the starting frame and the ending frame of multiple valid voice signal segments in the standard voice signal are determined by the multiple voice activity values ​​corresponding to the continuous multiple frames of signals in the standard voice signal, thereby dividing the standard voice signal according to the starting frame and the ending frame of the multiple valid voice signal segments in the standard voice signal to obtain multiple standard signal segments; similarly, dividing the participating voice signal of the participating channel into multiple participating signal segments is also determined by VAD to determine the voice activity value of each frame signal in the participating voice signal, so that the starting frame and the ending frame of multiple valid voice signal segments in the participating voice signal are determined by the multiple voice activity values ​​corresponding to the continuous multiple frames of signals in the participating voice signal, thereby dividing the participating voice signal according to the starting frame and the ending frame of multiple valid voice signal segments in the participating voice signal to obtain multiple participating signal segments. Therefore, the present application provides a possible implementation method, and S301 may include the following S3011-S3013 (not shown in the figure):

[0053] S3011: Perform voice activity detection on the standard voice signal to obtain the voice activity value of each frame signal in the standard voice signal.

[0054] S3012: Determine the start frame and the end frame of multiple valid voice signal segments in the standard voice signal according to multiple voice activity values ​​corresponding to multiple consecutive frame signals in the standard voice signal.

[0055] As an example, when the voice activity value of a frame signal is 0, it indicates that the frame signal is non-valid speech (small noise or silence), and when the voice activity value of a frame signal is 1, it indicates that the frame signal is valid speech; based on this, if the P voice activity values ​​corresponding to consecutive P frame signals in the standard voice signal are 0, followed by a voice activity value of 1 frame signal, then the 1 frame signal with the voice activity value of 1 is the starting frame of the valid voice signal segment in the standard voice signal; if the voice activity value of 1 frame signal is 1, followed by Q voice activity values ​​corresponding to consecutive Q frame signals are 0, then the 1 frame signal with the voice activity value of 1 is the end frame of the valid voice signal segment in the standard voice signal.

[0056] S3013: According to the start frames and end frames of multiple valid voice signal segments in the standard voice signal, the standard voice signal is segmented to obtain multiple standard signal segments.

[0057] S3014: Perform voice activity detection on the participating voice signal to obtain a voice activity value of each frame signal in the participating voice signal.

[0058] S3015: Determine the start frame and the end frame of multiple valid speech signal segments in the participating speech signal according to multiple speech activity values ​​corresponding to multiple consecutive frame signals in the participating speech signal.

[0059] As an example, based on the above example, if the P voice activity values ​​corresponding to the consecutive P frame signals in the participating voice signal are 0, and the voice activity value of the subsequent 1 frame signal is 1, then the 1 frame signal with the voice activity value of 1 is the starting frame of the valid voice signal segment in the participating voice signal; if the voice activity value of 1 frame signal is 1, and the Q voice activity values ​​corresponding to the consecutive Q frame signals are 0, then the 1 frame signal with the voice activity value of 1 is the ending frame of the valid voice signal segment in the participating voice signal.

[0060] S3016: According to the start frames and end frames of multiple valid voice signal segments in the participating voice signal, the participating voice signal is segmented to obtain multiple participating signal segments.

[0061] Among them, considering that it is necessary to determine the two most similar signal segments in speech from multiple standard signal segments and multiple participating signal segments through the pitch period calculation method and the dynamic time warping method, and the pitch period and dynamic time warping are performed on the signal segments with shorter duration to determine the two most similar signal segments in speech, which is easy to lead to low similarity credibility; therefore, the standard speech is divided according to the start frame and the end frame of multiple valid speech signal segments in the standard speech signal to directly obtain multiple first signal segments, and the first signal segments less than the preset duration in the multiple first signal segments need to be filtered to obtain multiple standard signal segments greater than and equal to the preset duration; similarly, the participating speech is divided according to the start frame and the end frame of multiple valid speech signal segments in the participating speech signal to directly obtain multiple second signal segments, and the second signal segments less than the preset duration in the multiple second signal segments need to be filtered to obtain multiple participating signal segments greater than and equal to the preset duration. This filtering method can ensure the credibility of determining the two most similar signal segments in speech from multiple standard signal segments and multiple participating signal segments through the pitch period calculation method and the dynamic time warping method. That is, the present application provides a possible implementation method, and S3013 may include the following S1-S2 (not shown in the figure):

[0062] S1: According to the start frames and end frames of multiple valid voice signal segments in the standard voice signal, the standard voice signal is segmented to obtain multiple first signal segments.

[0063] S2: Filter the first signal segments shorter than a preset duration among the multiple first signal segments to obtain multiple standard signal segments.

[0064] Similarly, S3016 may include, for example, the following S3-S4 (not shown in the figure):

[0065] S3: According to the start frames and end frames of multiple valid voice signal segments in the participating voice signal, the participating voice signal is segmented to obtain multiple second signal segments.

[0066] S4: Filter the second signal segments shorter than a preset duration among the plurality of second signal segments to obtain a plurality of participating signal segments.

[0067] As an example, the voice alignment scenario is an online chorus scenario, the standard voice data is the standard singing data, and the participating voice data is the participating singing data. The standard singing data is decoded to obtain the standard voice signal as Sig_std, and the participating singing data is decoded to obtain the participating voice signal as Sig_join; on the one hand, the voice activity value of each frame signal in Sig_std is determined by VAD, so as to determine the start frame and end frame of multiple valid voice signal segments in Sig_std through multiple voice activity values ​​corresponding to multiple consecutive frames of signals in Sig_std, so as to divide Sig_std into multiple first signal segments according to the start frame and end frame of multiple valid voice signal segments in Sig_std; if the preset time length is 50 frames, the first signal segments less than 50 frames in the multiple first signal segments are filtered to obtain multiple standard signal segments as multiple Seg_std.

[0068] On the other hand, the voice activity value of each frame signal in Sig_join is determined by VAD method, so as to determine the start frame and end frame of multiple valid voice signal segments in Sig_join through multiple voice activity values ​​corresponding to multiple consecutive frame signals in Sig_join, so as to divide Sig_join into multiple second signal segments according to the start frame and end frame of multiple valid voice signal segments in Sig_join; and filter the second signal segments less than 50 frames in the multiple second signal segments to obtain multiple participating signal segments as multiple Seg_join.

[0069] S302: For multiple standard signal segments and multiple participating signal segments, calculate the pitch period for each frame signal to obtain multiple standard pitch period strings corresponding to the multiple standard signal segments and multiple participating pitch period strings corresponding to the multiple participating signal segments.

[0070] In the embodiment of the present application, after executing S301 to obtain multiple standard signal segments in the standard voice signal and multiple participating signal segments in the participating voice signal, considering that voice signal analysis is the premise and basis of voice signal processing, only by analyzing the parameters that can characterize the essential characteristics of the voice signal can it be possible to use these parameters for efficient voice processing, and the pitch period is one of the most important characteristic parameters of the voice signal; it is necessary to obtain the voice characteristics of each signal segment based on the pronunciation characteristics of the effective voice segment through the pitch period calculation method, so that the two signal segments that are most similar in voice can be determined from the multiple standard signal segments and the multiple participating signal segments through the dynamic time warping method, not only considering the transmission delay of the terminal, but also considering the processing delay of the terminal and the pronunciation delay of the voice, so as to determine the more accurate delay of the participating channel relative to the standard channel. Based on this, the pitch period is calculated for each frame of the signal in the multiple standard signal segments to obtain multiple standard pitch period strings corresponding to the multiple standard signal segments, and the pitch period is calculated for each frame of the signal in the multiple participating signal segments to obtain multiple participating pitch period strings corresponding to the multiple participating signal segments.

[0071] In the specific implementation of S302, for multiple standard signal segments and multiple participating signal segments, the fundamental pitch period of each frame signal is calculated to obtain the fundamental pitch period of each frame signal, thereby determining multiple standard fundamental pitch period strings corresponding to the multiple standard signal segments through the fundamental pitch period of each frame signal in the multiple standard signal segments; and determining multiple participating fundamental pitch period strings corresponding to the multiple participating signal segments through the fundamental pitch period of each frame signal in the multiple participating signal segments.

[0072] The fundamental period calculation for each frame signal can be performed by the autocorrelation method. Specifically, first, considering the values ​​of the number of sampling points of each frame signal corresponding to the multiple sampling signals of each frame signal, the maximum number of sampling points of each frame signal is determined by the sampling frequency and signal frame length of each frame signal, thereby obtaining multiple sampling signals of each frame signal; secondly, each sampling signal of each frame signal is normalized to obtain a normalized signal; then, the maximum value of the delayed sample number is determined by the aforementioned sampling frequency and the minimum fundamental frequency, so as to perform autocorrelation calculation based on the multiple normalized signals of each frame signal to obtain multiple first autocorrelation values ​​of each frame signal, and the multiple first autocorrelation values ​​correspond to the maximum value of the delayed sample number; finally, from the multiple delayed sample numbers corresponding to the multiple first autocorrelation values ​​of each frame signal, a delayed sample number is determined to further determine the fundamental period of each frame signal. Therefore, the present application provides a possible implementation method, and S302 may include the following S3021-S3026 (not shown in the figure):

[0073] S3021: for multiple standard signal segments and multiple participating signal segments, multiple sampling signals of each frame signal are obtained; the maximum number of sampling points corresponding to the multiple sampling signals is determined according to the sampling frequency and signal frame length of each frame signal.

[0074] S3022: Perform normalization processing on each sampling signal of each frame signal to obtain a normalized signal.

[0075] S3023: Perform autocorrelation calculation based on multiple normalized signals of each frame signal to obtain multiple first autocorrelation values ​​of each frame signal; the maximum value of the number of delayed sample points corresponding to the multiple first autocorrelation values ​​is determined according to the sampling frequency and the minimum fundamental frequency.

[0076] S3024: Determine the pitch period of each frame of the signal from a plurality of delayed sample points corresponding to a plurality of first autocorrelation values ​​of each frame of the signal.

[0077] Among them, when S3024 is specifically implemented, considering that the smaller autocorrelation value indicates no correlation, that is, the number of delayed sample points corresponding to the smaller autocorrelation value may not be the fundamental pitch period of each frame signal, it is necessary to filter the first autocorrelation value less than the preset correlation value among the multiple first autocorrelation values ​​of each frame signal to obtain multiple second autocorrelation values ​​greater than and equal to the preset correlation value for each frame signal; in addition, the multiple second autocorrelation values ​​are sorted from large to small to obtain a first sorting result, and the preset number of sample points determined by the preset coefficient and the number of delayed sample points corresponding to the first second autocorrelation value in the first sorting result may not be the fundamental pitch period of each frame signal, and it is also necessary to filter the second autocorrelation value whose number of delayed sample points meets the preset number of sample points among the multiple second autocorrelation values ​​of each frame signal to obtain multiple third autocorrelation values ​​greater than and equal to the preset correlation value and whose number of delayed sample points does not meet the preset number of sample points for each frame signal; based on this, the multiple third autocorrelation values ​​are sorted from large to small to obtain a second sorting result, and the fundamental pitch period of each frame signal can be determined by the number of delayed sample points corresponding to the first third autocorrelation value in the second sorting result. Therefore, the present application provides a possible implementation manner, and S3024 may include, for example, the following S5-S8 (not shown in the figure):

[0078] S5: Filter the first autocorrelation values ​​that are smaller than the preset correlation value among the multiple first autocorrelation values ​​of each frame signal to obtain multiple second autocorrelation values ​​of each frame signal.

[0079] S6: Filter the second autocorrelation values ​​whose delayed sample points meet the preset sample point number among the multiple second autocorrelation values ​​of each frame signal to obtain multiple third autocorrelation values ​​of each frame signal; the preset sample point number is determined by the delayed sample point number corresponding to the first second autocorrelation value in the first sorting result obtained by sorting the multiple second autocorrelation values ​​of each frame signal from large to small and the preset coefficient.

[0080] S7: sorting the multiple third autocorrelation values ​​of each frame signal from large to small to obtain a second sorting result.

[0081] S8: Determine the number of delayed sample points corresponding to the first third autocorrelation value in the second sorting result as the pitch period of each frame signal.

[0082] S3025: Determine a plurality of standard pitch period strings according to the pitch period of each frame signal in a plurality of standard signal segments.

[0083] S3026: Determine multiple participating pitch period strings according to the pitch period of each frame signal in the multiple participating signal segments.

[0084] As an example, based on the above example, the fundamental pitch period of each frame signal in multiple Seg_std and multiple Seg_join is calculated to obtain the fundamental pitch period of each frame signal; that is, first, the maximum number of sampling points of each frame signal is determined by the sampling frequency and signal frame length of each frame signal, so as to obtain multiple sampling signals of each frame signal as S(u), the value range of u is [1, L], L is the maximum number of sampling points of each frame signal, when the sampling frequency of each frame signal is fs=16000, and the signal frame length of each frame signal is 40ms, L=40÷1000×16000=640. Secondly, S(u) is normalized to obtain a normalized signal Sn(u). Then, the maximum number of delayed samples is determined by the aforementioned sampling frequency and minimum fundamental pitch frequency, and autocorrelation calculation is performed based on Sn(u), to obtain multiple first autocorrelation values ​​of each frame signal. The value range of k is [1, M], where M is the maximum value of the number of delayed samples. When the minimum fundamental frequency is 80 Hz, M=1÷80×16000=200. Finally, when the preset correlation value is R, filter R(k) less than R in R(k) to obtain R(k) greater than or equal to R; when the preset coefficient is q, sort R(k) greater than or equal to R from large to small to obtain a first sorting result, and determine the adjacent integers with a preset number of samples k0 / q by q and the number of delayed samples k0 corresponding to the first R(k0) in the first sorting result, where q is a positive integer, q=2, 3, 4, 5,...; filter R(k) whose adjacent integers k meets k0 / q in R(k) greater than or equal to R to obtain R(k) whose adjacent integers k meet k0 / q; sort R(k) whose adjacent integers greater than or equal to R and k do not meet k0 / q from large to small to obtain a second sorting result, and determine the fundamental frequency period of each frame signal by the number of delayed samples k corresponding to the first R(k) in the second sorting result. Based on this, through the pitch period of each frame signal in multiple Seg_std, multiple standard pitch period strings corresponding to multiple Seg_std are determined as multiple T_std; and through the pitch period of each frame signal in multiple Seg_join, multiple participating pitch period strings corresponding to multiple Seg_join are determined as multiple T_join.

[0085] S303: Dynamically time-warp the pitch cycle string to be matched among multiple standard pitch cycle strings and multiple candidate pitch cycle strings within a preset time period from a preset moment among multiple participating pitch cycle strings to obtain multiple distances between the pitch cycle string to be matched and the multiple candidate pitch cycle strings; the preset moment is determined based on multiple moments corresponding to the pitch cycle string to be matched.

[0086] In the embodiment of the present application, after executing S302 to obtain multiple standard pitch cycle strings corresponding to multiple standard signal segments and multiple participating pitch cycle strings corresponding to multiple participating signal segments, in order to align the participating speech signal with the standard speech signal, a pitch cycle string to be matched can be selected from multiple standard pitch cycle strings, a preset time can be determined from multiple time moments corresponding to the pitch cycle string to be matched, and multiple candidate pitch cycle strings within a preset time period from the preset time can be selected from multiple participating pitch cycle strings; multiple distances between the pitch cycle string to be matched and the multiple candidate pitch cycle strings can be determined by dynamic time warping. The smaller the distance, the more similar the participating signal segment corresponding to the candidate pitch cycle string is to the standard signal segment corresponding to the pitch cycle string to be matched.

[0087] When S303 is specifically implemented, the distance between the pitch cycle string to be matched and the candidate pitch cycle string is determined by dynamic time warping, which is actually to solve the dynamic time warping distance between the two one-dimensional data, which is used to describe the similarity between the two one-dimensional data. Specifically, for the pitch cycle string to be matched and each candidate pitch cycle string, the distances of multiple pitch cycles in the pitch cycle string to be matched and multiple pitch cycles in the candidate pitch cycle string are first calculated in sequence according to the sorting order of the pitch cycles, until the calculated distance between the last pitch cycle in the pitch cycle string to be matched and the last pitch cycle in the candidate pitch cycle string is obtained; then the multiple calculated distances between the last pitch cycle in the pitch cycle string to be matched and the last pitch cycle in the multiple candidate pitch cycle strings are used as the multiple distances between the pitch cycle string to be matched and the multiple candidate pitch cycle strings, thereby describing the multiple similarities between the pitch cycle string to be matched and the multiple candidate pitch cycle strings. Therefore, the present application provides a possible implementation method, and S303 may, for example, include the following S3031-S3032 (not shown in the figure):

[0088] S3031: For the to-be-matched pitch cycle string and each candidate pitch cycle string, dynamic time warping is performed on multiple pitch cycles in the to-be-matched pitch cycle string and multiple pitch cycles in the candidate pitch cycle string in sequence according to the sorting order of the pitch cycles, until the calculated distance between the last pitch cycle in the to-be-matched pitch cycle string and the last pitch cycle in the candidate pitch cycle string is obtained.

[0089] S3032: Determine multiple calculated distances between the last pitch cycle in the to-be-matched pitch cycle string and the last pitch cycle in the multiple candidate pitch cycle strings as multiple distances.

[0090] As an example, the preset moment is the current moment i, and the preset time period is t; based on the above example, a fundamental frequency period string to be matched is selected from multiple T_stds as T_std(i), and multiple candidate fundamental frequency period strings within a distance of t from i are selected from multiple participating fundamental frequency period strings, including T_join(i), T_join(i-1),..., T_join(iN), where N is a positive integer, and N+1 is the number of T_joins within a distance of t from i in multiple T_joins; dynamic time warping is performed on T_std(i) and T_join(ij), and the distance between T_std(i) and T_join(ij) is obtained as Dist(i, ij), where j is a natural number, j=0, 1, 2,..., N-1.

[0091] Wherein, the length of T_std(i) is p, that is, T_std(i) includes a1, a2, ..., ap, and the length of T_join(ij) is q, that is, T_join(ij) includes b1, b2, ..., bq; since Dist(m, n) = cost + min(Dist(m-1, n), Dist(m, n-1), Dist(m-1, n-1)); m is a positive integer, m = 1, 2, ..., p; n is a positive integer, n = 1, 2, ..., q, cost = |am-bn|^2; then Dist(i, ij) = Dist(p, q) = |ap-bq|^2 + min(Dist(p-1, q), Dist(p, q-1), Dist(p-1, q-1)). See Figure 4 , Figure 4 A schematic diagram of the distance between a pitch period string to be matched and a candidate pitch period string provided in an embodiment of the present application. When p=8 and q=6, the first row in the figure represents the 8 pitch periods included in T_std(i), the first column in the figure represents the 6 pitch periods included in T_join(ij), and the shaded portion represents the calculated distance between a pitch period in T_std(i) and a pitch period in T_join(ij).

[0092] S304: If the minimum distance among the multiple distances meets the preset similarity condition, determine the delay of the participating channel relative to the standard channel according to the first alignment point time of the pitch cycle string to be matched and the second alignment point time of the candidate pitch cycle string corresponding to the minimum distance.

[0093] In an embodiment of the present application, after executing S303 to determine multiple distances between the pitch cycle string to be matched and multiple candidate pitch cycle strings, in order to determine the two signal segments that are most similar in voice from the standard voice signal of the standard channel and the participating voice signal of the participating channel, considering that the smaller the distance, the more similar the participating signal segment corresponding to the candidate pitch cycle string is to the standard signal segment corresponding to the pitch cycle string to be matched, the minimum distance is first determined from multiple distances, and then it is determined whether the minimum distance meets the preset similarity condition; if so, it means that the participating signal segment corresponding to the candidate pitch cycle string corresponding to the minimum distance and the standard signal segment corresponding to the pitch cycle string to be matched are most similar in voice. Based on this, through the first alignment point time of the pitch cycle string to be matched and the second alignment point time of the candidate pitch cycle string corresponding to the minimum distance, not only the transmission delay of the terminal is considered, but also the processing delay of the terminal and the pronunciation delay of the voice are considered, so as to determine a more accurate delay of the participating channel relative to the standard channel; so that the participating voice signal can be more accurately aligned with the standard voice signal through the delay later, thereby improving the voice alignment effect.

[0094] In the specific implementation of S304, considering that the standard signal segment corresponding to the pitch cycle string to be matched is obtained by segmenting from the standard speech signal, it is necessary to determine the first time that is most suitable for alignment in the standard speech signal by combining the first alignment point time of the pitch cycle string to be matched with the starting frame time of the standard speech signal; similarly, considering that the participating signal segment corresponding to the candidate pitch cycle string corresponding to the minimum distance is obtained by segmenting from the participating speech signal, it is necessary to determine the second time that is most suitable for alignment in the participating speech signal by combining the second alignment point time of the candidate pitch cycle string corresponding to the minimum distance with the starting frame time of the participating speech signal; thereby determining the delay of the participating channel relative to the standard channel by the difference between the second time and the first time. Therefore, the present application provides a possible implementation method, and S304 may include the following S3041-S3043 (not shown in the figure):

[0095] S3041: Determine a first time according to the start frame time of the standard signal segment corresponding to the pitch cycle string to be matched in the standard speech signal and the first alignment point time.

[0096] S3041: Determine the second time according to the starting frame time of the participating speech signal and the second alignment point time of the participating signal segment corresponding to the candidate pitch cycle string corresponding to the minimum distance.

[0097] S3043: Determine a delay according to a difference between the second time and the first time.

[0098] Among them, in order to determine the first alignment point time of the pitch cycle string to be matched, it is necessary to determine the more critical pitch cycle in the pitch cycle string to be matched, that is, the pitch cycle string to be matched is formed into a curve, and the point where the first-order derivative is zero in the curve is calculated as a stationary point, so as to determine the first stationary point in the pitch cycle string to be matched; the first alignment point time is determined by the time when the first stationary point is in the pitch cycle string to be matched. Similarly, in order to determine the second alignment point time of the candidate pitch cycle string corresponding to the minimum distance, it is necessary to determine the more critical pitch cycle in the candidate pitch cycle string corresponding to the minimum distance, that is, the candidate pitch cycle string corresponding to the minimum distance is formed into a curve, and the point where the first-order derivative is zero in the curve is calculated as a stationary point, so as to determine the second stationary point in the candidate pitch cycle string corresponding to the minimum distance; the second alignment point time is determined by the time when the second stationary point is in the candidate pitch cycle string corresponding to the minimum distance. Therefore, the present application provides a possible implementation method, and the steps of determining the first alignment point time and the second alignment point time may, for example, include the following S9-S12 (not shown in the figure):

[0099] S9: Perform stationary point calculation on the curve formed by the to-be-matched pitch cycle string to obtain the first stationary point in the to-be-matched pitch cycle string.

[0100] S10: Determine the time of the first alignment point according to the time when the first stationary point is in the pitch cycle string to be matched.

[0101] S11: Perform stationary point calculation on the curve formed by the candidate pitch cycle string corresponding to the minimum distance to obtain a second stationary point in the candidate pitch cycle string corresponding to the minimum distance.

[0102] S12: Determine the time of the second alignment point according to the time when the second stationary point is in the candidate pitch cycle string corresponding to the minimum distance.

[0103] As an example, the preset similarity condition is that the minimum distance is less than or equal to the preset distance Dist. Based on the above example, when the minimum distance Dist(i, iX) in Dist(i, ij) is less than Dist, X is a natural number, X is a value among 0, 1, 2, ..., N-1, T_std(i) is formed into a curve, and the point where the first-order derivative is zero in the curve is calculated as the stationary point, so as to determine that the first stationary point in T_std(i) includes the 3rd fundamental pitch period, the 5th fundamental pitch period and the 7th fundamental pitch period; the mean of the 3rd fundamental pitch period, the 5th fundamental pitch period and the 7th fundamental pitch period at times 3, 5 and 7 of T_std(i) is calculated, and the first alignment point time is 5 (center of gravity time); the starting frame time of the standard signal segment corresponding to T_std(i) in Sig_std is M1, combined with the first alignment point time 5 in T_std(i), it can be determined that the first time most suitable for alignment in Sig_std is M1+5. The candidate pitch cycle string T_join(iX) corresponding to the minimum distance is formed into a curve, and the point where the first-order derivative is zero in the curve is calculated as the stationary point, so as to determine that the second stationary point in T_join(iX) includes the 3rd pitch cycle, the 4th pitch cycle, the 5th pitch cycle and the 6th pitch cycle; the mean of the 3rd pitch cycle, the 4th pitch cycle, the 5th pitch cycle and the 6th pitch cycle at time 3, 4, 5 and 6 of T_join(iX) is calculated, and the second alignment point time is 4.5 (center of gravity time); the starting frame time of the participating signal segment corresponding to T_join(iX) in Sig_join is M2, combined with the first alignment point time 5 in T_join(iX), it can be determined that the second time to be aligned in Sig_join is M2+4.5. Through the difference between M2+4.5 and M1+5, the time delay of the participating channel relative to the standard channel is determined to be (M2+4.5)-(M1+5); among which M1, M2, 4.5 and 5 represent time in the form of serial numbers.

[0104] In addition, in an embodiment of the present application, in a voice alignment scenario, multiple voice data transmitted by multiple terminals are divided into standard voice data and participating voice data according to voice categories. For standard voice data and participating voice data, participating voice data needs to be aligned with standard voice data, and standard voice data is received through a standard channel and decoded into a standard voice signal, and participating voice data is received through a participating channel to be decoded into a participating voice signal; because the execution of S301-S304 not only considers the transmission delay of the terminal, but also considers the processing delay of the terminal and the pronunciation delay of the voice, etc., to determine the more accurate delay of the participating channel relative to the standard channel, the participating voice signal can be more accurately aligned with the standard voice signal through the delay; therefore, the participating voice signal is offset by the delay of the participating channel relative to the standard channel, and the offset voice signal aligned with the standard voice signal can be obtained, thereby improving the voice alignment effect. That is, the present application provides a possible implementation method. After S304, the method may also include S13 (not shown in the figure): time-shifting the participating voice signal according to the delay to obtain the offset voice signal; aligning the offset voice signal with the standard voice signal. Based on this, in the voice mixing scenario, the offset voice signal is used to replace the participating voice signal and mix with the standard voice signal.

[0105] As an example, based on the above example, Sig_join is offset by (M2+4.5)-(M1+5), and the offset voice signal aligned with Sig_std is obtained as Sig_join'. In the voice mixing scenario, Sig_join' is used to replace Sig_join and mix with Sig_std.

[0106] In addition, in an embodiment of the present application, in the speech alignment process of the speech alignment scenario, the standard speech data and the participating speech data are dynamically changing, that is, the standard speech signal and the participating speech signal are dynamically changing; on this basis, taking into account the influence of various factors, etc., the time delay of the participating channel relative to the standard channel is dynamically changing. Based on this, it is also necessary to update the time delay of the participating channel relative to the standard channel according to the preset update strategy to obtain an updated time delay, so that the updated time delay can be used when the participating speech signal of the participating channel is subsequently time-shifted to more accurately align the standard speech signal of the standard channel. Therefore, the present application provides a possible implementation method, and the method may also include the following S14 (not shown in the figure): updating the delay according to the preset update strategy to obtain the updated delay; correspondingly, S13 may specifically be, for example: time-shifting the participating speech signal according to the updated time delay to obtain the shifted speech signal.

[0107] In addition, in an embodiment of the present application, corresponding to S304, considering that the smaller the distance, the more similar the participating signal segment corresponding to the candidate pitch period string is to the standard signal segment corresponding to the pitch period string to be matched, the minimum distance is first determined from multiple distances, and then it is determined whether the minimum distance meets the preset similarity condition. If not, it means that among the multiple participating signal segments corresponding to the multiple candidate pitch period strings, there is no participating signal segment that is most similar in voice to the standard signal segment corresponding to the pitch period string to be matched, and the participating voice signal cannot be aligned with the standard voice signal; based on this, the participating voice signal is discarded during the mixing process of the voice mixing scene, that is, the participating voice signal does not participate in the mixing process to avoid interference and affect the voice mixing effect, thereby ensuring the voice alignment effect after voice mixing. Therefore, the present application provides a possible implementation method, and the method may also include the following S15 (not shown in the figure): in the voice mixing scene, if the minimum distance does not meet the preset similarity condition, it is determined that the participating voice signal does not participate in the mixing process.

[0108] As an example, based on the above example, when the minimum distance Dist(i, iX) in Dist(i, ij) is greater than Dist, Sig_join is discarded in the mixing process of the voice mixing scene, that is, Sig_join does not participate in the mixing process.

[0109] In summary, see the relevant instructions for S301-S304. Figure 5 , Figure 5A schematic diagram of the overall process of speech alignment provided for an embodiment of the present application. Step 1: Receive standard speech data through a standard channel and decode it into a standard speech signal; receive participating speech data through a participating channel and decode it into a participating speech signal. Step 2: Divide the standard speech signal of the standard channel into multiple standard signal segments; divide the participating speech signal of the participating channel into multiple participating signal segments. Step 3: Calculate the pitch period of each frame signal in the multiple standard signal segments to obtain multiple standard pitch period strings corresponding to the multiple standard signal segments; Calculate the pitch period of each frame signal in the multiple participating signal segments to obtain multiple participating pitch period strings corresponding to the multiple participating signal segments. Step 4: Select a pitch period string to be matched from multiple standard pitch period strings, determine a preset moment from multiple moments corresponding to the pitch period string to be matched, select multiple candidate pitch period strings within a preset time period from the multiple participating pitch period strings; determine multiple distances between the pitch period string to be matched and multiple candidate pitch period strings by dynamic time regularization. Step 5: Determine the minimum distance from multiple distances, and then determine whether the minimum distance meets the preset similarity condition; if so, determine the first alignment point time of the pitch cycle string to be matched and the second alignment point time of the candidate pitch cycle string corresponding to the minimum distance. Step 6: Determine the time delay of the participating channel relative to the standard channel through the first alignment point time and the second alignment point time. Step 7: Shift the participating voice signal by the time delay of the participating channel relative to the standard channel to obtain the shifted voice signal aligned with the standard voice signal.

[0110] It can be seen from the above technical scheme that in the speech alignment scenario, first, the standard speech signal of the standard channel is divided into multiple standard signal segments, and the participating speech signal of the participating channel is divided into multiple participating signal segments; secondly, the pitch period is calculated for each frame signal in the multiple standard signal segments to obtain multiple standard pitch period strings corresponding to the multiple standard signal segments, and the pitch period is calculated for each frame signal in the multiple participating signal segments to obtain multiple participating pitch period strings corresponding to the multiple participating signal segments; the pitch period calculation method obtains the speech features of each signal segment based on the pronunciation features of the valid speech segment. Then, the pitch period string to be matched is selected from the multiple standard pitch period strings, the preset time is determined by the multiple moments corresponding to the pitch period string to be matched, and multiple candidate pitch period strings within the preset time period from the multiple participating pitch period strings are selected; the pitch period string to be matched and the multiple candidate pitch period strings are dynamically time-warped to obtain multiple distances between the pitch period string to be matched and the multiple candidate pitch period strings; based on the speech features of each signal segment, the dynamic time warping method provides a data basis for determining the signal segment of the participating channel that is most similar in speech to the standard channel. Finally, when the minimum distance among multiple distances meets the preset similarity condition, the first alignment point time of the fundamental pitch period string to be matched and the second alignment point time of the candidate fundamental pitch period string corresponding to the minimum distance are used to consider not only the transmission delay of the terminal, but also the processing delay of the terminal and the pronunciation delay of the voice, so as to determine a more accurate delay of the participating channel relative to the standard channel; based on this, the participating voice signal can be more accurately aligned with the standard voice signal through this delay, thereby improving the voice alignment effect.

[0111] It should be noted that, based on the implementation methods provided in the above aspects, this application can also be further combined to provide more implementation methods.

[0112] See also Figure 6 , Figure 6 A schematic diagram of a speech alignment scenario provided for an embodiment of the present application. The delay determination method based on speech alignment provided in an embodiment of the present application is mainly applied to speech alignment scenarios such as online chorus scenarios, so that the speech alignment effect of online chorus is better and more ideal. The online chorus scenario refers to that multiple users can sing online after connecting to the Internet through multiple terminals, and multiple singing data corresponding to multiple terminals need to be transmitted to the same device for speech alignment to be mixed into chorus data. Specifically, the online chorus scenario can be an online accompaniment chorus scenario or an online concert chorus scenario, etc.

[0113] based on Figure 2 Corresponding to the method for determining the time delay based on speech alignment provided in the embodiment, the embodiment of the present application also provides a device for determining the time delay based on speech alignment, see Figure 7 , Figure 7A structural diagram of a time delay determination device based on speech alignment provided in an embodiment of the present application, the time delay determination device based on speech alignment 700 comprises: a segmentation unit 701, a calculation unit 702, a regularization unit 703 and a determination unit 704;

[0114] A segmentation unit 701 is used to segment the standard speech signal of the standard channel and the participating speech signal of the participating channel in the speech alignment scenario to obtain a plurality of standard signal segments and a plurality of participating signal segments;

[0115] A calculation unit 702 is used to calculate the pitch period of each frame signal for the multiple standard signal segments and the multiple participating signal segments, and obtain a plurality of standard pitch period strings corresponding to the multiple standard signal segments and a plurality of participating pitch period strings corresponding to the multiple participating signal segments;

[0116] The regularization unit 703 is used to perform dynamic time regularization on the pitch cycle string to be matched in the multiple standard pitch cycle strings and the multiple candidate pitch cycle strings within a preset time period from the preset moment in the multiple participating pitch cycle strings to obtain multiple distances between the pitch cycle string to be matched and the multiple candidate pitch cycle strings; the preset moment is determined according to the multiple moments corresponding to the pitch cycle string to be matched;

[0117] The determination unit 704 is used to determine the delay of the participating channel relative to the standard channel according to the first alignment point time of the fundamental pitch cycle string to be matched and the second alignment point time of the candidate fundamental pitch cycle string corresponding to the minimum distance if the minimum distance among the multiple distances meets the preset similarity condition.

[0118] In a possible implementation, the computing unit 702 is specifically configured to:

[0119] For multiple standard signal segments and multiple participating signal segments, multiple sampling signals of each frame signal are obtained; the maximum number of sampling points corresponding to the multiple sampling signals is determined according to the sampling frequency and signal frame length of each frame signal;

[0120] Performing normalization processing on each sampling signal of each frame signal to obtain a normalized signal;

[0121] Autocorrelation calculation is performed according to multiple normalized signals of each frame signal to obtain multiple first autocorrelation values ​​of each frame signal; the maximum value of the number of delayed sample points corresponding to the multiple first autocorrelation values ​​is determined according to the sampling frequency and the minimum fundamental frequency;

[0122] Determine the pitch period of each frame of signal from a plurality of delayed sample points corresponding to a plurality of first autocorrelation values ​​of each frame of signal;

[0123] Determine a plurality of standard pitch period strings according to the pitch period of each frame signal in a plurality of standard signal segments;

[0124] A plurality of participating pitch period strings are determined according to the pitch period of each frame signal in the plurality of participating signal segments.

[0125] In a possible implementation, the computing unit 702 is specifically configured to:

[0126] Filtering the first autocorrelation values ​​that are smaller than the preset correlation value among the multiple first autocorrelation values ​​of each frame signal to obtain multiple second autocorrelation values ​​of each frame signal;

[0127] Filter the second autocorrelation values ​​whose delayed sample points meet the preset sample point number among the multiple second autocorrelation values ​​of each frame signal to obtain multiple third autocorrelation values ​​of each frame signal; the preset sample point number is determined by the delayed sample point number corresponding to the first second autocorrelation value in the first sorting result obtained by sorting the multiple second autocorrelation values ​​of each frame signal from large to small and the preset coefficient;

[0128] Sort the multiple third autocorrelation values ​​of each frame signal from large to small to obtain a second sorting result;

[0129] The pitch period of each frame signal is determined according to the number of delayed sample points corresponding to the first third autocorrelation value in the second sorting result.

[0130] In a possible implementation, the regularization unit 703 is specifically configured to:

[0131] For the pitch cycle string to be matched and each candidate pitch cycle string, dynamic time warping is performed on multiple pitch cycles in the pitch cycle string to be matched and multiple pitch cycles in the candidate pitch cycle string in sequence according to the sorting order of the pitch cycles, until the calculated distance between the last pitch cycle in the pitch cycle string to be matched and the last pitch cycle in the candidate pitch cycle string is obtained;

[0132] A plurality of calculated distances between a last pitch cycle in the to-be-matched pitch cycle string and a last pitch cycle in the plurality of candidate pitch cycle strings are determined as a plurality of distances.

[0133] In a possible implementation manner, the determining unit 704 is specifically configured to:

[0134] Determine the first time according to the start frame time of the standard voice signal and the first alignment point time of the standard signal segment corresponding to the pitch cycle string to be matched;

[0135] Determine the second time according to the starting frame time of the participating speech signal and the second alignment point time of the participating signal segment corresponding to the candidate pitch cycle string corresponding to the minimum distance;

[0136] The time delay is determined according to the difference between the second time and the first time.

[0137] In a possible implementation manner, the determining unit 704 is specifically configured to:

[0138] Perform stationary point calculation on the curve formed by the to-be-matched pitch cycle string to obtain the first stationary point in the to-be-matched pitch cycle string;

[0139] Determine the time of the first alignment point according to the time of the first stationary point in the pitch cycle string to be matched;

[0140] Performing stationary point calculation on the curve formed by the candidate pitch cycle string corresponding to the minimum distance to obtain a second stationary point in the candidate pitch cycle string corresponding to the minimum distance;

[0141] The second alignment point time is determined according to the time when the second stationary point is in the candidate pitch cycle string corresponding to the minimum distance.

[0142] In a possible implementation, the segmentation unit 701 is specifically configured to:

[0143] Performing voice activity detection on the standard voice signal to obtain the voice activity value of each frame signal in the standard voice signal;

[0144] Determining the start frame and the end frame of multiple valid voice signal segments in the standard voice signal according to multiple voice activity values ​​corresponding to multiple consecutive frames of signals in the standard voice signal;

[0145] According to the start frames and end frames of multiple valid voice signal segments in the standard voice signal, the standard voice signal is segmented to obtain multiple standard signal segments;

[0146] Performing voice activity detection on the participating voice signal to obtain a voice activity value of each frame signal in the participating voice signal;

[0147] Determining the start frame and the end frame of multiple valid speech signal segments in the participating speech signal according to multiple speech activity values ​​corresponding to multiple consecutive frame signals in the participating speech signal;

[0148] According to the start frames and end frames of multiple valid voice signal segments in the participating voice signal, the participating voice signal is segmented to obtain multiple participating signal segments.

[0149] In a possible implementation, the segmentation unit 701 is specifically configured to:

[0150] According to the start frames and end frames of the multiple valid voice signal segments in the standard voice signal, the standard voice signal is segmented to obtain multiple first signal segments;

[0151] Filtering first signal segments shorter than a preset time length among the plurality of first signal segments to obtain a plurality of standard signal segments;

[0152] According to the start frames and end frames of multiple valid voice signal segments in the participating voice signal, the participating voice signal is segmented to obtain multiple participating signal segments, including:

[0153] According to the start frames and end frames of the multiple valid voice signal segments in the participating voice signal, the participating voice signal is segmented to obtain multiple second signal segments;

[0154] Second signal segments shorter than a preset duration among the plurality of second signal segments are filtered to obtain a plurality of participating signal segments.

[0155] In a possible implementation, the device further includes: an offset unit;

[0156] The offset unit is used to time-shift the participating voice signal according to the time delay to obtain a shifted voice signal; the shifted voice signal is aligned with the standard voice signal.

[0157] In a possible implementation, the device further includes: an updating unit;

[0158] An updating unit, used to update the delay according to a preset updating strategy to obtain an updated delay;

[0159] Offset unit, specifically used for:

[0160] The participating speech signal is time-shifted according to the updated time delay to obtain a shifted speech signal.

[0161] In a possible implementation manner, the determining unit 704 is further configured to:

[0162] In the voice mixing scenario, if the minimum distance does not meet the preset similarity condition, it is determined that the participating voice signal does not participate in the mixing process.

[0163] It can be seen from the above technical scheme that in the speech alignment scenario, first, the standard speech signal of the standard channel is divided into multiple standard signal segments, and the participating speech signal of the participating channel is divided into multiple participating signal segments; secondly, the pitch period is calculated for each frame signal in the multiple standard signal segments to obtain multiple standard pitch period strings corresponding to the multiple standard signal segments, and the pitch period is calculated for each frame signal in the multiple participating signal segments to obtain multiple participating pitch period strings corresponding to the multiple participating signal segments; the pitch period calculation method obtains the speech features of each signal segment based on the pronunciation features of the valid speech segment. Then, the pitch period string to be matched is selected from the multiple standard pitch period strings, the preset time is determined by the multiple moments corresponding to the pitch period string to be matched, and multiple candidate pitch period strings within the preset time period from the multiple participating pitch period strings are selected; the pitch period string to be matched and the multiple candidate pitch period strings are dynamically time-warped to obtain multiple distances between the pitch period string to be matched and the multiple candidate pitch period strings; based on the speech features of each signal segment, the dynamic time warping method provides a data basis for determining the signal segment of the participating channel that is most similar in speech to the standard channel. Finally, when the minimum distance among multiple distances meets the preset similarity condition, the first alignment point time of the fundamental pitch period string to be matched and the second alignment point time of the candidate fundamental pitch period string corresponding to the minimum distance are used to consider not only the transmission delay of the terminal, but also the processing delay of the terminal and the pronunciation delay of the voice, so as to determine a more accurate delay of the participating channel relative to the standard channel; based on this, the participating voice signal can be more accurately aligned with the standard voice signal through this delay, thereby improving the voice alignment effect.

[0164] The present application also provides a computer device, which may be a server. Figure 8 , Figure 8 A structural diagram of a server provided in an embodiment of the present application, the server 800 may have relatively large differences due to different configurations or performances, and may include one or more processors, such as a central processing unit (CPU) 822, and a memory 832, one or more storage media 830 (such as one or more mass storage devices) storing application programs 842 or data 844. Among them, the memory 832 and the storage medium 830 can be short-term storage or permanent storage. The program stored in the storage medium 830 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the server. Furthermore, the central processing unit 822 can be configured to communicate with the storage medium 830 and execute a series of instruction operations in the storage medium 830 on the server 800.

[0165] The server 800 may also include one or more power supplies 826, one or more wired or wireless network interfaces 850, one or more input and output interfaces 858, and / or one or more operating systems 841, such as Windows Server 2000. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM etc.

[0166] In this embodiment, the central processor 822 in the server 800 may perform the following steps:

[0167] In the speech alignment scenario, the standard speech signal of the standard channel and the participating speech signal of the participating channel are segmented to obtain multiple standard signal segments and multiple participating signal segments;

[0168] For multiple standard signal segments and multiple participating signal segments, calculate the pitch period of each frame signal to obtain multiple standard pitch period strings corresponding to the multiple standard signal segments and multiple participating pitch period strings corresponding to the multiple participating signal segments;

[0169] Dynamic time warping is performed on a pitch cycle string to be matched among multiple standard pitch cycle strings and multiple candidate pitch cycle strings within a preset time period from a preset moment among multiple participating pitch cycle strings to obtain multiple distances between the pitch cycle string to be matched and the multiple candidate pitch cycle strings; the preset moment is determined according to multiple moments corresponding to the pitch cycle string to be matched;

[0170] If the minimum distance among the multiple distances meets the preset similarity condition, the delay of the participating channel relative to the standard channel is determined according to the first alignment point time of the fundamental pitch cycle string to be matched and the second alignment point time of the candidate fundamental pitch cycle string corresponding to the minimum distance.

[0171] The computer device provided in the embodiment of the present application may also be a terminal. Fig. 9 , Fig. 9 A structural diagram of a terminal provided in an embodiment of the present application. Taking a smartphone as an example, the smartphone includes components such as a radio frequency (RF) circuit 910, a memory 920, an input unit 930, a display unit 940, a sensor 950, an audio circuit 960, a wireless fidelity (WiFi) module 970, a processor 980, and a power supply 990. The input unit 930 may include a touch panel 931 and other input devices 932, the display unit 940 may include a display panel 941, and the audio circuit 960 may include a speaker 961 and a microphone 962. Those skilled in the art will appreciate that Fig. 9 The structure of the smartphone shown in the figure does not constitute a limitation of the smartphone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0172] The memory 920 can be used to store software programs and modules. The processor 980 executes various functional applications and data processing of the smartphone by running the software programs and modules stored in the memory 920. The memory 920 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the smartphone (such as audio data, a phone book, etc.), etc. In addition, the memory 920 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0173] The processor 980 is the control center of the smartphone, which uses various interfaces and lines to connect various parts of the entire smartphone, and executes various functions of the smartphone and processes data by running or executing software programs and / or modules stored in the memory 920, and calling data stored in the memory 920. Optionally, the processor 980 may include one or more processing units; preferably, the processor 980 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 980.

[0174] In this embodiment, the processor 980 in the smartphone may perform the following steps:

[0175] In the speech alignment scenario, the standard speech signal of the standard channel and the participating speech signal of the participating channel are segmented to obtain multiple standard signal segments and multiple participating signal segments;

[0176] For multiple standard signal segments and multiple participating signal segments, calculate the pitch period of each frame signal to obtain multiple standard pitch period strings corresponding to the multiple standard signal segments and multiple participating pitch period strings corresponding to the multiple participating signal segments;

[0177] Dynamic time warping is performed on a pitch cycle string to be matched among multiple standard pitch cycle strings and multiple candidate pitch cycle strings within a preset time period from a preset moment among multiple participating pitch cycle strings to obtain multiple distances between the pitch cycle string to be matched and the multiple candidate pitch cycle strings; the preset moment is determined according to multiple moments corresponding to the pitch cycle string to be matched;

[0178] If the minimum distance among the multiple distances meets the preset similarity condition, the delay of the participating channel relative to the standard channel is determined according to the first alignment point time of the fundamental pitch cycle string to be matched and the second alignment point time of the candidate fundamental pitch cycle string corresponding to the minimum distance.

[0179] According to one aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium is used to store a computer program. When the computer program is executed on a computer device, the computer device executes the methods provided in various optional implementations of the above embodiments.

[0180] According to one aspect of the present application, a computer program product is provided, the computer program product comprising a computer program, the computer program being stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the method provided in various optional implementations of the above-mentioned embodiments.

[0181] The descriptions of the processes or structures corresponding to the above-mentioned figures have different emphases. For parts that are not described in detail in a certain process or structure, please refer to the relevant descriptions of other processes or structures.

[0182] The terms "first", "second", etc. in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable in appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, the process, method, system, product or equipment comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0183] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0184] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0185] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0186] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, or all or part of the technical solution. The computer software product is stored in a storage medium, including several instructions for a computer device to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc. Various media that can store computer programs.

[0187] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, ordinary technical members in the art should understand that they can still modify the technical solutions recorded in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for determining time delay based on speech alignment, characterized in that: The method comprises: In the speech alignment scenario, the standard speech signal of the standard channel and the participating speech signal of the participating channel are segmented to obtain multiple standard signal segments and multiple participating signal segments; For the plurality of standard signal segments and the plurality of participating signal segments, the pitch period calculation is performed on each frame signal to obtain a plurality of standard pitch period strings corresponding to the plurality of standard signal segments and a plurality of participating pitch period strings corresponding to the plurality of participating signal segments; Dynamic time warping is performed on a pitch cycle string to be matched among the plurality of standard pitch cycle strings and a plurality of candidate pitch cycle strings within a preset time period from a preset moment among the plurality of participating pitch cycle strings, to obtain a plurality of distances between the pitch cycle string to be matched and the plurality of candidate pitch cycle strings; the preset moment is determined according to a plurality of moments corresponding to the pitch cycle string to be matched; If the minimum distance among the multiple distances meets the preset similarity condition, the delay of the participating channel relative to the standard channel is determined according to the first alignment point time of the fundamental pitch cycle string to be matched and the second alignment point time of the candidate fundamental pitch cycle string corresponding to the minimum distance.

2. The method according to claim 1, characterized in that The step of calculating the pitch period of each frame signal for the plurality of standard signal segments and the plurality of participating signal segments to obtain a plurality of standard pitch period strings corresponding to the plurality of standard signal segments and a plurality of participating pitch period strings corresponding to the plurality of participating signal segments includes: For the plurality of standard signal segments and the plurality of participating signal segments, a plurality of sampling signals of each frame signal are obtained; the maximum number of sampling points corresponding to the plurality of sampling signals is determined according to the sampling frequency and signal frame length of each frame signal; Performing normalization processing on each sampling signal of each frame signal to obtain a normalized signal; Performing autocorrelation calculations on the multiple normalized signals of each frame of the signal to obtain multiple first autocorrelation values ​​of each frame of the signal; the maximum value of the number of delayed sample points corresponding to the multiple first autocorrelation values ​​is determined according to the sampling frequency and the minimum fundamental frequency; Determine the pitch period of each frame of the signal from a plurality of delayed sample points corresponding to a plurality of first autocorrelation values ​​of each frame of the signal; Determine a plurality of standard pitch period strings according to the pitch period of each frame signal in a plurality of the standard signal segments; A plurality of participating pitch period strings are determined according to the pitch period of each frame signal in the plurality of participating signal segments.

3. The method according to claim 2, characterized in that Determining the pitch period of each frame of the signal from a plurality of delayed sample points corresponding to a plurality of first autocorrelation values ​​of each frame of the signal comprises: Filtering the first autocorrelation values ​​that are smaller than the preset correlation value among the multiple first autocorrelation values ​​of each frame signal to obtain multiple second autocorrelation values ​​of each frame signal; Filter the second autocorrelation values ​​whose delayed sample points meet the preset sample point number among the multiple second autocorrelation values ​​of each frame signal to obtain multiple third autocorrelation values ​​of each frame signal; the preset sample point number is determined by the delayed sample point number corresponding to the first second autocorrelation value in the first sorting result obtained by sorting the multiple second autocorrelation values ​​of each frame signal from large to small and the preset coefficient; Sorting the plurality of third autocorrelation values ​​of each frame signal from large to small to obtain a second sorting result; The pitch period of each frame signal is determined according to the number of delayed sample points corresponding to the first third autocorrelation value in the second sorting result.

4. The method according to claim 1, characterized in that The method of performing dynamic time warping on a to-be-matched pitch cycle string among the plurality of standard pitch cycle strings and a plurality of candidate pitch cycle strings within a preset time period from a preset moment among the plurality of participating pitch cycle strings to obtain a plurality of distances between the to-be-matched pitch cycle string and the plurality of candidate pitch cycle strings comprises: For the pitch cycle string to be matched and each of the candidate pitch cycle strings, dynamically time warping multiple pitch cycles in the pitch cycle string to be matched and multiple pitch cycles in the candidate pitch cycle string in sequence according to the sorting order of the pitch cycles until a calculated distance between the last pitch cycle in the pitch cycle string to be matched and the last pitch cycle in the candidate pitch cycle string is obtained; The multiple calculated distances between the last pitch cycle in the to-be-matched pitch cycle string and the last pitch cycle in the multiple candidate pitch cycle strings are determined as the multiple distances.

5. The method according to claim 1, characterized in that The determining, according to the first alignment point time of the pitch cycle string to be matched and the second alignment point time of the candidate pitch cycle string corresponding to the minimum distance, the delay of the participating channel relative to the standard channel includes: Determine a first time according to the standard signal segment corresponding to the pitch cycle string to be matched and the start frame time of the standard speech signal and the first alignment point time; Determine the second time according to the participating signal segment corresponding to the candidate pitch cycle string corresponding to the minimum distance and the start frame time of the participating speech signal and the second alignment point time; The time delay is determined according to a difference between the second time and the first time.

6. The method according to claim 1, characterized in that The step of determining the first alignment point time and the second alignment point time comprises: Performing stationary point calculation on the curve formed by the to-be-matched pitch cycle string to obtain the first stationary point in the to-be-matched pitch cycle string; Determine the first alignment point time according to the time when the first stationary point is in the pitch cycle string to be matched; Performing stationary point calculation on a curve formed by the candidate pitch cycle string corresponding to the minimum distance to obtain a second stationary point in the candidate pitch cycle string corresponding to the minimum distance; The second alignment point time is determined according to the time when the second stationary point is in the candidate pitch cycle string corresponding to the minimum distance.

7. The method according to claim 1, characterized in that The signal segmentation of the standard voice signal of the standard channel and the participating voice signal of the participating channel to obtain a plurality of standard signal segments and a plurality of participating signal segments includes: Performing voice activity detection on the standard voice signal to obtain a voice activity value of each frame signal in the standard voice signal; Determining the start frame and the end frame of multiple valid voice signal segments in the standard voice signal according to multiple voice activity values ​​corresponding to multiple consecutive frames of signals in the standard voice signal; According to the start frame and the end frame of the multiple valid voice signal segments in the standard voice signal, the standard voice signal is segmented to obtain multiple standard signal segments; Performing voice activity detection on the participating voice signal to obtain a voice activity value of each frame signal in the participating voice signal; Determining the start frame and the end frame of multiple valid voice signal segments in the participating voice signal according to multiple voice activity values ​​corresponding to multiple consecutive frames of signals in the participating voice signal; According to the start frames and end frames of a plurality of valid voice signal segments in the participating voice signal, the participating voice signal is segmented to obtain a plurality of participating signal segments.

8. The method according to claim 7, characterized in that The method of performing signal segmentation on the standard voice signal according to the start frame and the end frame of the plurality of valid voice signal segments in the standard voice signal to obtain the plurality of the standard signal segments comprises: According to the start frames and end frames of the multiple valid voice signal segments in the standard voice signal, the standard voice signal is segmented to obtain multiple first signal segments; Filtering the first signal segments shorter than a preset time length among the plurality of the first signal segments to obtain a plurality of the standard signal segments; The step of performing signal segmentation on the participating voice signal according to the start frames and the end frames of the multiple valid voice signal segments in the participating voice signal to obtain the multiple participating signal segments comprises: According to the start frames and end frames of the multiple valid voice signal segments in the participating voice signal, the participating voice signal is segmented to obtain multiple second signal segments; Second signal segments shorter than the preset duration among the plurality of second signal segments are filtered to obtain the plurality of participating signal segments.

9. The method according to any one of claims 1 to 8, characterized in that The method further comprises: The participating speech signal is time-shifted according to the time delay to obtain a shifted speech signal; and the shifted speech signal is aligned with the standard speech signal.

10. The method according to claim 9, characterized in that The method further comprises: Updating the delay according to a preset update strategy to obtain an updated delay; The performing time shifting on the participating voice signal according to the time delay to obtain the shifted voice signal is specifically: The participating speech signal is time-shifted according to the updated time delay to obtain the shifted speech signal.

11. The method according to claim 1, characterized in that: The method further comprises: In a voice mixing scenario, if the minimum distance does not meet the preset similarity condition, it is determined that the participating voice signal does not participate in the mixing process.

12. A delay determination device based on speech alignment, characterized in that: The device comprises: a segmentation unit, a calculation unit, a regularization unit and a determination unit; The segmentation unit is used to perform signal segmentation on the standard speech signal of the standard channel and the participating speech signal of the participating channel in the speech alignment scenario to obtain multiple standard signal segments and multiple participating signal segments; The calculation unit is used to calculate the pitch period of each frame signal for the multiple standard signal segments and the multiple participating signal segments, and obtain a plurality of standard pitch period strings corresponding to the multiple standard signal segments and a plurality of participating pitch period strings corresponding to the multiple participating signal segments; The regularization unit is used to perform dynamic time regularization on the pitch cycle string to be matched in the multiple standard pitch cycle strings and the multiple candidate pitch cycle strings within a preset time period from the preset moment in the multiple participating pitch cycle strings, so as to obtain multiple distances between the pitch cycle string to be matched and the multiple candidate pitch cycle strings; the preset moment is determined according to the multiple moments corresponding to the pitch cycle string to be matched; The determination unit is used to determine the delay of the participating channel relative to the standard channel according to the first alignment point time of the fundamental pitch cycle string to be matched and the second alignment point time of the candidate fundamental pitch cycle string corresponding to the minimum distance if the minimum distance among the multiple distances meets the preset similarity condition.

13. A computer device, characterized in that: The computer device comprises a processor and a memory: The memory is used to store a computer program and transmit the computer program to the processor; The processor is configured to execute the method according to any one of claims 1 to 11 according to the instructions in the computer program.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and when the computer program is executed on a computer device, the computer device executes the method according to any one of claims 1 to 11.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed on a computer device, the computer device is caused to execute the method according to any one of claims 1 to 11.

Images