Speech processing method and apparatus, electronic device, and storage medium

Abstract

The application discloses a speech processing method and device, electronic equipment and a storage medium. The method comprises: an electronic device divides a first audio to obtain sub-audios corresponding to N time periods, one time period corresponds to one sub-audio, and N is a positive integer; in the case that the first sub-audio corresponding to the first time period is interrupted by abnormal speech, if it is determined that the second sub-audio corresponding to the second time period can be used to fill the first sub-audio, the first sub-audio is filled based on the second sub-audio to obtain a third sub-audio corresponding to the first time period; the third sub-audio corresponding to the first time period and the sub-audios corresponding to other time periods in the N time periods are combined to obtain a second audio; wherein the first time period is at least one time period in the N time periods; the second time period includes at least one of the following: a time period adjacent to the first time period before the first time period, and a time period adjacent to the first time period after the first time period.

Description

Technical Field

[0001] This application belongs to the field of audio technology, specifically relating to a speech processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the development of electronic devices, many electronic device applications have the function of sending voice messages, and chatting or commenting by transmitting real-time or non-real-time voice messages is becoming increasingly popular.

[0003] However, the environments in which users send voice messages are diverse and complex. Entering an elevator, switching networks, or other factors can cause the voice message to be interrupted or choppy. This can result in the loss of voice information sent by electronic devices. Summary of the Invention

[0004] The purpose of this application is to provide a voice processing method, apparatus, electronic device, and storage medium that can solve the problem of missing voice information sent by electronic devices.

[0005] In a first aspect, embodiments of this application provide a speech processing method, the method comprising: dividing a first audio segment to obtain N sub-audio segments corresponding to time periods, each time period corresponding to one sub-audio segment, where N is a positive integer; if an abnormal speech interruption occurs in the first sub-audio segment corresponding to the first time period, and if it is determined that a second sub-audio segment corresponding to a second time period can be used to complete the first sub-audio segment corresponding to the first time period, then completing the first sub-audio segment based on the second sub-audio segment to obtain a third sub-audio segment corresponding to the first time period; combining the third sub-audio segment corresponding to the first time period with sub-audio segments corresponding to other time periods among the N time periods to obtain a second audio segment; wherein the first time period is at least one of the N time periods; the second time period includes at least one of the following: the time period adjacent to the first time period, and the time period adjacent to the first time period.

[0006] Secondly, embodiments of this application provide a voice processing device, which includes a processing module and a combination module. The processing module is used to divide a first audio segment into N sub-audio segments corresponding to time periods, where one time period corresponds to one sub-audio segment, and N is a positive integer. If an abnormal voice interruption occurs in the first sub-audio segment corresponding to the first time period, and if it is determined that a second sub-audio segment corresponding to a second time period can be used to complete the first sub-audio segment corresponding to the first time period, then the first sub-audio segment is completed based on the second sub-audio segment to obtain a third sub-audio segment corresponding to the first time period. The combination module is used to combine the third sub-audio segment corresponding to the first time period with sub-audio segments corresponding to other time periods among the N time periods to obtain a second audio segment. The first time period is at least one of the N time periods; the second time period includes at least one of the following: a time period adjacent to the first time period, and a time period adjacent to the first time period.

[0007] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, wherein the memory stores programs or instructions executable on the processor, and the programs or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0008] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0009] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.

[0010] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method described in the first aspect.

[0011] In this embodiment, the electronic device can divide the first audio into N sub-audio segments corresponding to different time periods. If there is an abnormal voice interruption in the first sub-audio segment corresponding to the first time period among the N time periods, and if it is determined that the second sub-audio segment corresponding to the second time period can be used to complete the first sub-audio segment, the electronic device completes the first sub-audio segment based on the second sub-audio segment to obtain the third sub-audio segment corresponding to the first time period. Then, the electronic device can combine the third sub-audio segment corresponding to the first time period with the sub-audio segments corresponding to other time periods among the N time periods to obtain the second audio segment. In this scheme, the electronic device can divide the first audio to be sent into N sub-audio segments corresponding to time periods. Then, it detects whether there are abnormal voice interruptions in the sub-audio segments corresponding to the N time periods. When an abnormal voice interruption is detected in the first sub-audio segment corresponding to the first time period, it determines whether the second sub-audio segment corresponding to the second time period can be used to complete the first sub-audio segment. If the second sub-audio segment can be used to complete the first sub-audio segment, the electronic device can complete the first sub-audio segment based on the second sub-audio segment to obtain the processed first sub-audio segment, i.e., the third sub-audio segment corresponding to the first time period. Then, the electronic device can combine the third sub-audio segment with the sub-audio segments corresponding to other time periods in the N time periods to obtain the processed first audio segment (i.e., the second audio segment), so that the electronic device can send complete voice information. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of a speech processing method provided in an embodiment of this application;

[0013] Figure 2 This is a schematic flowchart of a speech processing method provided in an embodiment of this application;

[0014] Figure 3 This is a schematic diagram of the structure of a voice processing device provided in an embodiment of this application;

[0015] Figure 4 This is one of the hardware structure diagrams of an electronic device provided in the embodiments of this application;

[0016] Figure 5 This is a second schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0018] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0019] The speech processing method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0020] The speech processing method in this application embodiment can be applied to scenarios where interrupted speech needs to be completed.

[0021] Currently, many electronic devices have voice message sending capabilities. Users are in diverse and complex environments when sending voice messages. Issues such as entering elevators, switching networks, or sudden environmental noise can cause voice messages to be intermittent, have impurities, or experience pauses and repetitions, affecting the recipient's comprehension and listening experience. Related technologies can improve voice intelligibility by eliminating environmental noise and repairing voice information quality after detecting distortion. However, these existing methods still have limitations when voice messages are interrupted or discontinuous due to environmental or network factors. For example, eliminating environmental noise cannot solve the problem of voice interruption and may even degrade sound quality to some extent; while methods for detecting and repairing voice distortion can recover the distorted audio, detecting and restoring interrupted voice messages is difficult. This can result in missing voice information sent by electronic devices.

[0022] In the solution provided in this application embodiment, the electronic device can divide the first audio into N sub-audio segments corresponding to time periods. If there is an abnormal voice interruption in the first sub-audio segment corresponding to the first time period in the N time periods, and if it is determined that the second sub-audio segment corresponding to the second time period can be used to complete the first sub-audio segment, the electronic device completes the first sub-audio segment based on the second sub-audio segment to obtain the third sub-audio segment corresponding to the first time period. Then, the electronic device can combine the third sub-audio segment corresponding to the first time period with the sub-audio segments corresponding to other time periods in the N time periods to obtain the second audio segment. In this scheme, the electronic device can divide the first audio to be sent into N sub-audio segments corresponding to time periods. Then, it detects whether there are abnormal voice interruptions in the sub-audio segments corresponding to the N time periods. When an abnormal voice interruption is detected in the first sub-audio segment corresponding to the first time period, it determines whether the second sub-audio segment corresponding to the second time period can be used to complete the first sub-audio segment. If the second sub-audio segment can be used to complete the first sub-audio segment, the electronic device can complete the first sub-audio segment based on the second sub-audio segment to obtain the processed first sub-audio segment, i.e., the third sub-audio segment corresponding to the first time period. Then, the electronic device can combine the third sub-audio segment with the sub-audio segments corresponding to other time periods in the N time periods to obtain the processed first audio segment (i.e., the second audio segment), so that the electronic device can send complete voice information.

[0023] This application provides a speech processing method. Figure 1 A flowchart illustrating a speech processing method provided in an embodiment of this application is shown, which can be applied to electronic devices. Figure 1 As shown, the voice processing method provided in this application embodiment may include the following steps 201 to 203.

[0024] Step 201: The electronic device divides the first audio into N sub-audio segments corresponding to different time periods.

[0025] In this embodiment of the application, a time period corresponds to a sub-audio, and N is a positive integer.

[0026] Optionally, in this embodiment of the application, the first audio can be the audio corresponding to an instant voice message or a non-instant voice message sent by the user on an electronic device application.

[0027] Optionally, in this embodiment of the application, the duration of a time period can be t1, and the value of t1 can be 10ms to 20ms.

[0028] Optionally, in this embodiment of the application, after step 201 above, the speech processing method provided in this embodiment of the application further includes the following steps 301 and 302.

[0029] Step 301: The electronic device acquires the first time-domain audio signal corresponding to the sub-audio of N time periods.

[0030] In this embodiment of the application, a time period corresponds to a first time-domain audio signal.

[0031] In this embodiment of the application, after the electronic device obtains sub-audio frequencies of multiple time periods, it can acquire the first time-domain audio signal corresponding to each sub-audio frequency.

[0032] Step 302: The electronic device performs frequency domain conversion on the first time-domain audio signals corresponding to N time periods to obtain the first frequency-domain audio signals in N frequency domains.

[0033] Optionally, in this embodiment of the application, the electronic device can window and perform Fourier transform on the first time-domain audio signal corresponding to each time period to obtain the first frequency-domain audio signal in N frequency domains.

[0034] Optionally, in the embodiments of this application, since the audio signal energy is more concentrated in the low frequency part of 0 to 4000 Hz, and the speech harmonics are generally spaced between 50 and 100 Hz, the electronic device can take a frequency band every 50 Hz, so 80 frequency bands can be taken in the range of 0 to 4000 Hz, that is, each first frequency domain audio signal can be a frequency domain audio signal in 80 frequency bands.

[0035] For example, the range of a first frequency domain audio signal in the kth frequency band can be [(k-1)*50,k*50]Hz.

[0036] For example, the first frequency domain audio signal in the k-th frequency band corresponding to the l-th time period can be represented as Y(k,l).

[0037] Step 202: If there is an abnormal voice interruption in the first sub-audio corresponding to the first time period, and if it is determined that the second sub-audio corresponding to the second time period can be used to complete the first sub-audio corresponding to the first time period, the electronic device completes the first sub-audio based on the second sub-audio to obtain the third sub-audio corresponding to the first time period.

[0038] The first time period is at least one of the N time periods.

[0039] In this embodiment of the application, the second time period includes at least one of the following: the time period adjacent to the first time period, and the time period adjacent to the first time period.

[0040] Optionally, in this embodiment of the application, the second time period includes the time period adjacent to the first time period, which may be one or more time periods adjacent to the first time period; similarly, the second time period includes the time period adjacent to the first time period, which may be one or more time periods adjacent to the first time period.

[0041] Optionally, in this embodiment of the application, the step 202 above, "if it is determined that the second sub-audio corresponding to the second time period can be used to complete the first sub-audio corresponding to the first time period, the electronic device completes the first sub-audio based on the second sub-audio to obtain the third sub-audio corresponding to the first time period", can be specifically implemented through the following step 202a.

[0042] Step 202a: If it is determined that the first frequency domain audio signal corresponding to the second time period can be used to complete the first frequency domain audio signal corresponding to the first time period, the electronic device completes the first frequency domain audio signal corresponding to the first time period based on the first frequency domain audio signal corresponding to the second time period to obtain the second frequency domain audio signal.

[0043] Optionally, in this embodiment of the application, since each first frequency domain signal is a frequency domain audio signal in multiple frequency bands, the electronic device can supplement the first frequency domain audio signals in all or part of the frequency bands corresponding to the first time period based on the first frequency domain audio signal corresponding to the second time period. It can be understood that the part of the frequency bands corresponding to the first time period is the frequency band where the signal energy of the first frequency domain audio signal is concentrated.

[0044] It should be noted that the specific implementation method for the electronic device to obtain the second frequency domain audio signal will be described in the following embodiments, and will not be repeated here.

[0045] Optionally, in the embodiments of this application, the above step 202a can be specifically implemented by the following steps 202a1 to 202a3.

[0046] Step 202a1: For each frequency band corresponding to the first time period, if the minimum energy difference in the energy difference set corresponding to a frequency band is less than the first threshold, the electronic device determines that the first frequency domain audio signal corresponding to the second time period can be used to complete the first frequency domain audio signal corresponding to a frequency band in the first time period.

[0047] In this set of energy differences, each energy difference corresponds to a set of frequency bands.

[0048] Among them, one energy difference in the energy difference set is: the difference between the first energy value of the first frequency domain audio signal corresponding to X time periods before the first time period in the target frequency band and the second energy value of the first frequency domain audio signal corresponding to X time periods after the first time period in the target frequency band. The target frequency band includes any one of the following: the aforementioned frequency band, the first frequency band, or the second frequency band; the frequency band group corresponding to one energy difference includes any two of the following: the aforementioned frequency band, the first frequency band, or the second frequency band, wherein the first frequency band and the second frequency band are frequency bands adjacent to each other.

[0049] Optionally, in this embodiment of the application, the first frequency domain audio signal corresponding to a frequency band can be the first frequency domain audio signal of any frequency band corresponding to a time period.

[0050] Optionally, in the embodiments of this application, the first frequency band can be a frequency band adjacent to the first frequency band, and the second frequency band can be a frequency band adjacent to the second frequency band.

[0051] It is understood that each first frequency domain signal is a frequency domain audio signal in multiple frequency bands. When there is an abnormal voice interruption in the first sub-audio corresponding to the first time period, the electronic device can determine whether the first frequency domain audio signal in a frequency band corresponding to the first time period can be supplemented by the first frequency domain audio signal in the target frequency band corresponding to the X time periods before and after the first time period. When the first frequency domain audio signal in the target frequency band in the X time periods before and after the first time period can supplement the first frequency domain audio signal in a frequency band corresponding to the first time period, the first frequency domain audio signal in the target frequency band corresponding to the first time period is supplemented based on the first frequency domain audio signal in the target frequency band in the X time periods before and after the first time period.

[0052] Optionally, in this embodiment, the formula for calculating an energy difference is as follows:

[0053]

[0054] k1,k2s.t.min k-1≤k1,k2≤k+1 DY(k1,k2,l0)

[0055] Wherein, l0 represents the first time period; k1 and k2 represent the target frequency band; t0 represents the X time periods before or after the first time period; Y(k1,l) represents the first frequency domain audio signal in the k1 frequency band during one of the X time periods before the first time period; Y(k2,l) represents the first frequency domain audio signal in the k2 frequency band during one of the X time periods after the first time period; k-1 represents the first frequency band; and k+1 represents the second frequency band.

[0056] For example, when one of the above frequency bands is the 10th frequency band in the target frequency band, the first frequency band is the 9th frequency band in the target frequency band, and the second frequency band is the 11th frequency band in the target frequency band; then the target frequency band can be any of the following: the 9th frequency band, the 10th frequency band, and the 11th frequency band; then a set of frequency bands (k1, k2) can be any of the following: (9th frequency band, 9th frequency band), (9th frequency band, 10th frequency band), (9th frequency band, 11th frequency band), (10th frequency band, 9th frequency band), (10th frequency band, 10th frequency band), (10th frequency band, 11th frequency band), (11th frequency band, 9th frequency band), (11th frequency band, 10th frequency band) and (11th frequency band, 11th frequency band).

[0057] It can be understood that each of the above frequency bands includes two frequency bands: one frequency band corresponds to the X time periods before the first time period; the other frequency band corresponds to the X time periods after the first time period.

[0058] For example, by substituting a set of frequency bands (the 10th frequency band, the 9th frequency band) into the above formula, we can obtain an energy difference value: the difference between the first energy value of the first frequency domain audio signal in the 10th frequency band corresponding to the X time periods before the first time period, and the second energy value of the first frequency domain audio signal in the 9th frequency band corresponding to the X time periods after the first time period.

[0059] It is understandable that for multiple different frequency band groups, i.e. different k1 and k2, electronic devices can obtain multiple different energy differences, thus obtaining a set of energy differences.

[0060] Optionally, in this embodiment of the application, the electronic device can compare the minimum energy difference in the energy difference set with a first threshold. When the minimum energy difference is less than the first threshold, it can confirm that the first frequency domain audio signal in a frequency band corresponding to the first time period can be supplemented according to the first frequency domain audio signal corresponding to the second time period.

[0061] For example, I(k,l0) can be used to indicate whether the minimum energy difference is less than the first threshold thre_minDY. When the minimum energy difference is less than the first threshold, I(k,l0) = 1; when the minimum energy difference is greater than the first threshold, I(k,l0) = 0.

[0062] Step 202a2: The electronic device supplements the first frequency domain audio signal corresponding to a frequency band in the first time period based on the first frequency domain audio signal corresponding to the first time period in the first frequency band group, so as to obtain the target first frequency domain audio signal.

[0063] In this embodiment of the application, the first frequency band group is the frequency band group corresponding to the minimum energy difference.

[0064] Optionally, in this embodiment of the application, the electronic device can obtain a set of frequency bands corresponding to the minimum energy difference, that is, obtain k1 and k2, and record k1 and k2 corresponding to the minimum energy difference as k1_opt and k2_opt; then calculate the target first frequency domain audio signal corresponding to a frequency band in the first time period based on the first frequency domain audio signal corresponding to k1_opt and k2_opt.

[0065] For example, the calculation formula for the target first frequency domain audio signal corresponding to a frequency band in the first time period is as follows:

[0066]

[0067]

[0068] Where Y(k,l0) represents the first frequency domain audio signal in a frequency band during the first time period; Gain(l,l0) represents the signal gain; Y(k1) represents the signal gain. opt Y(k2) represents the first frequency domain audio signal in the k1_opt frequency band during one of the X time periods preceding the first time period; opt ,l) represents the first frequency domain audio signal in the k2_opt frequency band during one of the X time periods following the first time period.

[0069] Step 202a3: The electronic device replaces the first frequency domain audio signal corresponding to a frequency band in the first time period with the target first frequency domain audio signal to obtain the second frequency domain audio signal.

[0070] It is understandable that after the electronic device obtains the target first frequency domain audio signal corresponding to a frequency band in the first time period, it can replace the first frequency domain audio signal corresponding to a frequency band in the first time period with the target first frequency domain audio signal. Then, for the first frequency domain audio signal in all or some frequency bands corresponding to the first time period, the electronic device needs to execute the above step 202a2 multiple times to obtain the target first frequency domain audio signal corresponding to the first time period in all or some frequency bands respectively, thereby obtaining the second frequency domain audio signal corresponding to the first time period.

[0071] Step 203: The electronic device combines the third sub-audio corresponding to the first time period with the sub-audio corresponding to other time periods in the N time periods to obtain the second audio.

[0072] In this embodiment of the application, after obtaining the third sub-audio corresponding to the first time period, the electronic device can combine the third sub-audio corresponding to the first time period with the sub-audio corresponding to other time periods in N time periods to obtain the first audio after speech recovery, namely the second audio.

[0073] Optionally, in the embodiments of this application, step 203 can be implemented by steps 203a and 203b as described below.

[0074] Step 203a: The electronic device performs time-domain conversion on the frequency domain audio signals corresponding to N time periods to obtain the second time-domain audio signals in N time domains.

[0075] In this embodiment of the application, the electronic device can perform time-domain conversion (i.e., windowing and inverse Fourier transform) on the second frequency domain audio signal corresponding to the first time period and the frequency domain audio signals corresponding to the time periods other than the first time period in N time periods to obtain the second time domain audio signal in N time domains.

[0076] Step 203b: The electronic device combines the second time-domain audio signals in the N time domains according to the time sequence corresponding to the N time periods to obtain the second audio.

[0077] In this embodiment of the application, after the electronic device obtains N second time-domain audio signals in the time domain, it can combine the N second time-domain audio signals in the time order corresponding to the N time periods to obtain the second audio.

[0078] This application provides a voice processing method. An electronic device can divide a first audio segment to be transmitted into N sub-audio segments corresponding to different time periods. It then detects whether any abnormal voice interruptions exist in the sub-audio segments corresponding to the N time periods. When an abnormal voice interruption is detected in the first sub-audio segment corresponding to the first time period, it determines whether a second sub-audio segment corresponding to the second time period can be used to complete the first sub-audio segment. If the second sub-audio segment can be used to complete the first sub-audio segment, the electronic device can complete the first sub-audio segment based on the second sub-audio segment to obtain the processed first sub-audio segment, i.e., the third sub-audio segment corresponding to the first time period. The electronic device can then combine the third sub-audio segment with the sub-audio segments corresponding to other time periods within the N time periods to obtain the processed first audio segment (i.e., the second audio segment), thereby enabling the electronic device to send complete voice information.

[0079] Optionally, in this embodiment of the application, before step 202 above, the speech processing method provided in this embodiment of the application may further include the following step 401.

[0080] Step 401: If the audio time-domain energy parameters corresponding to the first sub-audio satisfy the first condition, the electronic device determines that the first sub-audio contains speech information.

[0081] The audio time-domain energy parameters include: the audio time-domain energy of the first sub-audio, the average audio energy of the sub-audio in the first sub-audio set, the minimum audio time-domain energy among the audio time-domain energies of the sub-audio corresponding to N time periods, and the minimum audio time-domain energy among the audio time-domain energies of the sub-audio in the first sub-audio set.

[0082] In this embodiment of the application, the first sub-audio set includes: a first sub-audio, and M sub-audio adjacent to the first sub-audio.

[0083] Optionally, in this embodiment of the application, the audio time-domain energy of the first sub-audio is used to indicate the energy information of the first audio over a short period of time.

[0084] For example, the formula for calculating the audio temporal energy of the first sub-audio is as follows:

[0085]

[0086] Where l represents the first time period; E(l) represents the audio temporal energy of the first sub-audio corresponding to the first time period; t1 represents the duration of a time period; and e(t) represents the audio temporal energy at time t.

[0087] For example, suppose the first time period is the first time period in N time periods, and the duration of each time period is 10ms; then the audio time domain energy of the first sub-audio corresponding to the first time period is the sum of the audio time domain energy from the 1ms to the 10ms.

[0088] Optionally, in this embodiment of the application, the average audio energy of the sub-audio in the first sub-audio set is used to track the energy level change of the first audio.

[0089] For example, the M sub-audio sets adjacent to the first sub-audio can be sub-audio corresponding to a time period within a predetermined duration preceding the first time period. The predetermined duration can be t2, and the value of t2 can be 50ms to 100ms.

[0090] For example, suppose the duration of a time period is 10ms and t2 is 50ms; then the above M sub-audio frequencies are the sub-audio frequencies corresponding to the time periods within the adjacent 50ms before the first time period, that is, the 5 sub-audio frequencies corresponding to the 5 time periods adjacent to the first time period.

[0091] For example, the formula for calculating the average audio energy of the sub-audio in the first sub-audio set is as follows:

[0092] When l*t1 < t2

[0093] When l*t1≥t2

[0094] Where l represents the first time period; E(i) represents the audio temporal energy of the i-th time period divided by t1; t1 represents the duration of a time period; and t2 represents the total duration of the M time periods corresponding to the M sub-audio segments.

[0095] For example, suppose the first time period is the third time period out of N time periods, the duration of a time period is 10ms, and t2 is 50ms; then l*t1<t2; if the audio time domain energy of the first time period out of N time periods is 20, the audio time domain energy of the second time period is 30, and the audio time domain energy of the third time period is 30, then the average audio energy of the sub-audio in the first sub-audio set is: (20+30+40) / 3=30.

[0096] Optionally, in this embodiment, the formula for calculating the minimum audio time-domain energy among the sub-audio energies corresponding to the above N time periods is as follows:

[0097] minE_global=min{E(1),E(2),...,E(N)}

[0098] Optionally, in this embodiment, the formula for calculating the minimum audio time-domain energy among the sub-audio energies in the first sub-audio set is as follows:

[0099] minE_local(l)=min{E(l-t3+1),E(l-t3+2),...,E(l)}

[0100] Where l represents the first time period; t3 represents the total duration of the M time periods corresponding to the M sub-audio segments.

[0101] For example, the value of t3 can be 50ms to 100ms. It can be understood that when the values ​​of t2 and t3 are the same, the first sub-audio set includes the same M sub-audio; when the values ​​of t2 and t3 are different, the first sub-audio set includes different M sub-audio.

[0102] In this embodiment of the application, the first condition includes: the audio time-domain energy of the first sub-audio is greater than the average audio energy of the sub-audio in the first sub-audio set; or, the audio time-domain energy of the first sub-audio is greater than the minimum audio time-domain energy among the audio time-domain energies of the sub-audio corresponding to N time periods, and is greater than the minimum audio time-domain energy among the audio time-domain energies of the sub-audio in the first sub-audio set.

[0103] Optionally, in the embodiments of this application, I(l) = 1 can be used to indicate that the first sub-audio contains speech information, and I(l) = 0 can be used to indicate that the first sub-audio does not contain speech information.

[0104] For example, the audio time-domain energy of the first sub-audio is greater than the average audio energy of the sub-audio in the first sub-audio set, i.e., E(l) > E. mean(l) At that time, the electronic device can determine that the first sub-audio is a transition segment from noise to speech, and thus determine that the first sub-audio contains speech information, i.e., I(l) = 1.

[0105] For example, when the audio time-domain energy of the first sub-audio is greater than the minimum audio time-domain energy among the audio time-domain energies of the sub-audio corresponding to the N time periods, and is also greater than the minimum audio time-domain energy among the audio time-domain energies of the sub-audio in the first sub-audio set, i.e., E(l) > minE_global and E(l) > minE_local(l), the electronic device can determine that the first sub-audio is a speech mid-segment, thereby determining that the first sub-audio contains speech information, i.e., I(l) = 1.

[0106] For example, when the difference between the audio time-domain energy of the first sub-audio and the minimum audio time-domain energy of the sub-audio in the first sub-audio set is less than a preset threshold, i.e., E(l)-minE_local(l)<thre1, the electronic device can determine that the first sub-audio is a noise segment, thereby determining that the first sub-audio does not contain speech information, i.e., I(l)=0.

[0107] Where thre1 is a preset threshold.

[0108] For example, when the audio time-domain energy of the first sub-audio is less than the minimum audio time-domain energy among the sub-audio energies in the first sub-audio set, i.e., E(l) < minE_local(l), the electronic device can determine that the first sub-audio is a transition from the speech end segment to the noise segment, thereby determining that the first sub-audio does not contain speech information, i.e., I(l) = 0.

[0109] For example, when an electronic device continuously detects that the sub-audio corresponding to multiple time periods does not contain speech information, i.e., I(l) = 0, the sub-audio corresponding to these multiple time periods may be speech intervals. Therefore, when the electronic device detects the sub-audio corresponding to the next M time periods, it can correct the I(l) result to 1 to ensure that the speech intervals are not cut off.

[0110] Optionally, in the embodiments of this application, step 202 above can be specifically implemented by step 202b below.

[0111] Step 202b: If the first sub-audio corresponding to the first time period contains voice information and there is an abnormal voice interruption in the first sub-audio, and it is determined that the second sub-audio corresponding to the second time period can be used to complete the first sub-audio, then the electronic device completes the first sub-audio based on the second sub-audio to obtain the third sub-audio corresponding to the first time period.

[0112] In this embodiment of the application, after the electronic device acquires the first audio and divides the first audio into N sub-audio segments corresponding to time periods, it can first determine whether the first sub-audio segment corresponding to the first time period contains voice information; then, if the first sub-audio segment corresponding to the first time period contains voice information, it can further determine whether there is an abnormal voice interruption in the first sub-audio segment; so that when it is determined that there is a voice interruption in the first sub-audio segment, if it is determined that the second sub-audio segment corresponding to the second time period can be used to complete the first sub-audio segment, the electronic device can complete the first sub-audio segment based on the second sub-audio segment to obtain the third sub-audio segment corresponding to the first time period.

[0113] Optionally, in this embodiment of the application, after step 201 above, the speech processing method provided in this embodiment of the application further includes the following step 501.

[0114] Step 501: If the mean of the first variable corresponding to all frequency bands in the target frequency band is greater than the second threshold, and the mean of the second variable corresponding to all frequency bands in the target frequency band is less than the third threshold, the electronic device determines that there is an abnormal voice interruption in the first sub-audio corresponding to the first time period.

[0115] Among them, a first variable is used to indicate the spectral difference between the first frequency domain audio signal corresponding to the first time period and the first frequency domain audio signal corresponding to the third time period in a frequency band of the target frequency band; a second variable is used to indicate the spectral similarity between the first frequency domain audio signal corresponding to the first time period and the first frequency domain audio signal corresponding to the third time period in a frequency band of the target frequency band.

[0116] The third time period includes: the time period within the predetermined duration before the first time period, and the time period within the predetermined duration after the first time period.

[0117] In this embodiment of the application, for the first frequency domain audio signal corresponding to the first time period, the electronic device can detect whether the first frequency domain audio signal corresponding to the third time period is smooth compared to the first frequency domain audio signal corresponding to the first time period.

[0118] Optionally, in this embodiment of the application, the predetermined duration before the first time period and the predetermined duration after the first time period can be 100-200ms.

[0119] For example, suppose the duration of a time period is 10ms, and the predetermined duration before the first time period and the predetermined duration after the first time period are 100ms; then the aforementioned third time period may include: the 10 adjacent time periods before the first time period and the 10 adjacent time periods after the first time period.

[0120] For example, the formula for calculating one of the first variables is as follows:

[0121]

[0122] Where k represents the k-th frequency band; l represents the first time period; t_local represents the predetermined duration before or after the first time period; Y(k,l) represents the first frequency domain audio signal in the k-th frequency band corresponding to the l-th time period; Y(k,lt) represents the first frequency domain audio signal in the k-th frequency band corresponding to a time period within the predetermined duration before the l-th time period; and Y(k,l+t) represents the first frequency domain audio signal in the k-th frequency band corresponding to a time period within the predetermined duration after the l-th time period.

[0123] For example, the smaller the value of the first variable, the smoother the frequency energy transition between the first frequency domain audio signal corresponding to the first time period and the first frequency domain audio signal corresponding to the third time period in the Kth frequency band of the target frequency band.

[0124] Optionally, in this embodiment of the application, the width of the target frequency band can be 0 to 4000 Hz. Since voice harmonics are generally spaced between 50 and 100 Hz, the electronic device can take a frequency band every 50 Hz, so the target frequency band can include 80 frequency bands.

[0125] Optionally, in this embodiment of the application, the electronic device needs to calculate the spectral differences between the first frequency domain audio signal corresponding to the first time period and the first frequency domain audio signal corresponding to the third time period in each frequency band of the target frequency band, so as to obtain the first variable corresponding to all frequency bands in the target frequency band.

[0126] Optionally, in this embodiment, the calculation formula for the aforementioned second variable is as follows:

[0127]

[0128] Where X*Y * This indicates the conjugate multiplication of complex numbers.

[0129] It should be noted that the descriptions of other parameters in the above formulas can be found in the descriptions in the above embodiments, and will not be repeated here.

[0130] For example, the larger the value of the second variable, the smoother the frequency energy transition between the first frequency domain audio signal corresponding to the first time period and the first frequency domain audio signal corresponding to the third time period in the Kth frequency band of the target frequency band.

[0131] For example, the electronic device needs to calculate the spectral similarity between the first frequency domain audio signal corresponding to the first time period and the first frequency domain audio signal corresponding to the third time period in each frequency band of the target frequency band, so as to obtain the second variable corresponding to all frequency bands in the target frequency band.

[0132] Optionally, in this embodiment of the application, the electronic device obtains the first variables and second variables corresponding to all frequency bands in the target frequency band. That is, after obtaining multiple first variables and multiple second variables, the mean of the multiple first variables and the mean of the multiple second variables can be calculated.

[0133] For example, if the mean of the first variable corresponding to all frequency bands in the target frequency band is greater than the second threshold, and the mean of the second variable corresponding to all frequency bands in the target frequency band is less than the third threshold, that is, when the mean of SD(k,l) is greater than the threshold thre_SD, and the mean of SR(k,l) is less than the threshold thre_SR, the electronic device can determine that there is an abnormal voice interruption in the first sub-audio corresponding to the first time period.

[0134] Optionally, in this embodiment of the application, before step 501 above, the speech processing method provided in this embodiment of the application further includes the following step 501a.

[0135] Step 501a: If the mean of the third variable corresponding to all frequency bands in the target frequency band is greater than the fourth threshold, and the mean of the fourth variable corresponding to all frequency bands in the target frequency band is less than the fifth threshold, the electronic device determines that there is an abnormal voice interruption or voice interval in the first sub-audio corresponding to the first time period.

[0136] Among them, a third variable is used to indicate the spectral difference between the first frequency domain audio signal corresponding to the first time period and the first frequency domain audio signal corresponding to the fourth time period in a frequency band of the target frequency band; a fourth variable is used to indicate the spectral similarity between the first frequency domain audio signal corresponding to the first time period and the first frequency domain audio signal corresponding to the fourth time period in a frequency band of the target frequency band.

[0137] The fourth time period includes: the time period within the predetermined duration before the first time period, and the time period within the predetermined duration after the first time period.

[0138] Optionally, in this embodiment of the application, since the duration of a typical speech interval is 50ms to 100ms, the predetermined duration before the first time period and the predetermined duration after the first time period can be 50ms to 100ms.

[0139] For example, if the duration of a time period is 10ms, and the predetermined duration before the first time period and the predetermined duration after the first time period are 50ms, the aforementioned third time period may include: the 5 adjacent time periods before the first time period and the 5 adjacent time periods after the first time period.

[0140] It is understandable that electronic devices can determine whether there is an abnormal voice interruption or voice gap in the first sub-audio corresponding to the first time period based on a small number of time periods adjacent to the first time period before determining whether there is an abnormal voice interruption or voice gap in the first sub-audio corresponding to the first time period. Then, when it is determined that there is an abnormal voice interruption or voice gap in the first sub-audio corresponding to the first time period, it can further determine whether there is an abnormal voice interruption in the first sub-audio corresponding to the first time period based on multiple time periods adjacent to the first time period.

[0141] For example, the electronic device can calculate the spectral differences and spectral similarities between the first frequency domain audio signal corresponding to the first time period and the first frequency domain audio signal corresponding to the fourth time period in each frequency band of the target frequency band, thereby obtaining the third and fourth variables corresponding to all frequency bands in the target frequency band; it is understood that the third variable is calculated in the same way as the first variable, and the fourth variable is calculated in the same way as the second variable, the difference being the different values ​​of the parameter t_local.

[0142] Then, the electronic device can calculate the mean of the third variable corresponding to all frequency bands in the target frequency band and the mean of the fourth variable corresponding to all frequency bands in the target frequency band;

[0143] Finally, when the mean of the third variable corresponding to all frequency bands in the target frequency band is greater than the fourth threshold, and the mean of the fourth variable corresponding to all frequency bands in the target frequency band is less than the fifth threshold, that is, when the mean of SD2(k,l) is greater than the threshold thre_SD2, and the mean of SR2(k,l) is less than the threshold thre_SR2, the electronic device determines that there is an abnormal voice interruption or voice interval in the first sub-audio corresponding to the first time period.

[0144] It should be noted that the calculation formulas for the third and fourth variables, as well as the descriptions of the parameters involved in the calculation formulas, can be found in the calculation formulas for the first and second variables and the descriptions of the parameters involved in the calculation formulas in the above embodiments, and will not be repeated here.

[0145] In the embodiments of this application, such as Figure 2 As shown, the speech processing method provided in this application embodiment may include the following steps S1 to S7.

[0146] Step S1: The electronic device divides the first audio into N sub-audio segments corresponding to different time periods.

[0147] Step S2: The electronic device determines whether the first sub-audio corresponding to the first time period contains voice information. If yes, proceed to step S3; otherwise, proceed to step S4.

[0148] Step S3: The electronic device determines whether the audio energy transition between the first sub-audio and the sub-audio corresponding to the time period within a predetermined duration before and after the first time period is smooth. If yes, proceed to step S4; otherwise, proceed to step S5.

[0149] Step S4: The electronic device does not process the first sub-audio.

[0150] Step S5: The electronic device determines whether there is an abnormal voice interruption in the first sub-audio. If yes, proceed to step S6; otherwise, proceed to step S4.

[0151] Step S6: The electronic device predicts and completes the first sub-audio corresponding to the first time period to obtain the third sub-audio.

[0152] Step S7: The electronic device combines the third sub-audio with the sub-audio corresponding to other time periods in the N time periods to obtain the second audio, and sends it to the receiving electronic device.

[0153] It should be noted that the speech processing method provided in this application embodiment can be executed by a speech processing device. This application embodiment uses a speech processing device executing a speech processing method as an example to illustrate the speech processing device provided in this application embodiment.

[0154] Figure 3 A schematic diagram of a possible structure of the voice processing device involved in an embodiment of this application is shown. For example... Figure 3 As shown, the voice processing device 70 may include a processing module 71 and a combination module 72.

[0155] The processing module 71 is used to divide the first audio into N sub-audio segments, each segment corresponding to one sub-audio segment, where N is a positive integer. If an abnormal audio interruption occurs in the first sub-audio segment corresponding to the first segment, and if it is determined that the second sub-audio segment corresponding to the second segment can be used to complete the first sub-audio segment, then the first sub-audio segment is completed based on the second sub-audio segment to obtain the third sub-audio segment corresponding to the first segment. The combination module 72 is used to combine the third sub-audio segment corresponding to the first segment with the sub-audio segments corresponding to other segments in the N segment groups to obtain the second audio. The first segment group is at least one of the N segment groups; the second segment group includes at least one of the following: the segment group adjacent to the first segment group, and the segment group adjacent to the first segment group.

[0156] This application provides a voice processing device. The voice processing device can divide a first audio segment to be transmitted into N sub-audio segments corresponding to different time periods. It then detects whether any abnormal voice interruptions exist in the sub-audio segments corresponding to the N time periods. When an abnormal voice interruption is detected in the first sub-audio segment corresponding to the first time period, the device determines whether a second sub-audio segment corresponding to the second time period can be used to complete the first sub-audio segment. If the second sub-audio segment can be used to complete the first sub-audio segment, the voice processing device can complete the first sub-audio segment based on the second sub-audio segment to obtain the processed first sub-audio segment, i.e., the third sub-audio segment corresponding to the first time period. The voice processing device can then combine the third sub-audio segment with the sub-audio segments corresponding to other time periods within the N time periods to obtain the processed first audio segment (i.e., the second audio segment), thereby enabling the electronic device to send complete voice information.

[0157] In one possible implementation, the voice processing apparatus 70 provided in this application embodiment further includes: an acquisition module and a conversion module. The acquisition module is used to acquire first time-domain audio signals corresponding to the N time-segment sub-audio signals after the processing module 71 divides the first audio into N time-segment sub-audio signals, with one time-segment corresponding to one first time-domain audio signal. The conversion module is used to perform frequency-domain conversion on the first time-domain audio signals corresponding to the N time-segments acquired by the acquisition module to obtain first frequency-domain audio signals in N frequency domains. Specifically, the processing module is used to, if it is determined that the first frequency-domain audio signal corresponding to the second time-segment can be used to complete the first frequency-domain audio signal corresponding to the first time-segment, complete the first frequency-domain audio signal corresponding to the first time-segment based on the first frequency-domain audio signal corresponding to the second time-segment to obtain the second frequency-domain audio signal.

[0158] In one possible implementation, the voice processing device further includes: a determining module; the determining module is configured to determine, for each frequency band corresponding to the first time period, if the minimum energy difference in the energy difference set corresponding to a frequency band is less than a first threshold, that the first frequency domain audio signal corresponding to the second time period can be used to complete the first frequency domain audio signal corresponding to the first time period in a frequency band. The processing module is specifically used to supplement the first frequency domain audio signal corresponding to a frequency band in the first time period based on the first frequency domain audio signal corresponding to the first frequency band group in the second time period, so as to obtain a target first frequency domain audio signal; the first frequency band group is the frequency band group corresponding to the minimum energy difference; and replace the first frequency domain audio signal corresponding to the first time period in a frequency band with the target first frequency domain audio signal to obtain a second frequency domain audio signal; wherein, each energy difference in the energy difference set corresponds to a set of frequency bands; one energy difference in the energy difference set is: the difference between the first energy value of the first frequency domain audio signal corresponding to X time periods before the first time period in the target frequency band and the second energy value of the first frequency domain audio signal corresponding to X time periods after the first time period in the target frequency band, the target frequency band includes any one of the following: the one frequency band, the first frequency band or the second frequency band; the frequency band group corresponding to an energy difference includes any two of the following: the one frequency band, the first frequency band or the second frequency band, the first frequency band and the second frequency band are frequency bands adjacent to one frequency band.

[0159] In one possible implementation, the conversion module is further configured to perform time-domain conversion on the frequency-domain audio signals corresponding to the N time periods to obtain N second time-domain audio signals. The combination module 72 is specifically configured to combine the N second time-domain audio signals according to the time sequence corresponding to the N time periods to obtain the second audio signal.

[0160] In one possible implementation, the speech processing apparatus 70 provided in this application embodiment further includes: a determining module. The determining module is configured to, when the processing module 71 has an abnormal speech interruption in the first sub-audio corresponding to the first time period, and if it is determined that the second sub-audio corresponding to the second time period can be used to complete the first sub-audio corresponding to the first time period, before completing the first sub-audio based on the second sub-audio to obtain the third sub-audio corresponding to the first time period, determine that the first sub-audio contains speech information if the audio time-domain energy parameters corresponding to the first sub-audio satisfy a first condition; wherein the audio time-domain energy parameters include: the audio time-domain energy of the first sub-audio, the average audio energy of the sub-audio in the first sub-audio set, and N time periods. The minimum audio temporal energy of the corresponding sub-audio, and the minimum audio temporal energy of the sub-audio in the first sub-audio set; the first sub-audio set includes: the first sub-audio, and M sub-audio adjacent to the first sub-audio; the first condition includes: the audio temporal energy of the first sub-audio is greater than the average audio energy of the sub-audio in the first sub-audio set; or, the audio temporal energy of the first sub-audio is greater than the minimum audio temporal energy of the sub-audio in the N time periods, and greater than the minimum audio temporal energy of the sub-audio in the first sub-audio set. Processing module 71 is specifically used to, when the first sub-audio corresponding to the first time period contains speech information and there is an abnormal speech interruption in the first sub-audio, if it is determined that the second sub-audio corresponding to the second time period can be used to complete the first sub-audio, then complete the first sub-audio based on the second sub-audio to obtain the third sub-audio corresponding to the first time period.

[0161] In one possible implementation, the speech processing apparatus 70 provided in this application embodiment further includes: a determination module. The determination module is configured to determine that an abnormal speech interruption exists in the first sub-audio corresponding to the first time period when, after the processing module 71 divides the first audio into N sub-audio segments corresponding to time periods, the mean of a first variable corresponding to all frequency bands in the target frequency band is greater than a second threshold, and the mean of a second variable corresponding to all frequency bands in the target frequency band is less than a third threshold. Specifically, a first variable indicates the spectral difference between the first frequency domain audio signal corresponding to the first time period and the first frequency domain audio signal corresponding to the third time period in a frequency band within the target frequency band; a second variable indicates the spectral similarity between the first frequency domain audio signal corresponding to the first time period and the first frequency domain audio signal corresponding to the third time period in a frequency band within the target frequency band; the third time period includes: a time period within a predetermined duration before the first time period and a time period within a predetermined duration after the first time period.

[0162] The voice processing device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television set (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the scope of the device.

[0163] The voice processing device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system used.

[0164] The voice processing device provided in this application embodiment can implement the various processes implemented in the above method embodiments, and will not be described again here to avoid repetition.

[0165] Optionally, such as Figure 4 As shown, this application embodiment also provides an electronic device 900, including a processor 901 and a memory 902. The memory 902 stores a program or instructions that can run on the processor 901. When the program or instructions are executed by the processor 901, they implement the various steps of the above method embodiments and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0166] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0167] Figure 5 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0168] The electronic device 100 includes, but is not limited to, components such as: radio frequency unit 101, network module 102, audio output unit 103, input unit 104, sensor 105, display unit 106, user input unit 107, interface unit 108, memory 109, and processor 110.

[0169] Those skilled in the art will understand that the electronic device 100 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 110 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 5 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0170] The processor 110 is used to divide the first audio into N sub-audio segments corresponding to time periods, with one sub-audio segment corresponding to one time period, where N is a positive integer. If an abnormal audio interruption occurs in the first sub-audio segment corresponding to the first time period, and if it is determined that the second sub-audio segment corresponding to the second time period can be used to complete the first sub-audio segment corresponding to the first time period, then the first sub-audio segment is completed based on the second sub-audio segment to obtain the third sub-audio segment corresponding to the first time period. The processor 110 also combines the third sub-audio segment corresponding to the first time period with the sub-audio segments corresponding to other time periods among the N time periods to obtain the second audio. The first time period is at least one of the N time periods. The second time period includes at least one of the following: the time period adjacent to the first time period, and the time period adjacent to the first time period.

[0171] This application provides an electronic device that can divide a first audio to be transmitted into N sub-audio segments corresponding to time periods. It then detects whether any abnormal voice interruptions exist in the sub-audio segments corresponding to the N time periods. When an abnormal voice interruption is detected in the first sub-audio segment corresponding to the first time period, it determines whether a second sub-audio segment corresponding to the second time period can be used to complete the first sub-audio segment. If the second sub-audio segment can be used to complete the first sub-audio segment, the electronic device can complete the first sub-audio segment based on the second sub-audio segment to obtain the processed first sub-audio segment, i.e., the third sub-audio segment corresponding to the first time period. The electronic device can then combine the third sub-audio segment with the sub-audio segments corresponding to other time periods within the N time periods to obtain the processed first audio segment (i.e., the second audio segment), thereby enabling the electronic device to transmit complete voice information.

[0172] Optionally, the processor 110 is further configured to, after dividing the first audio into N sub-audio segments corresponding to the time segments, acquire the first time-domain audio signals corresponding to the sub-audio segments corresponding to the N time segments, with one time segment corresponding to one first time-domain audio signal; and perform frequency domain conversion on the acquired first time-domain audio signals corresponding to the N time segments to obtain the first frequency-domain audio signals in the N frequency domains.

[0173] The processor 110 is specifically configured to, if it is determined that the first frequency domain audio signal corresponding to the second time period can be used to complete the first frequency domain audio signal corresponding to the first time period, complete the first frequency domain audio signal corresponding to the first time period based on the first frequency domain audio signal corresponding to the second time period to obtain the second frequency domain audio signal.

[0174] Optionally, the processor 110 is further configured to determine, for each frequency band corresponding to the first time period, if the minimum energy difference in the energy difference set corresponding to a frequency band is less than a first threshold, that the first frequency domain audio signal corresponding to the second time period can be used to complete the first frequency domain audio signal corresponding to a frequency band in the first time period.

[0175] The processor 110 is specifically configured to, based on the first frequency domain audio signal corresponding to the first frequency domain in a frequency band during the second time period, supplement the first frequency domain audio signal corresponding to the first frequency domain in a frequency band group during the second time period to obtain a target first frequency domain audio signal; the first frequency band group is the frequency band group corresponding to the minimum energy difference; and replace the first frequency domain audio signal corresponding to the first frequency domain in a frequency band during the first time period with the target first frequency domain audio signal to obtain a second frequency domain audio signal; wherein, each energy difference in the energy difference set corresponds to a set of frequency bands; one energy difference in the energy difference set is: the difference between the first energy value of the first frequency domain audio signal corresponding to X time periods before the first time period in the target frequency band and the second energy value of the first frequency domain audio signal corresponding to X time periods after the first time period in the target frequency band, the target frequency band including any one of the following: the one frequency band, the first frequency band, or the second frequency band; the frequency band group corresponding to an energy difference includes any two of the following: the one frequency band, the first frequency band, or the second frequency band, the first frequency band and the second frequency band being frequency bands adjacent to one frequency band.

[0176] Optionally, the processor 110 is also used to perform time-domain conversion on the frequency-domain audio signals corresponding to N time periods to obtain a second time-domain audio signal in N time domains.

[0177] The processor 110 is specifically used to combine the second time-domain audio signals in the N time domains according to the time sequence corresponding to the N time periods to obtain the second audio.

[0178] Optionally, the processor 110 is further configured to, in the event of an abnormal speech interruption in the first sub-audio corresponding to the first time period, if it is determined that the second sub-audio corresponding to the second time period can be used to complete the first sub-audio corresponding to the first time period, before completing the first sub-audio based on the second sub-audio to obtain the third sub-audio corresponding to the first time period, determine that the first sub-audio contains speech information if the audio time-domain energy parameter corresponding to the first sub-audio satisfies a first condition; wherein, the audio time-domain energy parameter includes: the audio time-domain energy of the first sub-audio, the average audio energy of the sub-audio in the set of first sub-audio, and N time periods. The minimum audio temporal energy among the corresponding sub-audio frequencies, and the minimum audio temporal energy among the audio temporal energies of the sub-audio frequencies in the first sub-audio set; the first sub-audio set includes: the first sub-audio frequency, and M sub-audio frequencies adjacent to the first sub-audio frequency; the first condition includes: the audio temporal energy of the first sub-audio frequency is greater than the average audio energy of the sub-audio frequencies in the first sub-audio set; or, the audio temporal energy of the first sub-audio frequency is greater than the minimum audio temporal energy among the audio temporal energies of the sub-audio frequencies corresponding to N time periods, and is greater than the minimum audio temporal energy among the audio temporal energies of the sub-audio frequencies in the first sub-audio set.

[0179] The processor 110 is specifically used to, when the first sub-audio corresponding to the first time period contains voice information and there is an abnormal voice interruption in the first sub-audio, if it is determined that the second sub-audio corresponding to the second time period can be used to complete the first sub-audio, then complete the first sub-audio based on the second sub-audio to obtain the third sub-audio corresponding to the first time period.

[0180] Optionally, the processor 110 is further configured to, after dividing the first audio into N sub-audio segments corresponding to time periods, determine that there is an abnormal voice interruption in the first sub-audio segment corresponding to the first time period if the mean of the first variable corresponding to all frequency bands in the target frequency band is greater than a second threshold and the mean of the second variable corresponding to all frequency bands in the target frequency band is less than a third threshold; wherein, a first variable is used to indicate the spectral difference between the first frequency domain audio signal corresponding to the first time period and the first frequency domain audio signal corresponding to the third time period in a frequency band within the target frequency band; a second variable is used to indicate the spectral similarity between the first frequency domain audio signal corresponding to the first time period and the first frequency domain audio signal corresponding to the third time period in a frequency band within the target frequency band; the third time period includes: a time period within a predetermined duration before the first time period and a time period within a predetermined duration after the first time period.

[0181] The electronic device provided in this application embodiment can implement the various processes implemented in the above method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0182] For details on the beneficial effects of the various implementation methods in this embodiment, please refer to the beneficial effects of the corresponding implementation methods in the above method embodiments. To avoid repetition, these will not be repeated here.

[0183] It should be understood that, in this embodiment, the input unit 104 may include a graphics processing unit (GPU) 1041 and a microphone 1042. The GPU 1041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 106 may include a display panel 1061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 107 includes at least one of a touch panel 1071 and other input devices 1072. The touch panel 1071 is also called a touch screen. The touch panel 1071 may include a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0184] The memory 109 can be used to store software programs and various data. The memory 109 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 109 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 109 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0185] Processor 110 may include one or more processing units; optionally, processor 110 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 110.

[0186] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0187] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0188] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0189] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0190] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above method embodiments and achieve the same technical effects. To avoid repetition, it will not be described again here.

[0191] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0192] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0193] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A speech processing method, characterized in that, The method includes: The first audio segment is divided into N sub-audio segments, with one sub-audio segment corresponding to one time segment, where N is a positive integer. If there is an abnormal voice interruption in the first sub-audio corresponding to the first time period, and if it is determined that the second sub-audio corresponding to the second time period can be used to complete the first sub-audio corresponding to the first time period, then the first sub-audio is completed based on the second sub-audio to obtain the third sub-audio corresponding to the first time period. The third sub-audio corresponding to the first time period is combined with the sub-audio corresponding to other time periods among the N time periods to obtain the second audio. Wherein, the first time period is at least one of the N time periods; The second time period includes at least one of the following: the time period adjacent to the first time period, and the time period adjacent to the first time period; After dividing the first audio into N sub-audio segments corresponding to different time periods, the method further includes: If the mean of the first variable corresponding to all frequency bands in the target frequency band is greater than the second threshold, and the mean of the second variable corresponding to all frequency bands in the target frequency band is less than the third threshold, it is determined that there is an abnormal voice interruption in the first sub-audio corresponding to the first time period. Among them, a first variable is used to indicate the spectral difference between the first frequency domain audio signal corresponding to the first time period and the first frequency domain audio signal corresponding to the third time period in a frequency band of the target frequency band; A second variable is used to indicate the spectral similarity between the first frequency domain audio signal corresponding to the first time period and the first frequency domain audio signal corresponding to the third time period in a frequency band of the target frequency band. The third time period includes: the time period within a predetermined duration before the first time period, and the time period within the predetermined duration after the first time period.

2. The method according to claim 1, characterized in that, After dividing the first audio into N sub-audio segments corresponding to different time periods, the method further includes: Obtain the first time-domain audio signal corresponding to the sub-audio of the N time periods, with one time period corresponding to one first time-domain audio signal; The first time-domain audio signals corresponding to the N time periods are converted into the frequency domain to obtain the first frequency-domain audio signals in the N frequency domains; If it is determined that the second sub-audio corresponding to the second time period can be used to complete the first sub-audio corresponding to the first time period, then the first sub-audio is completed based on the second sub-audio to obtain the third sub-audio corresponding to the first time period, including: If it is determined that the first frequency domain audio signal corresponding to the second time period can be used to complete the first frequency domain audio signal corresponding to the first time period, then the first frequency domain audio signal corresponding to the first time period is completed based on the first frequency domain audio signal corresponding to the second time period to obtain the second frequency domain audio signal.

3. The method according to claim 2, characterized in that, If it is determined that the first frequency domain audio signal corresponding to the second time period can be used to complete the first frequency domain audio signal corresponding to the first time period, then the first frequency domain audio signal corresponding to the first time period is completed based on the first frequency domain audio signal corresponding to the second time period to obtain the second frequency domain audio signal, including: For each frequency band corresponding to the first time period, if the minimum energy difference in the energy difference set corresponding to a frequency band is less than a first threshold, it is determined that the first frequency domain audio signal corresponding to the second time period can be used to complete the first frequency domain audio signal corresponding to the first time period in the frequency band. Based on the first frequency domain audio signal corresponding to the second time period in the first frequency band group, the first frequency domain audio signal corresponding to the first time period in the first frequency band is supplemented to obtain the target first frequency domain audio signal; the first frequency band group is the frequency band group corresponding to the minimum energy difference. The first frequency domain audio signal corresponding to the first time period in the frequency band is replaced with the target first frequency domain audio signal to obtain the second frequency domain audio signal; In this set of energy differences, each energy difference corresponds to a set of frequency bands; One of the energy difference values ​​in the energy difference set is: the difference between the first energy value of the first frequency domain audio signal corresponding to X time periods before the first time period in the target frequency band and the second energy value of the first frequency domain audio signal corresponding to X time periods after the first time period in the target frequency band. The target frequency band includes any one of the following: the one frequency band, the first frequency band, or the second frequency band. The frequency band group corresponding to the one energy difference value includes any two of the following: the one frequency band, the first frequency band, or the second frequency band, wherein the first frequency band and the second frequency band are adjacent to the one frequency band.

4. The method according to claim 2, characterized in that, The step of combining the third sub-audio corresponding to the first time period with the sub-audio corresponding to other time periods among the N time periods to obtain the second audio includes: The frequency domain audio signals corresponding to the N time periods are converted into the time domain to obtain the second time domain audio signals in the N time domains; According to the time sequence corresponding to the N time periods, the second time-domain audio signals in the N time domains are combined to obtain the second audio.

5. The method according to claim 1, characterized in that, In the event of an abnormal voice interruption in the first sub-audio corresponding to the first time period, if it is determined that the second sub-audio corresponding to the second time period can be used to complete the first sub-audio corresponding to the first time period, before completing the first sub-audio based on the second sub-audio to obtain the third sub-audio corresponding to the first time period, the method further includes: If the audio time-domain energy parameters corresponding to the first sub-audio satisfy the first condition, it is determined that the first sub-audio contains speech information; The audio time-domain energy parameters include: the audio time-domain energy of the first sub-audio, the average audio energy of the sub-audio in the first sub-audio set, the minimum audio time-domain energy among the audio time-domain energies of the sub-audio corresponding to the N time periods, and the minimum audio time-domain energy among the audio time-domain energies of the sub-audio in the first sub-audio set. The first sub-audio set includes: the first sub-audio, and M sub-audio adjacent to the first sub-audio; The first condition includes: the audio time-domain energy of the first sub-audio is greater than the average audio energy of the sub-audio in the first sub-audio set; or, the audio time-domain energy of the first sub-audio is greater than the minimum audio time-domain energy among the audio time-domain energies of the sub-audio corresponding to the N time periods, and is greater than the minimum audio time-domain energy among the audio time-domain energies of the sub-audio in the first sub-audio set. In the case where there is an abnormal voice interruption in the first sub-audio corresponding to the first time period, if it is determined that the second sub-audio corresponding to the second time period can be used to complete the first sub-audio corresponding to the first time period, then the first sub-audio is completed based on the second sub-audio to obtain the third sub-audio corresponding to the first time period, including: If the first sub-audio corresponding to the first time period contains voice information and there is an abnormal voice interruption in the first sub-audio, and it is determined that the second sub-audio corresponding to the second time period can be used to complete the first sub-audio, then the first sub-audio is completed based on the second sub-audio to obtain the third sub-audio corresponding to the first time period.

6. A voice processing device, characterized in that, The voice processing device includes: a processing module and a combination module; The processing module is used to divide the first audio into N sub-audio segments corresponding to time periods, with one sub-audio segment corresponding to one time period, where N is a positive integer; and if there is an abnormal voice interruption in the first sub-audio segment corresponding to the first time period, if it is determined that the second sub-audio segment corresponding to the second time period can be used to complete the first sub-audio segment corresponding to the first time period, then the first sub-audio segment is completed based on the second sub-audio segment to obtain the third sub-audio segment corresponding to the first time period. The combination module is used to combine the third sub-audio corresponding to the first time period with the sub-audio corresponding to other time periods in the N time periods to obtain the second audio. Wherein, the first time period is at least one of the N time periods; The second time period includes at least one of the following: the time period adjacent to the first time period, and the time period adjacent to the first time period; The voice processing device further includes: a determination module; The determining module is used to determine that there is an abnormal voice interruption in the first sub-audio corresponding to the first time period after the processing module divides the first audio into N sub-audio segments corresponding to time periods, and the mean of the first variable corresponding to all frequency bands in the target frequency band is greater than a second threshold and the mean of the second variable corresponding to all frequency bands in the target frequency band is less than a third threshold. Among them, a first variable is used to indicate the spectral difference between the first frequency domain audio signal corresponding to the first time period and the first frequency domain audio signal corresponding to the third time period in a frequency band of the target frequency band; A second variable is used to indicate the spectral similarity between the first frequency domain audio signal corresponding to the first time period and the first frequency domain audio signal corresponding to the third time period in a frequency band of the target frequency band. The third time period includes: the time period within a predetermined duration before the first time period, and the time period within the predetermined duration after the first time period.

7. The apparatus according to claim 6, characterized in that, The voice processing device further includes: an acquisition module and a conversion module; The acquisition module is used to acquire the first time-domain audio signal corresponding to the sub-audio signals corresponding to the N time-segments after the processing module divides the first audio into N time-segments and obtains the sub-audio signals corresponding to the N time-segments. One time-segment corresponds to one first time-domain audio signal. The conversion module is used to perform frequency domain conversion on the first time-domain audio signals corresponding to the N time periods acquired by the acquisition module to obtain the first frequency-domain audio signals in the N frequency domains. The processing module is specifically configured to, if it is determined that the first frequency domain audio signal corresponding to the second time period can be used to complete the first frequency domain audio signal corresponding to the first time period, complete the first frequency domain audio signal corresponding to the first time period based on the first frequency domain audio signal corresponding to the second time period to obtain the second frequency domain audio signal.

8. The apparatus according to claim 7, characterized in that, The voice processing device further includes: a determination module; The determining module is used to determine, for each frequency band corresponding to the first time period, if the minimum energy difference in the energy difference set corresponding to a frequency band is less than a first threshold, that the first frequency domain audio signal corresponding to the second time period can be used to complete the first frequency domain audio signal corresponding to the first time period in the frequency band. The processing module is specifically used to supplement the first frequency domain audio signal corresponding to the first time period in the first frequency band group based on the first frequency domain audio signal corresponding to the second time period in the first frequency band group, so as to obtain a target first frequency domain audio signal; the first frequency band group is the frequency band group corresponding to the minimum energy difference; and replace the first frequency domain audio signal corresponding to the first time period in the first frequency band with the target first frequency domain audio signal to obtain a second frequency domain audio signal; In this set of energy differences, each energy difference corresponds to a set of frequency bands; One of the energy difference values ​​in the energy difference set is: the difference between the first energy value of the first frequency domain audio signal corresponding to X time periods before the first time period in the target frequency band and the second energy value of the first frequency domain audio signal corresponding to X time periods after the first time period in the target frequency band. The target frequency band includes any one of the following: the one frequency band, the first frequency band, or the second frequency band. The frequency band group corresponding to the one energy difference value includes any two of the following: the one frequency band, the first frequency band, or the second frequency band, wherein the first frequency band and the second frequency band are adjacent to the one frequency band.

9. The apparatus according to claim 7, characterized in that, The conversion module is also used to perform time-domain conversion on the frequency domain audio signals corresponding to the N time periods to obtain the second time-domain audio signals in the N time domains; The combination module is specifically used to combine the second time-domain audio signals in the N time domains according to the time sequence corresponding to the N time periods to obtain the second audio.

10. The apparatus according to claim 6, characterized in that, The voice processing device further includes: a determination module; The determining module is configured to determine that the first sub-audio contains speech information if the processing module determines that the second sub-audio corresponding to the second time period can be used to complete the first sub-audio corresponding to the first time period, and before completing the first sub-audio based on the second sub-audio to obtain the third sub-audio corresponding to the first time period, the first sub-audio contains speech information if the audio time-domain energy parameter corresponding to the first sub-audio satisfies the first condition. The audio time-domain energy parameters include: the audio time-domain energy of the first sub-audio, the average audio energy of the sub-audio in the first sub-audio set, the minimum audio time-domain energy among the audio time-domain energies of the sub-audio corresponding to the N time periods, and the minimum audio time-domain energy among the audio time-domain energies of the sub-audio in the first sub-audio set. The first sub-audio set includes: the first sub-audio, and M sub-audio adjacent to the first sub-audio; The first condition includes: the audio time-domain energy of the first sub-audio is greater than the average audio energy of the sub-audio in the first sub-audio set; or, the audio time-domain energy of the first sub-audio is greater than the minimum audio time-domain energy among the audio time-domain energies of the sub-audio corresponding to the N time periods, and is greater than the minimum audio time-domain energy among the audio time-domain energies of the sub-audio in the first sub-audio set. The processing module is specifically configured to, when the first sub-audio corresponding to the first time period contains voice information and the first sub-audio has an abnormal voice interruption, if it is determined that the second sub-audio corresponding to the second time period can be used to complete the first sub-audio, then complete the first sub-audio based on the second sub-audio to obtain the third sub-audio corresponding to the first time period.

11. An electronic device, characterized in that, It includes a processor and a memory, the memory storing programs or instructions that can run on the processor, the programs or instructions being executed by the processor to implement the steps of the speech processing method as described in any one of claims 1-5.

12. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the speech processing method as described in any one of claims 1-5.

Images