Speech segmentation method, device, computer equipment and readable storage medium

Abstract

The present application relates to a speech segmentation method, device, computer equipment and readable storage medium. The method divides the waveform of the audio data to be segmented according to the initial phoneme segmentation position of the audio data to be segmented, and determines the segmentation waveform; inputs the segmentation waveform into an offset determination module to obtain an offset; corrects the initial phoneme segmentation position according to the offset to obtain a target phoneme segmentation position in the segmentation waveform; determines the phoneme segmentation position in the waveform of the audio data to be segmented according to the target phoneme segmentation position, the segmentation waveform and the initial phoneme segmentation position. The speech segmentation method provided by the present application can improve the accuracy of the determined phoneme segmentation position in the waveform of the audio data to be segmented.

Description

Technical Field

[0001] The present application relates to the field of speech recognition technology, and in particular to a speech segmentation method, apparatus, computer device and readable storage medium. Background Art

[0002] In the scene of generating animation based on speech and image, the speech needs to be aligned with the image in the animation scene, that is, the sound of the animation should correspond to the facial shape of the person or animal in the image. When the speech is misaligned, that is, the speech and image are not aligned, there will be a significant difference between the image display and the speech, which will affect the effect of the animation and thus affect the viewer's experience.

[0003] In traditional technology, Hidden Markov Model (HMM) and Gaussian Mixture Model (GMM) are used to achieve speech and image alignment. However, due to the limitations of HMM and GMM training data and the technology itself, inaccurate speech segmentation will occur when achieving speech and image alignment, resulting in misalignment between speech and image. Summary of the invention

[0004] Based on this, it is necessary to provide a speech segmentation method, apparatus, computer device and readable storage medium to address the above technical problems.

[0005] In a first aspect, an embodiment of the present application provides a speech segmentation method, comprising:

[0006] Dividing the waveform graph of the audio data to be segmented according to the initial phoneme segmentation position of the audio data to be segmented, and determining the segmentation waveform graph; the segmentation waveform graph includes a waveform within a preset range centered on the initial phoneme segmentation position;

[0007] The segmented waveform is input into an offset determination model to obtain an offset; the offset determination model is obtained by training an initial offset determination model based on a plurality of segmented waveform samples and accurate phoneme segmentation positions corresponding to the plurality of segmented waveform samples;

[0008] Correcting the initial phoneme segmentation position according to the offset to obtain the target phoneme segmentation position in the segmentation waveform;

[0009] The phoneme segmentation position in the waveform of the audio data to be segmented is determined according to the target phoneme segmentation position, the segmentation waveform and the initial phoneme segmentation position.

[0010] In one embodiment, determining the phoneme segmentation position in the waveform of the audio data to be segmented according to the target phoneme segmentation position, the segmentation waveform and the initial phoneme segmentation position includes:

[0011] According to the initial phoneme segmentation position, a sub-waveform represented by the segmentation waveform graph is matched with the waveform graph of the audio data to be segmented, and the position of the sub-waveform in the waveform graph of the audio data to be segmented is determined to obtain the waveform position;

[0012] According to the waveform position and the target phoneme segmentation position, the phoneme segmentation position in the waveform graph of the audio data to be segmented is determined.

[0013] In one embodiment, determining the phoneme segmentation position in the waveform graph of the audio data to be segmented according to the waveform position and the target phoneme segmentation position includes:

[0014] According to the waveform position, determine the position of the starting position of the sub-waveform in the waveform diagram of the audio data to be segmented, and obtain the waveform starting position;

[0015] The waveform start position and the target phoneme segmentation position are processed by an AND operation to obtain the phoneme segmentation position in the waveform diagram of the audio data to be segmented.

[0016] In one embodiment, dividing the waveform graph of the audio data to be segmented according to the initial phoneme segmentation position of the audio data to be segmented to determine the segmented waveform graph includes:

[0017] Based on the hidden Markov model and the mixed Gaussian model, according to the audio data to be segmented, determining the initial phoneme segmentation position of the audio data to be segmented;

[0018] Taking the initial phoneme segmentation position as the center, a waveform graph having a preset distance from the initial phoneme segmentation position is obtained in the waveform graph of the audio data to be segmented to obtain a segmentation waveform graph.

[0019] In one embodiment, the speech segmentation method further includes:

[0020] The accurate phoneme segmentation positions corresponding to the multiple segmentation waveform samples are determined according to the characteristics of the segmentation waveforms in the multiple segmentation waveform samples.

[0021] In one embodiment, the speech segmentation method further includes:

[0022] For each segmented waveform sample, obtain an audio data waveform corresponding to the segmented waveform sample; the segmented waveform sample is a waveform formed by dividing the waveform in the audio data waveform;

[0023] Based on the accurate phoneme segmentation position corresponding to the segmented waveform sample, the waveform range included in the segmented waveform sample is adjusted to obtain a new segmented waveform, and the new segmented waveform is added to the multiple segmented waveform samples.

[0024] In one embodiment, the offset determination model is obtained by training the initial offset determination model according to a plurality of segmented waveform samples and accurate phoneme segmentation positions corresponding to the plurality of segmented waveform samples, including:

[0025] The initial offset determination model is trained according to the extracted image features of different channels of each segmented waveform sample and the accurate phoneme segmentation position corresponding to each segmented waveform sample to obtain the offset determination model.

[0026] In one of the embodiments, the offset determination model includes a lightweight network model and a regression loss function.

[0027] In a second aspect, an embodiment of the present application provides a speech segmentation method and apparatus, including:

[0028] A segmentation waveform determination module is used to divide the waveform of the audio data to be segmented according to the initial phoneme segmentation position of the audio data to be segmented, and determine the segmentation waveform; the segmentation waveform includes a waveform within a preset range centered on the initial phoneme segmentation position;

[0029] The offset determination module is used to input the segmented waveform into the offset determination model to obtain the offset; the offset model is obtained by training the initial offset determination model based on multiple segmented waveform samples and the accurate phoneme segmentation positions corresponding to the multiple segmented waveform samples.

[0030] A first phoneme segmentation position determination module is used to correct the initial phoneme segmentation position according to the offset to obtain a target phoneme segmentation position in the segmentation waveform;

[0031] The second phoneme segmentation position determination module is used to determine the phoneme segmentation position in the waveform of the audio data to be segmented according to the target phoneme segmentation position, the segmentation waveform and the initial phoneme segmentation point.

[0032] In a third aspect, an embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method provided in the above embodiment are implemented.

[0033] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the method provided in the above embodiment when the computer program is executed by a processor.

[0034] The embodiments of the present application provide a method, apparatus, computer device and readable storage medium for speech segmentation, which divides the waveform of the audio data to be segmented according to the initial phoneme segmentation position of the audio data to be segmented, and determines the segmented waveform; inputs the segmented waveform into an offset determination model to obtain an offset; corrects the initial phoneme segmentation position according to the offset to obtain a target phoneme segmentation position in the segmented waveform; determines the phoneme segmentation position in the waveform of the audio data to be segmented according to the target phoneme segmentation position, the segmented waveform and the initial phoneme segmentation position. The speech segmentation method provided by the embodiments of the present application corrects the initial phoneme segmentation position based on the offset obtained by the offset determination model, and can obtain a more accurate phoneme segmentation position in the waveform of the audio data to be segmented, and can achieve accurate segmentation of the audio data to be segmented, so that the speech and the image can be aligned without misalignment. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the conventional technology, the drawings required for use in the embodiments or the conventional technology descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For different technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0036] Figure 1 A flowchart of the steps of a speech segmentation method provided in one embodiment of the present application;

[0037] Figure 2 A flowchart of the steps of a speech segmentation method provided in one embodiment of the present application;

[0038] Figure 3 A flowchart of the steps of a speech segmentation method provided in one embodiment of the present application;

[0039] Figure 4 A flowchart of the steps of a speech segmentation method provided in one embodiment of the present application;

[0040] Figure 5 A flowchart of the steps of a speech segmentation method provided in one embodiment of the present application;

[0041] Figure 6 A schematic diagram of the structure of a speech segmentation device provided in one embodiment of the present application;

[0042] Figure 7 A schematic diagram of the structure of a computer device provided for one embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present application, so the present application is not limited by the specific embodiments disclosed below.

[0044] The serial numbers assigned to the components in this article, such as "first", "second", etc., are only used to distinguish the objects described and do not have any order or technical meaning.

[0045] In an animation scene that includes sound, the voice needs to be aligned with the image in the animation scene, that is, the sound produced in the animation scene must correspond to the facial shape of the person or animal in the image. When the voice is misaligned, that is, the voice and image are not aligned, there will be a significant difference between the image display and the voice, which will affect the effect of the animation and, in turn, the experience of the viewer. Taking a 25fps animation video as an example, ordinary people can perceive animation deviations within 2 frames (80ms), and professionals can perceive deviations of about 1 frame (40ms). Therefore, deviations of 1 frame or more will have a more obvious impact on the effect of the animation.

[0046] In the conventional technology, the Hidden Markov Model (HMM) and the Gaussian Mixture Model (GMM) are used to realize the alignment of speech and image. However, due to the limitation of training data and the technology itself, the effect of speech and image alignment (segmentation of whole word pronunciation) in the conventional technology is not ideal, and the problem of inaccurate speech segmentation usually occurs. In order to solve the problem of speech segmentation, it is necessary to retrain the HMM and GMM, and the amount of work is very large. Therefore, in view of the problem of inaccurate speech segmentation, the present application provides a speech segmentation method.

[0047] The speech segmentation method provided in the present application can be implemented by a computer device. The computer device includes but is not limited to a control chip, a personal computer, a laptop computer, a smart phone, a tablet computer, and a portable wearable device. The method provided in the present application can be implemented by JAVA software, and can also be applied to other software.

[0048] The technical solution of the present application and how the technical solution of the present application solves the technical problem are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0049] See also Figure 1An embodiment of the present application provides a method for speech segmentation. This embodiment uses a computer device as an execution subject to illustrate the method for speech segmentation. The specific steps of the method include:

[0050] Step 100: divide the waveform graph of the audio data to be segmented according to the initial phoneme segmentation position of the audio data to be segmented, and determine the segmentation waveform graph; the segmentation waveform graph includes a waveform within a preset range centered on the initial phoneme segmentation position.

[0051] The computer device obtains the audio data to be segmented, determines the waveform of the audio data to be segmented, and the initial phoneme segmentation position of the audio data to be segmented. The waveform of the audio data to be segmented is divided according to the initial phoneme segmentation position to obtain a waveform within a preset range centered on the initial phoneme segmentation position, i.e., a segmentation waveform. The waveform of the audio data to be segmented refers to a waveform composed of the waveforms of each phoneme in the audio data to be segmented. The initial phoneme segmentation position refers to the coordinates of the initial phoneme segmentation point in the audio data to be segmented. This embodiment does not impose any restrictions on the method of obtaining the audio data to be segmented and the method of determining the waveform of the audio data to be segmented, as long as its functions can be achieved.

[0052] In an optional embodiment, the computer device first obtains the text data, and then determines the audio data to be segmented corresponding to the text data according to the text data.

[0053] In an optional embodiment, the initial phoneme segmentation position is the segmentation position between words in the audio data to be segmented corresponding to the text data.

[0054] Step 110, input the segmented waveform into the offset determination model to obtain the offset; the offset determination model is obtained by training the initial offset determination model based on multiple segmented waveform samples and the accurate phoneme segmentation positions corresponding to the multiple segmented waveform samples.

[0055] Step 120: Correct the initial phoneme segmentation position according to the offset to obtain the target phoneme segmentation position in the segmentation waveform.

[0056] After obtaining the segmented waveform, the computer device inputs it into the offset determination model to obtain the offset. The offset determination model is obtained by the staff training the initial offset determination model based on multiple segmented waveform samples and the accurate phoneme segmentation positions corresponding to the multiple segmented waveform samples. During the training process, the offset refers to the deviation between the accurate phoneme segmentation positions corresponding to the multiple segmented waveform samples and the initial phoneme segmentation positions in the multiple segmented waveform samples. The offset obtained by the offset determination model can determine the deviation between the accurate phoneme segmentation position corresponding to the segmented waveform and the initial phoneme segmentation position. Then, the initial phoneme phoneme segmentation position is corrected according to the offset, and the accurate phoneme segmentation position in the segmented waveform, that is, the target phoneme segmentation position, can be obtained. The target phoneme segmentation position refers to the coordinate value of the target phoneme segmentation point in the segmented waveform. This embodiment does not limit the specific structure of the offset determination model, as long as its function can be achieved.

[0057] In an optional embodiment, the size of the segmentation waveform image input to the offset determination model can be converted according to the size of the image that the offset determination model can receive. For example, if the size of the segmentation waveform image determined by the computer device is 64x64 pixels, but the offset determination model can only receive an image of 96x96 pixels, the computer device needs to first perform image conversion on the segmentation waveform image to convert it into a waveform image of 96x96 pixels.

[0058] Step 130: Determine the phoneme segmentation position in the waveform of the audio data to be segmented according to the target phoneme segmentation position, the segmentation waveform and the initial phoneme segmentation position.

[0059] After obtaining the target phoneme segmentation position, the segmentation waveform and the initial phoneme segmentation position, the computer device compares the segmentation waveform and the waveform of the audio data to be segmented, and determines the phoneme segmentation position in the waveform of the audio data to be segmented according to the target phoneme segmentation position in the segmentation waveform. The phoneme segmentation position in the waveform of the audio data to be segmented refers to the coordinate value of the phoneme segmentation point in the waveform of the audio data to be segmented. In other words, the coordinates of the target phoneme segmentation point in the waveform of the audio data to be segmented are determined according to the coordinate value of the target phoneme segmentation point in the segmentation waveform and the coordinates of the phoneme segmentation point of the initial phoneme segmentation point in the waveform of the audio data to be segmented. This embodiment does not impose any restrictions on the specific method for determining the phoneme segmentation position in the waveform of the audio data to be segmented, as long as its function can be achieved.

[0060] The embodiment of the present application provides a method for speech segmentation, which divides the waveform of the audio data to be segmented according to the initial phoneme segmentation position of the audio data to be segmented, and determines the segmented waveform; inputs the segmented waveform into the offset determination model to obtain the offset; corrects the initial phoneme segmentation position according to the offset to obtain the target phoneme segmentation position in the segmented waveform; determines the phoneme segmentation position in the waveform of the audio data to be segmented according to the target phoneme segmentation position, the segmented waveform and the initial phoneme segmentation position. The method for speech segmentation provided by the embodiment of the present application corrects the initial phoneme segmentation position based on the offset obtained by the offset determination model, and can obtain a more accurate phoneme segmentation position in the waveform of the audio data to be segmented, and can achieve accurate segmentation of the audio data to be segmented, so that the speech and the image can be aligned without misalignment, thereby improving the viewer's experience.

[0061] See also Figure 2 In one embodiment, a method for determining a phoneme segmentation position in a waveform of audio data to be segmented according to a target phoneme segmentation position, a segmentation waveform and an initial phoneme segmentation position is provided, including:

[0062] Step 200: Match the sub-waveform represented by the segmentation waveform with the waveform of the audio data to be segmented according to the initial phoneme segmentation position, determine the position of the sub-waveform in the waveform of the audio data to be segmented, and obtain the waveform position.

[0063] After the computer device obtains the initial phoneme segmentation position, the segmentation waveform and the waveform of the audio data to be segmented, since the segmentation waveform is obtained by dividing the waveform of the audio data to be segmented according to the initial phoneme segmentation position, the waveform in the segmentation waveform is a partial waveform in the waveform of the audio data to be segmented. According to the initial phoneme segmentation position, the sub-waveform represented by the segmentation waveform is matched with the waveform of the audio data to be segmented, and the position of the sub-waveform in the waveform of the audio data to be segmented, that is, the waveform position, can be determined. In other words, the coordinate value of the sub-waveform represented by the segmentation waveform in the waveform of the audio data to be segmented can be obtained according to the coordinate value of the initial phoneme segmentation point in the waveform of the audio data to be segmented.

[0064] Step 210: Determine the phoneme segmentation position in the waveform graph of the audio data to be segmented according to the waveform position and the target phoneme segmentation position.

[0065] After the computer device locates the waveform, it can determine the coordinates of the target phoneme segmentation point in the waveform of the audio data to be segmented, that is, the phoneme segmentation position in the waveform of the audio data to be segmented, based on the coordinates of the target phoneme segmentation point in the segmentation waveform.

[0066] The method for determining the phoneme segmentation position in the waveform diagram of the audio data to be segmented provided in this embodiment is simple, easy to understand, and easy to implement.

[0067] See also Figure 3 In one embodiment, determining the phoneme segmentation position in the waveform graph of the audio data to be segmented according to the waveform position and the target phoneme segmentation position includes:

[0068] Step 300: Determine the position of the starting position of the sub-waveform in the waveform diagram of the audio data to be segmented according to the waveform position, and obtain the waveform starting position.

[0069] After obtaining the waveform position, that is, the position of the sub-waveform represented by the segmentation waveform in the waveform of the audio data to be segmented, the computer device determines the coordinate value of the starting point of the sub-waveform (the starting position of the sub-waveform from left to right, that is, the leftmost position of the sub-waveform) in the waveform of the audio data to be segmented, and obtains the starting position of the waveform.

[0070] Step 310: Perform an AND operation on the waveform start position and the target phoneme segmentation position to obtain the phoneme segmentation position in the waveform diagram of the audio data to be segmented.

[0071] After determining the starting position of the waveform, that is, the coordinate value of the starting point of the waveform, the computer device calculates the sum of the coordinate value and the target phoneme segmentation position (the coordinate value of the target phoneme segmentation point in the segmentation waveform diagram), and can obtain the phoneme segmentation position in the waveform diagram of the audio data to be segmented (the coordinate value of the phoneme segmentation point).

[0072] The method provided in this embodiment for finally determining the phoneme segmentation position in the waveform diagram of the audio data to be segmented according to the waveform position is simple and easy to understand.

[0073] In a specific embodiment, assuming that the coordinate value of the initial phoneme segmentation point is (a, 0), the coordinate value of the target phoneme segmentation point is (b, 0), the size of the segmentation waveform is 64x64 pixels, the target phoneme segmentation position in the segmentation waveform is in the middle position of the segmentation waveform, and the target phoneme segmentation position is W away from the starting position of the segmentation waveform. Then, the sub-waveform represented by the segmentation waveform is matched with the waveform of the audio data to be segmented, and the coordinate value of the starting point of the sub-waveform represented by the segmentation waveform in the waveform of the audio data to be segmented is (aW, 0). Then, the coordinate value of the phoneme segmentation point in the waveform of the audio data to be segmented is (a-W+b, 0).

[0074] See also Figure 4 In one embodiment, a method for dividing the waveform graph of the audio data to be segmented according to the initial phoneme segmentation position of the audio data to be segmented and determining the segmented waveform graph is provided, including:

[0075] Step 400: Based on the hidden Markov model and the mixed Gaussian model, the initial phoneme segmentation position of the audio data to be segmented is determined according to the audio data to be segmented.

[0076] After obtaining the audio data to be segmented, the computer device can determine the initial phoneme segmentation position of the audio data to be segmented based on the pre-trained Hidden Markov Model (HMM) and Gaussian Mixture Model (GMM). Specifically, HMM and GMM calculate the probability of each phoneme in the audio data to be segmented (i.e., the probability that each phoneme in the audio data to be segmented belongs to each phoneme in all phonemes), and determine the initial phoneme segmentation position of the audio data to be segmented according to the probability of each phoneme. This embodiment does not limit the specific training process of the HMM and GMM models, as long as their functions can be achieved.

[0077] Step 410: With the initial phoneme segmentation position as the center, a waveform graph having a preset distance from the initial phoneme segmentation position is obtained in the waveform graph of the audio data to be segmented to obtain a segmentation waveform graph.

[0078] After determining the initial phoneme segmentation position, the computer device can divide the sub-waveform of the segmentation waveform in the waveform of the audio data to be segmented with the initial phoneme segmentation position as the center to form a segmentation waveform. The sub-waveform is a waveform within a preset distance from the initial phoneme segmentation position in the waveform of the audio data to be segmented. The preset distance can be set by the user, and this embodiment does not limit this.

[0079] In this embodiment, a method for determining the initial phoneme segmentation position and a method for determining the segmentation waveform are specifically described. The method is simple, convenient and easy to implement.

[0080] See also Figure 5 In one embodiment, the speech segmentation method further includes:

[0081] Step 500: Determine accurate phoneme segmentation positions corresponding to the plurality of segmentation waveform samples according to the characteristics of the segmentation waveforms in the plurality of segmentation waveform samples.

[0082] The method for obtaining the segmented waveform sample can refer to the method for obtaining the segmented waveform in the above embodiment, which will not be repeated here. For each segmented waveform sample, after obtaining the segmented waveform sample, the computer device can determine the accurate phoneme segmentation position corresponding to the segmented waveform sample according to the characteristics of the segmented waveform in the segmented waveform sample. The accurate phoneme segmentation position refers to the coordinate value of the accurate phoneme segmentation point in the segmented waveform sample. The segmented waveform in each segmented waveform sample is a partial waveform of each phoneme, and the waveform of each phoneme has different characteristics. Specifically, the starting position of the waveform of the initial consonant has a more obvious image feature, and the ending position of the waveform of the final has a more obvious image feature.

[0083] In this embodiment, the method of determining the accurate phoneme segmentation position corresponding to the segmented waveform sample according to the characteristics of the segmented waveform can improve the accuracy of the determined accurate phoneme segmentation position.

[0084] In an optional embodiment, the computer device may determine the accurate phoneme segmentation position corresponding to the segmented waveform sample by analyzing and identifying the segmented waveform in each segmented waveform sample.

[0085] In another optional embodiment, the computer may display each segmented waveform sample, and the staff may determine the accurate phoneme segmentation position corresponding to the segmented waveform sample based on the characteristics of the segmented waveform.

[0086] Please continue to see Figure 5 In one embodiment, the speech segmentation method further includes:

[0087] Step 510: For each segmented waveform sample, obtain the audio data waveform corresponding to the segmented waveform sample; the segmented waveform sample is a waveform formed by dividing the waveform in the audio data waveform.

[0088] The segmented waveform sample is a waveform formed by dividing the waveform in the audio data waveform. The computer device can determine the audio data waveform corresponding to each segmented waveform sample based on the segmented waveform sample.

[0089] In an optional embodiment, a plurality of segmented waveform samples may correspond to the same audio data waveform, that is, a plurality of segmented waveform samples are divided out from one audio data waveform.

[0090] Step 520: Based on the accurate phoneme segmentation position corresponding to the segmented waveform sample, adjust the waveform area included in the segmented waveform sample to obtain a new segmented waveform, and add the new segmented waveform to the multiple segmented waveform samples.

[0091] After obtaining the accurate phoneme segmentation position, the computer device adjusts the audio data waveform area included in the segmentation waveform sample based on the position, and a new segmentation waveform can be obtained. In other words, a waveform with the same size as the segmentation waveform sample is obtained in the audio data waveform, but the waveform must include the accurate phoneme segmentation position. The computer device adds the obtained new segmentation waveform to multiple segmentation waveform samples, which can increase the number of samples of the training offset determination model.

[0092] In this embodiment, by increasing the number of segmented waveform samples, the accuracy of the offset determination model can be improved, so that a more accurate offset can be obtained, and then the phoneme segmentation position in the waveform of the audio data to be segmented can be determined.

[0093] In an optional embodiment, the computer device may add the segmented waveform and the determined target phoneme position in the segmented waveform to the training samples of the offset determination model to increase the number of training samples.

[0094] In one embodiment, the offset determination model is obtained by training the initial offset determination model according to a plurality of segmented waveform samples and accurate phoneme segmentation positions corresponding to the plurality of segmented waveform samples, including:

[0095] The initial offset determination model is trained according to the extracted image features of different channels of each segmented waveform sample and the accurate phoneme segmentation position corresponding to each segmented waveform sample to obtain the offset determination model.

[0096] In the process of training the initial offset determination model, each segmented waveform sample includes images of three channels of RGB. The computer device trains the image of each channel by extracting the image features of different channels in the segmented waveform sample and the accurate phoneme segmentation position corresponding to the segmented waveform sample. Finally, the results of the training of the image of each channel are unified to finally obtain the offset determination model.

[0097] In this embodiment, different channels correspond to different image features. By training the segmentation waveform samples by channel, the accuracy of the offset determination model can be improved. That is to say, a more accurate offset can be obtained according to the offset model, thereby improving the accuracy of determining the target phoneme segmentation position in the segmentation waveform.

[0098] In a specific embodiment, assuming that W1 in the segmented waveform is the ending phoneme and W2 is the starting phoneme, the multi-channel prediction mode of the offset determination model is shown in the following table:

[0099]

[0100] In one embodiment, the offset determination model includes a lightweight network model and a regression loss function.

[0101] The offset determination model uses a lightweight network model (MobileNet V2) as the backbone network, which is mainly used to obtain the features of the segmentation waveform. The lightweight network model generally includes two modes: head (classification prediction head) or heaaless (excluding classification prediction head). In this embodiment, the lightweight network model adopts the heaaless mode, in which fine-tuning is used, that is, two Dense layers are added after the feature layer of the lightweight network model, and finally the regression loss function is used as the output of the offset determination model. In this embodiment, a lightweight network model with low computational cost is used, and by adding two Dense layers after the feature layer of the lightweight network model, the features of the acquired segmentation waveform and the final output are combined in one network layer, and an accurate offset determination model can be obtained by training the initial offset determination model.

[0102] It should be understood that, although the various steps in the flowchart in the figure are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the figure may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0103] See also Figure 6 An embodiment of the present application provides a speech segmentation device 10, which includes a segmentation waveform determination module 11, an offset determination module 12, a first phoneme segmentation position determination module 13, and a second phoneme segmentation position determination module 14.

[0104] The segmentation waveform determining module 11 is used to divide the waveform of the audio data to be segmented according to the initial phoneme segmentation position of the audio data to be segmented, and determine the segmentation waveform; the segmentation waveform includes a waveform within a preset range centered on the initial phoneme segmentation position;

[0105] The offset determination module 12 is used to input the segmented waveform into the offset determination model to obtain the offset; the offset model is obtained by training the initial offset determination model based on multiple segmented waveform samples and the accurate phoneme segmentation positions corresponding to the multiple segmented waveform samples.

[0106] The first phoneme segmentation position determination module 13 is used to correct the initial phoneme segmentation position according to the offset to obtain the target phoneme segmentation position in the segmentation waveform;

[0107] The second phoneme segmentation position determination module 14 is used to determine the phoneme segmentation position in the waveform of the audio data to be segmented according to the target phoneme segmentation position, the segmentation waveform and the initial phoneme segmentation point.

[0108] In one embodiment, the second phoneme segmentation position determination module 14 includes a first determination unit and a second determination unit. The first determination unit is used to match the sub-waveform represented by the segmentation waveform with the waveform of the audio data to be segmented according to the initial phoneme segmentation position, determine the position of the sub-waveform in the waveform of the audio data to be segmented, and obtain the waveform position; the second determination unit is used to determine the phoneme segmentation position in the waveform of the audio data to be segmented according to the waveform position and the target phoneme segmentation position.

[0109] In one embodiment, the second determination unit is specifically used to determine the starting position of the sub-waveform in the waveform graph of the audio data to be segmented based on the waveform position, and obtain the waveform starting position; and perform an AND operation on the waveform starting position and the target phoneme segmentation position to obtain the phoneme segmentation position in the waveform graph of the audio data to be segmented.

[0110] In one embodiment, the segmentation waveform determination module 11 includes a third determination unit and a fourth determination unit. The first determination unit is used to determine the initial phoneme segmentation position of the audio data to be segmented based on the hidden Markov model and the mixed Gaussian model according to the audio data to be segmented; the second determination unit is used to obtain a waveform at a preset distance from the initial phoneme segmentation position in the waveform of the audio data to be segmented, with the initial phoneme segmentation position as the center, to obtain a segmentation waveform.

[0111] In one embodiment, the speech segmentation device 10 further includes an accurate phoneme segmentation position determination module, which is used to determine the accurate phoneme segmentation positions corresponding to the multiple segmentation waveform samples according to the characteristics of the segmentation waveforms in the multiple segmentation waveform samples.

[0112] In one embodiment, the speech segmentation device 10 further includes a first acquisition module and a second acquisition module. The first acquisition module is used to acquire, for each segmentation waveform sample, an audio data waveform corresponding to the segmentation waveform sample; the segmentation waveform sample is a waveform formed by a waveform divided from the audio data waveform; the second acquisition module is used to adjust the waveform range included in the segmentation waveform sample based on the accurate phoneme segmentation position corresponding to the segmentation waveform sample, obtain a new segmentation waveform, and add the new segmentation waveform to multiple segmentation waveform samples.

[0113] In one embodiment, the offset determination module 12 is specifically used to train the initial offset determination model according to the extracted image features of different channels of each segmented waveform sample and the accurate phoneme segmentation position corresponding to each segmented waveform sample to obtain the offset determination model.

[0114] In one embodiment, the offset determination model includes a lightweight network model and a regression loss function.

[0115] For the specific definition of the above-mentioned speech segmentation device 10, please refer to the definition of the speech segmentation method in the above text, which will not be repeated here. The various modules in the speech segmentation device 10 can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned devices, modules or units can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above-mentioned devices or modules.

[0116] See also Figure 7 In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 7 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store an offset determination model and a segmentation waveform diagram, etc. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer device is executed by the processor, a speech segmentation method is implemented.

[0117] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0118] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0119] Dividing the waveform graph of the audio data to be segmented according to the initial phoneme segmentation position of the audio data to be segmented, and determining the segmentation waveform graph; the segmentation waveform graph includes a waveform within a preset range centered on the initial phoneme segmentation position;

[0120] The segmented waveform is input into an offset determination model to obtain an offset; the offset determination model is obtained by training an initial offset determination model based on a plurality of segmented waveform samples and accurate phoneme segmentation positions corresponding to the plurality of segmented waveform samples;

[0121] Correcting the initial phoneme segmentation position according to the offset to obtain the target phoneme segmentation position in the segmentation waveform;

[0122] The phoneme segmentation position in the waveform of the audio data to be segmented is determined according to the target phoneme segmentation position, the segmentation waveform and the initial phoneme segmentation position.

[0123] In one embodiment, when the processor executes the computer program, the following steps are also implemented: according to the initial phoneme segmentation position, the sub-waveform represented by the segmentation waveform graph is matched with the waveform graph of the audio data to be segmented, the position of the sub-waveform in the waveform graph of the audio data to be segmented is determined, and the waveform position is obtained; according to the waveform position and the target phoneme segmentation position, the phoneme segmentation position in the waveform graph of the audio data to be segmented is determined.

[0124] In one embodiment, when the processor executes the computer program, the following steps are also implemented: according to the waveform position, the starting position of the sub-waveform is determined in the waveform graph of the audio data to be segmented to obtain the waveform starting position; and the waveform starting position and the target phoneme segmentation position are processed by an AND operation to obtain the phoneme segmentation position in the waveform graph of the audio data to be segmented.

[0125] In one embodiment, when the processor executes the computer program, the following steps are also implemented: based on the hidden Markov model and the mixed Gaussian model, according to the audio data to be segmented, the initial phoneme segmentation position of the audio data to be segmented is determined; with the initial phoneme segmentation position as the center, a waveform graph with a preset distance from the initial phoneme segmentation position is obtained in the waveform graph of the audio data to be segmented to obtain a segmented waveform graph.

[0126] In one embodiment, when the processor executes the computer program, the following steps are further implemented: determining accurate phoneme segmentation positions corresponding to the plurality of segmented waveform samples according to the features of the segmented waveforms in the plurality of segmented waveform samples.

[0127] In one embodiment, the processor also implements the following steps when executing the computer program: for each segmented waveform sample, obtain the audio data waveform corresponding to the segmented waveform sample; the segmented waveform sample is a waveform formed by the waveform divided in the audio data waveform; based on the accurate phoneme segmentation position corresponding to the segmented waveform sample, adjust the waveform range included in the segmented waveform sample to obtain a new segmented waveform, and add the new segmented waveform to multiple segmented waveform samples.

[0128] In one embodiment, when the processor executes the computer program, the following steps are also implemented: the initial offset determination model is trained according to the image features of different channels of each segmented waveform sample extracted, and the accurate phoneme segmentation position corresponding to each segmented waveform sample, to obtain the offset determination model.

[0129] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0130] Dividing the waveform graph of the audio data to be segmented according to the initial phoneme segmentation position of the audio data to be segmented, and determining the segmentation waveform graph; the segmentation waveform graph includes a waveform within a preset range centered on the initial phoneme segmentation position;

[0131] The segmented waveform is input into an offset determination model to obtain an offset; the offset determination model is obtained by training an initial offset determination model based on a plurality of segmented waveform samples and accurate phoneme segmentation positions corresponding to the plurality of segmented waveform samples;

[0132] Correcting the initial phoneme segmentation position according to the offset to obtain the target phoneme segmentation position in the segmentation waveform;

[0133] The phoneme segmentation position in the waveform of the audio data to be segmented is determined according to the target phoneme segmentation position, the segmentation waveform and the initial phoneme segmentation position.

[0134] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: according to the initial phoneme segmentation position, the sub-waveform represented by the segmentation waveform graph is matched with the waveform graph of the audio data to be segmented, the position of the sub-waveform in the waveform graph of the audio data to be segmented is determined, and the waveform position is obtained; according to the waveform position and the target phoneme segmentation position, the phoneme segmentation position in the waveform graph of the audio data to be segmented is determined.

[0135] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: based on the waveform position, the starting position of the sub-waveform is determined in the waveform graph of the audio data to be segmented to obtain the waveform starting position; and the waveform starting position and the target phoneme segmentation position are processed by an AND operation to obtain the phoneme segmentation position in the waveform graph of the audio data to be segmented.

[0136] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: based on the hidden Markov model and the mixed Gaussian model, the initial phoneme segmentation position of the audio data to be segmented is determined according to the audio data to be segmented; with the initial phoneme segmentation position as the center, a waveform graph with a preset distance from the initial phoneme segmentation position is obtained in the waveform graph of the audio data to be segmented to obtain a segmented waveform graph.

[0137] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: determining accurate phoneme segmentation positions corresponding to the plurality of segmented waveform samples according to the features of the segmented waveforms in the plurality of segmented waveform samples.

[0138] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: for each segmented waveform sample, an audio data waveform corresponding to the segmented waveform sample is obtained; the segmented waveform sample is a waveform formed by the waveform divided in the audio data waveform; based on the accurate phoneme segmentation position corresponding to the segmented waveform sample, the waveform range included in the segmented waveform sample is adjusted to obtain a new segmented waveform, and the new segmented waveform is added to multiple segmented waveform samples.

[0139] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: the initial offset determination model is trained according to the image features of different channels of each segmented waveform sample extracted, and the accurate phoneme segmentation position corresponding to each segmented waveform sample, to obtain the offset determination model.

[0140] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0141] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0142] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A speech segmentation method, characterized in that: include: The waveform graph of the audio data to be segmented is divided according to the initial phoneme segmentation position of the audio data to be segmented, and a segmentation waveform graph is determined; the segmentation waveform graph includes a waveform within a preset range centered on the initial phoneme segmentation position; the initial phoneme segmentation position refers to the coordinates of the initial phoneme segmentation point in the audio data to be segmented; Inputting the segmented waveform into an offset determination model to obtain an offset; the offset determination model is obtained by training an initial offset determination model based on a plurality of segmented waveform samples and accurate phoneme segmentation positions corresponding to the plurality of segmented waveform samples; Correcting the initial phoneme segmentation position according to the offset to obtain a target phoneme segmentation position in the segmentation waveform; Matching the sub-waveform represented by the segmentation waveform graph with the waveform graph of the audio data to be segmented according to the initial phoneme segmentation position, determining the position of the sub-waveform in the waveform graph of the audio data to be segmented, and obtaining the waveform position; According to the waveform position, determining the position of the starting position of the sub-waveform in the waveform diagram of the audio data to be segmented, and obtaining the waveform starting position; The waveform starting position and the target phoneme segmentation position are subjected to an AND operation to obtain the phoneme segmentation position in the waveform diagram of the audio data to be segmented.

2. The speech segmentation method according to claim 1, characterized in that: The step of dividing the waveform graph of the audio data to be segmented according to the initial phoneme segmentation positions of the audio data to be segmented to determine the segmented waveform graphs includes: Based on the hidden Markov model and the mixed Gaussian model, determining the initial phoneme segmentation position of the audio data to be segmented according to the audio data to be segmented; Taking the initial phoneme segmentation position as the center, a waveform graph having a preset distance from the initial phoneme segmentation position is obtained in the waveform graph of the audio data to be segmented to obtain the segmentation waveform graph.

3. The speech segmentation method according to claim 1, characterized in that: The method further comprises: The accurate phoneme segmentation positions corresponding to the plurality of segmentation waveform samples are determined according to the characteristics of the segmentation waveforms in the plurality of segmentation waveform samples.

4. The speech segmentation method according to claim 1, characterized in that: The method further comprises: For each of the segmented waveform samples, obtaining an audio data waveform corresponding to the segmented waveform sample; the segmented waveform sample is a waveform formed by dividing the waveform in the audio data waveform; Based on the accurate phoneme segmentation position corresponding to the segmented waveform sample, the waveform range included in the segmented waveform sample is adjusted to obtain a new segmented waveform, and the new segmented waveform is added to the multiple segmented waveform samples.

5. The speech segmentation method according to claim 1, characterized in that: The offset determination model is obtained by training an initial offset determination model according to a plurality of segmented waveform samples and accurate phoneme segmentation positions corresponding to the plurality of segmented waveform samples, and includes: The initial offset determination model is trained according to the extracted image features of different channels of each segmented waveform sample and the accurate phoneme segmentation position corresponding to each segmented waveform sample to obtain the offset determination model.

6. The speech segmentation method according to claim 1, characterized in that: The offset determination model includes a lightweight network model and a regression loss function.

7. A speech segmentation device, characterized in that: include: A segmentation waveform determination module is used to divide the waveform of the audio data to be segmented according to the initial phoneme segmentation position of the audio data to be segmented, and determine the segmentation waveform; the segmentation waveform includes a waveform within a preset range centered on the initial phoneme segmentation position; the initial phoneme segmentation position refers to the coordinates of the initial phoneme segmentation point in the audio data to be segmented; An offset determination module, used for inputting the segmented waveform into an offset determination model to obtain an offset; the offset determination model is obtained by training an initial offset determination model based on a plurality of segmented waveform samples and accurate phoneme segmentation positions corresponding to the plurality of segmented waveform samples; A first phoneme segmentation position determination module, configured to correct the initial phoneme segmentation position according to the offset to obtain a target phoneme segmentation position in the segmentation waveform; The second phoneme segmentation position determination module is used to match the sub-waveform represented by the segmentation waveform with the waveform of the audio data to be segmented according to the initial phoneme segmentation position, determine the position of the sub-waveform in the waveform of the audio data to be segmented, and obtain the waveform position; determine the position of the starting position of the sub-waveform in the waveform of the audio data to be segmented according to the waveform position, and obtain the waveform starting position; and perform an AND operation on the waveform starting position and the target phoneme segmentation position to obtain the phoneme segmentation position in the waveform of the audio data to be segmented.

8. The device according to claim 7, characterized in that The speech segmentation device further comprises an accurate phoneme segmentation position determination module, which is used for: The accurate phoneme segmentation positions corresponding to the plurality of segmentation waveform samples are determined according to the characteristics of the segmentation waveforms in the plurality of segmentation waveform samples.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images