Model training method and device, phoneme segmentation method and device, and storage medium

Abstract

The embodiment of the application provides a model training method and device, a phoneme segmentation method, equipment and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: acquiring a speech sample sequence and inputting the speech sample sequence into a preset phoneme segmentation model; performing hidden variable conversion on the speech sample sequence through an encoder to obtain an original speech feature sequence; performing mask processing on the original speech feature sequence through a decoder to obtain phoneme reference data; quantizing the original speech feature sequence according to a preset phoneme codebook through a phoneme quantization network to obtain a target phoneme feature sequence; performing phoneme boundary segmentation on the target phoneme feature sequence through a nearest neighbor clustering network to obtain phoneme segmentation data; performing loss calculation on the phoneme reference data and the phoneme segmentation data through a preset loss model to obtain a segmentation loss value; and adjusting parameters of the phoneme segmentation model according to the segmentation loss value. The embodiment of the application can construct a phoneme segmentation model more accurate in phoneme segmentation.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a model training method and apparatus, a phoneme segmentation method, device and storage medium. Background Technology

[0002] Phoneme segmentation is a prerequisite task for speech or audio product applications, with the main goal of automatically dividing the input speech audio into phoneme boundaries in a temporal sequence. In related technologies, phoneme segmentation models are mainly trained using a large amount of labeled data, which includes the start and end times of each phoneme and its label. Therefore, it requires a lot of manpower for annotation, and the annotators need to have professional knowledge in the speech domain, which increases the difficulty of training the phoneme segmentation model. Summary of the Invention

[0003] The main objective of this application is to propose a model training method and apparatus, a phoneme segmentation method, a device, and a storage medium, with the aim of constructing a phoneme segmentation model with higher accuracy.

[0004] To achieve the above objectives, a first aspect of this application proposes a model training method, the method comprising:

[0005] Obtain speech sample sequences;

[0006] The speech sample sequence is input into a preset phoneme segmentation model; wherein, the phoneme segmentation model includes: an encoder, a decoder, a phoneme quantization network, and a nearest neighbor clustering network;

[0007] The encoder performs latent variable transformation on the speech sample sequence to obtain the original speech feature sequence;

[0008] The original speech feature sequence is masked by the decoder to obtain phoneme reference data.

[0009] The original speech feature sequence is quantized by the phoneme quantization network according to a preset phoneme codebook to obtain the target phoneme feature sequence.

[0010] The target phoneme feature sequence is segmented by phoneme boundaries using the nearest neighbor clustering network to obtain phoneme segmentation data.

[0011] The phoneme reference data and the phoneme segmentation data are subjected to loss calculation using a preset loss model to obtain the segmentation loss value;

[0012] The phoneme segmentation model is adjusted based on the segmentation loss value.

[0013] In some embodiments, the step of quantizing the original speech feature sequence according to a preset phoneme codebook using the phoneme quantization network to obtain the target phoneme feature sequence includes:

[0014] The feature vector of the original speech feature sequence is obtained by the phoneme quantization network, thus obtaining the speech feature vector;

[0015] The phoneme entry vector is obtained by using the phoneme quantization network to obtain the entry vector of the phoneme codebook.

[0016] The target distance is obtained by calculating the distance between the speech feature vector and the phoneme entry vector through the phoneme quantization network.

[0017] The phoneme quantization network filters the phoneme entry vectors based on the target distance to obtain the target phoneme feature sequence.

[0018] In some embodiments, the step of segmenting the target phoneme feature sequence through the nearest neighbor clustering network to obtain phoneme segmentation data includes:

[0019] The target phoneme feature sequence is anomaly detected by the nearest neighbor clustering network to obtain the original anomaly value;

[0020] The original outliers are filtered for peak values ​​using the nearest neighbor clustering network to obtain the target outliers.

[0021] The phoneme segmentation data is generated based on the target outlier.

[0022] In some embodiments, the target phoneme feature sequence includes at least two phoneme feature sub-sequences; the step of performing anomaly detection on the target phoneme feature sequence through the nearest neighbor clustering network to obtain the original anomaly value includes:

[0023] The target feature subsequence is obtained by acquiring any one of the phoneme feature subsequences in the target phoneme feature sequence through the nearest neighbor clustering network;

[0024] The original outlier value is obtained by calculating the minimum distance between the target feature subsequence and the phoneme feature subsequence using the nearest neighbor clustering network.

[0025] In some embodiments, the step of performing peak filtering on the original outliers through the nearest neighbor clustering network to obtain target outliers includes:

[0026] The original outliers are filtered by the nearest neighbor clustering network according to a preset threshold to obtain candidate outliers;

[0027] The nearest neighbor clustering network is used to obtain any one of the candidate outliers to obtain the selected outlier;

[0028] Based on the selected outlier, obtain the previous candidate outlier to obtain the previous outlier; based on the selected outlier, obtain the next candidate outlier to obtain the next outlier.

[0029] If the selected outlier is greater than the previous outlier and the next outlier, the selected outlier is taken as the target outlier.

[0030] In some embodiments, the loss model includes: a context network and a target loss function; the step of calculating the segmentation loss value by performing loss calculation on the phoneme reference data and the phoneme segmentation data using a preset loss model includes:

[0031] Contextual features are extracted from the phoneme reference data using the context network to obtain candidate feature information;

[0032] The context network is used to extract context features from the phoneme segmentation data to obtain target feature information;

[0033] The candidate feature information is selected by selecting positive and negative samples using the target loss function to obtain reference feature information;

[0034] The segmentation loss value is obtained by calculating the loss between the reference feature information and the target feature information using the target loss function.

[0035] To achieve the above objectives, a second aspect of this application proposes a phoneme segmentation method, the method comprising:

[0036] Acquire target speech data;

[0037] The target speech data is input into the target phoneme segmentation model for phoneme segmentation processing to obtain target phoneme segmentation information; wherein, the target phoneme segmentation model is obtained by the model training method described in the first aspect.

[0038] To achieve the above objectives, a third aspect of this application provides a model training apparatus, the apparatus comprising:

[0039] The data acquisition module is used to acquire speech sample sequences;

[0040] An input module is used to input the speech sample sequence into a preset phoneme segmentation model; wherein, the phoneme segmentation model includes: an encoder, a decoder, a phoneme quantization network, and a nearest neighbor clustering network;

[0041] The encoder performs latent variable transformation on the speech sample sequence to obtain the original speech feature sequence;

[0042] The original speech feature sequence is masked by the decoder to obtain phoneme reference data.

[0043] The original speech feature sequence is quantized by the phoneme quantization network according to a preset phoneme codebook to obtain the target phoneme feature sequence.

[0044] The nearest neighbor clustering network is used to select the phoneme segmentation boundary of the target phoneme feature sequence to obtain phoneme segmentation data.

[0045] The loss calculation module is used to perform loss calculation on the phoneme reference data and the phoneme segmentation data through a preset loss model to obtain the segmentation loss value;

[0046] The parameter adjustment module is used to adjust the parameters of the phoneme segmentation model according to the segmentation loss value.

[0047] To achieve the above objectives, a fourth aspect of the present application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the model training method described in the first aspect or the phoneme segmentation method described in the second aspect.

[0048] To achieve the above objectives, a fifth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the model training method described in the first aspect or the phoneme segmentation method described in the second aspect.

[0049] The model training method, apparatus, phoneme segmentation method, device, and storage medium proposed in this application establish a phoneme segmentation model including a phoneme quantization network and a nearest neighbor clustering network. The phoneme quantization network uses a pre-set phoneme codebook to represent the input original speech feature sequence as corresponding phonemes. Then, the nearest neighbor clustering network is used to segment phoneme boundaries to obtain more accurate phoneme segmentation data. At the same time, the segmentation loss value is calculated, and the parameters of the phoneme segmentation model are adjusted according to the segmentation loss value to train a more accurate phoneme segmentation model and improve the operation of downstream tasks after phoneme segmentation. Attached Figure Description

[0050] Figure 1 This is an example application scenario diagram of the model training method provided in the embodiments of this application;

[0051] Figure 2 This is a flowchart of the model training method provided in the embodiments of this application;

[0052] Figure 3 This is a schematic diagram of the phoneme segmentation model framework in the model training method provided in the embodiments of this application;

[0053] Figure 4 yes Figure 2 The flowchart of step S205 in the text;

[0054] Figure 5 yes Figure 2 The flowchart of step S206 in the text;

[0055] Figure 6 yes Figure 5 The flowchart of step S501 in the process;

[0056] Figure 7 yes Figure 5 The flowchart of step S502 in the document;

[0057] Figure 8 yes Figure 2 The flowchart of step S207 in the text;

[0058] Figure 9 This is a flowchart of the phoneme segmentation method provided in the embodiments of this application;

[0059] Figure 10 This is a schematic diagram of the structure of the model training device provided in the embodiments of this application;

[0060] Figure 11 This is a schematic diagram of the phoneme segmentation device provided in the embodiments of this application;

[0061] Figure 12 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0063] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0065] First, let's analyze some of the terms used in this application:

[0066] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0067] Speech processing, also known as speech signal processing or human voice processing, aims to generate desired signals for further speech recognition and application in mobile phone interfaces and everyday life, enabling communication between humans and computers. The key technologies of speech signal processing are based on general digital signal processing theory, and its main research areas are speech coding and speech compression techniques.

[0068] Phoneme: A phoneme is the smallest unit of speech determined by the natural properties of sound, and is divided into two main categories: vowels and consonants. From an acoustic perspective, a phoneme is the smallest unit of speech determined by its sound quality. From a physiological perspective, one articulation action forms one phoneme. For example, [ma] contains two articulation actions, [m] and [a], and are two phonemes. Sounds produced by the same articulation action are the same phoneme, and sounds produced by different articulation actions are different phonemes.

[0069] Codebook: A codebook is a collection of available codes that are grouped together to form a table. Different codes are then numbered with serial numbers. The codebook is a collection of codes that can be searched for directly in the table based on the serial number.

[0070] Latent variable approach: The basic idea of ​​the latent variable approach is to treat total factor productivity as a latent variable, i.e., an unobserved variable, and then use the state-space model to give an estimate of total factor productivity using maximum likelihood estimation.

[0071] Mask: A mask is a string of binary code that performs a bitwise AND operation on a target field to mask the current input bits. Masks can be categorized into subnet masks and interrupt masks, etc. New operands are obtained by performing bitwise or logical operations between the source code and the mask.

[0072] Quantization: In digital signal processing, quantization refers to the process of approximating a continuous signal value (or a large number of possible discrete values) to a finite number (or a small number) of discrete values. Quantization is primarily used in the conversion from continuous signals to digital signals. Continuous signals are sampled to become discrete signals, and discrete signals are quantized to become digital signals. Note that discrete signals do not necessarily require quantization. Quantization converts the waveform of analog sound into digital data; the number of binary bits representing the sampled value determines the quantization precision. The quantization process involves first dividing the entire amplitude into a finite set of small amplitudes (quantization steps), grouping samples falling within a certain step into a single category, and assigning them the same quantization value.

[0073] K-Nearest Neighbors (KNN) is a basic classification and regression algorithm. The input to KNN is the feature vector of an instance, corresponding to a point in the feature space; the output is the instance's class, which can be multiple classes. KNN assumes a training dataset with predefined instance classes. During classification, for a new instance, it predicts its class based on the classes of its k nearest neighbor training instances, using methods such as majority voting. Simply put, given a training dataset, for a new input instance, it finds the k nearest neighbors in the training set, and the class of the input instance is determined by the majority of these k neighbors. This is the origin of the 'k' in KNN; typically, k is an integer no greater than 20.

[0074] Phoneme segmentation is a crucial prerequisite for many speech or audio product applications, such as speech recognition, keyword detection, and voiceprint recognition. Its main goal is to automatically delineate phoneme boundaries in the temporal sequence of input speech audio, thereby improving the performance of downstream tasks. Among related technologies, phoneme segmentation primarily employs supervised phoneme segmentation models. However, supervised phoneme segmentation models are trained on large amounts of labeled data, with annotations including the start and end times and phoneme labels for each phoneme. This requires significant human and time investment and expertise in the speech domain. Furthermore, the lack of labeled trees for less common languages ​​makes it unsuitable for phoneme segmentation across multiple language types.

[0075] Based on this, embodiments of this application provide a model training method and apparatus, a phoneme segmentation method, device, and storage medium. The method involves inputting a speech sample sequence into a preset phoneme segmentation model. The speech segmentation model includes an encoder, a decoder, a phoneme quantization network, and a nearest neighbor clustering network. The encoder performs latent variable processing on the speech sample sequence to obtain an original speech feature sequence. The decoder performs masking processing on the original speech feature sequence to obtain phoneme reference data. The phoneme quantization network then quantizes the original speech feature sequence according to a preset phoneme codebook to obtain a target phoneme feature sequence. The nearest neighbor clustering network performs phoneme boundary segmentation on the target phoneme feature sequence to obtain phoneme segmentation data. A preset loss model calculates the loss between the phoneme reference data and the phoneme segmentation data to obtain a segmentation loss value. The parameters of the phoneme segmentation model are adjusted based on the segmentation loss value until the segmentation loss value converges, thus completing the training of the phoneme segmentation model. Therefore, by using a phoneme quantization network to represent the original speech feature sequence as a target phoneme feature sequence corresponding to the phonemes, and then using a nearest neighbor clustering network to determine the phoneme boundaries of the target phoneme feature sequence to obtain phoneme segmentation data, phoneme segmentation becomes more accurate. At the same time, the parameters of the phoneme segmentation model are adjusted based on the segmentation loss value to construct a more accurate phoneme segmentation model.

[0076] The model training method and apparatus, phoneme segmentation method, device and storage medium provided in the embodiments of this application are specifically described through the following embodiments. First, the model training method in the embodiments of this application is described.

[0077] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0078] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0079] The model training method provided in this application relates to the field of artificial intelligence technology. The model training method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the model training method, but is not limited to the above forms.

[0080] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0081] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0082] Please refer to Figure 1 , Figure 1This diagram illustrates an example application scenario of the model training method provided in this application. The scenario includes a data acquisition terminal 101 and a server 102. The model training method provided in this application can be deployed on the server 102. The data acquisition terminal 101 collects speech data, and the speech data is input into a preset database to obtain a speech sample sequence. Then, the server 102 trains the model based on the speech sample sequence to construct a phoneme segmentation model capable of accurate phoneme segmentation. The server 102 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, full-service, CDN, and big data and artificial intelligence platforms. Furthermore, it should be understood that the server 102 is only shown as an example. In practice, other devices or combinations of devices with computing and storage capabilities can be used alternatively or additionally to provide the corresponding services.

[0083] Figure 2 This is an optional flowchart of the model training method provided in the embodiments of this application. Figure 2 The method may include, but is not limited to, steps S201 to S208.

[0084] Step S201: Obtain the speech sample sequence;

[0085] Step S202: Input the speech sample sequence into a preset phoneme segmentation model; wherein, the phoneme segmentation model includes: an encoder, a decoder, a phoneme quantization network, and a nearest neighbor clustering network;

[0086] Step S203: The speech sample sequence is transformed by latent variables through the encoder to obtain the original speech feature sequence;

[0087] Step S204: The original speech feature sequence is masked by the decoder to obtain phoneme reference data;

[0088] Step S205: The original speech feature sequence is quantized by a phoneme quantization network according to a preset phoneme codebook to obtain the target phoneme feature sequence.

[0089] Step S206: The target phoneme feature sequence is segmented by phoneme boundary using a nearest neighbor clustering network to obtain phoneme segmentation data;

[0090] Step S207: Calculate the loss of phoneme reference data and phoneme segmentation data using a preset loss model to obtain the segmentation loss value;

[0091] Step S208: Adjust the parameters of the phoneme segmentation model based on the segmentation loss value.

[0092] Steps S201 to S208 of the embodiments of this application involve constructing a phoneme segmentation model including an encoder, a decoder, a phoneme quantization network, and a nearest neighbor clustering network. The speech sample sequence is input into the phoneme segmentation model. First, the encoder performs latent variable transformation on the speech sample sequence to obtain the original speech feature sequence. Then, the decoder performs masking processing on the original speech feature sequence to obtain phoneme reference data. Simultaneously, the phoneme quantization network quantizes the original speech feature sequence according to a preset phoneme codebook to obtain the target phoneme feature sequence. The nearest neighbor clustering network performs phoneme boundary segmentation on the target phoneme feature sequence to obtain phoneme segmentation data. Finally, a preset loss model is used to calculate the loss between the phoneme reference data and the phoneme segmentation data to obtain a segmentation loss value. The parameters of the phoneme segmentation model are then adjusted based on the segmentation loss value. Therefore, a phoneme segmentation model including a phoneme quantization network and a nearest neighbor clustering network is set up. The phoneme quantization network uses a pre-set phoneme codebook to represent the input original speech feature sequence as the corresponding phonemes. Then, the nearest neighbor clustering network is used to segment the phoneme boundaries to obtain more accurate phoneme segmentation data. At the same time, the segmentation loss value is calculated, and the parameters of the phoneme segmentation model are adjusted according to the segmentation loss value to train a more accurate phoneme segmentation model, thereby improving the operation of downstream tasks after phoneme segmentation.

[0093] In step S201 of some embodiments, a speech sample sequence can be obtained from a preset sample database, or by directly listening to user speech data to obtain a speech sample sequence, or by other means, not limited to these. The speech sample sequence is an unlabeled sample sequence, used to extract phonemes from the speech sample sequence using an unsupervised phoneme segmentation model.

[0094] In step S202 of some embodiments, a phoneme segmentation model is constructed, comprising an encoder, a decoder, a phoneme quantization network, and a nearest neighbor clustering network. In this embodiment, the phoneme segmentation model is an improvement upon the classic unsupervised speech pre-training model, specifically the wav2vec2.0 model. The core idea of ​​the wav2vec2.0 model is to construct a self-supervised training objective through vector quantization, and then train the model using a loss function based on comparison learning after extensive masking of the input. This embodiment's phoneme segmentation model retains vector quantization and optimizes the traditional quantization network. This embodiment constructs a phoneme quantization network, which pre-sets a phoneme codebook to represent speech features as phoneme features. Furthermore, this embodiment's phoneme segmentation model adds a nearest neighbor clustering network, which can determine phoneme boundaries to achieve phoneme segmentation. Therefore, this embodiment constructs a phoneme segmentation model including an encoder, a decoder, a phoneme quantization network, and a nearest neighbor clustering network, constructs speech features into phoneme feature sequences, and directly judges phoneme boundaries from the phoneme feature sequences, making the phoneme segmentation operation more accurate.

[0095] Specifically, please refer to Figure 3 , Figure 3 This is a schematic diagram of the phoneme segmentation model framework in the model training method provided in this application embodiment. The encoder is used to encode the input speech sample sequence to obtain the original speech feature sequence z of the latent variables. t Then, the phoneme quantization network processes the original speech feature sequence z according to a preset phoneme codebook. t Quantization is performed to map the original speech feature sequence to quantized latent variables to obtain the target phoneme feature sequence. Z t Meanwhile, the original speech feature sequence z t The input decoder performs masking processing to obtain phoneme reference data, and the target phoneme feature sequence. Z t The input is fed into a nearest neighbor clustering network to perform phoneme boundary segmentation, resulting in phoneme segmentation data S. t Finally, the loss module is used to calculate the segmentation loss value from the phoneme reference data and the phoneme segmentation data. Based on the segmentation loss value, the parameters of the phoneme segmentation model are adjusted to construct a more accurate phoneme segmentation model.

[0096] In step S203 of some embodiments, the speech sample sequence is input into the encoder. The encoder performs latent variable processing on the speech sample sequence, that is, it embeds the speech sample sequence according to each time step t to obtain the original speech feature sequence represented by latent variables. In other words, it represents the speech features in the speech sample sequence in vector form to facilitate the operation on the speech sample sequence. For example, if the speech sample sequence is X = (x1, ..., x... t ,…,xT The encoder embeds latent variables into the speech sample sequence at time step t to obtain the original speech feature sequence Z = (z1, ..., z2). t ,…,z T ).

[0097] Specifically, the encoder is a neural network with six convolutional blocks and a kernel stride of 2. The six convolutional blocks with a kernel stride of 2 perform latent variable transformation on the speech sample sequence, representing the speech sample features as vectors to obtain the original speech feature sequence. The original speech feature sequence includes at least two speech feature vectors, and each speech feature vector represents a speech feature.

[0098] In step S204 of some embodiments, the decoder is a masked transformer decoder, and the masked transformer decoder decodes only one word at a time. After outputting the word, it is put into the input of the decoder, and the operation is repeated until the decoding is completed to complete the sentence output and obtain phoneme reference data. The phoneme reference data serves as reference data for judging the accuracy of phoneme segmentation data. In this embodiment, the decoder combines the original speech feature sequences output by multiple time-step encoders, first performs masking processing on the original speech feature sequences, and then calculates global information to obtain phoneme reference data. The masking processing also involves decoding the original speech feature sequences to convert them into phoneme representations to obtain phoneme reference data. Therefore, by using the decoder to perform masking processing on the original speech feature sequences output by the encoder to obtain phoneme reference data, the processing of phoneme reference data is simplified.

[0099] Please see Figure 4 In some embodiments, step S205 may include, but is not limited to, steps S401 to S404:

[0100] Step S401: Obtain the feature vector of the original speech feature sequence through the phoneme quantization network to obtain the speech feature vector;

[0101] Step S402: Obtain the entry vector of the phoneme codebook through the phoneme quantization network to obtain the phoneme entry vector;

[0102] Step S403: The distance between the speech feature vector and the phoneme entry vector is calculated using a phoneme quantization network to obtain the target distance;

[0103] Step S404: The phoneme entry vectors are filtered based on the target distance using a phoneme quantization network to obtain the target phoneme feature sequence.

[0104] In step S401 of some embodiments, the original speech feature sequence includes at least two speech feature vectors. Then, a phoneme quantization network is used to obtain the speech feature vectors of the original speech feature sequence, so that each speech feature vector can be processed. Specifically, by obtaining the speech feature vectors, which represent speech features, the speech features can be determined based on the speech feature vectors, and the corresponding phonemes can be determined based on the speech feature vectors.

[0105] In step S402 of some embodiments, in order to convert the original speech feature sequence into a phoneme representation sequence, the entry vectors of the phoneme codebook are obtained to obtain phoneme entry vectors. The phoneme codebook is a codebook representing phonemes, and the speech feature vectors are represented by the phoneme entry vectors in the phoneme codebook, making phoneme segmentation more accurate. In this embodiment, the phoneme codebook includes 40 phoneme entry vectors, with 39 phoneme entry vectors representing one phoneme each; that is, 39 phoneme entry vectors represent 39 basic phonemes, and the remaining phoneme entry vector represents silence. Therefore, by obtaining 40 phoneme entry vectors and representing phoneme features with 40 phoneme entry vectors, phoneme segmentation is more accurate.

[0106] In step S403 of some embodiments, a target distance is obtained by calculating the distance between the speech feature vector and the phoneme entry vector through a phoneme quantization network. Specifically, the target distance is obtained by calculating the Euclidean distance between the speech feature vector and the phoneme entry vector. Based on the target distance, the phoneme entry vector that is closest to the speech feature vector is found. The closest phoneme entry vector is then used as the feature vector in the target phoneme feature sequence. The phoneme entry vector represents the speech features to obtain the target phoneme feature sequence, which makes the phoneme segmentation more accurate.

[0107] In step S404 of some embodiments, the phoneme entry vectors are filtered according to the target distance, that is, the phoneme entry vector with the smallest target distance is obtained, and this phoneme entry vector is replaced with a speech feature vector to obtain the target phoneme feature sequence. By replacing the speech feature vector in the original speech feature sequence with the phoneme entry vector, the original speech feature sequence is represented in the form of a phoneme feature vector to obtain the target phoneme feature sequence, which makes phoneme boundary judgment easier.

[0108] It should be noted that if the original speech feature sequence is Z = (z1, ..., z...), then... t ,…,z T ), and the phoneme codebook is E = (e1, ..., e k ,…,e 40Then, the Euclidean distance between each speech feature vector and the phoneme entry vector is calculated to obtain the target distance. Based on the target distance, the phoneme entry vector with the closest speech feature vector is selected, and the speech feature vector is replaced with the phoneme entry vector to obtain the target phoneme feature sequence, which is... To obtain the target phoneme feature sequence Replace the original speech feature sequence z t By representing speech feature sequences through phoneme vectors, phoneme boundary lookup becomes more accurate.

[0109] In steps S401 to S404 of the embodiments of this application, the speech feature vector of the original speech feature sequence is obtained through a phoneme quantization network, and then the phoneme entry vector of the phoneme codebook is obtained. The Euclidean distance between the phoneme entry vector and the speech feature vector is then calculated to obtain the target distance. Based on the target distance, the phoneme entry vector with the closest distance to the speech feature vector is selected, and the phoneme entry vector replaces the speech feature vector in the original speech feature sequence to obtain the target phoneme feature sequence. Therefore, by representing speech features as corresponding phoneme features, phoneme segmentation becomes more accurate.

[0110] Please see Figure 5 In some embodiments, step S206 may include, but is not limited to, steps S501 to S503:

[0111] Step S501: Anomaly detection is performed on the target phoneme feature sequence through a nearest neighbor clustering network to obtain the original anomaly value;

[0112] Step S502: Peak filtering is performed on the original outliers using a nearest neighbor clustering network to obtain the target outliers;

[0113] Step S503: Generate phoneme segmentation data based on the target outlier.

[0114] In step S501 of some embodiments, anomaly detection is performed on the target phoneme feature sequence through a nearest neighbor clustering network to obtain the original anomaly value. The anomaly detection involves calculating the distance between each phoneme feature vector in the target phoneme feature sequence to obtain the original anomaly value. The original anomaly value is used to characterize the abnormal relationship between phoneme features, and the original anomaly value is proportional to the abnormal relationship between phoneme features. The original anomaly value is used as the judgment standard for phoneme boundaries, making the judgment of phoneme boundaries easier.

[0115] In step S502 of some embodiments, target outliers are obtained by peak filtering of the original outliers using a nearest neighbor clustering network. Peak detection involves sorting the original outliers in order to form an outlier curve, and then determining the peak value of the outlier curve. The target outlier is used as the peak value of the original outliers to determine the phoneme segmentation boundary, making the phoneme segmentation boundary search easier.

[0116] In step S503 of some embodiments, since the target outlier represents the most likely phoneme segmentation boundary, phoneme segmentation data is directly generated based on the target outlier to determine the phoneme segmentation boundary through the phoneme segmentation data, thus making the phoneme segmentation operation simpler.

[0117] In steps S501 to S503 of the embodiments of this application, anomaly detection is performed on the target phoneme feature sequence using a nearest neighbor clustering network to obtain original outlier values. Then, peak filtering is applied to the original outlier values ​​to obtain target outlier values, which are used to generate phoneme segmentation data. Therefore, calculating the original outlier values ​​using nearest neighbor clustering as reference values ​​for judging phoneme boundaries makes phoneme boundary judgment easier and the generated phoneme segmentation data more accurate.

[0118] Please see Figure 6 In some embodiments, the target phoneme feature sequence includes at least two phoneme feature sub-sequences, and step S501 may include, but is not limited to, steps S601 to S602:

[0119] Step S601: Obtain any one phoneme feature subsequence from the target phoneme feature sequence through the nearest neighbor clustering network to obtain the target feature subsequence;

[0120] Step S602: The minimum distance between the target feature subsequence and the phoneme feature subsequence is calculated using a nearest neighbor clustering network to obtain the original outlier value.

[0121] In step S601 of some embodiments, the target phoneme feature sequence includes at least two phoneme feature subsequences, and each phoneme feature subsequence includes at least two phoneme feature vectors. Therefore, the target feature subsequence is obtained by using a nearest neighbor clustering network to obtain any one of the phoneme feature subsequences in the target phoneme feature sequence, making the acquisition of the target feature subsequence simple.

[0122] Specifically, first, the feature subsequence centered at a preset data frame t is obtained from the target phoneme feature sequence. Then, the phoneme feature vectors in the feature subsequence are concatenated to obtain the phoneme feature subsequence, making the acquisition of the phoneme feature subsequence simple. Among them, the phoneme feature subsequence with a window size of 10 frames in the obtained target phoneme feature sequence is used to obtain the target feature subsequence.

[0123] In step S602 of some embodiments, the minimum distance between the target feature subsequence and the phoneme feature subsequence is calculated using a nearest neighbor clustering network to define the original outliers. Since the target phoneme feature sequence includes at least two phoneme feature subsequences, at least two original outliers are calculated, which is to obtain the original outlier sequence.

[0124] Specifically, the target phoneme feature sequence is The target feature subsequence is Phoneme feature subsequence is The minimum distance between the target feature subsequence and the phoneme feature subsequence is then calculated as follows: To obtain at least two original outliers, that is, to obtain the sequence of original outliers. Therefore, the most likely phoneme segmentation boundaries can be determined by the original outlier sequence.

[0125] In steps S601 to S602 as illustrated in the embodiments of this application, a target feature subsequence is obtained by acquiring any one phoneme feature subsequence from the target phoneme feature sequence, and then the other phoneme feature subsequences and the target feature subsequence are subjected to minimum distance calculation, which is also the calculation of the nearest neighbor distance F. kNN At least two raw outliers are obtained to determine the correlation between phoneme features, thereby selecting phoneme segmentation boundaries and making phoneme segmentation easier.

[0126] Please see Figure 7 In some embodiments, step S502 may include, but is not limited to, steps S701 to S704:

[0127] Step S701: The original outliers are filtered by the nearest neighbor clustering network according to a preset threshold to obtain candidate outliers;

[0128] Step S702: Obtain any candidate outlier through the nearest neighbor clustering network to obtain the selected outlier;

[0129] Step S703: Obtain the previous candidate outlier based on the selected outlier to obtain the previous outlier; obtain the next candidate outlier based on the selected outlier to obtain the next outlier.

[0130] Step S704: If the selected outlier is greater than the previous outlier and the next outlier, the selected outlier will be used as the target outlier.

[0131] In step S701 of some embodiments, candidate outliers are selected from at least two original outliers using a nearest neighbor clustering network, with outliers greater than a preset threshold, to initially filter out outliers that do not belong to peak points. Specifically, for original outliers with a preset threshold less than the lowest peak value, an outlier curve is first constructed based on the at least two original outliers. The derivative of the outlier curve is then calculated to obtain the derivative value. Points between negative and positive derivative values ​​are used to determine peak values, and the original outlier with the smallest peak value is obtained as the minimum outlier. The preset threshold is then determined based on the minimum outlier value. Therefore, by filtering out original outliers below the preset threshold to obtain candidate outliers, and then selecting peak values ​​from these candidate outliers, the amount of data computation is saved.

[0132] In step S702 of some embodiments, any candidate outlier is obtained to obtain a selected outlier, and the selected outlier is any selected candidate outlier, so as to compare it with other candidate outliers to filter out the target outlier with the peak value.

[0133] In step S703 of some embodiments, after obtaining the selected outlier, the previous candidate outlier is obtained to obtain the previous outlier, and then the next candidate outlier is obtained to obtain the next outlier, so as to perform peak detection based on the selected outlier, the previous outlier, and the next outlier, making peak detection more accurate.

[0134] In step S704 of some embodiments, a selected outlier greater than both the previous and next outlier values ​​is obtained; that is, the selected outlier value is determined to be the peak value to obtain the target outlier, thus simplifying the selection of the target outlier. For example, if the previous outlier value is P... t-1 Select an anomaly value as P t The latter outlier is P t+1 If P t-1 <P t >P t+1 Then determine the selected outlier P. t The target outliers are selected, and the phoneme features corresponding to the target outliers are used as the most likely phoneme segmentation boundaries. Phoneme segmentation data is then generated based on the target outliers, making the phoneme segmentation operation simpler and more accurate.

[0135] In steps S701 to S704 of this embodiment, candidate outliers are obtained by using a nearest neighbor clustering network to acquire original outliers greater than a preset threshold. Then, an arbitrary candidate outlier is selected as the chosen outlier. The preceding candidate outlier is selected as the previous outlier, and the following candidate outlier is selected as the next outlier. Finally, the chosen outlier, which is greater than both the preceding and next outliers, is selected as the target outlier. This simplifies the selection of target outliers, allowing for the determination of phoneme segmentation boundaries based on the target outliers, thus making the phoneme segmentation operation simpler and more accurate.

[0136] Please see Figure 8 In some embodiments, the loss model includes: a context network and a target loss function; step S207 includes, but is not limited to, steps S801 to S804:

[0137] Step S801: Extract contextual features from phoneme reference data using a context network to obtain candidate feature information;

[0138] Step S802: Extract contextual features from the phoneme segmentation data using a context network to obtain target feature information;

[0139] Step S803: Select positive and negative samples from candidate feature information using the target loss function to obtain reference feature information;

[0140] Step S804: Calculate the loss of the reference feature information and the target feature information using the target loss function to obtain the segmentation loss value.

[0141] In step S801 of some embodiments, phoneme reference data is input into a context network to extract context features to obtain candidate feature information. The candidate feature information is used as a reference standard for judging the training of phoneme segmentation data to train the phoneme segmentation model and obtain more accurate phoneme segmentation data.

[0142] In step S802 of some embodiments, the phoneme segmentation data is input into the context network to extract context features and obtain target feature information. To analyze the segmentation accuracy of the phoneme segmentation data, it is necessary to obtain target feature information and candidate feature information of the same data type for analysis so that the phoneme segmentation accuracy can be judged more accurately.

[0143] In step S803 of some embodiments, positive and negative samples are selected from candidate feature information using a target loss function to obtain reference feature information. To make the target feature information comparison more accurate, and since the core of unsupervised learning is to narrow the distance between positive examples of the phoneme segmentation data and the phoneme reference data, and widen the distance between them and negative examples, positive and negative samples are obtained from the candidate feature information to obtain reference feature information. This means dividing the candidate feature information into reference feature information corresponding to positive and negative samples, and then calculating the loss based on the reference feature information of the negative samples and the target feature information to improve the accuracy of feature analysis.

[0144] In step S804 of some embodiments, a segmentation loss value is obtained by calculating the loss between the reference feature information and the target feature information using a target loss function. This segmentation loss value characterizes the distance between the phoneme segmentation data and the phoneme reference data. Since the reference feature information is the candidate feature information of the negative samples, the segmentation loss value is calculated. Therefore, the parameters of the phoneme segmentation model are adjusted based on the segmentation loss value, and steps S201 to S207 are repeated until the segmentation loss value converges, completing the training of the phoneme segmentation model. Specifically, since the reference feature information is the reference feature information corresponding to the positive and negative samples, and the target loss function is the contrastive loss function of contrastive learning, the loss value between the reference feature information and the target feature information corresponding to the positive and negative samples is calculated using the contrastive loss function to obtain the segmentation loss value. This segmentation loss value is then used to adjust the parameters of the phoneme segmentation model, resulting in better training performance and the construction of a more accurate phoneme segmentation model.

[0145] In steps S801 to S804 of this embodiment, candidate feature information is obtained by extracting context features from the phoneme reference data using a context network. Then, target feature information is obtained by extracting upper and lower features from the phoneme segmentation data using the context network. Reference feature information is obtained by selecting positive and negative samples from the candidate feature information using a target loss function. Finally, a segmentation loss value is obtained by calculating the loss between the reference and target feature information using the target loss function. Therefore, by using the reference and target feature information corresponding to positive and negative samples to calculate the segmentation loss value, the parameters of the phoneme segmentation model can be adjusted more accurately based on the segmentation loss value, resulting in better model training performance for the phoneme segmentation model.

[0146] Please see Figure 9 This application also provides a phoneme segmentation method, which may include, but is not limited to, steps S901 to S902:

[0147] Step S901: Obtain target speech data;

[0148] Step S902: Input the target speech data into the target phoneme segmentation model for phoneme segmentation processing to obtain target phoneme segmentation information; wherein, the target phoneme segmentation model is obtained by the model training method described above.

[0149] In steps S901 to S902 of this embodiment, target speech data is acquired and input into the target phoneme segmentation model trained by the above-described model training method. The target phoneme segmentation model is obtained after multiple parameter adjustments by the above-described model training method and the segmentation loss value converges. Therefore, by performing phoneme segmentation processing on the target speech data using the target phoneme segmentation model to obtain target phoneme segmentation information, the phoneme segmentation operation is simple and accurate.

[0150] In step S901 of some embodiments, audio data is collected, and the speech data of the target object in the audio data is extracted to obtain target speech data. If the audio data includes speech data of multiple objects, the speech data is divided according to the objects, and the timbre information of the speech data is obtained to obtain candidate timbre information. Then, speech data that matches the candidate timbre information with preset timbre information is obtained to obtain target speech data.

[0151] In step S902 of some embodiments, the target phoneme segmentation model includes an encoder, a phoneme quantization network, and a nearest neighbor clustering network. The encoder performs latent variable processing on the target speech data to obtain a target speech feature sequence. Then, the phoneme quantization network quantizes the target speech feature sequence according to a preset phoneme codebook to obtain a candidate phoneme feature sequence. The nearest neighbor clustering network performs phoneme boundary segmentation on the candidate phoneme feature sequence to obtain target segmentation data. Finally, the target speech data is segmented based on the target segmentation data to obtain target phoneme segmentation information. Therefore, phoneme segmentation using the encoder, phoneme quantization network, and nearest neighbor clustering network makes the phoneme segmentation operation both simple and accurate. After obtaining the target phoneme segmentation information, if the downstream task is speech recognition, keyword detection, etc., the downstream task performed based on the target phoneme segmentation information is more accurate.

[0152] Please refer to Figure 3This application embodiment acquires a speech sample sequence and inputs it into an encoder. The encoder performs latent variable transformation on the speech sample sequence to obtain the original speech feature sequence. The encoder then performs masking processing on the original speech feature sequence to obtain phoneme reference data. Simultaneously, a phoneme quantization network is used to obtain the speech feature vector of the original speech feature sequence, and then the phoneme entry vector of the phoneme codebook is obtained. The Euclidean distance between the phoneme entry vector and the speech feature vector is calculated to obtain the target distance. Based on the target distance, the phoneme entry vector with the closest speech feature vector is selected, and the phoneme entry vector replaces the speech feature vector in the original speech feature sequence to obtain the target phoneme feature sequence. A nearest neighbor clustering network is used to perform anomaly detection on the target phoneme feature sequence to obtain the original anomaly value. Peak filtering processing is then performed on the original anomaly value to obtain the target anomaly value, which is used to generate phoneme segmentation data. A context network is used to extract contextual features from the phoneme reference data to obtain candidate feature information. Then, contextual feature extraction is performed on the phoneme segmentation data to obtain the target feature information. A target loss function is used to select positive and negative samples from the candidate feature information to obtain reference feature information. Finally, a loss calculation is performed on the reference feature information and the target feature information to obtain the segmentation loss value. Finally, the parameters of the phoneme segmentation model are adjusted based on the segmentation loss value until the segmentation loss value converges, at which point the parameter adjustment of the phoneme segmentation model is stopped, thus obtaining the target phoneme segmentation model. By acquiring the target speech data and inputting it into the target phoneme segmentation model for phoneme segmentation processing, the target phoneme segmentation information is obtained, making the phoneme segmentation operation simple and accurate.

[0153] Please see Figure 10 This application also provides a model training apparatus that can implement the above-described model training method. The apparatus includes:

[0154] Data acquisition module 1001 is used to acquire speech sample sequences;

[0155] The input module 1002 is used to input the speech sample sequence into a preset phoneme segmentation model; wherein, the phoneme segmentation model includes: encoder 1003, decoder 1004, phoneme quantization network 1005 and nearest neighbor clustering network 1006;

[0156] The speech sample sequence is transformed using encoder 1003 to obtain the original speech feature sequence;

[0157] The original speech feature sequence is masked by decoder 1004 to obtain phoneme reference data;

[0158] The phoneme feature sequence is obtained by quantizing the phoneme feature sequence according to the preset phoneme codebook through the phoneme quantization network 1005.

[0159] Phoneme segmentation data is obtained by selecting the phoneme segmentation boundary of the target phoneme feature sequence through the nearest neighbor clustering network 1006.

[0160] The loss calculation module 1007 is used to calculate the loss of phoneme reference data and phoneme segmentation data through a preset loss model to obtain the segmentation loss value.

[0161] The parameter adjustment module 1008 is used to adjust the parameters of the phoneme segmentation model based on the segmentation loss value.

[0162] The specific implementation of this model training device is basically the same as the specific implementation of the above-mentioned model training method, and will not be described again here.

[0163] Please see Figure 11 This application also provides a phoneme segmentation apparatus that can implement the above-described phoneme segmentation method. The apparatus includes:

[0164] Target acquisition module 1101 is used to acquire target speech data;

[0165] The phoneme segmentation module 1102 is used to input the target speech data into the target phoneme segmentation model for phoneme segmentation processing to obtain target phoneme segmentation information; wherein, the target phoneme segmentation model is obtained by the model training method described above.

[0166] The specific implementation of this phoneme segmentation device is basically the same as the specific implementation of the phoneme segmentation method described above, and will not be repeated here.

[0167] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described model training method or the above-described phoneme segmentation method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0168] Please see Figure 12 , Figure 12 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0169] The processor 1201 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0170] The memory 1202 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1202 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1202 and is called and executed by the processor 1201 to execute the model training method or phoneme segmentation method of the embodiments of this application.

[0171] The input / output interface 1203 is used to implement information input and output;

[0172] The communication interface 1204 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0173] Bus 1205 transmits information between various components of the device (e.g., processor 1201, memory 1202, input / output interface 1203, and communication interface 1204);

[0174] The processor 1201, memory 1202, input / output interface 1203 and communication interface 1204 are connected to each other within the device via bus 1205.

[0175] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described model training method or the above-described phoneme segmentation method.

[0176] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0177] The model training method and apparatus, phoneme segmentation method, device, and storage medium provided in this application embodiment obtain original speech feature sequences by performing latent variable transformation on speech sample sequences through an encoder, and obtain phoneme reference data by masking the original speech feature sequences through a decoder. A phoneme quantization network quantizes the original speech feature sequences according to a preset phoneme codebook to obtain target phoneme feature sequences. A nearest neighbor clustering network performs phoneme boundary segmentation on the target phoneme feature sequences to obtain phoneme segmentation data. Then, a preset loss model calculates the loss between the phoneme reference data and the phoneme segmentation data to obtain a segmentation loss value. The parameters of the phoneme segmentation model are adjusted based on the segmentation loss value to continuously optimize the phoneme segmentation model and construct a more accurate phoneme segmentation model. Therefore, the phoneme codebook preset by the phoneme quantization network is used to represent the input original speech feature sequences as corresponding phonemes, and the nearest neighbor clustering network performs phoneme boundary segmentation to obtain more accurate phoneme segmentation data.

[0178] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0179] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0180] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0181] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0182] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0183] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0184] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0185] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0186] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0187] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0188] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A model training method, characterized in that, The method includes: Obtain speech sample sequences; The speech sample sequence is input into a preset phoneme segmentation model; wherein, the phoneme segmentation model includes: an encoder, a decoder, a phoneme quantization network, and a nearest neighbor clustering network; The encoder performs latent variable transformation on the speech sample sequence to obtain the original speech feature sequence; The original speech feature sequence is masked by the decoder to obtain phoneme reference data. The original speech feature sequence is quantized by the phoneme quantization network according to a preset phoneme codebook to obtain the target phoneme feature sequence. The target phoneme feature sequence is segmented by phoneme boundaries using the nearest neighbor clustering network to obtain phoneme segmentation data. The phoneme reference data and the phoneme segmentation data are subjected to loss calculation using a preset loss model to obtain the segmentation loss value; The parameters of the phoneme segmentation model are adjusted based on the segmentation loss value; The step of segmenting the target phoneme feature sequence by the nearest neighbor clustering network to obtain phoneme segmentation data includes: Anomaly detection is performed on the target phoneme feature sequence using the phoneme segmentation model of the nearest neighbor clustering network to obtain the original anomaly value; wherein, the anomaly detection is performed by calculating the distance between each phoneme feature vector in the target phoneme feature sequence to obtain the original anomaly value; The original outliers are processed by peak filtering using the phoneme segmentation model of the nearest neighbor clustering network to obtain the target outliers; The phoneme segmentation data is generated based on the target outlier.

2. The method according to claim 1, characterized in that, The step of quantizing the original speech feature sequence using the phoneme quantization network according to a preset phoneme codebook to obtain the target phoneme feature sequence includes: The feature vector of the original speech feature sequence is obtained by the phoneme quantization network, thus obtaining the speech feature vector; The phoneme entry vector is obtained by using the phoneme quantization network to obtain the entry vector of the phoneme codebook. The target distance is obtained by calculating the distance between the speech feature vector and the phoneme entry vector through the phoneme quantization network. The phoneme quantization network filters the phoneme entry vectors based on the target distance to obtain the target phoneme feature sequence.

3. The method according to claim 1, characterized in that, The target phoneme feature sequence includes at least two phoneme feature sub-sequences; the step of performing anomaly detection on the target phoneme feature sequence through the nearest neighbor clustering network to obtain the original anomaly value includes: The target feature subsequence is obtained by acquiring any one of the phoneme feature subsequences in the target phoneme feature sequence through the nearest neighbor clustering network; The original outlier value is obtained by calculating the minimum distance between the target feature subsequence and the phoneme feature subsequence using the nearest neighbor clustering network.

4. The method according to claim 1, characterized in that, The step of performing peak filtering on the original outliers using the nearest neighbor clustering network to obtain target outliers includes: The original outliers are filtered by the nearest neighbor clustering network according to a preset threshold to obtain candidate outliers; The nearest neighbor clustering network is used to obtain any one of the candidate outliers to obtain the selected outlier; Based on the selected outlier, obtain the previous candidate outlier to obtain the previous outlier; based on the selected outlier, obtain the next candidate outlier to obtain the next outlier. If the selected outlier is greater than the previous outlier and the next outlier, the selected outlier is taken as the target outlier.

5. The method according to any one of claims 1 to 4, characterized in that, The loss model includes a context network and a target loss function; the step of calculating the segmentation loss value by applying the preset loss model to the phoneme reference data and the phoneme segmentation data includes: Contextual features are extracted from the phoneme reference data using the context network to obtain candidate feature information; The context network is used to extract context features from the phoneme segmentation data to obtain target feature information; The candidate feature information is selected by selecting positive and negative samples using the target loss function to obtain reference feature information; The segmentation loss value is obtained by calculating the loss between the reference feature information and the target feature information using the target loss function.

6. A phoneme segmentation method, characterized in that, The method includes: Acquire target speech data; The target speech data is input into the target phoneme segmentation model for phoneme segmentation processing to obtain target phoneme segmentation information; wherein, the target phoneme segmentation model is obtained by the model training method according to any one of claims 1 to 5.

7. A model training device, characterized in that, The device includes: The data acquisition module is used to acquire speech sample sequences; An input module is used to input the speech sample sequence into a preset phoneme segmentation model; wherein, the phoneme segmentation model includes: an encoder, a decoder, a phoneme quantization network, and a nearest neighbor clustering network; The encoder performs latent variable transformation on the speech sample sequence to obtain the original speech feature sequence; The original speech feature sequence is masked by the decoder to obtain phoneme reference data. The original speech feature sequence is quantized by the phoneme quantization network according to a preset phoneme codebook to obtain the target phoneme feature sequence. The nearest neighbor clustering network is used to select the phoneme segmentation boundary of the target phoneme feature sequence to obtain phoneme segmentation data. The loss calculation module is used to perform loss calculation on the phoneme reference data and the phoneme segmentation data through a preset loss model to obtain the segmentation loss value; The parameter adjustment module is used to adjust the parameters of the phoneme segmentation model according to the segmentation loss value; The step of segmenting the target phoneme feature sequence by the nearest neighbor clustering network to obtain phoneme segmentation data includes: Anomaly detection is performed on the target phoneme feature sequence using the phoneme segmentation model of the nearest neighbor clustering network to obtain the original anomaly value; wherein, the anomaly detection is performed by calculating the distance between each phoneme feature vector in the target phoneme feature sequence to obtain the original anomaly value; The original outliers are processed by peak filtering using the phoneme segmentation model of the nearest neighbor clustering network to obtain the target outliers; The phoneme segmentation data is generated based on the target outlier.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the model training method according to any one of claims 1 to 5, or the phoneme segmentation method according to claim 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the model training method according to any one of claims 1 to 5, or the phoneme segmentation method according to claim 6.

Images