Model training method, speech processing method, device, equipment and medium

Abstract

The present disclosure provides a model training method, a speech processing method, an apparatus, a device and a medium, which relate to the field of artificial intelligence, and in particular to the field of speech technology. The implementation scheme is: determining multiple speech segments in sample speech data, wherein the sample speech data contains at least one transition point where a speaker change occurs, and each of the multiple speech segments has corresponding acoustic features and content features, as well as a first label for indicating whether the speech segment contains a transition point; inputting the acoustic features and content features corresponding to each of the multiple speech segments into a prediction module to obtain a first prediction result corresponding to each of the multiple speech segments; and adjusting the parameters of the speech processing model for any one of the multiple speech segments, at least based on the first label and the first prediction result of the speech segment.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the field of speech technology, and specifically to a training method for a speech processing model, a speech processing method, an apparatus, a model, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] Artificial intelligence is a discipline that studies how to use computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing. Artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0003] The methods described in this section are not necessarily methods that have been previously conceived or used. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention

[0004] The present disclosure provides a training method for a speech processing model, a speech processing method, an apparatus, a model, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] According to one aspect of the present disclosure, a training method for a speech processing model is provided, wherein the speech processing model includes at least a prediction module, and the method includes: determining multiple speech segments in sample speech data, wherein the sample speech data contains at least one transition point where a speaker change occurs, and each of the multiple speech segments has corresponding acoustic features and content features, and a first label for indicating whether the speech segment contains a transition point; inputting the acoustic features and content features corresponding to each of the multiple speech segments into the prediction module to obtain a first prediction result corresponding to each of the multiple speech segments; and adjusting the parameters of the speech processing model for any one of the multiple speech segments, at least based on the first label and the first prediction result of the speech segment.

[0006] According to another aspect of the present disclosure, a speech processing method is provided, comprising: determining a plurality of speech segments in speech data to be processed, wherein the speech data to be processed contains at least one transition point where a speaker change occurs, and each of the plurality of speech segments has a corresponding acoustic feature; and inputting the acoustic feature corresponding to each of the plurality of speech segments into a speech processing model to obtain a prediction result corresponding to each of the plurality of speech segments, wherein the prediction result corresponding to each speech segment can be used to indicate whether the speech segment contains a transition point, wherein the speech processing model is trained based on the above method.

[0007] According to another aspect of the present disclosure, a training device for a speech processing model is provided, wherein the speech processing model includes at least a prediction module, and the device includes: a first determination unit, configured to determine multiple speech segments in sample speech data, wherein the sample speech data contains at least one transition point where a speaker change occurs, and each of the multiple speech segments has corresponding acoustic features and content features, and a first label for indicating whether the speech segment contains a transition point; a first acquisition unit, configured to input the acoustic features and content features corresponding to each of the multiple speech segments into the prediction module to obtain a first prediction result corresponding to each of the multiple speech segments; and an adjustment unit, configured to adjust the parameters of the speech processing model for any one of the multiple speech segments, at least based on the first label and the first prediction result of the speech segment.

[0008] According to another aspect of the present disclosure, a speech processing model is provided, wherein the speech processing model is trained based on the above-mentioned training method.

[0009] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of the above methods.

[0010] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute any one of the above methods.

[0011] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program implements any one of the above methods when executed by a processor.

[0012] According to one or more embodiments of the present disclosure, the speech processing model can obtain richer feature information during the training process, so that the prediction accuracy of the trained speech processing model is improved.

[0013] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0015] Figure 1 A schematic diagram showing monophonic speech data is shown;

[0016] Figure 2 A schematic diagram showing an exemplary system in which the various methods described herein may be implemented according to an embodiment of the present disclosure;

[0017] Figure 3 A flow chart of a method for training a speech processing model according to an embodiment of the present disclosure is shown;

[0018] Figure 4A A schematic diagram showing word position information according to an embodiment of the present disclosure;

[0019] Figure 4B A schematic diagram showing word boundary information according to an embodiment of the present disclosure is shown;

[0020] Figure 5 A schematic diagram of a fusion submodule according to an embodiment of the present disclosure is shown;

[0021] Figure 6 A schematic diagram showing a method for training a speech processing model according to an embodiment of the present disclosure is shown;

[0022] Figure 7 A flow chart of a speech processing method according to an embodiment of the present disclosure is shown;

[0023] Figure 8 A structural block diagram of a training device for a speech processing model according to an embodiment of the present disclosure is shown; and

[0024] Fig. 9 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0025] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0026] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, temporal relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

[0027] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations.

[0028] In scenarios such as intelligent customer service, conference discussions, interviews, and variety shows, there are often multiple speakers on a single channel. Figure 1 FIG. 1 shows a schematic diagram of monophonic speech data. Figure 1 As shown, the speech data 100 includes speech data 110 from a first speaker, speech data 120 from a second speaker, and speech data 130 from a third speaker in sequence. In order to be able to process speech data of different speakers in a targeted manner, it is necessary to train a reliable speech processing model to perform human voice separation processing on the speech data, so that each segment of speech data separated from the speech data contains only speech data of a single speaker.

[0029] In the related art, the training of the above-mentioned speech processing model depends on the acoustic features of the speech, which results in the model learning relatively single information during the training process and makes it difficult to improve the prediction accuracy.

[0030] Based on this, the present disclosure proposes a method for training a speech processing model. For multiple speech segments in sample speech data, the speech processing model is trained based on both the acoustic features and content features of each speech segment, so that the speech processing model can obtain richer feature information from two dimensions, namely the sound dimension and the content dimension, during the training process. For the acoustic features and content features corresponding to each speech segment, the speech processing model can deepen the analysis and understanding capabilities of the corresponding acoustic features during the training process based on the reference provided by the content features. In practical applications, even if content information is not input into the trained speech processing model, the speech processing model can extract effective information for human voice separation from the acoustic features of the speech by relying on the analysis and understanding capabilities learned during training, thereby improving the prediction accuracy.

[0031] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0032] Figure 2 FIG. 2 is a schematic diagram of an exemplary system 200 in which various methods and apparatuses described herein may be implemented according to an embodiment of the present disclosure. Figure 2 , the system 200 includes one or more client devices 201, 202, 203, 204, 205, and 206, a server 220, and one or more communication networks 210 coupling the one or more client devices to the server 220. The client devices 201, 202, 203, 204, 205, and 206 may be configured to execute one or more applications.

[0033] In an embodiment of the present disclosure, the server 220 may run one or more services or software applications that enable execution of a training method of a speech processing model or a speech processing method.

[0034] In some embodiments, server 220 may also provide other services or software applications that may include non-virtualized environments and virtualized environments. In some embodiments, these services may be provided as web-based services or cloud services, such as provided to users of client devices 201, 202, 203, 204, 205, and / or 206 under a software as a service (SaaS) model.

[0035] exist Figure 2In the configuration shown, the server 220 may include one or more components that implement the functions performed by the server 220. These components may include software components, hardware components, or a combination thereof that can be executed by one or more processors. Users operating client devices 201, 202, 203, 204, 205, and / or 206 may in turn utilize one or more client applications to interact with the server 220 to utilize the services provided by these components. It should be understood that a variety of different system configurations are possible, which may differ from the system 200. Therefore, Figure 2 is one example of a system for implementing the various methods described herein and is not intended to be limiting.

[0036] The user may use client devices 201, 202, 203, 204, 205 and / or 206 to obtain or transmit voice data to be processed. The client device may provide an interface that enables the user of the client device to interact with the client device. The client device may also output information to the user via the interface. Figure 2 Only six client devices are depicted, but one skilled in the art will appreciate that the present disclosure may support any number of client devices.

[0037] Client devices 201, 202, 203, 204, 205 and / or 206 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptop computers), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, game systems, thin clients, various messaging devices, sensors or other sensing devices, etc. These computer devices may run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, Linux or Linux-like operating systems; or include various mobile operating systems, such as MICROSOFT Windows Mobile OS, iOS, Windows Phone, Android. Portable handheld devices may include cellular phones, smart phones, tablet computers, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Game systems may include various handheld game devices, Internet-enabled game devices, etc. Client devices are capable of executing various different applications, such as various Internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and may use various communication protocols.

[0038] The network 210 may be any type of network known to those skilled in the art that may support data communications using any of a variety of available protocols, including but not limited to TCP / IP, SNA, IPX, etc. By way of example only, the one or more networks 210 may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0039] Server 220 may include one or more general purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 220 may include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that may be virtualized to maintain a server's virtual storage device). In various embodiments, server 220 may run one or more services or software applications that provide the functionality described below.

[0040] The computing units in the server 220 may run one or more operating systems including any of the above operating systems and any commercially available server operating systems. The server 220 may also run any of a variety of additional server applications and / or middle-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0041] In some implementations, server 220 may include one or more applications to analyze and consolidate data feeds and / or event updates received from users of client devices 201, 202, 203, 204, 205, and / or 206. Server 220 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client devices 201, 202, 203, 204, 205, and / or 206.

[0042] In some embodiments, server 220 may be a server of a distributed system, or a server combined with a blockchain. Server 220 may also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in a cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services.

[0043] The system 200 may also include one or more databases 230. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 230 may be used to store information such as voice files and video files. The databases 230 may reside in various locations. For example, the database used by the server 220 may be local to the server 220, or may be remote from the server 220 and may communicate with the server 220 via a network-based or dedicated connection. The databases 230 may be of different types. In some embodiments, the databases used by the server 220 may be, for example, relational databases. One or more of these databases may store, update, and retrieve data from the databases in response to commands.

[0044] In some embodiments, one or more of the databases 230 may also be used by an application to store application data. The databases used by the application may be different types of databases, such as a key-value store, an object store, or a conventional store backed by a file system.

[0045] Figure 2 The system 200 may be configured and operated in various ways to enable the application of various methods and apparatuses described in the present disclosure.

[0046] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0047] Figure 3 A flowchart of a method for training a speech processing model according to an exemplary embodiment of the present disclosure is shown, wherein the speech processing model includes at least a prediction module, and the method 300 includes: step S301, determining multiple speech segments in sample speech data, wherein the sample speech data includes at least one transition point where a speaker change occurs, and each of the multiple speech segments has corresponding acoustic features and content features, and a first label for indicating whether the speech segment contains a transition point; step S302, inputting the acoustic features and content features corresponding to each of the multiple speech segments into the prediction module to obtain a first prediction result corresponding to each of the multiple speech segments; and step S303, for any one of the multiple speech segments, adjusting the parameters of the speech processing model based at least on the first label and the first prediction result of the speech segment.

[0048] Therefore, during the training process of the speech processing model, the speech processing model can obtain richer feature information from two dimensions, namely the sound dimension and the content dimension. For the acoustic features and content features corresponding to each speech segment, the speech processing model can deepen its analysis and understanding capabilities of the corresponding acoustic features during the training process with the help of the reference provided by the content features. In practical applications, even if content information is not input into the trained speech processing model, the speech processing model can rely on this analysis and understanding capability to extract effective information for human voice separation from the acoustic features of the speech, thereby improving the prediction accuracy.

[0049] Regarding step S301, the transition point where the speaker change occurs in the sample voice data can be a transition point from the voice data of one speaker to the voice data of another speaker, or a transition point from a silent segment to the voice data of a certain speaker, or a transition point from the voice data of a certain speaker to a silent segment. Figure 1 For example, Figure 1 Each of the time points 101 to 105 in the illustrated speech data may be used as a transition point at which a speaker change occurs as described in the present disclosure.

[0050] According to some embodiments, the acoustic features corresponding to each of the multiple speech segments can be determined based on at least one of the following: Mel-frequency cepstral coefficients (MFCC) of the speech segment, perceptual linear prediction coefficients (PLP) of the speech segment, and filter bank coefficients (Fbank). In particular, the acoustic features of the speech segment can be represented by combining the above-mentioned different types of coefficients.

[0051] According to some embodiments, the content feature corresponding to each of the plurality of speech segments is determined based at least on the phoneme corresponding to the speech segment.

[0052] Phonemes, for example, can be Chinese vowels and vowels, or English phonetic symbols. Based on the phonemes corresponding to different speech segments, different speech segments can be distinguished in terms of speech content.

[0053] According to some embodiments, a pre-trained speech recognition feature extraction model may be used to determine the phonemes corresponding to each speech segment.

[0054] Exemplarily, the training of the speech recognition feature extraction model can be performed through the following steps: determining a plurality of speech segments in the initial speech data for training, each speech segment having a corresponding phoneme label; for each of the plurality of speech segments, extracting the acoustic features corresponding to the speech segment, for example, one or more of MFCC, PLP, and Fbank; inputting the acoustic features of each of the plurality of speech segments into the speech recognition feature extraction model to be trained to obtain the predicted phoneme corresponding to each of the plurality of speech segments; and adjusting the parameters of the speech recognition feature extraction model based on the phoneme label and the predicted phoneme of any one of the plurality of speech segments.

[0055] According to some embodiments, the content features of each of the plurality of speech segments include one or more of phoneme information, word position information, and word boundary information, and wherein the word position information includes information for indicating the pronunciation position of the phoneme corresponding to the speech segment in the word, and the word boundary information includes information for indicating the positional relationship between the phoneme corresponding to the speech segment and the pronunciation edge of the word.

[0056] Herein, the "word" can be understood as a Chinese character or an English word.

[0057] In one implementation, the word position information can be inferred based on the phonemes respectively corresponding to a plurality of consecutive speech segments. Exemplarily, the word position information can be classified into 5 categories, namely, silent segment, monosyllabic word, word start, word middle, and word end.

[0058] Figure 4A FIG. shows a schematic diagram of the word position information according to an example of the present disclosure. As Figure 4A shown, in the speech data 410 containing "I love my home", the word position information corresponding to the speech segment in the speech data 411 is a silent segment; the word position information corresponding to the speech segment in the speech data 414 corresponding to the monosyllabic word "love" is a monosyllabic word; the word position information corresponding to the speech segment in the speech data 412 corresponding to the initial consonant "w" of the disyllabic word "I" is the word start, and the word position information corresponding to the speech segment in the speech data 413 corresponding to the final vowel "o" is the word end; the word position information corresponding to the speech segment in the speech data 415 corresponding to the initial consonant "j" of the trisyllabic word "home" is the word start, the word position information corresponding to the speech segment in the speech data 416 corresponding to the medial vowel "i" is the word middle, and the word position information corresponding to the speech segment in the speech data 417 corresponding to the final vowel "a" is the word end.

[0059] It can be understood that in Chinese, the above five types of word position information can be used to represent all possible pronunciation position information of the phonemes of a speech segment in a word. However, in English, a word may be composed of more than three phonetic symbols. Accordingly, the information of the pronunciation position of the phonemes corresponding to the speech segment in the word can be indicated by adding categories of word position information.

[0060] In one embodiment, word boundary information can be inferred based on the phonemes corresponding to multiple consecutive speech segments. For example, word boundary information can be divided into five categories, namely, silence segment, intra-word segment, silence-speech segment, speech-silence segment, and inter-word segment.

[0061] Figure 4B FIG. 4 shows a schematic diagram of exemplary word boundary information according to the present disclosure. Figure 4B As shown, in the speech data 420 containing "Howare you", the word boundary information corresponding to the speech segment in the speech data 422 is a silent segment, the word boundary information corresponding to the speech segment in the speech data 425 is an intra-word segment, the word boundary information corresponding to the speech segment in the speech data 423 is a silent-speech segment, the word boundary information corresponding to the speech segment in the speech data 421 is a speech-silent segment, and the word boundary information corresponding to the speech segment in the speech data 424 is an inter-word segment.

[0062] The phoneme information, word position information and word boundary information obtained based on the phonemes of the speech segment can provide an effective reference for voice separation, so that the speech processing model can refer to these content features to deepen the understanding of the corresponding acoustic features during the training process.

[0063] For example, if the phoneme information corresponding to each of multiple consecutive speech segments in the speech data is the same, then the speech in these speech segments is likely to come from the same speaker; for example, if the word position information of the speech segment is the beginning of a word, or the word boundary information of the speech segment is an inter-word segment, then the probability of containing a transition point in the speech segment is relatively small; for example, if the word boundary information of the speech segment is a silence-speech segment, then the probability of containing a transition point in the speech segment is relatively large.

[0064] According to some embodiments, when the content features of a speech segment include at least two types of information, the at least two types of information are jointly represented to facilitate input into a speech processing model during training and application.

[0065] In one embodiment, each of the at least two types of information corresponding to the speech segment can be represented separately, and then the representation results of each of the at least two types of information can be connected as the content feature of the speech segment. For example, there are 48 optional categories for phoneme information, 5 optional categories for word position information, and 5 optional categories for word boundary information. The phoneme information is represented independently to obtain a 48-dimensional vector, wherein the position corresponding to the phoneme information of the speech segment in the vector is set to 1, and the other positions of the vector are set to 0; the word position information is represented independently to obtain a 5-dimensional vector, wherein the position corresponding to the word position information of the speech segment in the vector is set to 1, and the other positions of the vector are set to 0; the word boundary information is represented independently to obtain a 5-dimensional vector, wherein the position corresponding to the word boundary information of the speech segment in the vector is set to 1, and the other positions of the vector are set to 0. Finally, the 48-dimensional vector corresponding to the above-mentioned phoneme information, the 5-dimensional vector corresponding to the word position information, and the 5-dimensional vector corresponding to the word boundary information are spliced, and the splicing result is used as the content feature of the speech segment.

[0066] In another embodiment, the categories corresponding to each of the at least two types of information corresponding to the speech segment can be taken as a whole to characterize the whole. Still taking the example that the phoneme information has 48 optional categories, the word position information has 5 optional categories, and the word boundary information has 5 optional categories, a 48*5*5 dimensional vector is set, and one position of the phoneme information, word position information, and word boundary information corresponding to the speech segment in the 48*5*5 dimensional vector is set to 1, and the other positions of the vector are all set to 0.

[0067] The “multiple speech segments” used for inputting into the prediction module may be determined by directly extracting from sample speech data, or by performing downsampling on a plurality of speech frames.

[0068] According to some embodiments, the speech processing model may further include an acoustic feature extraction module, and the method may further include: before determining a plurality of speech segments in the sample speech data, extracting a plurality of speech frames from the sample speech data, wherein each of the plurality of speech frames has a corresponding acoustic sub-feature; and inputting the acoustic sub-feature corresponding to each of the plurality of speech frames into the acoustic feature extraction module to obtain an acoustic feature corresponding to each of the plurality of speech segments, wherein the acoustic feature of each speech segment is determined based on a convolution calculation of the acoustic feature extraction module on the plurality of acoustic sub-features corresponding to at least some of the plurality of speech frames.

[0069] Among them, for the acoustic features of each speech segment, multiple acoustic sub-features used to calculate the acoustic features of the speech segment are determined based on the size of the convolution kernel of the convolution calculation and the number of convolution calculations.

[0070] It can be understood that the time range covered by the speech frame in the sample speech data is smaller than the time range covered by the speech segment in the sample speech data. At the same time, the number of multiple speech segments input into the prediction module can be smaller than the number of multiple speech frames extracted from the sample speech data.

[0071] Due to the nature of the convolution calculation itself, compared with the input data of the convolution calculation, the output data of the convolution calculation not only has a smaller data volume, but also has a stronger representation ability. In this way, the acoustic features of the speech segment determined based on the convolution calculation of multiple acoustic sub-features not only cover the information in the multiple acoustic sub-features corresponding to at least part of the speech frames, but also have a stronger representation ability, so that without losing the information extracted from the sample speech data, the amount of data input to the prediction module can be reduced, computing resources can be saved, and computing efficiency can be improved.

[0072] According to some embodiments, the multiple speech frames in the sample speech data may be multiple speech frames sequentially connected in the time domain. In particular, the sample speech data may be framed using a preset time window and a preset step size to obtain multiple speech frames sequentially connected in the time domain.

[0073] According to some embodiments, before the acoustic sub-features corresponding to each of the multiple speech frames are input into the acoustic feature extraction module, the acoustic sub-features corresponding to each of the multiple speech frames are enhanced to increase the difficulty of training the speech processing model and enhance the robustness of the trained speech processing model.

[0074] In order to match the multiple acoustic features determined by performing convolution downsampling using the acoustic feature extraction module, the multiple content sub-features corresponding to the multiple speech frames can be downsampled so that the number of content features and the number of acoustic features input to the prediction model are equal.

[0075] According to some embodiments, each of the multiple speech frames may also have a corresponding content sub-feature, and the method may further include: determining the initial content feature corresponding to each of the multiple speech segments based on the content sub-feature corresponding to each of the multiple speech frames, wherein the initial content feature of each speech segment is one of the multiple content sub-features respectively corresponding to the multiple speech frames in the speech segment; and inputting the initial content feature of each of the multiple speech segments into a content feature extraction module for assisting speech processing model training to obtain the content feature corresponding to each of the multiple speech segments, wherein the content feature of each speech segment is determined by the fusion calculation of the initial content features of each of the multiple speech segments by the content feature extraction module.

[0076] Therefore, the number of content features input into the prediction model can be equal to the number of acoustic features. The acoustic features and content features corresponding to the same speech segment form a combination, which jointly characterize the characteristics of a speech segment in two different dimensions, namely, the sound dimension and the content dimension. This allows the prediction module of the speech processing model to refer to the content features of the speech segment to enhance its understanding and analysis capabilities of the acoustic features of the speech segment during the training process.

[0077] According to some embodiments, the content feature extraction module can use a self-attention mechanism to implement the fusion calculation of the initial content features of each of the multiple speech segments. In particular, the self-attention mechanism in the content feature extraction module can be implemented based on a transformer network.

[0078] It can be understood that the content feature extraction module does not belong to the speech processing model and can only be used to assist the training of the speech processing model during the training process. However, during the training iteration, the parameters in the content feature extraction module can be adjusted together with the parameters in the speech processing model. Therefore, as the number of training iterations increases, the auxiliary ability of the content feature extraction module for training can be gradually improved, so that the training effect of each round of training is continuously improved.

[0079] Regarding step S202, according to some embodiments, the prediction module may include a fusion submodule and a prediction submodule, and wherein, inputting the acoustic features and content features corresponding to each of the multiple speech segments into the prediction module to obtain a first prediction result corresponding to each of the multiple speech segments may include: inputting the acoustic features and content features corresponding to each of the multiple speech segments into the fusion submodule to obtain a first fusion feature of each of the multiple speech segments; for each of the multiple speech segments, determining a second fusion feature for inputting into the prediction submodule at least based on the first fusion feature of the speech segment; and inputting the second fusion feature of each of the multiple speech segments into the prediction submodule to obtain a first prediction result corresponding to each of the multiple speech segments.

[0080] Therefore, by using the fusion submodule in the prediction module, the input acoustic features and content features can be fused with each other, so that during the training process, the speech processing model can make full use of the content features to improve the prediction ability.

[0081] According to some embodiments, the fusion submodule may utilize a self-attention mechanism to achieve fusion of acoustic features and content features. In particular, the self-attention mechanism in the fusion submodule may be implemented based on a convolution-enhanced attention sequence network, such as a Conformer network.

[0082] In one embodiment, the acoustic features and content features corresponding to each of the multiple speech segments can be input into the fusion submodule for sorting after being spliced. In this way, the fusion of the acoustic features and content features of the speech segments performed by the fusion submodule can be expressed by the following formula:

[0083] X Concat =Concatenate(X Acoustic , X ASR )

[0084] E = ConformerEncoder(X Concat )

[0085] Among them, X Acoustic , and X ASR They respectively represent the acoustic features and content features corresponding to each speech segment, and the dimensions of the two are the same.

[0086] The acoustic feature X corresponding to each speech segment is concatenated by the Concatenate function Acoustic and content feature X ASR Perform splicing to obtain the splicing result X of the two Concat . The splicing result X Concat The convolution-enhanced attention sequence model ConformerEncoder is input, and the first fusion feature E of each of the multiple speech segments can be obtained through multiple nonlinear transformations inside ConformerEncoder.

[0087] Figure 5 A schematic diagram of a fusion submodule according to an exemplary embodiment of the present disclosure is shown, and the fusion submodule 500 may include multiple layers, wherein each layer is composed of a first feed-forward network (Feed-forward) 510, a context-based self-attention module (Contextualized self-attention) 520, a convolution module (Convolution) 530, a second feed-forward network (Feed-forward) 540, and a normalization module (layernorm) 550 connected in sequence. Now, the processing process of one layer in the fusion submodule is taken as an example for description.

[0088] like Figure 5As shown, the input data 501 of the layer is determined, wherein the input data 501 includes content features and acoustic features corresponding to each of the multiple speech segments. If the layer is the first layer in the fusion submodule, the content features and acoustic features in the input data 501 are the content features and acoustic features of the input fusion submodule respectively; if the layer is not the first layer in the fusion submodule, the content features in the input data 501 are the content features of the input fusion submodule, and the acoustic features are the updated acoustic features output by the previous layer.

[0089] The input data 501 is input into the forward network 510, which performs matrix calculation on the content features and acoustic features in the input data 501 to obtain high-level content features 502 and high-level acoustic features 503 for each of the multiple speech segments, and inputs the high-level content features 502 and high-level acoustic features 503 into the content context-based self-attention module 520.

[0090] The content context-based self-attention module 520 includes a splicing module 521, a calculation module 522 and an acoustic feature fusion module 523.

[0091] In the content-context-based self-attention module 520, for each of the multiple speech segments, the high-level content features 502 and high-level acoustic features 503 of the speech segment are both input into the splicing module 521, and the splicing results of the high-level content features 502 and high-level acoustic features 503 are used as the query vector Q and keyword vector K corresponding to the speech segment, that is, the query vector Q and keyword vector K used for performing self-attention calculation for each speech segment contain information in two dimensions, content and acoustics. At the same time, the high-level acoustic features 503 of the speech segment are used as the value vector V corresponding to the speech segment, that is, the value vector V used for performing self-attention calculation for each speech segment only contains information in the acoustic dimension.

[0092] The query vector Q and the keyword vector K are input to the calculation module 522, and the calculation module 522 performs matrix calculation, coefficient multiplication and normalization calculation on the query vector Q and the keyword vector K to obtain a similarity matrix. The similarity matrix includes the calculated similarity values ​​between each two speech segments in the multiple speech segments.

[0093] The similarity matrix and the value vector V corresponding to the speech segment are input into the acoustic feature fusion module 523. In the acoustic feature fusion module 523, for each of the multiple speech segments, the weighted sum of the value vectors V corresponding to each speech segment is calculated to obtain the fused acoustic feature corresponding to the speech segment, wherein the weight corresponding to each speech segment is determined according to the similarity value between each speech segment and the speech segment in the similarity matrix. The fused acoustic feature of each of the multiple speech segments constitutes the output data 504.

[0094] After the output data 504 is processed by the convolution calculation of the convolution module 530, the matrix calculation of the second forward network 540, and the normalization module 550, the updated acoustic features 505 corresponding to each of the multiple speech segments are finally obtained. After multiple layers of iterative calculations in the fusion submodule, the first fusion feature of each of the multiple speech segments can be finally obtained.

[0095] Wherein, illustratively, ignoring the influence of other modules in the fusion submodule, the calculation performed by the content context-based self-attention module 520 can be expressed by the following formula:

[0096] V i =E i-1 (1≤i≤n; E 0 =X Acoustic )

[0097] K i =Q i =Concatenate(X ASR , E i-1 ) (1≤i≤n)

[0098] E i =ConformerEncoderLayer(Q i , K i , V i ) (1≤i≤n)

[0099] Where n represents the number of layers of the fusion submodule, E i represents the updated acoustic features output by the fusion submodule layer i, Q i represents the query vector Q, K of the i-th layer i Represents the keyword vector K, V of the i-th layer i Represents the value vector V of the i-th layer.

[0100] In the calculation of the i-th layer of the content-context based self-attention module 520, V i The updated acoustic features from the previous layer output, that is, E output from the i-1th layer i-1 , initial value E 0 Equal to the acoustic feature X of the input fusion submodule Acoustic . K i and Q i The values ​​are equal to the content feature X of the input fusion submodule through the Concatenate function ASR and E of the output of layer i-1 i-1 The result of splicing is Figure 5The ConformerEncoderLayer function represents the encoding layer based on the Conformer network, by performing the query vector Q i , keyword vector K i Sum value vector V i Perform self-attention calculation to obtain the updated acoustic feature E of the i-th layer output i ,Right now Figure 5 The processing performed by both the calculation module 522 and the acoustic feature fusion module 523.

[0101] According to some embodiments, for each of a plurality of speech segments, determining a second fusion feature for inputting into a prediction submodule based at least on a first fusion feature of the speech segment may include: for each of a plurality of speech segments, concatenating the first fusion feature of the speech segment and a content feature of the speech segment to determine a second fusion feature.

[0102] Compared with the rich information in acoustic features, the amount of information in content features is relatively small. In order to prevent the content features from being overwhelmed in the processing of the fusion sub-module, resulting in the prediction sub-module being unable to obtain relevant information in the content features, the first fusion features and content features output by the fusion sub-module are spliced, so that the prediction sub-module can obtain information on the content dimension during the training process, thereby improving the accuracy of training.

[0103] The first fusion feature and the content feature are concatenated, and the concatenated result is input into the prediction submodule for further calculation. This process can be expressed by the following formula:

[0104] E concat =Concatenate(E, X ASR )

[0105] Z=σ(Linear(E concat ))

[0106] Among them, E represents the first fusion feature, X ASR represents content features, and σ represents the sigmoid function.

[0107] Through the Concatenate function, the first fusion feature E and the content feature X ASR Splicing is performed to obtain the second fusion feature E for input prediction submodule concat After the matrix linear transformation performed by the Linear function and the sigmod function on the second fusion feature E concat The processing, that is, the processing performed by the prediction submodule, can obtain a first prediction result Z corresponding to each of the multiple speech segments.

[0108] Regarding step S203, according to some embodiments, each of the multiple speech segments may also have a second label for representing the phoneme corresponding to the speech segment, and wherein, based at least on the first label and the first prediction result of the speech segment, adjusting the parameters of the speech processing model may include: inputting the second fusion feature of each of the multiple speech segments into an additional prediction module for assisting speech processing model training to obtain a second prediction result corresponding to each of the multiple speech segments; and adjusting the parameters of the speech processing model for any one of the multiple speech segments based on the first label and the first prediction result corresponding to the speech segment, and the second label and the second prediction result corresponding to the speech segment.

[0109] In the training process, the prediction branch constructed by the additional prediction module is introduced. Based on the constraint effect of the branch in the training, the role of content features in various parts of the model can be strengthened during the training process. Even if the amount of information of the content features is small, it will not be lost during the training process, thereby improving the training effect of the speech processing model.

[0110] Figure 6 A schematic diagram of a method for training a speech processing model according to an exemplary embodiment of the present disclosure is shown.

[0111] like Figure 6 As shown, firstly, a plurality of speech frames are extracted from the sample speech data, and the content sub-features of each of the plurality of speech frames constitute the input data 601 , and the acoustic sub-features of each of the plurality of speech frames constitute the input data 602 .

[0112] The input data 601 is input into the sampling and downsampling module 611, so that the sampling and downsampling module 611 can extract a certain proportion of content sub-features from the input data 601. The extracted content sub-features are input into the content feature extraction module 612 to obtain content features 603 corresponding to each of the multiple speech segments, wherein the content feature extraction module 612 may include a transformer network.

[0113] The input data 602 is input into the enhancement processing module 621 to obtain the acoustic sub-features after enhancement processing. The acoustic sub-features after enhancement processing are input into the acoustic feature extraction module 622 to obtain the acoustic features 604 corresponding to each of the multiple speech segments.

[0114] The content features 603 of the multiple speech segments and the acoustic features 604 of the multiple speech segments are input into the fusion submodule 630 to obtain the first fusion feature 605 of each of the multiple speech segments. The first fusion features 605 of the multiple speech segments and the content features 603 of the multiple speech segments are concatenated, and the concatenated results are respectively input into the prediction submodule 641 and the additional prediction module 642.

[0115] The prediction submodule 641 uses the Sigmod nonlinear activation function to calculate a first prediction result 606 for each of the multiple speech segments. The first prediction result of each speech segment can be used to predict whether the speech segment contains a transition point of speaker change. Based on the difference between the first prediction result and the first label indicating whether the speech segment has a transition point, the parameters in the speech processing model can be adjusted inversely.

[0116] The additional prediction module 642 uses the Softmax activation function to calculate the second prediction result 607 of each of the multiple speech segments, and the second prediction result of each speech segment can be used to predict the phoneme corresponding to the speech segment. Based on the difference between the second prediction result and the second label indicating the phoneme corresponding to the speech segment, the parameters in the speech processing model can be adjusted inversely.

[0117] Figure 7 A flowchart of a speech processing method according to an exemplary embodiment of the present disclosure is shown, the method 700 comprising: step S701, determining a plurality of speech segments in speech data to be processed, wherein the speech data to be processed contains at least one transition point where a speaker change occurs, and each of the plurality of speech segments has a corresponding acoustic feature; and step S702, inputting the acoustic feature corresponding to each of the plurality of speech segments into a speech processing model to obtain a prediction result corresponding to each of the plurality of speech segments, wherein the prediction result corresponding to each speech segment can be used to indicate whether the speech segment contains a transition point, wherein the speech processing model is trained based on any of the above methods.

[0118] Since the speech processing model has learned how to use acoustic features to characterize the speaker's characteristics by referring to the content features during the training process of the speech processing model, during the testing and application phase, only the acoustic features need to be input to obtain the ideal prediction effect.

[0119] It can be understood that, in the application process, the acoustic features corresponding to each of the multiple speech segments can be input into the prediction module in the speech processing model to perform speech processing. Among them, for the speech processing model including the acoustic feature extraction module, the acoustic features corresponding to each of the multiple speech segments can also be input into the acoustic feature extraction module in the speech processing model to perform speech processing.

[0120] Figure 8 A training device for a speech processing model according to an exemplary embodiment of the present disclosure is shown, wherein the speech processing model includes at least a prediction module, and the device 800 includes: a first determination unit 801, configured to determine multiple speech segments in sample speech data, wherein the sample speech data contains at least one transition point where a speaker change occurs, and each of the multiple speech segments has corresponding acoustic features and content features, and a first label for indicating whether the speech segment contains a transition point; a first acquisition unit 802, configured to input the acoustic features and content features corresponding to each of the multiple speech segments into the prediction module to obtain a first prediction result corresponding to each of the multiple speech segments; and an adjustment unit 803, configured to adjust the parameters of the speech processing model for any one of the multiple speech segments, at least based on the first label and the first prediction result of the speech segment.

[0121] According to some embodiments, the acoustic feature corresponding to each of the plurality of speech segments is determined based on at least one of: a Mel-cepstral coefficient of the speech segment; a perceptual linear prediction coefficient of the speech segment; and a filter bank coefficient.

[0122] According to some embodiments, the content feature corresponding to each of the plurality of speech segments is determined based at least on the phoneme corresponding to the speech segment.

[0123] According to some embodiments, the content features of each of the multiple speech segments include one or more of phoneme information, word position information and word boundary information, and the word position information includes information for indicating the pronunciation position of the phoneme corresponding to the speech segment in the word, and the word boundary information includes information for indicating the positional relationship between the phoneme corresponding to the speech segment and the pronunciation edge of the word.

[0124] According to some embodiments, the prediction module includes a fusion submodule and a prediction submodule, and wherein the first acquisition unit includes: a first acquisition subunit, configured to input the acoustic features and content features corresponding to each of the multiple speech segments into the fusion submodule to obtain a first fusion feature of each of the multiple speech segments; a determination subunit, configured to determine, for each of the multiple speech segments, at least based on the first fusion feature of the speech segment, a second fusion feature for inputting into the prediction submodule; and a second acquisition subunit, configured to input the second fusion feature of each of the multiple speech segments into the prediction submodule to obtain a first prediction result corresponding to each of the multiple speech segments.

[0125] According to some embodiments, determining the sub-unit includes: for each of the plurality of speech segments, concatenating the first fused feature of the speech segment and the content feature of the speech segment to determine the sub-unit of the second fused feature.

[0126] According to some embodiments, each of the multiple speech segments also has a second label for representing the phoneme corresponding to the speech segment, and wherein the adjustment unit includes: a third acquisition subunit, configured to input the second fusion feature of each of the multiple speech segments into an additional prediction module for assisting speech processing model training to obtain a second prediction result corresponding to each of the multiple speech segments; and an adjustment subunit, configured to adjust the parameters of the speech processing model for any one of the multiple speech segments based on the first label and the first prediction result corresponding to the speech segment, and the second label and the second prediction result corresponding to the speech segment.

[0127] According to some embodiments, the speech processing model also includes an acoustic feature extraction module, and the device also includes: an extraction unit, configured to extract multiple speech frames from the sample speech data before determining multiple speech segments in the sample speech data, wherein each of the multiple speech frames has a corresponding acoustic sub-feature; and a second acquisition unit, configured to input the acoustic sub-feature corresponding to each of the multiple speech frames into the acoustic feature extraction module to obtain the acoustic feature corresponding to each of the multiple speech segments, wherein the acoustic feature of each speech segment is determined based on the convolution calculation of the acoustic feature extraction module on the multiple acoustic sub-features corresponding to at least some of the multiple speech frames.

[0128] According to some embodiments, each of the multiple speech frames also has a corresponding content sub-feature, and the device also includes: a second determination unit, configured to determine the initial content feature corresponding to each of the multiple speech segments based on the content sub-feature corresponding to each of the multiple speech frames, wherein the initial content feature of each speech segment is one of the multiple content sub-features respectively corresponding to the multiple speech frames in the speech segment; and a third acquisition unit, configured to input the initial content feature of each of the multiple speech segments into a content feature extraction module for assisting speech processing model training to obtain the content feature corresponding to each of the multiple speech segments, wherein the content feature of each speech segment is determined by the content feature extraction module through a fusion calculation of the initial content features of each of the multiple speech segments.

[0129] According to an embodiment of the present disclosure, a speech processing model is also provided, wherein the speech processing model is trained based on any one of the above-mentioned training methods.

[0130] According to an embodiment of the present disclosure, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of the above methods.

[0131] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is also provided, wherein the computer instructions are used to enable a computer to execute any one of the above methods.

[0132] According to an embodiment of the present disclosure, a computer program product is also provided, including a computer program, wherein the computer program implements any one of the above methods when executed by a processor.

[0133] refer to Fig. 9 , a block diagram of an electronic device 900 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0134] like Fig. 9 As shown, the electronic device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0135] Multiple components in the electronic device 900 are connected to the I / O interface 905, including: an input unit 906, an output unit 907, a storage unit 908, and a communication unit 909. The input unit 906 can be any type of device that can input information to the electronic device 900. The input unit 906 can receive input digital or character information and generate key signal input related to user settings and / or function control of the electronic device, and can include but is not limited to a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone, and / or a remote controller. The output unit 907 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / voice output terminal, a vibrator, and / or a printer. The storage unit 908 can include but is not limited to a disk, an optical disk. The communication unit 909 allows the electronic device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0136] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, for example, a training method or a speech processing method for a speech processing model. For example, in some embodiments, the training method or the speech processing method for a speech processing model may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the training method or the speech processing method for the speech processing model described above may be executed. Alternatively, in other embodiments, the computing unit 901 may be configured to execute the training method of the speech processing model or the speech processing method in any other suitable manner (for example, by means of firmware).

[0137] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0138] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0139] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0140] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0141] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0142] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0143] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0144] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples can be omitted or replaced by their equivalent elements. In addition, each step can be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples can be combined in various ways. It is important that with the evolution of technology, many elements described herein can be replaced by equivalent elements that appear after the present disclosure.

Claims

1. A method for training a speech processing model, wherein: The speech processing model includes at least a prediction module, and the method includes: Determine a plurality of speech segments in sample speech data, wherein the sample speech data contains at least one transition point where a speaker change occurs, each of the plurality of speech segments has a corresponding acoustic feature and content feature, and a first label for indicating whether the speech segment contains the transition point, and the content feature of each of the plurality of speech segments includes one or more of phoneme information, word position information, and word boundary information; Inputting the acoustic features and content features corresponding to each of the plurality of speech segments into the prediction module to obtain a first prediction result corresponding to each of the plurality of speech segments; and For any one of the plurality of speech segments, adjusting the parameters of the speech processing model based at least on the first label and the first prediction result of the speech segment, The prediction module includes a fusion submodule and a prediction submodule, and inputting the acoustic features and content features corresponding to each of the multiple speech segments into the prediction module includes: Inputting the acoustic features and content features corresponding to each of the plurality of speech segments into the fusion submodule to obtain a first fusion feature of each of the plurality of speech segments; For each of the plurality of speech segments, determining a second fused feature for inputting into the prediction submodule based at least on the first fused feature of the speech segment; and The second fusion feature of each of the multiple speech segments is input into the prediction submodule to obtain a first prediction result corresponding to each of the multiple speech segments.

2. The method according to claim 1, wherein: The acoustic feature corresponding to each of the plurality of speech segments is determined based on at least one of the following: Mel-frequency cepstral coefficient of the speech segment; The perceptual linear prediction coefficients for the speech segment; and Filter bank coefficients.

3. The method according to claim 1 or 2, wherein: The content feature corresponding to each of the plurality of speech segments is determined based at least on the phoneme corresponding to the speech segment.

4. The method according to claim 3, wherein: The word position information includes information for indicating the pronunciation position of the phoneme corresponding to the speech segment in the word, and the word boundary information includes information for indicating the positional relationship between the phoneme corresponding to the speech segment and the pronunciation edge of the word.

5. The method according to claim 1, wherein: The determining, for each of the plurality of speech segments, at least based on the first fused feature of the speech segment, a second fused feature for inputting into the prediction submodule comprises: For each of the multiple speech segments, the first fusion feature of the speech segment and the content feature of the speech segment are concatenated to determine the second fusion feature.

6. The method according to claim 5, wherein: Each of the plurality of speech segments also has a second label for indicating the phoneme corresponding to the speech segment. And wherein, adjusting the parameters of the speech processing model based at least on the first label and the first prediction result of the speech segment includes: Inputting the second fused feature of each of the plurality of speech segments into an additional prediction module for assisting the training of the speech processing model to obtain a second prediction result corresponding to each of the plurality of speech segments; and For any one of the multiple speech segments, the parameters of the speech processing model are adjusted based on the first label and the first prediction result corresponding to the speech segment, and the second label and the second prediction result corresponding to the speech segment.

7. The method according to claim 1, wherein: The speech processing model also includes an acoustic feature extraction module, and the method also includes: Before determining a plurality of speech segments in the sample speech data, extracting a plurality of speech frames from the sample speech data, wherein each of the plurality of speech frames has a corresponding acoustic sub-feature; and Inputting the acoustic sub-feature corresponding to each of the plurality of speech frames into the acoustic feature extraction module to obtain the acoustic feature corresponding to each of the plurality of speech segments, The acoustic features of each speech segment are determined based on the convolution calculation of multiple acoustic sub-features corresponding to at least some of the multiple speech frames by the acoustic feature extraction module.

8. The method according to claim 7, wherein: Each of the plurality of speech frames also has a corresponding content sub-feature, and the method further comprises: Determining an initial content feature corresponding to each of the plurality of speech segments based on the content sub-feature corresponding to each of the plurality of speech frames, wherein the initial content feature of each speech segment is one of the plurality of content sub-features respectively corresponding to the plurality of speech frames in the speech segment; and Inputting the initial content feature of each of the plurality of speech segments into a content feature extraction module for assisting the training of the speech processing model to obtain the content feature corresponding to each of the plurality of speech segments, The content feature of each speech segment is determined by the content feature extraction module through a fusion calculation of the initial content features of each of the multiple speech segments.

9. A speech processing method, comprising: Determining a plurality of speech segments in speech data to be processed, wherein the speech data to be processed includes at least one transition point where a speaker change occurs, and each of the plurality of speech segments has a corresponding acoustic feature; and Inputting the acoustic features corresponding to each of the plurality of speech segments into a speech processing model to obtain a prediction result corresponding to each of the plurality of speech segments, wherein the prediction result corresponding to each speech segment can be used to indicate whether the speech segment contains the transition point, Wherein, the speech processing model is trained based on the method described in any one of claims 1 to 8.

10. A training device for a speech processing model, wherein: The speech processing model includes at least a prediction module, and the device includes: a first determining unit configured to determine a plurality of speech segments in the sample speech data, wherein the sample speech data includes at least one transition point where a speaker change occurs, each of the plurality of speech segments has a corresponding acoustic feature and content feature, and a first label for indicating whether the speech segment contains the transition point, and the content feature of each of the plurality of speech segments includes one or more of phoneme information, word position information, and word boundary information; a first acquisition unit configured to input the acoustic features and content features corresponding to each of the plurality of speech segments into the prediction module to obtain a first prediction result corresponding to each of the plurality of speech segments; and an adjusting unit configured to adjust, for any one of the plurality of speech segments, parameters of the speech processing model based at least on the first label and the first prediction result of the speech segment, The prediction module includes a fusion submodule and a prediction submodule, and the first acquisition unit includes: a first acquisition subunit configured to input the acoustic feature and the content feature corresponding to each of the plurality of speech segments into the fusion submodule to obtain a first fusion feature of each of the plurality of speech segments; a determination subunit configured to determine, for each of the plurality of speech segments, a second fused feature for inputting into the prediction submodule based at least on the first fused feature of the speech segment; and The second acquisition subunit is configured to input the second fusion feature of each of the multiple speech segments into the prediction submodule to obtain a first prediction result corresponding to each of the multiple speech segments.

11. The device according to claim 10, wherein: The acoustic feature corresponding to each of the plurality of speech segments is determined based on at least one of the following: Mel-frequency cepstral coefficient of the speech segment; The perceptual linear prediction coefficients for the speech segment; and Filter bank coefficients.

12. The device according to claim 10 or 11, wherein: The content feature corresponding to each of the plurality of speech segments is determined based at least on the phoneme corresponding to the speech segment.

13. The device according to claim 12, wherein: The word position information includes information for indicating the pronunciation position of the phoneme corresponding to the speech segment in the word, and the word boundary information includes information for indicating the positional relationship between the phoneme corresponding to the speech segment and the pronunciation edge of the word.

14. The device according to claim 10, wherein: The determining subunit comprises: For each of the plurality of speech segments, the first fusion feature of the speech segment and the content feature of the speech segment are concatenated to determine a subunit of the second fusion feature.

15. The device according to claim 14, wherein: Each of the plurality of speech segments also has a second label for indicating the phoneme corresponding to the speech segment. And wherein, the adjustment unit comprises: a third acquisition subunit configured to input the second fusion feature of each of the plurality of speech segments into an additional prediction module for assisting the training of the speech processing model, so as to obtain a second prediction result corresponding to each of the plurality of speech segments; and The adjustment subunit is configured to adjust the parameters of the speech processing model for any one of the multiple speech segments based on the first label and the first prediction result corresponding to the speech segment, and the second label and the second prediction result corresponding to the speech segment.

16. The device according to claim 10, wherein: The speech processing model also includes an acoustic feature extraction module, and the device also includes: an extracting unit configured to extract a plurality of speech frames from the sample speech data before determining a plurality of speech segments in the sample speech data, wherein each of the plurality of speech frames has a corresponding acoustic sub-feature; and a second acquisition unit configured to input the acoustic sub-feature corresponding to each of the plurality of speech frames into the acoustic feature extraction module to obtain the acoustic feature corresponding to each of the plurality of speech segments, The acoustic features of each speech segment are determined based on the convolution calculation of multiple acoustic sub-features corresponding to at least some of the multiple speech frames by the acoustic feature extraction module.

17. The device according to claim 16, wherein: Each of the plurality of speech frames also has a corresponding content sub-feature, and the apparatus further comprises: a second determining unit configured to determine an initial content feature corresponding to each of the plurality of speech segments based on a content sub-feature corresponding to each of the plurality of speech frames, wherein the initial content feature of each speech segment is one of a plurality of content sub-features respectively corresponding to a plurality of speech frames in the speech segment; and a third acquisition unit configured to input the initial content feature of each of the plurality of speech segments into a content feature extraction module for assisting the training of the speech processing model, so as to obtain the content feature corresponding to each of the plurality of speech segments; The content feature of each speech segment is determined by the content feature extraction module through a fusion calculation of the initial content features of each of the multiple speech segments.

18. A speech processing model, wherein: The speech processing model is trained based on the method described in any one of claims 1 to 8.

19. An electronic device comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; in The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to make a computer execute the method according to any one of claims 1-9.

21. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images