Speech processing method, speech processing model training method and related apparatus

Abstract

A speech processing method, comprising: acquiring speech data, the speech data comprising speech of at least one speaker, and the speech data being picked up by means of at least one speech channel; coding the speech data to obtain basic speech features; determining speaker features of each of the at least one speaker; and obtaining independent speech features of each speaker on the basis of the speaker features and the basic speech features of each speaker. By means of a speech processing model, the present application can extract independent speech features of each speaker from speech features of mixed speech, thereby solving the problem of processing mixed speech of a plurality of speakers.

Description

Speech processing methods, training methods for speech processing models, and related devices

[0001] This application claims priority to Chinese Patent Application No. 2024109255669, filed on July 10, 2024, entitled "Speech Processing Method, Training Method for Speech Processing Model and Related Apparatus", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence technology, and in particular to speech processing methods, training methods for speech processing models, and related devices. Background Technology

[0003] In the field of artificial intelligence, for speech processing problems, pre-trained neural network models are used for pre-training, and then fine-tuned to obtain a neural network model that matches the specific task. However, for mixed speech processing problems, modeling and processing are generally performed on the speech of a single speaker, and it is impossible to solve the problem of processing mixed speech from multiple speakers. Summary of the Invention

[0004] To address the aforementioned issues, embodiments of this application provide a speech processing method, a speech processing model training method, and related apparatus, which can extract the independent speech features of each speaker from the speech features of mixed speech, thus solving the problem of processing mixed speech from multiple speakers.

[0005] Therefore, the following technical solutions are adopted in the embodiments of this application:

[0006] In a first aspect, embodiments of this application provide a speech processing method, which mainly includes the following steps: acquiring speech data, the speech data including the speech of at least one speaker, the speech data being acquired through at least one speech channel; encoding the speech data to obtain basic speech features; determining the speaker features of each speaker among the at least one speaker; and obtaining the independent speech features of each speaker based on the speaker features of each speaker and the basic speech features.

[0007] In the technical solution of this application, exemplarily, the speech processing method of this application embodiment can be implemented through a speech processing model. Speech data is input into the speech processing model, and the speech processing model outputs independent speech features for each speaker. Therefore, downstream tasks can use the independent speech features of each speaker to carry out corresponding downstream tasks. The speech processing model of this application can directly separate the speech of different speakers, obtaining independent speech features for each speaker. Each speaker's independent speech feature also incorporates the corresponding speaker's independent speech features and speaker features. Therefore, downstream tasks can obtain each speaker's speech features and speaker features, and can directly carry out downstream tasks, resulting in higher efficiency in speech analysis. Furthermore, since each speaker's independent speech features incorporate each speaker's speech features and speaker features, it can be directly applied to different downstream tasks. That is, for different downstream tasks, the speech processing model of this application does not require further fine-tuning, exhibiting high robustness. Furthermore, the speech processing model of this application can, on the one hand, encode speech data to obtain basic speech features; on the other hand, it can determine the speaker features of each speaker among at least one speaker based on the speech data; finally, it can obtain the independent speech features of each speaker based on the speaker features and basic speech features. In other words, the speech processing model of this application can solve the speech processing problems of single speaker and multiple speakers, and the output independent speech features of each speaker, which integrates the speech features and speaker features of each speaker, can be directly applied to a variety of downstream tasks.

[0008] In one possible implementation, the above-mentioned determination of the speaker features of each speaker among at least one speaker includes: performing voiceprint recognition on the speech data, matching the reference speech of each speaker among at least one speaker from the registered speech database, and encoding the reference speech of each speaker to obtain the speaker features of each speaker.

[0009] This implementation provides a method for obtaining speaker features when the speaker is a registered user. The method combines voiceprint recognition technology to obtain speaker recognition results from mixed speech, and combines this with a registered speech database to match and obtain the speaker's reference speech; the reference speech is then encoded to obtain the speaker features.

[0010] In one possible implementation, the above-mentioned determination of the speaker features of each speaker among at least one speaker includes: identifying the target speaker and extracting speech to obtain the reference speech of the target speaker.

[0011] This implementation provides a method for obtaining speaker features when the speaker is not a registered user. When the reference speech of the target speaker can be matched with the registered speech database, the reference speech can be obtained directly. When the reference speech cannot be matched with the registered speech database, the target speaker can be identified using video recognition, and then speech extraction can be performed on the identified target speaker, that is, extracting a segment of the target speaker speaking alone as the reference speech. Optionally, the extracted reference speech can be added to the registered speech database. This method solves the problem of speakers not being pre-registered, enabling dynamic registration and online registration for unregistered users. In other words, the entire registration process is automatically completed by the backend system without the user's awareness, improving the user experience.

[0012] In one possible implementation, the aforementioned speech data is obtained through multiple channels. The speech data is then encoded to obtain basic speech features, including: encoding the speech data from multiple channels to obtain local speech features for each channel; and fusing the local speech features of each channel through an attention mechanism to obtain the basic speech features.

[0013] This implementation provides a speech processing method for multi-channel input speech. First, the speech data from multiple channels is encoded to obtain local speech features for each channel; then, these features are fused using an attention mechanism to obtain basic speech features. In other words, the speech processing model of this application can process single-channel or multi-channel input speech data, thereby addressing scenarios with multi-channel input speech. For example, in a smart cockpit system, multiple microphones may be distributed to better capture user speech data. In this case, the speech data is multi-channel input, and the speech processing model of this application can be applied to the smart cockpit system to process multi-channel input speech data.

[0014] In one possible implementation, the method further includes: fusing the speech features of each speaker to obtain fused language features, which are used as input to at least one downstream task model so that the downstream task model obtains its inference result based on the fused language features.

[0015] This implementation provides a method for inputting downstream tasks. By fusing the speech features of each speaker, a fused language feature is obtained. Downstream tasks can then directly perform speaker logging, speech separation, and speech recognition based on this fused speech feature without further processing, thus improving efficiency.

[0016] Secondly, embodiments of this application provide a training method for a speech processing model, wherein the speech processing model includes a basic encoding module and an independent speech feature extraction module. The training method includes: acquiring a training dataset, the training dataset including multiple training sample pairs, each training sample pair including mixed speech data and multiple independent speech data, the mixed speech data being obtained by mixing multiple independent speech data, each independent speech data being the speech data of one speaker among multiple speakers; using the mixed speech data as input to the basic encoding module and outputting a second basic speech feature; using the mixed speech data as input to a speaker feature determination module and outputting a second speaker feature for each speaker; using the second basic speech feature and the second speaker feature as input to an independent speech feature extraction module and outputting a second independent speech feature for each speaker; using multiple independent speech data as input to a reference speech encoding module and outputting a third independent speech feature corresponding to each independent speech; calculating the loss values ​​of the second independent speech feature and the third independent speech feature; and adjusting the parameters of the basic encoding module, the independent speech feature extraction module, and the reference speech encoding module with the objective of minimizing the loss value.

[0017] In the technical solution of this application, the speech features of independent speech data of multiple speakers are used as labels, and the mixed language data after mixing the independent speech data of multiple speakers is used as training samples to train the speech processing model to extract the speech features corresponding to the speaker features from the mixed speech. This makes the independent speech features of the speaker output by the speech processing model include the speech features separated by the speaker and the speaker features, so that downstream tasks can directly work based on the independent speech features of the speaker.

[0018] In one possible implementation, before extracting the second independent speech features of each speaker from the second basic speech features based on the second speaker features of each speaker, the method further includes: adjusting the order of speakers corresponding to the second speaker features of multiple speakers; and / or, when the number of multiple speakers is less than a threshold, setting the speaker features of the default part of the speakers to a silence representation.

[0019] In this implementation, during the training of the speech processing model, the missing speaker features can be represented by shuffling the order of speaker features and / or by setting silence representations. This makes the speech processing model more robust to the input order of speaker features and the absence of speakers, thereby improving the training effect of the speech processing model.

[0020] Thirdly, embodiments of this application provide a speech processing system, including: a feature extraction module, used to input speech data of at least one speaker, execute the aforementioned speech processing method, and output independent speech features for each speaker; and a multi-speech task processing module, used to execute at least one downstream task based on the independent speech features of each speaker. Optionally, the at least one downstream task includes at least one of speaker logging, speech separation, and speech recognition.

[0021] In the technical solution of this application, in the speech processing system, the feature extraction module extracts independent speech features for each speaker from the speech data of at least one speaker; the multi-speech task processing module executes multiple downstream tasks based on the independent speech features of each speaker. The independent speech features of each speaker output by the feature extraction module are more accurate speech features, which can be directly used in the multi-speech task processing module without needing to fine-tune the feature extraction module according to different tasks, thus improving the efficiency of the speech processing system in processing speech data.

[0022] In one possible implementation, the aforementioned speech processing system includes at least an independent mode and a high-precision mode. In the independent mode, the independent speech features of each speaker are used as input to each downstream task, and the result of each downstream task is output. In the high-precision mode, the independent speech features of each speaker and the output results of other downstream tasks are used as input to the target downstream task, and the result of the target downstream task is output.

[0023] In this implementation, the speech processing system of this application can be understood as a multi-speech task processing framework that simultaneously supports two different working modes: "independent mode" and "high-precision mode." Users can choose the working mode according to their computing platform capabilities, business accuracy requirements, and other factors. In independent mode, each downstream task can directly use the independent speech features output by the speech processing model as input, without depending on the results of other downstream tasks, characterized by high computational efficiency, fast result return, and low computational resource consumption. In high-precision mode, according to task dependencies, it can fuse independent speech features and the output results of other downstream tasks on which the current task depends as input, comparing the results of the current task with the input, characterized by high output precision and accuracy.

[0024] Fourthly, embodiments of this application provide a speech processing apparatus, mainly comprising: an acquisition module for acquiring speech data, the speech data including the speech of at least one speaker, the speech data being acquired through at least one speech channel; a processing module for encoding the speech data to obtain basic speech features; determining the speaker features of each of the at least one speaker; and obtaining the independent speech features of each speaker based on the speaker features and the basic speech features.

[0025] In one possible implementation, the above processing module is specifically used to: perform voiceprint recognition on the speech data, and match at least one reference speech of each speaker from the registered speech database; and encode the reference speech of each speaker to obtain the speaker features of each speaker.

[0026] In one possible implementation, the aforementioned processing module is specifically used to: identify the target speaker and extract speech to obtain the target speaker's reference speech.

[0027] In one possible implementation, the aforementioned speech data is obtained through multiple channels. Specifically, the processing module is used to: encode the speech data from the multiple channels to obtain local speech features for each channel; and fuse the local speech features of each channel using an attention mechanism to obtain basic speech features.

[0028] In one possible implementation, the above processing module is further configured to: fuse the speech features of each speaker to obtain fused language features, which are used as input to at least one downstream task model so that the downstream task model obtains the inference result of the downstream task model based on the fused language features.

[0029] Fifthly, embodiments of this application provide a training apparatus for a speech processing model, mainly comprising: a training dataset acquisition module, used to acquire a training dataset, the training dataset including multiple training sample pairs, each training sample pair including mixed speech data and multiple independent speech data, the mixed speech data being obtained by mixing multiple independent speech data, each independent speech data being the speech data of one speaker among multiple speakers; a training module, used to take the mixed speech data as input to a basic encoding module and output a second basic speech feature; and take the mixed speech data as input to a speaker feature determination module and output a second speaker feature for each speaker; take the second basic speech feature and the second speaker feature as input to an independent speech feature extraction module and output a second independent speech feature for each speaker; and take multiple independent speech data as input to a reference speech encoding module and output a third independent speech feature corresponding to each independent speech; and calculate the loss values ​​of the second independent speech feature and the third independent speech feature; and adjust the parameters of the basic encoding module, the independent speech feature extraction module, and the reference speech encoding module with the objective of minimizing the loss value.

[0030] In one possible implementation, the training module described above is further configured to: adjust the order of speakers corresponding to the second speaker features of multiple speakers; and / or, when the number of speakers is less than a threshold, set the speaker features of the default part of the speakers to a silent representation.

[0031] In a sixth aspect, this application provides a chip system including a processor and a power supply circuit, the power supply circuit being used to supply power to the processor, and the processor being used to execute the method described in any possible implementation of the first aspect and / or the algorithmic functions embodied in the apparatus described in any possible implementation of the second aspect.

[0032] In a seventh aspect, this application provides a computing device including a processor and a memory. The processor is configured to execute instructions stored in the memory, causing the computing device to perform the methods described in any possible implementation of the first aspect and / or the algorithmic functions embodied by the apparatus described in any possible implementation of the second aspect.

[0033] Eighthly, this application provides a computing device cluster including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, such that the computing device cluster performs the method described in any possible implementation of the first aspect and / or the algorithmic function embodied by the apparatus described in any possible implementation of the second aspect.

[0034] Ninthly, this application provides a computer-readable storage medium including computer program instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method described in any possible implementation of the first aspect and / or the algorithmic function embodied by the apparatus described in any possible implementation of the second aspect, wherein the cluster of computing devices includes at least one computing device.

[0035] In a tenth aspect, this application provides a computer program product containing instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method described in any possible implementation of the first aspect and / or the algorithmic function embodied by the apparatus described in any possible implementation of the second aspect, wherein the cluster of computing devices includes at least one computing device.

[0036] It is understood that the beneficial effects of aspects two through ten above can be found in the relevant descriptions in aspect one above, and will not be repeated here. Attached Figure Description

[0037] The accompanying drawings used in the embodiments or technical description are briefly introduced below.

[0038] Figure 1 is a schematic diagram of the composition of a voice processing architecture provided in an embodiment of this application;

[0039] Figure 2 is a schematic diagram of an example of a voice processing architecture provided in an embodiment of this application;

[0040] Figure 3 is a schematic diagram of the training and inference phases of a speech processing model provided in an embodiment of this application;

[0041] Figure 4 is a schematic diagram of the inference stage of a speech processing model provided in an embodiment of this application;

[0042] Figure 5 is a schematic diagram of the composition of a voice processing system provided in an embodiment of this application;

[0043] Figure 6 is a flowchart illustrating a speech processing method provided in an embodiment of this application;

[0044] Figure 7 is a schematic diagram of a model training framework for a speech processing model provided in an embodiment of this application;

[0045] Figure 8 is a flowchart illustrating a training method for a speech processing model provided in an embodiment of this application;

[0046] Figure 9a is a schematic diagram of a voice processing device provided in an embodiment of this application;

[0047] Figure 9b is a schematic diagram of a training device for a speech processing model provided in an embodiment of this application;

[0048] Figure 10 is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0049] Figure 11 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;

[0050] Figure 12 is a schematic diagram of another computing device cluster provided in an embodiment of this application. Detailed Implementation

[0051] In this article, the term "and / or" describes the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three cases: A exists alone, A and B exist simultaneously, and B exists alone. The symbol " / " in this article indicates that the related objects have an "or" relationship; for example, A / B means A or B.

[0052] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.

[0053] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0054] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0055] To facilitate understanding of the solutions provided in the embodiments of this application, a brief introduction to some of the terms involved in this solution will be given first.

[0056] Speaker Diarization, also known as speaker logging, solves the problem of "who spoke when," that is, determining who was speaking at each point in time; that is, identifying the time period of each speaker's speech from mixed speech from multiple speakers.

[0057] Target Speaker Extraction, also known as speaker extraction, is used to extract and recover the speech of a specified target speaker from mixed speech of multiple speakers.

[0058] Automatic Speech Recognition (ASR), also known as speech content recognition, is used to automatically convert speech into corresponding text.

[0059] Speaker features are feature vectors that represent a speaker's timbre or some stylistic information. For example, speech mainly consists of three parts: content information; speaker identity features, generally a coarse-grained acoustic feature referring to timbre; and prosodic features related to speaking style, generally a fine-grained acoustic feature referring to suprasegmental features. Speaker features, in addition to content information, are feature vectors that can characterize the speaker's identity. For example, speaker features can be speaker embeddings extracted by a speaker encoder from a speaker's reference speech.

[0060] In the field of speech processing technology, multi-speaker mixed speech is a common scenario, such as voice conferencing and smart cockpits. The following illustrative examples of these two scenarios are provided to further illustrate the technical problems addressed by this application.

[0061] In voice conferencing scenarios, different speakers may speak at different times or simultaneously. Content understanding of mixed speech and speech separation between different speakers have always been very challenging problems in the field of speech. To solve the problem of multi-speaker mixed speech in this scenario, the Speaker Diarization task addresses the "who speaks when" problem. That is, for each point in time in the speech, it is necessary to determine who is speaking at that moment, thereby ultimately obtaining the timestamp of the speech segment corresponding to each speaker. This is particularly important for applications such as automated meeting logging.

[0062] In the context of smart cockpits, in addition to the presence of multiple speakers (which may include drivers, passengers, voice assistants, etc.), there is also the challenge of different input channels for speakers in different seats, i.e., multi-channel speech processing. In practical applications, the number of speakers in mixed speech may be unknown beforehand, and the speakers' identities may not be registered in advance. In this case, voiceprint segmentation and clustering, speech extraction from each speaker, and speech recognition become particularly difficult.

[0063] One solution employs self-supervised pre-training to develop a pre-trained model for mixed speech by setting corresponding reconstruction targets for different speech inputs. This model performs well in multi-speaker tasks. However, this solution cannot distinguish between different speakers and relies solely on single-channel information, failing to consider multi-channel speech inputs in environments such as in-vehicle settings.

[0064] In other words, although speech pre-trained models have been applied to many speech tasks, most existing speech pre-trained models focus on modeling and processing speech data from a single speaker or a single channel input. As a result, these speech pre-trained models are not suitable for feature representation modeling of mixed speech from multiple speakers or multiple channels, and therefore perform poorly in downstream tasks involving mixed speech.

[0065] In view of this, the embodiments of this application provide a speech processing technical solution. The speech data of different speakers have corresponding speaker features. Speaker features can better help distinguish different speakers from background noise. Therefore, the basic speech features of mixed speech can be extracted through the speaker features of different speakers to obtain independent speech features of each speaker. The independent speech features of each speaker can be directly input into the downstream speech processing task, thereby improving the efficiency of speech processing business.

[0066] For example, downstream speech processing tasks may include speech processing tasks such as speaker logging (SD), speech separation (SS), and speech recognition.

[0067] In some implementations, embodiments of this application further solve the speech processing problem of multi-channel input speech by fusing local speech features from different channels to obtain fused basic speech features. In other words, the technical solutions of embodiments of this application can be applied to speech interaction scenarios with multiple speakers and multi-channel input.

[0068] To make the technical solutions of the embodiments of this application clearer and easier to understand, the technical solutions of this application will be described by way of example below with reference to the accompanying drawings.

[0069] Referring to Figure 1, Figure 1 is a schematic diagram of the composition of a voice processing architecture provided in an embodiment of this application. In this embodiment, a voice processing architecture 100 deployed with this embodiment may include a terminal 110 and / or a server 120. The server 120 may include one or more servers (Figure 1 illustrates this with one server as an example), and the server 120 may provide the methods or apparatus provided in this embodiment to one or more terminals.

[0070] Optionally, a related application may be installed on the terminal 110. This application can receive the user's voice data through the terminal 110 and send the voice data to the server 120. The server 120 is equipped with the voice processing model of this application embodiment, which can process the voice data to obtain the independent voice features of each speaker.

[0071] Optionally, server 120 may also deploy downstream voice processing tasks, which can process voice data and obtain processing results. Optionally, the processing results of the downstream voice processing tasks may also be returned to terminal 110.

[0072] It should be understood that in some optional implementations, terminal 110 may also deploy the speech processing model and / or downstream speech processing tasks of this application embodiment. Terminal 110 can complete the speech processing work on its own to obtain the independent speech features of each speaker or the processing results of the downstream speech processing tasks, without the need for the cooperation of server 120. This application embodiment is not limited in this respect. In some optional implementations, server 120 may also be based on received speech data from other devices outside the terminal or locally stored, without the need for the cooperation of terminal 110. This application embodiment is not limited in this respect.

[0073] The product form of terminal 110 in Figure 1 will be described next. In this embodiment, terminal 110 can be a mobile phone, a speaker, a robot, a watch with voice function, a tablet computer, a wearable device, an in-vehicle device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc. This embodiment does not impose any limitations on these.

[0074] Furthermore, the voice processing architecture 100 shown in Figure 1 includes an entity (i.e., an end-side device) capable of outputting personalized voice. This can be a smart terminal (mobile phone, tablet, smart screen, etc.), a conferencing device, or a telepresence conference room, etc. These entities themselves possess certain computing capabilities and can complete certain model inference calculation tasks. The implementable devices in this application embodiment can be voice software on platforms such as desktop computers and mobile phones, intelligent voice assistants on terminal devices such as computers and mobile phones, or applications or software for smart speakers and human-computer interaction machines, etc.

[0075] Furthermore, the technical solution based on a speech processing model provided in this application embodiment can simultaneously perform multiple speech tasks such as speech extraction of the target speaker, voiceprint clustering, and speech recognition from mixed (single-channel input or multi-channel input) speech. This technical solution can be used in any possible voice interaction scenario, including on platforms such as work computers, mobile smart cockpits, voice conferencing systems, and smart screens.

[0076] The product form of server 120 in Figure 1 will be described next. It can be further understood that server 120 can be various types of servers, such as x86 architecture servers, specifically rack servers, blade servers, high-density servers, platform servers, or high-performance servers. In other words, this application embodiment does not specifically limit the specific type of server. Furthermore, it can be understood that the server structure shown in Figure 1 does not constitute a limitation on the server structure; a server may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0077] Furthermore, server 120 can be configured as an independent physical server, or as a server cluster or distributed system consisting of multiple physical servers. It can also be configured as a cloud server or cloud server cluster that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The cloud server cluster is deployed in several cloud data centers. The software can be an application that implements object control methods, but it is not limited to the above forms.

[0078] Next, the communication connection method between terminal 110 and server 120 is described. For example, terminal 110 and server 120 are connected via a network, enabling terminal 110 to access the cloud management platform deployed on the cloud server cluster. The network can be a wired network or a wireless network. For example, a wired network can be a cable network, fiber optic network, Digital Data Network (DDN), etc., while a wireless network can be a telecommunications network, intranet, Internet, Local Area Network (LAN), Wide Area Network (WAN), Wireless Local Area Network (WLAN), Metropolitan Area Network (MAN), Public Service Telephone Network (PSTN), Bluetooth network, ZigBee network, Global System for Mobile Communications (GSM), CDMA (Code Division Multiple Access) network, CPRS (General Packet Radio Service) network, etc., or any combination thereof.

[0079] Understandably, a network can use any known network communication protocol to enable communication between different terminal layers and gateways. These network communication protocols can be various wired or wireless communication protocols, such as Ethernet, Universal Serial Bus (USB), FireWire, Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), New Radio (NR), Bluetooth, Wireless Fidelity (Wi-Fi), and other communication protocols.

[0080] In one possible scenario, server 120 can function as a cloud platform (a software platform employing application virtualization technology, integrating multiple functions such as software search, download, use, management, and backup). In practical use, server 120 can deploy a cloud management platform and a data center, with terminal 110 interacting with the cloud through the cloud management platform. Additionally, the data center can deploy nodes, which can be virtual machine instances, container instances, physical servers, etc.

[0081] In another possible scenario, the method provided in this application embodiment can be implemented by software. The software has a terminal and a server; the terminal 110 runs the software, and the server 120 runs the software's server. While the terminal 110 is running the software, it can call the server running on the server 120 to implement the method provided in this application embodiment.

[0082] In other words, the method provided in this application embodiment can be applied to the terminal 110 or the server 120. In specific implementation, it can run as software on the terminal 110 or the server 120; for example, the software can be a service or an application. This application embodiment can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application embodiment can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can reside in local and remote computer storage media, including storage devices.

[0083] Referring to Figure 2, which is a schematic diagram of an example of a voice processing architecture provided in an embodiment of this application, the terminal 110 is a smart cockpit. The smart cockpit itself possesses certain computing power and can independently complete the voice processing method of this embodiment. Therefore, in this example, the voice processing architecture 100 does not include a server 120, that is, it does not include cloud devices. In the smart cockpit, multiple voice pickup devices 111 are distributed to better pick up the user's voice data, preventing data distortion and loss, and avoiding poor pickup performance due to distance.

[0084] Furthermore, in the context of smart cockpits, in addition to the characteristic of multiple speakers (which may include drivers, passengers, voice assistants, etc.), there is also the challenge of different input channels for speakers in different seats, i.e., multi-channel voice processing. In some implementations of this application, multi-channel input voice data can be processed. In practical applications, the number of speakers in mixed speech may be unknown beforehand, and the speakers' identities may not be registered in advance. In this case, voiceprint segmentation and clustering, speech extraction for each speaker, and speech recognition become particularly difficult. In some implementations of this application, methods such as video recognition of target speakers and extraction of reference speech from target speakers can be used to achieve online dynamic registration of speakers. Registration can be completed seamlessly for the user, improving the user experience; simultaneously, it solves the problem of voice processing for unregistered speakers.

[0085] Referring to Figure 3, Figure 3 is a schematic diagram of the training and inference stages of a speech processing model provided in an embodiment of this application. As shown in Figure 3, this embodiment of the application provides a speech processing model that can extract the independent speech features of each speaker from the speech of one or more speakers and input them into downstream tasks.

[0086] As shown in Figure 3, during the training phase, the training dataset is used to train the untrained speech processing model (the speech processing model that has not yet completed training) in this embodiment of the application. After training, the speech processing model is obtained. During the inference phase, the trained speech processing model is deployed by inputting speech data into the model. The model outputs the independent speech features of each speaker and inputs them into the downstream task. The technical solution for the inference phase of the speech processing model will be described in detail below.

[0087] Referring to Figure 4, Figure 4 is a schematic diagram of the inference stage of a speech processing model provided in an embodiment of this application. As shown in Figure 4, the speech processing model in this embodiment mainly includes the following modules:

[0088] The basic encoding module is used to encode speech data to obtain basic speech features.

[0089] The speaker feature determination module is used to determine the speaker features of each speaker in at least one speaker based on the speech data.

[0090] The independent speech feature extraction module is used to obtain the independent speech features of each speaker based on the speaker features and basic speech features of each speaker.

[0091] For example, as shown in Figure 4, in the speech processing model, based on the number of input channels, the input speech data can be single-channel (SC) speech or multi-channel (MC) speech. Based on the number of speakers, the input speech data can be single-speaker speech or multi-speaker speech.

[0092] Optionally, as shown in Figure 4, the speaker features of each speaker output in the speech processing model can be input into multiple downstream tasks. These downstream tasks can also have dependencies, resulting in upstream and downstream relationships among them. For example, the speaker log task is an upstream task of the speech separation task, and the speech separation task is an upstream task of the speech recognition task.

[0093] Optionally, as shown in Figure 4, the speech processing model also includes a channel attention module. When speech data is obtained through multiple channels, the basic encoding module encodes the speech data in multiple channels to obtain local speech features for each channel. The channel attention module fuses the local speech features of each channel through an attention mechanism to obtain basic speech features. In other words, the speech processing model of this application can process single-channel or multi-channel input speech data, thereby solving scenarios with multi-channel input speech, such as multi-channel input speech scenarios in smart cockpits.

[0094] In one example, the input speech first passes through a CNN Encoder module (an example of a basic encoding module) to obtain local speech features, while the reference speech of each speaker passes through a Speaker Encoder module (an example of a speaker feature determination module) to obtain speaker features for each speaker. Optionally, the Speaker Encoder is a mature encoder for speaker features, and its parameters can be frozen during the training of the speech processing model, i.e., the Speaker Encoder parameters are not adjusted.

[0095] In this example, one instance of a channel-assistance module is Spatio-Temporal Attention (STA) and Mean Pooling. Local speech features are then processed by Spatio-Temporal Attention (STA) and Mean Pooling to capture the relationships between different channels. For example, STA consists of cross-channel and cross-frame Transformer layers. The cross-channel Transformer layer uses cross-channel self-attention to capture the relationships between different channels, while the cross-frame Transformer layer uses cross-frame self-attention to capture the relationships between different speech frames.

[0096] In this example, one instance of the independent speech feature extraction module is the Speaker Adapted Transformer Encoder (SATE). Based on the provided speaker features, this local speech feature is fused with each speaker feature using the Speaker Adapted Transformer Encoder (SATE) to obtain independent speech features with fused speaker features. This ultimately yields k sets of C feature results (assuming modeling is being performed on k speakers). Independent speech features are also called contextual speech features, meaning that independent speech features are related to each other across different frames. For example, after obtaining independent speech features from the previous frame's speech data, the parameters of the speech processing model are adjusted, thus the independent speech features obtained from the subsequent frame's speech data are related to and influence the speech data from the previous frame.

[0097] In this example, optionally, the speech processing model also includes a Speaker Merge Block, or a fusion module is set after the speech processing model. This fusion module is used to fuse independent speech features from different speakers to improve the convenience of downstream tasks. For example, k sets of C features (independent speech features) corresponding to k speakers are processed through Concat (concatenation), Linear (linear layer transformation), and Transformer (Transformer layer transformation) to generate one set of T-dimensional (assuming the number of speech frames is T) intermediate result features I. PRE This feature will also be the target feature that the speech processing model of this application can directly use for downstream tasks.

[0098] Referring to Figure 5, Figure 5 is a schematic diagram of the composition of a speech processing system provided in an embodiment of this application. As shown in Figure 5, this embodiment of the application provides a speech processing system 500, which mainly includes:

[0099] The feature extraction module 510 includes a speech processing model as shown in Figure 4, which is used to input speech data of at least one speaker, execute the above speech processing method, and output independent speech features of each speaker; the multi-speech task processing module is used to execute multiple downstream tasks based on the independent speech features of each speaker, and the multiple downstream tasks include at least one of speaker logging, speech separation, and speech recognition.

[0100] In the technical solution of this application, in the speech processing system, the feature extraction module extracts independent speech features for each speaker from the speech data of at least one speaker; the multi-speech task processing module executes multiple downstream tasks based on the independent speech features of each speaker. The independent speech features of each speaker output by the feature extraction module are more accurate speech features. The independent speech features of each speaker are fused with the speech features of that speaker separated from the basic speech features, as well as the speaker features of that speaker. They can be directly used by the multi-speech task processing module to perform relevant tasks for different speakers, without the need to fine-tune the feature extraction module according to different tasks, thereby improving the efficiency of the speech processing system in processing speech data.

[0101] In one possible implementation, the aforementioned speech processing system includes at least an independent mode and a high-precision mode. In the independent mode, the independent speech features of each speaker are used as input to each downstream task, and the result of each downstream task is output. In the high-precision mode, the independent speech features of each speaker and the output results of other downstream tasks are used as input to a single downstream task, and the result of a single downstream task is output.

[0102] In this implementation, the speech processing system of this application can be understood as a multi-speech task processing framework that simultaneously supports two different working modes: "independent mode" and "high-precision mode." Users can choose the working mode according to their computing platform capabilities, business accuracy requirements, and other factors. In independent mode, each downstream task can directly use the independent speech features output by the speech processing model as input, without depending on the results of other downstream tasks, characterized by high computational efficiency, fast result return, and low computational resource consumption. In high-precision mode, according to task dependencies, it can fuse independent speech features and the output results of other downstream tasks on which the current task depends as input, comparing the results of the current task with the input, characterized by high output precision and accuracy.

[0103] In one example, downstream tasks include: Speaker Diarization (SD), Speech Separation (SS), and Automatic Speech Recognition (ASR). The input speech data (whether single-channel or multi-channel) first passes through the speech processing model of the feature extraction module to extract independent speech features (denoted as I). PRE ), I PRE It can flow to various specific tasks downstream to calculate specific recognition results.

[0104] In this example, independent mode refers to the feature I directly output by the speech processing model for each downstream speech task. PRE It takes feature I as input, without relying on the results of other upstream tasks. For example, for the ASR task, the ASR module directly uses feature I as input. PRE The recognition result is calculated without fusing the results of the SS module, using the input as input.

[0105] In this example, the high-precision mode refers to fusing feature I according to task dependencies. PRE The output of the previous task, along with the output of the previous task, serves as the input for the current task to calculate the final output. For example, for the SS task, its input would be the output O of the speaker log task. SD and Feature I PRE For the ASR task, its input depends on the result O of its upstream SS task. SS .

[0106] In this example, in the independent mode, each speech task is computed independently without relying on other results, resulting in high computational efficiency, fast result return, and relatively low computational resource requirements. However, the trade-off is that the accuracy of the independent mode is slightly lower than that of the high-precision mode. Therefore, users can choose different computation modes according to their specific needs.

[0107] In this example, in the Speaker Log (SD) task, the network model structure uses the TS-VAD network, I PRE It can be used directly as a representation of input speech, O SD This is the output of the SD task. The output of the SD task is O. SD It can be directly passed to the speech separation (SS) task.

[0108] In this example, for the speech separation (SS) task, the network model architecture can use a Conv-TasNet network, which integrates O... SD and I PRE The output features of the decoder are used as the input of the separator. SS It is the output of the speech separation (SS) task and can be directly used by the ASR model.

[0109] In this example, for the speech recognition (ASR) task, the network model structure uses a forward propagation network (Branchformer), while also incorporating O SS and I PRE As input. Because O SS It is composed of the voices of different speakers, and the result of ASR can be the voice recognition result of each channel / speaker.

[0110] Referring to Figure 6, which is a flowchart illustrating a speech processing method provided in an embodiment of this application, based on the speech processing model in Figure 4 and the speech processing system in Figure 5, as shown in Figure 6, this embodiment also provides a speech processing method, mainly including the following steps:

[0111] Step S610: Acquire voice data, which includes the voice of at least one speaker, and the voice data is acquired through at least one voice channel.

[0112] Step S620: Encode the speech data to obtain basic speech features.

[0113] Step S630: Determine the speaker characteristics of each speaker among at least one speaker.

[0114] Step S640: Based on the speaker features and basic speech features of each speaker, obtain the independent speech features of each speaker.

[0115] In one possible implementation, voiceprint recognition is performed on the speech data to obtain speaker recognition results; based on the speaker recognition results and the registered speech database, reference speech for each speaker in at least one speaker is determined; and the reference speech for each speaker is encoded to obtain speaker features for each speaker.

[0116] This implementation provides a method for generating speaker features for each speaker in the speech processing model of this application. The method combines voiceprint recognition technology to obtain speaker recognition results from mixed speech, and combines a registered speech database to match and obtain the speaker's reference speech; the reference speech is then encoded to obtain speaker features.

[0117] In one possible implementation, the target speaker is determined based on the speaker recognition result; it is then determined whether the target speaker belongs to the registered speakers in the registered speech database; if the target speaker belongs to the registered speakers in the registered speech database, the reference speech of the target speaker is extracted from the registered speech database; if the target speaker does not belong to the registered speakers in the registered speech database, video recognition and speech extraction are performed on the target speaker to obtain the reference speech of the target speaker.

[0118] This implementation discusses the case of matching the target speaker's reference speech from the registered speech database. When the target speaker's reference speech can be matched from the database, it can be obtained directly. When it cannot be matched, the target speaker can be identified using video recognition, and then a segment of the target speaker speaking alone can be extracted as the reference speech. This extracted reference speech can then be added to the registered speech database. This method solves the problem of speakers not being pre-registered, enabling dynamic and online registration for unregistered users. In other words, the entire registration process is completed automatically by the backend system without the user's awareness, improving the user experience.

[0119] For example, on the one hand, when the reference speech of the target speaker can be matched from the registered speech database, the current mode can be set to "seen speaker" mode. When "seen speaker" mode is executed, the reference speech of the target speaker can be obtained directly. On the other hand, when the reference speech of the target speaker cannot be matched from the registered speech database, the current mode can be set to "unseen speaker" mode. When "unseen speaker" mode is executed, the target speaker can be identified through video recognition, and then speech extraction can be performed on the identified target speaker, that is, extracting a segment of the target speaker speaking alone as the reference speech.

[0120] In one possible implementation, speech data from multiple channels are encoded to obtain local speech features for each channel; then, through an attention mechanism, the local speech features of each channel are fused to obtain basic speech features.

[0121] This implementation provides a speech processing method for multi-channel input speech. First, the speech data from multiple channels is encoded to obtain local speech features for each channel; then, these features are fused using an attention mechanism to obtain basic speech features. In other words, the speech processing model of this application can process single-channel or multi-channel input speech data, thereby addressing scenarios with multi-channel input speech. For example, in a smart cockpit system, multiple microphones may be distributed to better capture user speech data. In this case, the speech data is multi-channel input, and the speech processing model of this application can be applied to the smart cockpit system to process multi-channel input speech data.

[0122] In one possible implementation, the method further includes: fusing the speech features of each speaker to obtain fused language features, the fused language features being used for at least one downstream task, the downstream task including at least one of speaker logging, speech separation, and speech recognition.

[0123] This implementation provides a method for inputting downstream tasks. By fusing the speech features of each speaker, a fused language feature is obtained. Downstream tasks can directly perform tasks such as speaker logging, speech separation, and speech recognition based on the fused speech feature without further processing of the fused speech feature, thus improving work efficiency.

[0124] In this embodiment of the application, the speech processing model can be considered as a multi-channel speech pre-training model that integrates speaker identity features. This speech processing model supports simultaneous feature representation learning for input speech from multiple speakers / multiple channels.

[0125] In this embodiment, the speech processing system that combines the speech processing model with downstream tasks can be considered a unified framework for multiple speech tasks in multi-person speech scenarios (such as smart cockpit scenarios, conference scenarios, etc.), and different computing modes can be set according to business needs, such as independent mode and high-precision mode. This solution can greatly improve the performance of tasks such as speaker clustering, speech separation, and speech recognition in complex scenarios such as smart cockpits and multi-person conferences.

[0126] The following describes the technical solution of the speech processing model in the training phase of this application embodiment.

[0127] Referring to Figure 7, which is a schematic diagram of a model training framework for a speech processing model provided in an embodiment of this application. Based on the speech processing model in Figure 4, as shown in Figure 7, during the training process of the speech processing model, the model training framework includes a prediction module, which is used to predict the independent speech features of each speaker output by the speech processing model, compare them with the reference speech of each speaker, and adjust the parameters of the neural network model in the speech processing model and the prediction module.

[0128] Optionally, the speech processing model's training framework is deployed on a training server for pre-training the required speech processing model. During training, the parameters of both the speech processing model and the prediction module need to be adjusted. After training, the prediction module is discarded, and the trained speech processing model is deployed. In the inference phase, the speech processing model runs on a human-computer interaction terminal device or a cloud server. After receiving input speech data, the speech processing model extracts the independent speech features of each speaker from the speech data. Subsequently, the multi-speech task processing module decodes the independent speech features of each speaker according to the task and outputs the corresponding recognition result.

[0129] In one example, the prediction module consists of K Linear networks, meaning there is one Linear network for each speaker, such as Linear1, Linear2, ..., Linear-n as shown in Figure 7. Assuming the current model can support a maximum of 5 speakers simultaneously, then K = 5, and the current prediction module contains 5 Linear networks. The independent speech features of each speaker output by the fusion module or speech processing model are then processed through the K Linear networks to obtain K sets of speech features corresponding to the speaker.

[0130] In this example, the independent speech features of each speaker are processed by a Linear Network to obtain the final independent speech features Z for each speaker. These independent speech features correspond to the prediction targets obtained from the speaker's reference speech. The training loss is calculated using the cross-entropy loss function to train the parameters of each model shown in Figure 7.

[0131] In other words, each Linear layer in the prediction module predicts a pseudolabel corresponding to the speaker's speech features. The order of the prediction results is consistent with the concatenation order of the C values ​​of the independent speech features of multiple speakers in the speech processing model. The final loss function is the cross-entry between the predicted pseudolabels and the labels obtained from the actual speech frames of the corresponding speakers using K-means.

[0132] The following four examples illustrate the training methods for the aforementioned speech processing methods and speech processing models. In this embodiment, the processing flow for different numbers of speakers in real-world application scenarios will be described. Assuming the current speech processing model can support a maximum of K (K=5) different speakers, the specific modeling operations will be explained in different scenarios below.

[0133] In the first example, we assume that the speaker is a registered user, the speech database contains the speaker's registered speech, the current speech processing architecture is a single-channel input speech, and the scenario involves a single speaker.

[0134] In this example, during the training phase, the speech processing model, during pre-training, uses SATE (an example of an independent speech feature extraction module) to extract the corresponding feature representation c of the speaker (let's say k) from the input speech. k However, it also extracts (K-1) silent representations c s The above c k and (K-1) silence representations c s The output c after fusion module m .

[0135] In this example, during the inference phase, the speech output of the corresponding speaker is only calculated by the speech processing model, without going through the fusion module. That is, only the independent speech features of the corresponding speaker are calculated, while silence is used to replace the default speaker.

[0136] In the second example, assume the speaker is a registered user, the speech database contains the speaker's registered speech, the current speech processing architecture is a single-channel input speech, and there are multiple speakers in the scenario.

[0137] In this example, during the training phase, SATE uses speaker features e of speaker k∈{a,b,c,d,e}. k From mixed inputs m =∑s k Extract the speech representation c corresponding to speaker k. k If there are fewer than 5 speakers, silence is used to represent the default speaker. s Complete the sentence. The SMB (Speaker Merge Block, an example of a fusion module) receives K speech representations c. k Output c m .

[0138] In this example, during the inference phase, only the CNN Encoder (an example of the basic encoding module), STA (an example of the channel attention module), and SATE are needed to extract the representation c of speaker k, respectively. k .

[0139] In the third example, we assume that the speaker is a registered user, the speech database contains the speaker's registered speech, the current speech processing architecture is multi-channel input speech, and there are multiple speakers in the scenario.

[0140] In this example, during the training phase, speech signals from different channels are superimposed to form a mixed input s. mThe STA will learn the relationships between different channels, outputting a T×D×N representation, and finally obtain a T×D representation through mean pooling. The STA uses the speaker embedding code e of speaker k∈{a,b,c,d,e}. k From input s m =∑s k Extracting the representation c of speaker k k If there are fewer than 5 speakers, use silence to represent c. s Complete the representation c k The SMB receives K representations and outputs c. m .

[0141] In this example, during the inference phase, a CNN Encoder, STA, and SATE are needed to extract the representation c of speaker k. k .

[0142] In the fourth example, assume the speaker is an unregistered user, and the speech database does not contain any registered speech for that speaker. In the case of missing registered speech for the corresponding speaker, video data and mixed speech data are input into the ASD (Automatic Speech Detection) model. The ASD model outputs a speech segment of the speaker speaking alone, thus obtaining the corresponding speaker's reference speech. For example, [0,1,1,1,…,0,1,0] can be used as the output, where 1 indicates that the current frame corresponds to the speaker speaking alone. In other words, the speaker is determined using visual input and the ASD model, and then the speaker's reference speech is determined online to dynamically register the corresponding speaker's speech.

[0143] Referring to Figure 8, Figure 8 is a flowchart illustrating a training method for a speech processing model provided in an embodiment of this application. As shown in Figure 8, this embodiment of the application provides a training method for a speech processing model, mainly including the following steps:

[0144] Step S810: Obtain the training dataset. The training dataset includes multiple training sample pairs. Each training sample pair includes mixed speech data and multiple independent speech data. The mixed speech data is obtained by mixing multiple independent speech data. Each independent speech data is the speech data of one speaker among multiple speakers.

[0145] Step S820: The mixed speech data is used as input to the basic coding module, and the second basic speech features are output.

[0146] Step S830: The mixed speech data is used as input to the speaker feature determination module, and the second speaker feature of each speaker is output.

[0147] Step S840: Use the second basic speech features and the second speaker features as input to the independent speech feature extraction module, and output the second independent speech features for each speaker.

[0148] Step S850: Calculate the loss values ​​of the second independent speech feature and the third independent speech feature.

[0149] Step S860: Adjust the parameters of the base coding module, the independent speech feature extraction module, and the reference speech coding module with the goal of minimizing the loss value.

[0150] In the technical solution of this application, the speech features of independent speech data of multiple speakers are used as labels, and the mixed language data after mixing the independent speech data of multiple speakers is used as training samples to train the speech processing model to extract the speech features corresponding to the speaker features from the mixed speech. This makes the independent speech features of the speaker output by the speech processing model include both speech features and speaker features, so that downstream tasks can directly work based on the independent speech features of the speaker.

[0151] In one possible implementation, before extracting the second independent speech features of each speaker from the second basic speech features based on the second speaker features of each speaker, the method further includes: adjusting the order of speakers corresponding to the second speaker features of multiple speakers; and / or, when the number of speakers is less than the number threshold, setting the speaker features of the missing part of the speakers to a silence representation.

[0152] In this implementation, during the training of the speech processing model, the missing speaker features can be represented by shuffling the order of speaker features and / or by setting silence representations. This makes the speech processing model more robust to the input order of speaker features and the absence of speakers, thereby improving the training effect of the speech processing model.

[0153] In one example, to make the speech processing model more robust to the input order of speaker features and speaker omissions, the following two operations can be performed during training: randomly shuffling the order of speaker features and setting a learnable embedding code e. s This is used to indicate the absence of speaker characteristics.

[0154] In this example, for instance, the mixed speech consists of 5 speakers, s m =s 1 +s 2 +s 3 +s 4 +s 5 The corresponding speaker characteristic is: e 1 ,e 2 ,e 3 ,e 4 ,e5 During training, this can be achieved through, for example, Random_shuffle(e 1 ,e 2 ,e 3 ,e 4 ,e 5 ) = e 2 ,e 4 ,e 1 ,e 5 ,e 3 The randomization method shuffles the order of speaker features, creating multiple different feature combinations. Therefore, the order of speaker features after the fusion module concatenates them will also differ. If the number of speakers in the current input is less than 5, then e is used. s This replaces the missing speaker features; correspondingly, the predicted pseudo-label for a missing speaker is a label indicating silence.

[0155] Next, based on the methods in the above embodiments, a speech processing device and a speech processing model training device provided in this application will be introduced, and their software and hardware implementation methods will be described by way of example.

[0156] Referring to Figure 9a, Figure 9a is a schematic diagram of a voice processing device provided in an embodiment of this application. As shown in Figure 9a, this embodiment of the application provides a voice processing device 910, which mainly includes:

[0157] The acquisition module 911 is used to acquire voice data, which includes the voice of at least one speaker, and the voice data is acquired through at least one voice channel.

[0158] Processing module 912 encodes speech data to obtain basic speech features; and determines the speaker features of each speaker among at least one speaker; and, based on the speaker features of each speaker and the basic speech features, obtains the independent speech features of each speaker.

[0159] In one possible implementation, the aforementioned processing module 912 is specifically used to: identify the target speaker and extract speech to obtain the target speaker's reference speech.

[0160] In one possible implementation, the processing module 912 is specifically used to: perform voiceprint recognition on the speech data, and match the reference speech of each speaker from the registered speech database; and encode the reference speech of each speaker to obtain the speaker features of each speaker.

[0161] In one possible implementation, the aforementioned speech data is obtained through multiple channels. Specifically, the processing module 720 is used to: encode the speech data from the multiple channels to obtain local speech features for each channel; and fuse the local speech features of each channel through an attention mechanism to obtain basic speech features.

[0162] In one possible implementation, the processing module 912 is further configured to: fuse the speech features of each speaker to obtain fused language features, the fused language features being used as input to at least one downstream task model, so that the downstream task model obtains the inference result of the downstream task model based on the fused language features.

[0163] Referring to Figure 9b, Figure 9b is a schematic diagram of a speech processing model training device provided in an embodiment of this application. As shown in Figure 9b, this embodiment of the application provides a speech processing model training device 920, which mainly includes:

[0164] The training dataset acquisition module 921 is used to acquire the training dataset, which includes multiple training sample pairs. Each training sample pair includes mixed speech data and multiple independent speech data. The mixed speech data is obtained by mixing multiple independent speech data. Each independent speech data is the speech data of one speaker among multiple speakers.

[0165] Training module 922 is used to take mixed speech data as input to a basic coding module and output a second basic speech feature; and take mixed speech data as input to a speaker feature determination module and output a second speaker feature for each speaker; take the second basic speech feature and the second speaker feature as input to an independent speech feature extraction module and output a second independent speech feature for each speaker; take multiple independent speech data as input to a reference speech coding module and output a third independent speech feature corresponding to each independent speech; calculate the loss value of the second independent speech feature and the third independent speech feature; and adjust the parameters of the basic coding module, the independent speech feature extraction module, and the reference speech coding module with the goal of minimizing the loss value.

[0166] In one possible implementation, the training module 922 is further configured to: adjust the order of speakers corresponding to the second speaker features of multiple speakers; and / or, when the number of speakers is less than a threshold, set the speaker features of the default part of the speakers to a silent representation.

[0167] The software and hardware implementation of a speech processing device 910 as shown in Figure 9a and a speech processing model training device 920 as shown in Figure 9b (hereinafter referred to as: related devices of speech processing model) need to be further explained below.

[0168] As an example of a software functional unit, a module can include code running on a computing instance. A computing instance can include at least one of a physical host (computing device), a virtual machine, or a container. Furthermore, the aforementioned computing instance can be one or more. For example, a module can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code can be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code can be distributed within the same availability zone (AZ) or in different AZs, each AZ comprising one or more geographically proximate data centers. Typically, a region can include multiple AZs.

[0169] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0170] As an example of a hardware functional unit, a module may include at least one computing device, such as a server. Alternatively, a module may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The aforementioned PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0171] The multiple computing devices included in the module can be distributed within the same region or in different regions. Similarly, the multiple computing devices included in the module can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the module can be distributed within the same Virtual Private Cloud (VPC) or multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0172] It should be noted that, in other embodiments, the related apparatus of the speech processing model is additionally provided with one or more modules for performing any of the steps included in the above implementation. The steps implemented by one or more modules in the related apparatus of the speech processing model can be specified as needed, and more or fewer modules can be obtained than in the embodiments of this application to implement different steps in the above method, thereby realizing all the functions of the related apparatus of the speech processing model.

[0173] This application also provides a computing device 1000. As shown in FIG10, the computing device 1000 includes: a bus 1002, a processor 1004, a memory 1006, and a communication interface 1008. The processor 1004, the memory 1006, and the communication interface 1008 communicate with each other via the bus 1002. The computing device 1000 can be a server, such as a central server, an edge server, or a local server in a local data center, or it can be an electronic device such as a desktop computer, a laptop computer, or a smartphone. It should be understood that this application does not limit the number of processors and memories in the computing device 1000.

[0174] Bus 1002 can be a Peripheral Component Interconnect (PCI) bus, a Peripheral Component Interconnect Express (PCIe) bus, an Extended Industry Standard Architecture (EISA) bus, a Unified Bus (Ubus or UB), a Compute Express Link (CXL), a Cache Coherent Interconnect for Accelerators (CCIX), etc. The Unified Bus is also known as the Lingqu Bus. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 10 uses only one line, but this does not imply that there is only one bus or one type of bus. Bus 1004 can include pathways for transmitting information between various components of the computing device 1000 (e.g., memory 1006, processor 1004, communication interface 1008).

[0175] The processor 1004 may include any one or more of the following computing devices: central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP) or digital signal processor (DSP), ASIC, FPGA, CPLD, NPU, SoC, offload card, accelerator card, etc.

[0176] The memory 1006 may include volatile memory, such as random access memory (RAM). The processor 1004 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD). Furthermore, the memory 1006 may also be implemented using storage class memory (SCM), phase change memory (PCM), or other types of storage media.

[0177] It is worth noting that the same type of storage medium can be configured in the same computing device to realize the function of memory 1006, or two or more types of storage media can be configured to realize the function of memory 1006. This application does not limit this.

[0178] The memory 1006 stores executable program code, and the processor 1004 executes the executable program code to implement the functions of the relevant devices of the above-described speech processing model, thereby implementing the method described in the above embodiments. That is, the memory 1006 stores instructions for executing the method described in the above embodiments.

[0179] Alternatively, the memory 1006 stores executable code, which the processor 1004 executes to implement the functions of the relevant devices of the aforementioned speech processing model, thereby implementing the method described in the above embodiments. That is, the memory 1006 stores instructions for executing the method described in the above embodiments.

[0180] The communication interface 1008 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1000 and other devices or communication networks.

[0181] As one possible implementation, the computing device 1000 may also include a chip system, which includes a processor and a power supply circuit. The power supply circuit supplies power to the processor, and the processor executes the operation steps corresponding to the methods in the embodiments of this application. For simplicity, further details are omitted here. The processor can be implemented using a GPU, or it can be implemented using computing devices or AI chips such as a DPU, NPU, XPU, SoC, offloading card, or accelerator card.

[0182] As one possible implementation, the computing device 1000 may include various types of processors 1004, that is, the computing device 1000 is a heterogeneous device. For example, the computing device 1000 includes a CPU and a GPU, and at least one of the processors 1004 can execute the operation steps corresponding to the method of the embodiments of this application. For the sake of brevity, further details are omitted here.

[0183] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be an electronic device such as a desktop computer, a laptop computer, or a smartphone.

[0184] As shown in Figure 11, the computing device cluster includes at least one computing device 1000. The memory 1006 of one or more computing devices 1000 in the computing device cluster may store the same instructions for performing the methods described in the above embodiments.

[0185] In some possible implementations, the memory 1006 of one or more computing devices 1000 in the computing device cluster may also store partial instructions for executing the methods described in the above embodiments. In other words, a combination of one or more computing devices 1000 can jointly execute instructions for executing the methods described in the above embodiments.

[0186] It should be noted that the memory 1006 in different computing devices 1000 within the computing device cluster can store different instructions, which are used to execute certain functions of the related devices of the aforementioned speech processing model. That is, the instructions stored in the memory 1006 of different computing devices 1000 can implement the functions of one or more modules of the related devices of the aforementioned speech processing model.

[0187] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 12 illustrates one possible implementation. As shown in Figure 12, two computing devices 1000A and 1000B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this type of possible implementation, the memory 1006 in computing device 1000A stores instructions for the functions of one or more modules of the related devices of the aforementioned speech processing model. Simultaneously, the memory 1006 in computing device 1000B stores instructions for the functions of another one or more modules of the related devices of the aforementioned speech processing model.

[0188] It should be understood that the functions of computing device 1000A shown in Figure 12 can also be performed by multiple computing devices 1000. Similarly, the functions of computing device 1000B can also be performed by multiple computing devices 1000.

[0189] This application also provides another computing device cluster. The connection relationship between the computing devices in this computing device cluster can be similarly referred to the connection method of the computing device cluster described in Figures 11 and 12. The difference is that the memory 1006 of one or more computing devices 1000 in this computing device cluster can store the same instructions for executing the methods in the above embodiments.

[0190] In some possible implementations, the memory 1006 of one or more computing devices 1000 in the computing device cluster may also store partial instructions for executing the aforementioned data processing method. In other words, a combination of one or more computing devices 1000 can jointly execute the instructions for performing the aforementioned method.

[0191] Based on the methods in the above embodiments, this application provides a computer-readable storage medium including computer program instructions. When executed by a cluster of computing devices including at least one computing device, the computer program instructions cause the cluster of computing devices to perform the methods in the above embodiments. Exemplarily, the computer-readable storage medium can be any available medium capable of being stored in the cluster of computing devices or a data storage device such as a data center containing one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives).

[0192] Based on the methods in the above embodiments, this application provides a computer program product containing instructions. The computer program product may be software or program products containing instructions, capable of running on a computing device or stored on any available medium. When the instructions are executed by a cluster of computing devices containing at least one computing device, at least one computing device in the cluster of computing devices performs the methods in the above embodiments.

[0193] It is understood that the processor in the embodiments of this application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor may be a microprocessor or any conventional processor.

[0194] The method steps in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0195] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0196] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application.

[0197] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A speech processing method, characterized in that, include: Acquire voice data, the voice data including the voice of at least one speaker, the voice data being picked up through at least one voice channel; The speech data is encoded to obtain basic speech features; Determine the speaker characteristics of each of the at least one speakers; Based on the speaker characteristics and the basic speech characteristics of each speaker, the independent speech characteristics of each speaker are obtained.

2. The method according to claim 1, characterized in that, Determining the speaker characteristics of each of the at least one speaker includes: Voiceprint recognition is performed on the speech data, and reference speech of each speaker among the at least one speaker is obtained from the registered speech database. The reference speech of each speaker is encoded to obtain the speaker features of each speaker.

3. The method according to claim 1, characterized in that, Determining the speaker characteristics of each of the at least one speaker includes: The target speaker is identified and speech is extracted to obtain the reference speech of the target speaker.

4. The method according to any one of claims 1-3, characterized in that, The speech data is obtained through multiple channels; the encoding of the speech data to obtain basic speech features includes: The speech data in the multiple channels are encoded to obtain the local speech features of each channel; By using an attention mechanism, the local speech features of each channel are fused to obtain the basic speech features.

5. The method according to any one of claims 1-4, characterized in that, Also includes: The speech features of each speaker are fused to obtain fused language features, which are used as input to at least one downstream task model so that the downstream task model obtains its inference result based on the fused language features.

6. A method for training a speech processing model, characterized in that, The speech processing model includes a basic coding module and an independent speech feature extraction module, and the training method includes: Obtain a training dataset, which includes multiple training sample pairs. Each training sample pair includes mixed speech data and multiple independent speech data. The mixed speech data is obtained by mixing the multiple independent speech data. Each independent speech data is the speech data of one speaker among the multiple speakers. The mixed speech data is used as input to the basic coding module to output the second basic speech feature; The mixed speech data is used as input to the speaker feature determination module, which outputs the second speaker feature for each speaker. The second basic speech features and the second speaker features are used as inputs to the independent speech feature extraction module, and the second independent speech features of each speaker are output. The multiple independent speech data are used as input to the reference speech coding module, and the third independent speech feature corresponding to each independent speech is output. Calculate the loss values ​​of the second independent speech feature and the third independent speech feature; With the goal of minimizing the loss value, the parameters of the basic coding module, the independent speech feature extraction module, and the reference speech coding module are adjusted.

7. The method according to claim 6, characterized in that, Before inputting the second basic speech features and the second speaker features into the independent speech feature extraction module, the method further includes: Adjust the order of speakers corresponding to the second speaker features of multiple speakers; And / or, when the number of the multiple speakers is less than the number threshold, the speaker characteristics of the default part of the speakers are set to silent representation.

8. A voice processing system, characterized in that, include: The feature extraction module is used to input speech data of at least one speaker, execute the method as described in any one of claims 1-5, and output independent speech features for each speaker; A multi-speech task processing module is used to perform at least one downstream task based on the independent speech characteristics of each speaker.

9. The system according to claim 8, characterized in that, The speech processing system includes at least an independent mode and a high-precision mode; wherein, in the independent mode, the independent speech features of each speaker are used as input to each downstream task, and the result of each downstream task is output; in the high-precision mode, the independent speech features of each speaker and the output results of other downstream tasks are used as input to a target downstream task, and the result of the target downstream task is output.

10. A computing device, characterized in that, The computing device includes a processor and a memory; the processor is configured to execute instructions stored in the memory to cause the computing device to perform the method as described in any one of claims 1-5 or the method as described in any one of claims 6-7, or to implement the algorithmic functions embodied by the system as described in any one of claims 8-9.

Images