A voice processing method and related device

Abstract

The application provides a speech processing method and a related device, which can simultaneously support two types of tasks of voiceprint segmentation clustering and target speaker voice extraction, and is beneficial to reduce the design cost of speech processing. The method of the application comprises: obtaining mixed speech and reference speech of a target object, the mixed speech containing speech of the target object and speech of remaining objects except the target object; processing the mixed speech, the reference speech and intermediate output of a second model through a first model to obtain intermediate output of the first model and final output of the first model, the final output of the first model being used to obtain speech of the target object; processing the mixed speech and the intermediate output of the first model through a second model to obtain intermediate output of the second model and final output of the second model, the final output of the second model being used to determine the position of the speech of the target object in the mixed speech.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to a speech processing method and related equipment. Background Technology

[0002] Multiple speakers speaking separately or simultaneously in the same space, generating mixed speech, is a common scenario. Tasks such as content understanding of mixed speech and speech separation between different speakers have always been very challenging problems in the field of speech. These problems can be solved by utilizing neural network models in AI technology.

[0003] For example, the speaker diarization task primarily addresses the "who speaks when" problem. In this task, after processing mixed speech using a neural network model, the position of each speaker's voice within the mixed speech can be obtained, which is essentially the timestamp of each speaker's voice. Similarly, in the target speaker extraction task, after processing mixed speech using a neural network model, the target speaker's voice can be extracted from the mixed speech.

[0004] For the two tasks mentioned above, the relevant technologies design specific neural network models for different tasks, which leads to a high design cost for speech processing. Summary of the Invention

[0005] This application provides a speech processing method and related equipment that can simultaneously support two types of tasks: speakerprint segmentation and clustering, and target speaker speech extraction, which helps to reduce the design cost of speech processing.

[0006] A first aspect of this application provides a speech processing method, the method comprising:

[0007] When a user needs to perform speakerprint segmentation and clustering and target speaker extraction on mixed speech, the user-input mixed speech to be processed can be obtained first. The mixed speech contains the speech of the target object and the speech of other objects besides the target object. In other words, the mixed speech is a superposition of the speech corresponding to the target and the speech of other objects. After obtaining the mixed speech, the reference speech of the target object can also be obtained. Therefore, the reference speech corresponding to the target can be used to process the mixed speech.

[0008] After obtaining the mixed speech and the reference speech of the target object, these can be input into the target model. The first model in the target model processes the intermediate outputs of the mixed speech, the reference speech, and the second model, resulting in its intermediate and final outputs. Similarly, the second model in the target model processes the intermediate outputs of the mixed speech and the first model, resulting in its intermediate and final outputs. In this way, the final output of the first model can be used to obtain the target object's speech contained within the mixed speech, and the final output of the second model can be used to determine the position of the target object's speech within the mixed speech, i.e., its timestamp. This completes the speaker recognition and clustering of the mixed speech and the extraction of the target speaker's speech.

[0009] As can be seen from the above method, when processing mixed speech, the first step is to obtain the mixed speech and the reference speech of the target object. The mixed speech includes the speech of the target object and the speech of other objects besides the target object. Next, the mixed speech and the reference speech can be input into the target model. The first model in the target model processes the intermediate outputs of the mixed speech, the reference speech, and the second model to obtain the intermediate output and final output of the first model. Simultaneously, the second model in the target model processes the intermediate outputs of the mixed speech and the first model to obtain the intermediate output and the final output of the second model. Finally, the final output of the first model can be used to obtain the speech of the target object, and the final output of the second model can be used to determine the position of the target object's speech within the mixed speech. Based on the aforementioned process, it can be seen that the first and second models, as two branches of the target model, achieve cross-fusion of intermediate outputs during their respective processing of the mixed speech, thus jointly completing the tasks of speaker segmentation and clustering for mixed speech, and target speaker extraction. As can be seen, the new model framework provided in this application, namely the target model, can simultaneously support two types of tasks: voiceprint segmentation and clustering, and target speaker speech extraction, which helps to reduce the design cost of speech processing.

[0010] In one possible implementation, processing the mixed speech, reference speech, and intermediate output of the second model to obtain the intermediate output and final output of the first model includes: performing a first processing on the mixed speech and reference speech to obtain the intermediate output of the first model; and performing a second processing on the intermediate output of the first model and the intermediate output of the second model to obtain the final output of the first model. In the aforementioned implementation, after obtaining the mixed speech and the reference speech of the target object, the first model can first perform the first processing on the mixed speech and the reference speech of the target object to obtain the intermediate output of the first model, and then send the intermediate output of the first model to the second model. Subsequently, the first model can also receive the intermediate output of the second model, and then perform the second processing on the intermediate output of the first model and the intermediate output of the second model to obtain the final output of the first model.

[0011] In one possible implementation, processing the mixed speech and the intermediate output of the first model to obtain the intermediate output of the second model and the final output of the second model includes: performing a third processing on the mixed speech and the intermediate output of the first model to obtain the intermediate output of the second model; and performing a fourth processing on the intermediate output of the second model to obtain the final output of the second model. In the aforementioned implementation, after obtaining the intermediate output of the mixed speech and the first model, the second model can first perform the third processing on the mixed speech and the intermediate output of the first model to obtain the intermediate output of the second model, and then send the intermediate output of the second model to the first model. Subsequently, the second model can perform the fourth processing on the intermediate output of the second model to obtain the final output of the second model.

[0012] In one possible implementation, the first process includes at least one of the following: encoding and recurrent neural network-based processing; the second process includes at least one of the following: concatenation, recurrent neural network-based processing, mask prediction, and decoding. In the aforementioned implementation, the first model in the target model includes: a speaker encoder, an extraction encoder, a first dual-path recurrent neural network, a first concatenation module, a second dual-path recurrent neural network, a masking module, a multiplication module, and an extraction decoder. Then, after obtaining the mixed speech, the extraction encoder can encode the mixed speech to obtain a first feature of the mixed speech, and send the first feature to the first dual-path recurrent neural network. After obtaining the reference speech of the target object, the speaker encoder can encode the reference speech of the target object to obtain features of the reference speech of the target object, and send the features of the reference speech of the target object to the first dual-path recurrent neural network. After receiving the first feature of the mixed speech and the feature of the reference speech, the first dual-path recurrent neural network performs a series of processes on the first feature of the mixed speech and the feature of the reference speech to obtain the second feature of the mixed speech (i.e., the intermediate output of the first model). This second feature is then sent to the first and second concatenation modules. Notably, the first concatenation module can also receive the seventh feature of the mixed speech from the second bidirectional long short-term memory network. After obtaining the second and seventh features of the mixed speech, the first concatenation module concatenates them to obtain the third feature of the mixed speech, which is then sent to the second dual-path recurrent neural network. After obtaining the third feature, the second dual-path recurrent neural network performs a series of processes on the third feature to obtain the fourth feature of the mixed speech, which is then sent to the masking module. Based on the fourth feature, the masking module predicts the temporal mask of the mixed speech and sends the temporal mask to the multiplication module. After obtaining the time-domain mask, the multiplication module multiplies the first feature of the mixed speech with the time-domain mask. This process removes the features of other speech objects from the first feature of the mixed speech, yielding the features of the target speech object. These target speech features are then sent to the extraction decoder. Once the target speech features are obtained, the extraction decoder decodes them to obtain the target speech object, which is the final output of the first model.

[0013] In one possible implementation, the third processing includes at least one of the following: processing based on a bidirectional long short-term memory network (BSSN), and the fourth processing includes at least one of the following: concatenation, processing based on a bidirectional BSSN, and linear computation. In the aforementioned implementation, the second model in the target model includes: a first bidirectional BSSN network, a second concatenation module, a second bidirectional BSSN network, and a linear module. After obtaining the mixed speech, the first bidirectional BSSN network can perform a series of processes on the mixed speech to obtain a fifth feature of the mixed speech, and send the fifth feature of the mixed speech to the second concatenation module. Notably, the second concatenation module can also receive a second feature of the mixed speech from a first dual-path recurrent neural network. After obtaining the second and fifth features of the mixed speech, the second concatenation module can concatenate the second and fifth features of the mixed speech to obtain a sixth feature of the mixed speech, and send the sixth feature of the mixed speech to the second bidirectional BSSN network. After obtaining the sixth feature of the mixed speech, the second bidirectional long short-term memory network performs a series of processes on the sixth feature to obtain the seventh feature of the mixed speech, and sends the seventh feature to the first concatenation module and the linear module. After obtaining the seventh feature of the mixed speech, the linear module performs linear operations on the seventh feature to obtain the probability that each speech frame in the mixed speech belongs to the target object, which is the final output of the second model.

[0014] In one possible implementation, the method further includes: upsampling the intermediate output of the second model using a third model to obtain an upsampled intermediate output of the second model; processing the mixed speech, reference speech, and the intermediate output of the second model to obtain the intermediate output and final output of the first model includes: processing the upsampled intermediate output of the mixed speech, reference speech, and the second model to obtain the intermediate output and final output of the first model. In the aforementioned implementation, the target model includes a third model positioned between the first and second models. Therefore, after the second model sends its intermediate output to the third model, the third model upsamples the intermediate output of the second model to obtain an upsampled intermediate output of the second model, which is then sent to the first model. In this way, the first model can process both its intermediate output and the upsampled intermediate output of the second model to obtain its final output.

[0015] In one possible implementation, the method further includes: downsampling the intermediate output of the first model using a third model to obtain a downsampled intermediate output of the first model; and processing the mixed speech and the intermediate output of the first model to obtain the intermediate output of the second model. The final output of the second model includes processing the mixed speech and the downsampled intermediate output of the first model to obtain the intermediate output of the second model and the final output of the second model. In the aforementioned implementation, the target model includes a third model positioned between the first and second models. Therefore, after the first model sends its intermediate output to the third model, the third model downsamples the intermediate output of the first model to obtain a downsampled intermediate output of the first model, which is then sent to the second model. In this way, the second model can process the mixed speech and the downsampled intermediate output of the first model to obtain the intermediate output of the second model.

[0016] In one possible implementation, obtaining the reference speech of the target object includes: obtaining information about the target object, which includes at least one of the following: an image of the target object, text of the target object, and an identifier of the target object; and obtaining the reference speech of the target object corresponding to the information from a pre-set speech library. In the aforementioned implementation, if the user specifies a target object, the user-inputted information about the target object can be obtained. This information includes at least one of the following: an image of the target object, text of the target object, and an identifier of the target object. Then, a pre-set speech library can be opened. This speech library contains not only information about multiple objects but also the speech entries registered by multiple objects in the speech library. It is evident that there is a one-to-one correspondence between the information of multiple objects and the multiple registered speech entries. Next, the speech library is traversed using the information of the target object as an index to find the speech corresponding to the information of the target object, and this speech is used as the reference speech of the target object.

[0017] In one possible implementation, obtaining the reference speech of the target object includes: dividing the mixed speech into multiple speech segments, each of which contains a target speech segment; if the target speech segment corresponds to the same object, the object is identified as the target object, and the target speech segment is identified as the reference speech of the target object. In the aforementioned implementation, if the user does not specify a target object, the mixed speech can first be divided into multiple speech segments of equal length, forming a speech segment set. For a specific speech segment in this set, that speech segment can be called the target speech segment. The target speech segment can then be further divided into several sub-speech segments of equal length, and these sub-speech segments can be calculated to determine whether they belong to the same object. If so, that object is identified as the target object, and the target speech segment is identified as the reference speech of the target object.

[0018] A second aspect of this application provides a model training method, comprising: acquiring mixed speech and reference speech of a target object, wherein the mixed speech includes speech of the target object and speech of other objects besides the target object; processing the mixed speech, reference speech, and intermediate output of a second training model through a first training model to obtain intermediate output and final output of the first training model, wherein the final output of the first training model is used to acquire speech of the target object; processing the mixed speech and intermediate output of the first training model through a second training model to obtain intermediate output and final output of the second training model, wherein the final output of the second training model is used to determine the position of speech of the target object in the mixed speech; and training the first training model and the second training model based on the final output of the first training model and the final output of the second training model to obtain a first model and a second model.

[0019] The target model trained using the above method possesses speech processing capabilities (i.e., speaker segmentation and clustering, and target speaker extraction). Specifically, when processing mixed speech, the mixed speech and the reference speech of the target object are first obtained. The mixed speech includes the speech of the target object and the speech of other objects besides the target object. Next, the mixed speech and the reference speech are input into the target model. The first model in the target model processes the intermediate outputs of the mixed speech, the reference speech, and the second model to obtain the intermediate output and final output of the first model. Simultaneously, the second model in the target model processes the intermediate outputs of the mixed speech and the first model to obtain the intermediate output and the final output of the second model. Finally, the final output of the first model can be used to obtain the speech of the target object, and the final output of the second model can be used to determine the position of the target object's speech within the mixed speech. Based on the aforementioned process, it can be seen that the first and second models, as two branches of the target model, achieve cross-fusion of intermediate outputs during the processing of mixed speech, thus jointly completing the tasks of speaker segmentation and clustering for mixed speech and target speaker extraction. As can be seen, the new model framework provided in this application, namely the target model, can simultaneously support two types of tasks: voiceprint segmentation and clustering, and target speaker speech extraction, which helps to reduce the design cost of speech processing.

[0020] In one possible implementation, processing the mixed speech, the reference speech, and the intermediate output of the second training model to obtain the intermediate output of the first training model and the final output of the first training model includes: performing a first processing on the mixed speech and the reference speech to obtain the intermediate output of the first training model; and performing a second processing on the intermediate output of the first training model and the intermediate output of the second training model to obtain the final output of the first training model.

[0021] In one possible implementation, processing the mixed speech and the intermediate output of the first training model to obtain the intermediate output of the second training model and the final output of the second training model includes: performing a third processing on the mixed speech and the intermediate output of the first training model to obtain the intermediate output of the second training model; and performing a fourth processing on the intermediate output of the second training model to obtain the final output of the second training model.

[0022] In one possible implementation, the first process includes at least one of the following: encoding and recurrent neural network-based processing, and the second process includes at least one of the following: concatenation, recurrent neural network-based processing, mask prediction, and decoding.

[0023] In one possible implementation, the third processing includes at least one of the following: processing based on a bidirectional long short-term memory network, and the fourth processing includes at least one of the following: splicing, processing based on a bidirectional long short-term memory network, and linear computation.

[0024] In one possible implementation, the method further includes: upsampling the intermediate output of the second training model using the third training model to obtain the upsampled intermediate output of the second training model; and processing the mixed speech, reference speech, and intermediate output of the second training model to obtain the intermediate output of the first training model and the final output of the first training model, which includes processing the upsampled intermediate output of the mixed speech, reference speech, and second training model to obtain the intermediate output of the first training model and the final output of the first training model.

[0025] In one possible implementation, the method further includes: downsampling the intermediate output of the first training model using the third training model to obtain the downsampled intermediate output of the first training model; and processing the mixed speech and the intermediate output of the first training model to obtain the intermediate output of the second training model and the final output of the second training model, which includes: processing the mixed speech and the downsampled intermediate output of the first training model to obtain the intermediate output of the second training model and the final output of the second training model.

[0026] In one possible implementation, obtaining the reference speech of the target object includes: obtaining information about the target object, the information including at least one of the following: an image of the target object, text of the target object, and an identifier of the target object; and obtaining the reference speech of the target object corresponding to the information from a pre-set speech library.

[0027] In one possible implementation, obtaining the reference speech of the target object includes: dividing the mixed speech into multiple speech segments, each of which contains the target speech segment; if the target speech segment corresponds to the same object, identifying the object as the target object, and identifying the target speech segment as the reference speech of the target object.

[0028] In one possible implementation, training the first and second training models based on their final outputs to obtain the first and second models includes: obtaining a target loss based on the final outputs of the first and second training models, the target loss being used to indicate the difference between the final output of the first training model and its true output, and the difference between the final output of the second training model and its true output; and updating the parameters of the first and second training models based on the target loss until the model training conditions are met, thus obtaining the first and second models.

[0029] In one possible implementation, the method further includes: updating the parameters of the first model to be trained based on the target loss until the model training conditions are met, thereby obtaining the third model.

[0030] A third aspect of this application provides a speech processing apparatus, comprising: an acquisition module for acquiring mixed speech and reference speech of a target object, the mixed speech including speech of the target object and speech of other objects besides the target object; a first processing module for processing the mixed speech, reference speech, and intermediate output of a second model through a first model to obtain intermediate output and final output of the first model, the final output of the first model being used to acquire the speech of the target object; and a second processing module for processing the mixed speech and intermediate output of the first model through a second model to obtain intermediate output and final output of the second model, the final output of the second model being used to determine the position of the speech of the target object in the mixed speech.

[0031] As can be seen from the above device, when mixed speech needs to be processed, the mixed speech and the reference speech of the target object can be obtained first. The mixed speech includes the speech of the target object and the speech of other objects besides the target object. Next, the mixed speech and the reference speech can be input into the target model. Then, the first model in the target model can process the intermediate outputs of the mixed speech, the reference speech, and the second model to obtain the intermediate output and the final output of the first model. Simultaneously, the second model in the target model can process the intermediate outputs of the mixed speech and the first model to obtain the intermediate output and the final output of the second model. Finally, the final output of the first model can be used to obtain the speech of the target object, and the final output of the second model can be used to determine the position of the target object's speech in the mixed speech. Based on the aforementioned process, it can be seen that the first model and the second model, as two branches in the target model, achieve cross-fusion of intermediate outputs during the processing of the mixed speech, thus jointly completing the two tasks of speaker segmentation and clustering for mixed speech and target speaker extraction. As can be seen, the new model framework provided in this application, namely the target model, can simultaneously support two types of tasks: voiceprint segmentation and clustering, and target speaker speech extraction, which helps to reduce the design cost of speech processing.

[0032] In one possible implementation, the first processing module is configured to: perform a first processing on the mixed speech and the reference speech to obtain an intermediate output of the first model; and perform a second processing on the intermediate output of the first model and the intermediate output of the second model to obtain the final output of the first model.

[0033] In one possible implementation, the second processing module is used to: perform a third processing on the intermediate output of the mixed speech and the first model to obtain an intermediate output of the second model; and perform a fourth processing on the intermediate output of the second model to obtain the final output of the second model.

[0034] In one possible implementation, the first process includes at least one of the following: encoding and recurrent neural network-based processing, and the second process includes at least one of the following: concatenation, recurrent neural network-based processing, mask prediction, and decoding.

[0035] In one possible implementation, the third processing includes at least one of the following: processing based on a bidirectional long short-term memory network, and the fourth processing includes at least one of the following: splicing, processing based on a bidirectional long short-term memory network, and linear computation.

[0036] In one possible implementation, the device further includes: an upsampling module for upsampling the intermediate output of the second model using a third model to obtain an upsampled intermediate output of the second model; and a first processing module for processing the mixed speech, the reference speech, and the upsampled intermediate output of the second model to obtain an intermediate output of the first model and a final output of the first model.

[0037] In one possible implementation, the device further includes: a downsampling module for downsampling the intermediate output of the first model using a third model to obtain a downsampled intermediate output of the first model; and a second processing module for processing the mixed speech and the downsampled intermediate output of the first model to obtain an intermediate output of the second model and a final output of the second model.

[0038] In one possible implementation, the acquisition module is configured to: acquire information about a target object, the information including at least one of the following: an image of the target object, text of the target object, and an identifier of the target object; and acquire reference speech of the target object corresponding to the information from a pre-set speech library.

[0039] In one possible implementation, the acquisition module is configured to: divide the mixed speech into multiple speech segments, the multiple speech segments containing a target speech segment; if the target speech segment corresponds to the same object, identify the object as the target object, and identify the target speech segment as the reference speech of the target object.

[0040] A fourth aspect of this application provides a model training apparatus, comprising: an acquisition module for acquiring mixed speech and reference speech of a target object, the mixed speech including speech of the target object and speech of other objects besides the target object; a first processing module for processing the mixed speech, reference speech, and intermediate output of a second training model through a first training model to obtain intermediate output and final output of the first training model, the final output of the first training model being used to acquire speech of the target object; a second processing module for processing the mixed speech and intermediate output of the first training model through a second training model to obtain intermediate output and final output of the second training model, the final output of the second training model being used to determine the position of speech of the target object in the mixed speech; and a training module for training the first training model and the second training model based on the final output of the first training model and the final output of the second training model to obtain a first model and a second model.

[0041] The target model trained by the aforementioned device possesses speech processing capabilities (i.e., speaker segmentation and clustering, and target speaker extraction). Specifically, when processing mixed speech, the mixed speech and the reference speech of the target object are first obtained. The mixed speech includes the speech of the target object and the speech of other objects besides the target object. Next, the mixed speech and the reference speech are input into the target model. The first model in the target model processes the intermediate outputs of the mixed speech, the reference speech, and the second model to obtain the intermediate output and final output of the first model. Simultaneously, the second model in the target model processes the intermediate outputs of the mixed speech and the first model to obtain the intermediate output and the final output of the second model. Finally, the final output of the first model can be used to obtain the speech of the target object, and the final output of the second model can be used to determine the position of the target object's speech within the mixed speech. Based on the aforementioned process, it can be seen that the first and second models, as two branches of the target model, achieve cross-fusion of intermediate outputs during the processing of mixed speech, thus jointly completing the tasks of speaker segmentation and clustering for mixed speech and target speaker extraction. As can be seen, the new model framework provided in this application, namely the target model, can simultaneously support two types of tasks: voiceprint segmentation and clustering, and target speaker speech extraction, which helps to reduce the design cost of speech processing.

[0042] In one possible implementation, the first processing module is configured to: perform a first processing on the mixed speech and the reference speech to obtain an intermediate output of a first model to be trained; and perform a second processing on the intermediate output of the first model to be trained and the intermediate output of a second model to be trained to obtain the final output of the first model to be trained.

[0043] In one possible implementation, the second processing module is used to: perform a third processing on the intermediate output of the mixed speech and the first model to be trained to obtain an intermediate output of the second model to be trained; and perform a fourth processing on the intermediate output of the second model to be trained to obtain the final output of the second model to be trained.

[0044] In one possible implementation, the first process includes at least one of the following: encoding and recurrent neural network-based processing, and the second process includes at least one of the following: concatenation, recurrent neural network-based processing, mask prediction, and decoding.

[0045] In one possible implementation, the third processing includes at least one of the following: processing based on a bidirectional long short-term memory network, and the fourth processing includes at least one of the following: splicing, processing based on a bidirectional long short-term memory network, and linear computation.

[0046] In one possible implementation, the device further includes: an upsampling module for upsampling the intermediate output of the second training model using the third training model to obtain the upsampled intermediate output of the second training model; and a first processing module for processing the mixed speech, the reference speech, and the upsampled intermediate output of the second training model to obtain the intermediate output of the first training model and the final output of the first training model.

[0047] In one possible implementation, the device further includes: a downsampling module for downsampling the intermediate output of the first training model using a third training model to obtain a downsampled intermediate output of the first training model; and a second processing module for processing the mixed speech and the downsampled intermediate output of the first training model to obtain an intermediate output of the second training model and a final output of the second training model.

[0048] In one possible implementation, the acquisition module is configured to: acquire information about a target object, the information including at least one of the following: an image of the target object, text of the target object, and an identifier of the target object; and acquire reference speech of the target object corresponding to the information from a pre-set speech library.

[0049] In one possible implementation, the acquisition module is configured to: divide the mixed speech into multiple speech segments, the multiple speech segments containing a target speech segment; if the target speech segment corresponds to the same object, identify the object as the target object, and identify the target speech segment as the reference speech of the target object.

[0050] In one possible implementation, the training module is configured to: obtain a target loss based on the final output of the first model to be trained (i.e., the aforementioned first prediction processing result of the mixed speech), the real output of the first model to be trained (i.e., the aforementioned first real processing result of the mixed speech), the final output of the second model to be trained (i.e., the aforementioned second prediction processing result of the mixed speech), and the real output of the second model to be trained (i.e., the aforementioned second real processing result of the mixed speech); the target loss is used to indicate the difference between the final output of the first model to be trained and the real output of the first model to be trained, and the difference between the final output of the second model to be trained and the real output of the second model to be trained; and update the parameters of the first model to be trained and the parameters of the second model to be trained based on the target loss until the model training conditions are met, thereby obtaining the first model and the second model.

[0051] In one possible implementation, the training module is also used to update the parameters of the first model to be trained based on the target loss until the model training conditions are met, thus obtaining the third model.

[0052] A fifth aspect of this application provides a voice processing apparatus, which includes a memory and a processor; the memory stores code, and the processor is configured to execute the code, wherein when the code is executed, the voice processing apparatus performs the method described in the first aspect or any possible implementation thereof.

[0053] A sixth aspect of this application provides a model training apparatus, which includes a memory and a processor; the memory stores code, and the processor is configured to execute the code. When the code is executed, the model training apparatus performs the method described in the second aspect or any possible implementation thereof.

[0054] A seventh aspect of this application provides a circuit system including a processing circuit configured to perform the method described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.

[0055] An eighth aspect of this application provides a chip system including a processor for calling a computer program or computer instructions stored in a memory to cause the processor to perform the method as described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.

[0056] In one possible implementation, the processor is coupled to the memory via an interface.

[0057] In one possible implementation, the chip system also includes a memory that stores computer programs or computer instructions.

[0058] A ninth aspect of this application provides a computer storage medium storing a computer program that, when executed by a computer, causes the computer to perform the method as described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.

[0059] A tenth aspect of this application provides a computer program product storing instructions that, when executed by a computer, cause the computer to perform the method as described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.

[0060] In this embodiment, when mixed speech needs to be processed, the mixed speech and the reference speech of the target object can be obtained first. The mixed speech includes the speech of the target object and the speech of other objects besides the target object. Next, the mixed speech and the reference speech can be input into the target model. Then, the first model in the target model can process the intermediate outputs of the mixed speech, the reference speech, and the second model to obtain the intermediate output and the final output of the first model. Simultaneously, the second model in the target model can process the intermediate outputs of the mixed speech and the first model to obtain the intermediate output and the final output of the second model. Finally, the final output of the first model can be used to obtain the speech of the target object, and the final output of the second model can be used to determine the position of the target object's speech in the mixed speech. Based on the aforementioned process, it can be seen that the first model and the second model, as two branches in the target model, achieve cross-fusion of intermediate outputs during the processing of the mixed speech, thereby jointly completing the two tasks of speaker segmentation and clustering for mixed speech and target speaker extraction. As can be seen, the new model framework provided in this application, namely the target model, can simultaneously support two types of tasks: voiceprint segmentation and clustering, and target speaker speech extraction, which helps to reduce the design cost of speech processing. Attached Figure Description

[0061] Figure 1 A structural diagram illustrating the main framework of artificial intelligence;

[0062] Figure 2a A schematic diagram of the structure of the voice processing system provided in the embodiments of this application;

[0063] Figure 2b This is another structural schematic diagram of the voice processing system provided in the embodiments of this application;

[0064] Figure 2c A schematic diagram of a speech processing device provided in an embodiment of this application;

[0065] Figure 3 A schematic diagram of the system 100 architecture provided in the embodiments of this application;

[0066] Figure 4 A schematic diagram of the structure of the target model provided in the embodiments of this application;

[0067] Figure 5 A schematic flowchart of the speech processing method provided in the embodiments of this application;

[0068] Figure 6 Another structural schematic diagram of the target model provided in the embodiments of this application;

[0069] Figure 7 Another structural schematic diagram of the target model provided in the embodiments of this application;

[0070] Figure 8 A schematic flowchart of the model training method provided in the embodiments of this application;

[0071] Figure 9 A schematic diagram of the structure of the voice processing device provided in the embodiments of this application;

[0072] Figure 10 A schematic diagram of the structure of the model training apparatus provided in the embodiments of this application;

[0073] Figure 11 A schematic diagram of the structure of the execution device provided in the embodiments of this application;

[0074] Figure 12 A schematic diagram of the structure of the training device provided in the embodiments of this application;

[0075] Figure 13 This is a schematic diagram of the structure of a chip provided in an embodiment of this application. Detailed Implementation

[0076] This application provides a speech processing method and related equipment that can simultaneously support two types of tasks: speakerprint segmentation and clustering, and target speaker speech extraction, which helps to reduce the design cost of speech processing.

[0077] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0078] Multiple speakers speaking separately or simultaneously within the same space, generating mixed speech, is a common scenario. Examples include audio conferences where different speakers speak at different times or simultaneously. Content understanding of mixed speech and speech separation between different speakers have always been very challenging problems in the field of speech recognition. These problems can be addressed by utilizing neural network models in AI technology.

[0079] For example, speaker diarization primarily addresses the "who speaks when" problem. In a voice conference, it's necessary to determine which speaker spoke at which time to generate meeting minutes. In this task, by using a neural network model to process the mixed speech, the position of each speaker's voice within the mixed speech can be obtained, which is essentially the timestamp corresponding to each speaker's voice.

[0080] For example, the target speaker extraction task primarily addresses how to extract the speech of a target speaker. In a voice conference, for instance, it's necessary to extract the speech of a specific speaker to identify what they are saying. In this task, a neural network model is used to process mixed speech, allowing the extraction of the target speaker's voice from the mixed speech.

[0081] For the two tasks mentioned above, the relevant technologies design specific neural network models for different tasks, which leads to a high design cost for speech processing.

[0082] To address the aforementioned problems, this application provides an image processing method that can be implemented in conjunction with artificial intelligence (AI) technology. AI technology is a discipline that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence. AI technology achieves optimal results by perceiving the environment, acquiring knowledge, and using that knowledge. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can react in a way similar to human intelligence. Using artificial intelligence for data processing is a common application of AI.

[0083] First, the overall workflow of the artificial intelligence system is described; please refer to [link / reference]. Figure 1 , Figure 1 This is a structural diagram illustrating the main framework of artificial intelligence. The following explanation of the AI ​​framework is based on two dimensions: the "Intelligent Information Chain" (horizontal axis) and the "IT Value Chain" (vertical axis). The "Intelligent Information Chain" reflects a series of processes from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom." The "IT Value Chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence and information (provided and processed by technology) to the industrial ecosystem of the system.

[0084] (1) Infrastructure

[0085] Infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. This communication occurs through sensors; computing power is provided by intelligent chips (hardware acceleration chips such as CPUs, NPUs, GPUs, ASICs, and FPGAs); and the basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0086] (2) Data

[0087] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, as well as IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0088] (3) Data processing

[0089] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0090] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data by symbolizing and formalizing it.

[0091] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0092] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0093] (4) General ability

[0094] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0095] (5) Smart Products and Industry Applications

[0096] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Their application areas mainly include: intelligent terminals, intelligent transportation, intelligent healthcare, autonomous driving, smart cities, etc.

[0097] The following sections will introduce several application scenarios for this application.

[0098] Figure 2a This is a schematic diagram of a voice processing system provided in an embodiment of this application. The voice processing system includes a user device and a data processing device. The user device includes smart terminals such as mobile phones, personal computers, or information processing centers. The user device is the initiator of voice processing; as the initiator of voice processing requests, requests are typically initiated by the user through the user device.

[0099] The aforementioned data processing equipment can be cloud servers, network servers, application servers, management servers, or other devices or servers with data processing capabilities. The data processing equipment receives text processing requests from smart terminals through an interactive interface, and then performs text processing through a storage device for storing data and a processor for data processing, employing methods such as machine learning, deep learning, search, reasoning, and decision-making. The storage device in the data processing equipment can be a general term, including local storage and a database storing historical data. The database can be located on the data processing equipment or on other network servers.

[0100] exist Figure 2a In the illustrated speech processing system, a user equipment (UE) can receive user instructions. For example, the UE can acquire a voice input / selection by the user and then send a request to a data processing device. This causes the data processing device to perform speech processing on the image obtained by the UE, thereby obtaining a corresponding processing result for the voice. For instance, the UE can acquire a mixed voice input by the user and then send a mixed voice processing request to the data processing device. This causes the data processing device to perform a series of processes on the mixed image (e.g., speakerprint segmentation and clustering, and target speaker voice extraction, etc.), thereby obtaining a mixed voice processing result. This result can be used to acquire the target speaker's voice and determine the position of the target speaker's voice within the mixed voice.

[0101] exist Figure 2a In this context, the data processing device can execute the voice processing method of the embodiments of this application.

[0102] Figure 2b This is another schematic diagram of the speech processing system provided in the embodiments of this application. Figure 2bIn this context, the user equipment (UE) directly functions as a data processing device. This UE can directly acquire input from the user and process it directly through its own hardware. The specific process is similar to... Figure 2a Similar to the description above, it will not be repeated here.

[0103] exist Figure 2b In the speech processing system shown, the user equipment can receive user instructions. For example, the user equipment can acquire a mixed speech input by the user, and then perform a series of processing on the mixed image (e.g., speaker segmentation and clustering, and target speaker speech extraction, etc.) to obtain the processing result of the mixed speech. The processing result of the mixed speech can be used to acquire the speech of the target speaker and determine the position of the target speaker's speech in the mixed speech.

[0104] exist Figure 2b In this context, the user equipment itself can execute the voice processing method of the embodiments of this application.

[0105] Figure 2c This is a schematic diagram of a speech processing device provided in an embodiment of this application.

[0106] The above Figure 2a and Figure 2b The user equipment in the context can specifically be Figure 2c Local device 301 or local device 302 in the system. Figure 2a The data processing equipment in the middle can specifically be Figure 2c The execution device 210 in the process includes a data storage system 250 that can store the data to be processed by the execution device 210. The data storage system 250 can be integrated into the execution device 210 or set up in the cloud or on other network servers.

[0107] Figure 2a and Figure 2b The processor in the system can be trained on data using neural network models or other models (e.g., support vector machine-based models) for machine learning / deep learning, and then use the trained or learned models to perform speech processing applications on images to obtain the corresponding processing results.

[0108] Figure 3 A schematic diagram of the system 100 architecture provided in this application embodiment, in Figure 3 In the process, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with external devices. Users can input data to the I / O interface 112 through the client device 140. The input data in this embodiment may include various scheduled tasks, callable resources, and other parameters.

[0109] During the preprocessing of input data by the execution device 110, or during the calculation module 111 of the execution device 110 performing calculations and other related processing (such as implementing the neural network function in this application), the execution device 110 may call data, code, etc. in the data storage system 150 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 150.

[0110] Finally, I / O interface 112 returns the processing result to client device 140, thereby providing it to the user.

[0111] It is worth noting that the training device 120 can generate corresponding target models / rules based on different training data for different objectives or tasks. These target models / rules can then be used to achieve the aforementioned objectives or complete the aforementioned tasks, thereby providing the user with the required results. The training data can be stored in the database 130 and originates from training samples collected by the data acquisition device 160.

[0112] exist Figure 3 In the scenario shown, the user can manually provide input data, which can be done through the interface provided by I / O interface 112. Alternatively, the client device 140 can automatically send input data to I / O interface 112. If user authorization is required for the client device 140 to automatically send input data, the user can set the corresponding permissions in the client device 140. The user can view the output results of the execution device 110 on the client device 140, which can be presented in various forms such as display, sound, or animation. The client device 140 can also act as a data acquisition terminal, collecting the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130. Alternatively, data can be collected directly from the I / O interface 112 without going through the client device 140, using the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130.

[0113] It is worth noting that, Figure 3 This is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in Figure 3 In this context, the data storage system 150 is an external memory relative to the execution device 110. However, in other cases, the data storage system 150 can also be placed within the execution device 110. For example... Figure 3 As shown, a neural network can be trained using training device 120.

[0114] This application also provides a chip including a neural network processor (NPU). This chip can be configured as follows: Figure 3 The execution device 110 shown is used to perform the calculations of the calculation module 111. This chip can also be located in, for example... Figure 3 The training device 120 shown is used to complete the training work of the training device 120 and output the target model / rules.

[0115] The Neural Processing Unit (NPU) is a coprocessor mounted on the main central processing unit (CPU) (host CPU), where tasks are assigned by the CPU. The core of the NPU is the computation circuitry, which is controlled by a controller to retrieve data from memory (weight memory or input memory) and perform calculations.

[0116] In some implementations, the arithmetic circuitry includes multiple process engines (PEs). In some implementations, the arithmetic circuitry is a two-dimensional pulsating array. The arithmetic circuitry can also be a one-dimensional pulsating array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuitry is a general-purpose matrix processor.

[0117] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory and caches it in each PE (Process Equipment) of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory and performs matrix operations with matrix B. The partial or final result of the obtained matrix is ​​stored in the accumulator.

[0118] Vector computation units can further process the output of computational circuits, such as vector multiplication, vector addition, exponentiation, logarithmic operations, size comparisons, etc. For example, vector computation units can be used for computation in non-convolutional / non-FC layers of neural networks, such as pooling, batch normalization, and local response normalization.

[0119] In some implementations, the vector computation unit can store the processed output vector into a unified buffer. For example, the vector computation unit can apply a nonlinear function to the output of the arithmetic circuit, such as a vector of accumulated values, to generate activation values. In some implementations, the vector computation unit generates normalized values, merged values, or both. In some implementations, the processed output vector can be used as activation input to the arithmetic circuit, for example, for use in subsequent layers of a neural network.

[0120] The unified memory is used to store input data and output data.

[0121] The weight data is directly transferred from the external memory to the input memory and / or unified memory, stored in the weight memory, and stored in the unified memory to the external memory through the direct memory access controller (DMAC).

[0122] The bus interface unit (BIU) is used to enable interaction between the main CPU, DMAC, and instruction fetch memory via a bus.

[0123] The instruction fetch buffer, connected to the controller, is used to store the instructions used by the controller.

[0124] The controller is used to invoke instructions cached in the memory to control the operation of the computing accelerator.

[0125] Generally, the unified memory, input memory, weight memory, and instruction fetch memory are all on-chip memories, while external memory is memory outside the NPU. This external memory can be double data rate synchronous dynamic random access memory (DDRSDRAM), high bandwidth memory (HBM), or other readable and writable memories.

[0126] Since the embodiments of this application involve a large number of neural network applications, for ease of understanding, the relevant terms and concepts such as neural networks involved in the embodiments of this application will be introduced below.

[0127] (1) Neural Network

[0128] A neural network can be composed of neural units, which can be operational units that take xs and an intercept of 1 as inputs, and whose output can be:

[0129]

[0130] Where s = 1, 2, ..., n, where n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of this activation function can be used as the input of the next convolutional layer. The activation function can be the sigmoid function. A neural network is a network formed by connecting many of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0131] The work of each layer in a neural network can be described by the mathematical expression y = a(Wx + b). From a physical perspective, the work of each layer in a neural network can be understood as transforming the input space (the set of input vectors) to the output space (i.e., from the row space to the column space of a matrix) through five operations on the input space. These five operations include: 1. Dimensionality increase / decrease; 2. Magnification / scaling; 3. Rotation; 4. Translation; 5. "Bending". Operations 1, 2, and 3 are performed by Wx, operation 4 by +b, and operation 5 by a(). The term "space" is used here because the objects being classified are not individual things, but a class of things, and space refers to the set of all individuals of this class of things. Here, W is the weight vector, and each value in this vector represents the weight value of a neuron in that layer of the neural network. This vector W determines the spatial transformation from the input space to the output space mentioned above; that is, the weights W of each layer control how the space is transformed. The purpose of training a neural network is to ultimately obtain the weight matrix of all layers of the trained neural network (a weight matrix formed by the vectors W of many layers). Therefore, the training process of a neural network is essentially about learning how to control the transformation space, and more specifically, learning the weight matrix.

[0132] Because we want the output of the neural network to be as close as possible to the actual predicted value, we can compare the current network's prediction with the desired target value, and then update the weight vector of each layer of the neural network based on the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring the parameters of each layer in the neural network). For example, if the network's prediction is too high, the weight vector is adjusted to make it predict lower, and this adjustment is continued until the neural network can predict the actual target value. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value," which is the loss function or objective function. These are important equations used to measure the difference between the predicted value and the target value. Taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so training the neural network becomes the process of minimizing this loss as much as possible.

[0133] (2) Backpropagation algorithm

[0134] Neural networks can employ backpropagation (BP) to correct the parameters of the initial neural network model during training, thereby reducing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates error loss; this error loss information is then propagated back to update the parameters of the initial neural network model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the neural network model, such as the weight matrix.

[0135] The method provided in this application is described below from the perspectives of neural network training and neural network application.

[0136] The model training method provided in this application involves the processing of data sequences. Specifically, it can be applied to data training, machine learning, deep learning, and other methods. It performs symbolic and formal intelligent information modeling, extraction, preprocessing, and training on training data (e.g., mixed speech and reference speech of the target object in the model training method provided in this application), ultimately obtaining a trained neural network (e.g., the first model, second model, and third model in the model training method provided in this application). Furthermore, the speech processing method provided in this application can utilize the trained neural network to input input data (e.g., mixed speech and reference speech of the target object in the speech processing method provided in this application) into the trained neural network to obtain output data (e.g., the final output of the first model and the final output of the second model in the speech processing method provided in this application). It should be noted that the model training method and speech processing method provided in this application are inventions based on the same concept and can be understood as two parts of a system or two stages of an overall process: such as the model training stage and the model application stage.

[0137] The speech processing method provided in this application can be implemented using a target model. Figure 4 A schematic diagram of the structure of the target model provided in the embodiments of this application, such as Figure 4 As shown, the target model includes a first model and a second model. The first input terminals of the first model and the second model serve as the input terminals of the entire target model. The first output terminals of the first model and the second model serve as the output terminals of the entire target model. Within the target model, the second input terminal of the first model is connected to the second output terminal of the second model, and vice versa. To understand the workflow of the target model, the following section combines... Figure 5 The workflow of the target model will be introduced. Figure 5 A flowchart illustrating the speech processing method provided in this application embodiment is shown below. Figure 5 As shown, the method includes:

[0138] 501. Obtain the mixed speech and the reference speech of the target object. The mixed speech includes the speech of the target object and the speech of the other objects besides the target object.

[0139] In this embodiment, when a user needs to perform voiceprint segmentation and clustering and target speaker speech extraction on mixed speech, the user-input mixed speech to be processed can be obtained first. The mixed speech includes the speech of the target object (also known as the target speaker) and the speech of other objects (also known as other speakers). In other words, the mixed speech is speech formed by superimposing the speech corresponding to the target and the speech of other objects.

[0140] After obtaining the mixed speech, the reference speech of the target object can also be acquired. Therefore, the reference speech corresponding to the target object can be used to process the mixed speech. It should be noted that the speech of the target object included in the mixed speech can also be understood as the real-time speech of the target object. Similarly, the speech of other objects can also be understood as the real-time speech of the other objects.

[0141] Specifically, reference speech of the target object can be obtained through the following methods:

[0142] (1) If the user specifies a target object, the information of the target object input by the user can be obtained. The information of the target object includes at least one of the following: a segment of speech of the target object (this segment of speech is different from the speech of the target object contained in the mixed speech), an image of the target object (e.g., a portrait of the target object, etc.), text of the target object (e.g., some words and articles describing the target object, etc.), and the identifier of the target object (e.g., the identity document ID of the target object, etc.).

[0143] If the information of the target object is a segment of speech of the target object, this segment of speech can be directly used as the reference speech of the target object.

[0144] If the target object's information includes its image, text, and identifier, a pre-built voice library can be opened. This library contains not only information about multiple objects but also the voices registered by those objects. Therefore, there is a one-to-one correspondence between the information of the multiple objects and the registered voices. Next, the voice library is traversed using the target object's information as an index to find the voice corresponding to that information, and this voice is then used as the reference voice for the target object.

[0145] (2) If the user does not specify a target object, the mixed speech can first be divided into multiple speech segments of equal length to form a speech segment set. A specific speech segment in this set can be called the target speech segment. The target speech segment can then be further divided into several sub-speech segments of equal length, and a speaker verification algorithm can be used to calculate whether these sub-speech segments belong to the same object. If so, this object is identified as the target object, and the target speech segment is identified as the reference speech for the target object.

[0146] For example, for mixed speech, it can be divided into blocks of length L, A1, A2, A3, ... AM, according to temporal sequence. These blocks of length L form a set A. For block A1, A1 can be further divided into three equal parts: A11, A12, and A13. Then, a speaker verification model can be used to calculate whether {A11, A12, A13} belong to the same speaker. If so, the speaker can be identified as the target speaker, and A1 can be designated as the reference speech for the target speaker.

[0147] It should be understood that in this embodiment, the number of the remaining objects can be one or more, and there is no limit to the number of the remaining objects.

[0148] It should also be understood that in this embodiment, the information directly used as an index can be the image, text, and identifier of the target object. Of course, the information directly used as an index can be only the identifier of the target object. So, if the information of the target object input by the user is the image and text of the target object, the image and text of the target object can be converted into the identifier of the target object first, and then the identifier of the target object can be used to index the reference speech of the target object.

[0149] 502. The intermediate outputs of the mixed speech, reference speech, and second model are processed by the first model to obtain the intermediate output and final output of the first model. The final output of the first model is used to obtain the speech of the target object.

[0150] 503. The intermediate outputs of the mixed speech and the first model are processed by the second model to obtain the intermediate output of the second model and the final output of the second model. The final output of the second model is used to determine the position of the target object's speech in the mixed speech.

[0151] After obtaining the mixed speech and the reference speech of the target object, these can be input into the target model. The first model in the target model then processes the intermediate outputs of the mixed speech, the reference speech, and the second model, resulting in the intermediate output and final output of the first model. Similarly, the second model in the target model processes the intermediate outputs of the mixed speech and the first model, resulting in its own intermediate output and final output. It should be noted that the final output of the first model can also be called the first processing result of the mixed speech, which can be used to extract the speech of the target object contained within the mixed speech. The final output of the second model can also be called the second processing result of the mixed speech, which can be used to determine the position of the target object's speech within the mixed speech, i.e., the timestamp corresponding to the target object's speech within the mixed speech. This completes the speakerprint segmentation and clustering of the mixed speech and the extraction of the target speaker's speech.

[0152] Specifically, the first model and the second model can obtain intermediate and final outputs in the following ways:

[0153] After obtaining the mixed speech and the reference speech of the target object, the first model can first process the mixed speech and the reference speech of the target object to obtain the intermediate output of the first model, and then send the intermediate output of the first model to the second model. Subsequently, the first model can also receive the intermediate output of the second model, and then process the intermediate outputs of the first model and the intermediate outputs of the second model to obtain the final output of the first model.

[0154] After obtaining the mixed speech and the intermediate output of the first model, the second model can first perform a third processing on the mixed speech and the intermediate output of the first model to obtain the intermediate output of the second model, and then send the intermediate output of the second model to the first model. Subsequently, the second model can perform a fourth processing on its intermediate output to obtain the final output of the second model.

[0155] More specifically, the first model and the second model can also obtain intermediate and final outputs in the following ways:

[0156] like Figure 6 As shown ( Figure 6(This is another structural diagram of the target model provided in an embodiment of this application). The first model in the target model includes: a speaker encoder, an extraction encoder, a first dual-path recurrent neural network (DPRNN), a first concatenation module, a second dual-path recurrent neural network, a masking module, a multiplication module, and an extraction decoder. The second model in the target model includes: a first bidirectional long short-term memory network (BSL-LSM), a second concatenation module, a second BSL-LSM, and a linear module.

[0157] After obtaining the mixed speech, the encoder encodes the mixed speech to obtain its first feature, which is then sent to the first dual-path recurrent neural network. After obtaining the reference speech of the target object, the speaker encoder encodes the reference speech to obtain its features, which are then sent to the first dual-path recurrent neural network. Upon receiving the first feature of the mixed speech and the features of the reference speech, the first dual-path recurrent neural network performs a series of processes on these features (e.g., feature extraction) to obtain the second feature of the mixed speech (i.e., the intermediate output of the first model). This second feature is then sent to the first and second concatenation modules. Notably, the first concatenation module can also receive the seventh feature of the mixed speech from the second bidirectional long short-term memory network. After obtaining the second and seventh features of the mixed speech, the first concatenation module concatenates them to obtain the third feature, which is then sent to the second dual-path recurrent neural network. After obtaining the third feature of the mixed speech, the second dual-path recurrent neural network performs a series of processes on the third feature to obtain the fourth feature of the mixed speech, which is then sent to the masking module. Based on the fourth feature, the masking module predicts the temporal mask of the mixed speech and sends it to the multiplication module. After obtaining the temporal mask, the multiplication module multiplies the first feature of the mixed speech with the temporal mask, thereby removing features of other objects from the first feature of the mixed speech to obtain the features of the target object's speech, which is then sent to the extraction decoder. Finally, the extraction decoder decodes the features of the target object's speech to obtain the speech of the target object, which is the final output of the first model.

[0158] After obtaining the mixed speech, the first bidirectional long short-term memory network performs a series of processes on the mixed speech (e.g., feature extraction) to obtain the fifth feature of the mixed speech. This fifth feature is then sent to the second concatenation module. Notably, the second concatenation module can also receive the second feature of the mixed speech from the first dual-path recurrent neural network. After obtaining both the second and fifth features of the mixed speech, the second concatenation module concatenates them to obtain the sixth feature, which is then sent to the second bidirectional long short-term memory network. After obtaining the sixth feature, the second bidirectional long short-term memory network performs a series of processes on it to obtain the seventh feature, which is then sent to the first concatenation module and the linearization module. Finally, the linearization module performs linear operations on the seventh feature to obtain the probability that each speech frame in the mixed speech belongs to the target object, which is the final output of the second model.

[0159] Therefore, the first model ultimately outputs the speech of the target object, while the second module ultimately outputs the probability that each speech frame in the mixed speech belongs to the target object. Based on these probabilities, several speech frames belonging to the target object can be determined (these several speech frames are the speech of the target object). Since each speech frame corresponds to a timestamp in the mixed speech (because all speech frames are sorted by time, thus forming the mixed speech), the timestamps corresponding to these several speech frames in the mixed speech can be obtained, which is the timestamp of the target object's speech in the mixed speech (also called the position of the target object's speech in the mixed speech). For example, suppose the first speech frame in the mixed speech corresponds to the first time, the second speech frame corresponds to the second time, ..., the 1000th speech frame corresponds to the 1000th time. Among them, the first to the 200th speech frames all belong to the target object. Therefore, the timestamp of the target object's speech in the mixed speech is the time period between the first and the 200th time.

[0160] More specifically, if the frame sampling rate of the mixed speech in the time domain for the first model is different from that of the second model, the following processing can also be performed:

[0161] like Figure 7 As shown ( Figure 7 This is another structural schematic diagram of the target model provided in the embodiments of this application. Figure 7 Is Figure 6Based on the above, the target model may also include a third model, which includes a downsampling module and an upsampling module. The downsampling module is set between the first dual-path recurrent neural network and the second splicing module, and the upsampling module is set between the second bidirectional long short-term memory network and the first splicing module.

[0162] based on Figure 7 It is known that the second bidirectional long short-term memory network can send the seventh feature of the mixed speech to the upsampling module. Therefore, the upsampling module upsamples the seventh feature of the mixed speech to obtain the upsampled seventh feature of the mixed speech (i.e., the intermediate output of the second model after upsampling). The upsampled seventh feature of the mixed speech is then sent to the first concatenation module so that the first concatenation module concatenates the second feature of the mixed speech and the upsampled seventh feature of the mixed speech to obtain the third feature of the mixed speech.

[0163] Accordingly, the first dual-path recurrent neural network can send the second feature of the mixed speech to the downsampling module. Therefore, the downsampling module downsamples the second feature of the mixed speech to obtain the downsampled second feature of the mixed speech (i.e., the intermediate output of the downsampled first model). The downsampled second feature of the mixed speech is then sent to the second splicing module so that the second splicing module splices the downsampled second feature of the mixed speech with the fifth feature of the mixed speech to obtain the sixth feature of the mixed speech.

[0164] It should be understood that in this embodiment, the number of layers in the first dual-path recurrent neural network can be the same as or different from the number of layers in the second dual-path recurrent neural network. Similarly, the number of layers in the first bidirectional long short-term memory network can be the same as or different from the number of layers in the second bidirectional long short-term memory network.

[0165] It should also be understood that, in this embodiment, if the user does not specify a target object, after determining the position of the target object's speech within the mixed speech, all speech segments constituting the target object's speech (including the aforementioned target speech segment) can be found in the speech segment set based on that position, and these speech segments can be removed from the speech segment set. In this way, from the remaining speech segments, a reference speech for the next object can be selected, and it and the mixed speech can be input again into the target model to obtain the speech of the next object and its position within the mixed speech. This process will not be elaborated further. Continuing with the example above, let's assume that the timestamp corresponding to segment A1 in the mixed speech is the time between the 1st and 100th time, and the timestamp corresponding to segment A2 in the mixed speech is the time between the 100th and 200th time. Since the timestamp corresponding to the speech of the target object in the mixed speech is the time between the 1st and 200th time, segments A1 and A2 can be removed from set A, and the reference speech of the next object can be selected from the remaining segments. This allows us to continue obtaining the speech of the next object contained in the mixed speech and the timestamp corresponding to the speech of the next object in the mixed speech.

[0166] In this embodiment, when mixed speech needs to be processed, the mixed speech and the reference speech of the target object can be obtained first. The mixed speech includes the speech of the target object and the speech of other objects besides the target object. Next, the mixed speech and the reference speech can be input into the target model. Then, the first model in the target model can process the intermediate outputs of the mixed speech, the reference speech, and the second model to obtain the intermediate output and the final output of the first model. Simultaneously, the second model in the target model can process the intermediate outputs of the mixed speech and the first model to obtain the intermediate output and the final output of the second model. Finally, the final output of the first model can be used to obtain the speech of the target object, and the final output of the second model can be used to determine the position of the target object's speech in the mixed speech. Based on the aforementioned process, it can be seen that the first model and the second model, as two branches in the target model, achieve cross-fusion of intermediate outputs during the processing of the mixed speech, thereby jointly completing the two tasks of speaker segmentation and clustering for mixed speech and target speaker extraction. As can be seen, the new model framework provided in this application, namely the target model, can simultaneously support two types of tasks: voiceprint segmentation and clustering, and target speaker speech extraction, which helps to reduce the design cost of speech processing.

[0167] Furthermore, in this embodiment, the target model has a multi-task learning mechanism. That is, in the process of extracting the target speaker's speech and segmenting and clustering the voiceprint for mixed speech, the two branches (the first model and the second model) in the target model are fully interdependent, which can effectively improve the performance of the entire target model in these two types of tasks, that is, improve the accuracy of speech processing.

[0168] Furthermore, in this embodiment, a multimodal fusion approach can be used to model the target speaker, thereby allowing users to specify the target speaker using information from different modalities, such as images, speech, text, and IDs, making the target model flexibly applicable in numerous scenarios.

[0169] The above is a detailed description of the speech processing method provided in the embodiments of this application. The following will introduce the model training method provided in the embodiments of this application. Figure 8 A schematic flowchart of the model training method provided in the embodiments of this application is shown below. Figure 8 As shown, the method includes:

[0170] 801. Obtain the mixed speech and the reference speech of the target object. The mixed speech includes the speech of the target object and the speech of the other objects besides the target object.

[0171] In this embodiment, when training the first, second, and third training models is required, a batch of training data can be obtained first. This batch of training data includes mixed speech and reference speech of the target object. The mixed speech includes the speech of the target object and the speech of other objects besides the target object. It should be noted that, for the mixed speech, the first true processing result of the mixed speech (i.e., the speech of the real target object in the mixed speech) and the second true processing result of the mixed speech (i.e., the true probability of each speech frame in the mixed speech belonging to the target object) are known.

[0172] 802. The intermediate outputs of the mixed speech, reference speech, and second training model are processed by the first training model to obtain the intermediate output and the final output of the first training model. The final output of the first training model is used to obtain the speech of the target object.

[0173] 803. The intermediate outputs of the mixed speech and the first training model are processed by the second training model to obtain the intermediate output of the second training model and the final output of the second training model. The final output of the second training model is used to determine the position of the target object's speech in the mixed speech.

[0174] After obtaining the mixed speech and the reference speech of the target object, these can be input into the target model. The first training model in the target model then processes the intermediate outputs of the mixed speech, the reference speech, and the second training model, resulting in the intermediate output and final output of the first training model. Similarly, the second training model in the target model processes the intermediate outputs of the mixed speech and the first training model, resulting in its intermediate output and final output. It should be noted that the final output of the first training model can also be called the first prediction processing result of the mixed speech (i.e., the predicted speech of the target object from the mixed speech), which can be used to obtain the predicted speech of the target object contained in the mixed speech. The final output of the second training model can also be called the second prediction processing result of the mixed speech (i.e., the predicted probability that each speech frame in the mixed speech belongs to the target object), which can be used to determine the predicted position of the target object's speech in the mixed speech, that is, the prediction timestamp of the target object's speech in the mixed speech.

[0175] In one possible implementation, processing the mixed speech, the reference speech, and the intermediate output of the second training model to obtain the intermediate output of the first training model and the final output of the first training model includes: performing a first processing on the mixed speech and the reference speech to obtain the intermediate output of the first training model; and performing a second processing on the intermediate output of the first training model and the intermediate output of the second training model to obtain the final output of the first training model.

[0176] In one possible implementation, processing the mixed speech and the intermediate output of the first training model to obtain the intermediate output of the second training model and the final output of the second training model includes: performing a third processing on the mixed speech and the intermediate output of the first training model to obtain the intermediate output of the second training model; and performing a fourth processing on the intermediate output of the second training model to obtain the final output of the second training model.

[0177] In one possible implementation, the first process includes at least one of the following: encoding and recurrent neural network-based processing, and the second process includes at least one of the following: concatenation, recurrent neural network-based processing, mask prediction, and decoding.

[0178] In one possible implementation, the third processing includes at least one of the following: processing based on a bidirectional long short-term memory network, and the fourth processing includes at least one of the following: splicing, processing based on a bidirectional long short-term memory network, and linear computation.

[0179] In one possible implementation, the method further includes: upsampling the intermediate output of the second training model using the third training model to obtain the upsampled intermediate output of the second training model; and processing the mixed speech, reference speech, and intermediate output of the second training model to obtain the intermediate output of the first training model and the final output of the first training model, which includes processing the upsampled intermediate output of the mixed speech, reference speech, and second training model to obtain the intermediate output of the first training model and the final output of the first training model.

[0180] In one possible implementation, the method further includes: downsampling the intermediate output of the first training model using the third training model to obtain the downsampled intermediate output of the first training model; and processing the mixed speech and the intermediate output of the first training model to obtain the intermediate output of the second training model and the final output of the second training model, which includes: processing the mixed speech and the downsampled intermediate output of the first training model to obtain the intermediate output of the second training model and the final output of the second training model.

[0181] In one possible implementation, obtaining the reference speech of the target object includes: obtaining information about the target object, the information including at least one of the following: an image of the target object, text of the target object, and an identifier of the target object; and obtaining the reference speech of the target object corresponding to the information from a pre-set speech library.

[0182] In one possible implementation, obtaining the reference speech of the target object includes: dividing the mixed speech into multiple speech segments, each of which contains the target speech segment; if the target speech segment corresponds to the same object, identifying the object as the target object, and identifying the target speech segment as the reference speech of the target object.

[0183] It should be noted that for the descriptions of steps 802 and 803, please refer to [link / reference needed]. Figure 5 The relevant descriptions of steps 502 and 503 in the illustrated embodiment will not be repeated here.

[0184] 804. Based on the final output of the first model to be trained and the final output of the second model to be trained, train the first model to be trained and the second model to be trained to obtain the first model and the second model.

[0185] After obtaining the first prediction processing result and the second prediction processing result of the mixed speech, the first training model, the second training model and the third training model can be trained based on the first prediction processing result and the second prediction processing result of the mixed speech to obtain the first model, the second model and the third model.

[0186] Specifically, the first model, the second model, and the third model can be trained in the following ways:

[0187] (1) Since the first true processing result of the mixed speech is known, the first prediction processing result and the first true processing result of the mixed speech can be calculated by the first loss function to obtain the first loss. The first loss is used to indicate the difference between the first prediction processing result and the first true processing result of the mixed speech.

[0188] The first loss function is shown in the following formula:

[0189]

[0190] In the above formula, L SE The first loss, s is the first prediction result of the mixed speech (i.e., the predicted speech of the target object), s is the first real result of the mixed speech (i.e., the real speech of the target object), and ε is the offset parameter.

[0191] (2) Since the second true processing result of the mixed speech is known, the second predicted processing result and the second true processing result of the mixed speech can be calculated using the second loss function to obtain the second loss. The second loss is used to indicate the difference between the second predicted processing result and the second true processing result of the mixed speech. The second loss function is shown in the following formula:

[0192]

[0193] In the above formula, L SD As the second loss, y t This represents the second true processing result of the mixed speech (i.e., the true probability that each speech frame in the mixed speech belongs to the target object). This is the second prediction processing result of the mixed speech (i.e., the predicted probability that each speech frame in the mixed speech belongs to the target object).

[0194] (3) The third loss is calculated by using a third loss function to evaluate the features of the reference speech of the target object (derived from the speaker encoder in the first model) and the features of the predicted speech of the target object (derived from the multiplication module in the first model). The third loss is used to indicate the difference between the features of the reference speech of the target object and the features of the predicted speech of the target object. The third loss function is similar to the second loss function and can be expressed as L. spk .

[0195] (4) The first loss, the second loss, and the third loss are summed to obtain the target loss, i.e.:

[0196] L=λ1L SE +λ2L SD +λ3L spk (4)

[0197] In the above formula, L represents the target loss.

[0198] (5) After obtaining the target loss, the parameters of the first model to be trained, the model parameters of the second model to be trained, and the model parameters of the third model to be trained can be updated based on the target loss to obtain the updated first model to be trained, the updated second model to be trained, and the updated third model to be trained. Then, the updated first model to be trained, the updated second model to be trained, and the updated third model to be trained can be trained again using the next batch of training data (i.e., re-execute steps 801 to 804) until the model training conditions are met (e.g., the target loss converges, etc.). Figure 4 or Figure 7 The first model, the second model, and the third model shown are the target models.

[0199] The target model trained in this embodiment possesses speech processing capabilities (i.e., speaker segmentation and clustering, and target speaker extraction). Specifically, when processing mixed speech, the mixed speech and the reference speech of the target object are first obtained. The mixed speech includes the speech of the target object and the speech of other objects besides the target object. Next, the mixed speech and the reference speech are input into the target model. The first model in the target model processes the intermediate outputs of the mixed speech, the reference speech, and the second model to obtain the intermediate output and the final output of the first model. Simultaneously, the second model in the target model processes the intermediate outputs of the mixed speech and the first model to obtain the intermediate output and the final output of the second model. Finally, the final output of the first model can be used to obtain the speech of the target object, and the final output of the second model can be used to determine the position of the target object's speech within the mixed speech. Based on the aforementioned process, the first and second models, as two branches of the target model, achieve cross-fusion of intermediate outputs during the processing of the mixed speech, thus jointly completing the tasks of speaker segmentation and clustering for mixed speech and target speaker extraction. As can be seen, the new model framework provided in this application, namely the target model, can simultaneously support two types of tasks: voiceprint segmentation and clustering, and target speaker speech extraction, which helps to reduce the design cost of speech processing.

[0200] The above is a detailed description of the speech processing method and model training method provided in the embodiments of this application. The speech processing device and model training device provided in the embodiments of this application will be introduced below. Figure 9 A schematic diagram of the structure of the voice processing device provided in the embodiments of this application is shown below. Figure 9 As shown, the device includes:

[0201] The acquisition module 901 is used to acquire the mixed speech and the reference speech of the target object. The mixed speech includes the speech of the target object and the speech of other objects besides the target object.

[0202] The first processing module 902 is used to process the mixed speech, reference speech and intermediate output of the second model through the first model to obtain the intermediate output of the first model and the final output of the first model. The final output of the first model is used to obtain the speech of the target object.

[0203] The second processing module 903 is used to process the intermediate output of the mixed speech and the first model through the second model to obtain the intermediate output of the second model and the final output of the second model. The final output of the second model is used to determine the position of the speech of the target object in the mixed speech.

[0204] In this embodiment, when mixed speech needs to be processed, the mixed speech and the reference speech of the target object can be obtained first. The mixed speech includes the speech of the target object and the speech of other objects besides the target object. Next, the mixed speech and the reference speech can be input into the target model. Then, the first model in the target model can process the intermediate outputs of the mixed speech, the reference speech, and the second model to obtain the intermediate output and the final output of the first model. Simultaneously, the second model in the target model can process the intermediate outputs of the mixed speech and the first model to obtain the intermediate output and the final output of the second model. Finally, the final output of the first model can be used to obtain the speech of the target object, and the final output of the second model can be used to determine the position of the target object's speech in the mixed speech. Based on the aforementioned process, it can be seen that the first model and the second model, as two branches in the target model, achieve cross-fusion of intermediate outputs during the processing of the mixed speech, thereby jointly completing the two tasks of speaker segmentation and clustering for mixed speech and target speaker extraction. As can be seen, the new model framework provided in this application, namely the target model, can simultaneously support two types of tasks: voiceprint segmentation and clustering, and target speaker speech extraction, which helps to reduce the design cost of speech processing.

[0205] In one possible implementation, the first processing module 902 is configured to: perform a first processing on the mixed speech and the reference speech to obtain an intermediate output of the first model; and perform a second processing on the intermediate output of the first model and the intermediate output of the second model to obtain the final output of the first model.

[0206] In one possible implementation, the second processing module 903 is used to: perform a third processing on the intermediate output of the mixed speech and the first model to obtain an intermediate output of the second model; and perform a fourth processing on the intermediate output of the second model to obtain the final output of the second model.

[0207] In one possible implementation, the first process includes at least one of the following: encoding and recurrent neural network-based processing, and the second process includes at least one of the following: concatenation, recurrent neural network-based processing, mask prediction, and decoding.

[0208] In one possible implementation, the third processing includes at least one of the following: processing based on a bidirectional long short-term memory network, and the fourth processing includes at least one of the following: splicing, processing based on a bidirectional long short-term memory network, and linear computation.

[0209] In one possible implementation, the device further includes: an upsampling module for upsampling the intermediate output of the second model using a third model to obtain an upsampled intermediate output of the second model; and a first processing module 902 for processing the mixed speech, the reference speech, and the upsampled intermediate output of the second model to obtain an intermediate output of the first model and a final output of the first model.

[0210] In one possible implementation, the device further includes: a downsampling module for downsampling the intermediate output of the first model using a third model to obtain a downsampled intermediate output of the first model; and a second processing module 903 for processing the mixed speech and the downsampled intermediate output of the first model to obtain an intermediate output of the second model and a final output of the second model.

[0211] In one possible implementation, the acquisition module 901 is used to: acquire information about a target object, the information including at least one of the following: an image of the target object, text of the target object, and an identifier of the target object; and acquire reference speech of the target object corresponding to the information from a preset speech library.

[0212] In one possible implementation, the acquisition module 901 is configured to: divide the mixed speech into multiple speech segments, the multiple speech segments containing a target speech segment; if the target speech segment corresponds to the same object, identify the object as the target object, and identify the target speech segment as the reference speech of the target object.

[0213] Figure 10 A schematic diagram of the model training apparatus provided in the embodiments of this application is shown below. Figure 10 As shown, the device includes:

[0214] The acquisition module 1001 is used to acquire the mixed speech and the reference speech of the target object. The mixed speech includes the speech of the target object and the speech of other objects besides the target object.

[0215] The first processing module 1002 is used to process the mixed speech, reference speech and intermediate output of the second training model through the first training model to obtain the intermediate output of the first training model and the final output of the first training model. The final output of the first training model is used to obtain the speech of the target object.

[0216] The second processing module 1003 is used to process the intermediate output of the mixed speech and the first training model through the second training model to obtain the intermediate output of the second training model and the final output of the second training model. The final output of the second training model is used to determine the position of the speech of the target object in the mixed speech.

[0217] The training module 1004 is used to train the first training model and the second training model based on the final output of the first training model and the final output of the second training model to obtain the first model and the second model.

[0218] The target model trained in this embodiment possesses speech processing capabilities (i.e., speaker segmentation and clustering, and target speaker extraction). Specifically, when processing mixed speech, the mixed speech and the reference speech of the target object are first obtained. The mixed speech includes the speech of the target object and the speech of other objects besides the target object. Next, the mixed speech and the reference speech are input into the target model. The first model in the target model processes the intermediate outputs of the mixed speech, the reference speech, and the second model to obtain the intermediate output and the final output of the first model. Simultaneously, the second model in the target model processes the intermediate outputs of the mixed speech and the first model to obtain the intermediate output and the final output of the second model. Finally, the final output of the first model can be used to obtain the speech of the target object, and the final output of the second model can be used to determine the position of the target object's speech within the mixed speech. Based on the aforementioned process, the first and second models, as two branches of the target model, achieve cross-fusion of intermediate outputs during the processing of the mixed speech, thus jointly completing the tasks of speaker segmentation and clustering for mixed speech and target speaker extraction. As can be seen, the new model framework provided in this application, namely the target model, can simultaneously support two types of tasks: voiceprint segmentation and clustering, and target speaker speech extraction, which helps to reduce the design cost of speech processing.

[0219] In one possible implementation, the first processing module 1002 is configured to: perform a first processing on the mixed speech and the reference speech to obtain an intermediate output of a first model to be trained; and perform a second processing on the intermediate output of the first model to be trained and the intermediate output of a second model to be trained to obtain the final output of the first model to be trained.

[0220] In one possible implementation, the second processing module 1003 is used to: perform a third processing on the intermediate output of the mixed speech and the first model to be trained to obtain the intermediate output of the second model to be trained; and perform a fourth processing on the intermediate output of the second model to be trained to obtain the final output of the second model to be trained.

[0221] In one possible implementation, the first process includes at least one of the following: encoding and recurrent neural network-based processing, and the second process includes at least one of the following: concatenation, recurrent neural network-based processing, mask prediction, and decoding.

[0222] In one possible implementation, the third processing includes at least one of the following: processing based on a bidirectional long short-term memory network, and the fourth processing includes at least one of the following: splicing, processing based on a bidirectional long short-term memory network, and linear computation.

[0223] In one possible implementation, the device further includes: an upsampling module for upsampling the intermediate output of the second training model using the third training model to obtain the upsampled intermediate output of the second training model; and a first processing module 1002 for processing the mixed speech, the reference speech, and the upsampled intermediate output of the second training model to obtain the intermediate output of the first training model and the final output of the first training model.

[0224] In one possible implementation, the device further includes: a downsampling module for downsampling the intermediate output of the first training model using the third training model to obtain the downsampled intermediate output of the first training model; and a second processing module 1003 for processing the mixed speech and the downsampled intermediate output of the first training model to obtain the intermediate output of the second training model and the final output of the second training model.

[0225] In one possible implementation, the acquisition module 1001 is used to: acquire information about a target object, the information including at least one of the following: an image of the target object, text of the target object, and an identifier of the target object; and acquire reference speech of the target object corresponding to the information from a preset speech library.

[0226] In one possible implementation, the acquisition module 1001 is used to: divide the mixed speech into multiple speech segments, the multiple speech segments containing a target speech segment; if the target speech segment corresponds to the same object, determine the object as the target object, and determine the target speech segment as the reference speech of the target object.

[0227] In one possible implementation, the training module 1004 is configured to: obtain a target loss based on the final output of the first model to be trained (i.e., the aforementioned first prediction processing result of mixed speech), the real output of the first model to be trained (i.e., the aforementioned first real processing result of mixed speech), the final output of the second model to be trained (i.e., the aforementioned second prediction processing result of mixed speech), and the real output of the second model to be trained (i.e., the aforementioned second real processing result of mixed speech); the target loss is used to indicate the difference between the final output of the first model to be trained and the real output of the first model to be trained, and the difference between the final output of the second model to be trained and the real output of the second model to be trained; and update the parameters of the first model to be trained and the parameters of the second model to be trained based on the target loss until the model training conditions are met, thereby obtaining the first model and the second model.

[0228] In one possible implementation, the training module 1004 is also used to update the parameters of the first model to be trained based on the target loss until the model training conditions are met, thus obtaining the third model.

[0229] It should be noted that the information interaction and execution process between the modules / units of the above-mentioned device are based on the same concept as the method embodiment of this application, and the resulting technical effects are the same as those of the method embodiment of this application. For details, please refer to the description in the method embodiment shown above in the embodiment of this application, and it will not be repeated here.

[0230] This application also relates to an execution device. Figure 11 This is a schematic diagram of the execution device provided in an embodiment of this application. Figure 11 As shown, the execution device 1100 can specifically be a mobile phone, tablet, laptop, smart wearable device, server, etc., and is not limited here. Among them, the execution device 1100 may be deployed with... Figure 9 The speech processing device described in the corresponding embodiment is used to implement Figure 5 The corresponding embodiment describes the voice processing function. Specifically, the execution device 1100 includes: a receiver 1101, a transmitter 1102, a processor 1103, and a memory 1104 (wherein the execution device 1100 may have one or more processors 1103). Figure 11(Taking a processor as an example), processor 1103 may include application processor 11031 and communication processor 11032. In some embodiments of this application, receiver 1101, transmitter 1102, processor 1103 and memory 1104 may be connected via bus or other means.

[0231] Memory 1104 may include read-only memory and random access memory, and provides instructions and data to processor 1103. A portion of memory 1104 may also include non-volatile random access memory (NVRAM). Memory 1104 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0232] Processor 1103 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together through a bus system, which may include not only the data bus, but also power buses, control buses, and status signal buses. However, for clarity, all buses are referred to as the bus system in the diagram.

[0233] The methods disclosed in the embodiments of this application can be applied to or implemented by the processor 1103. The processor 1103 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 1103 or by instructions in software form. The processor 1103 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and may further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor 1103 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 1104. Processor 1103 reads the information in memory 1104 and, in conjunction with its hardware, completes the steps of the above method.

[0234] Receiver 1101 can be used to receive input digital or character information, and to generate signal inputs related to the settings and function control of the execution device. Transmitter 1102 can be used to output digital or character information through the first interface; transmitter 1102 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; transmitter 1102 may also include a display device such as a display screen.

[0235] In one embodiment of this application, the processor 1103 is used to... Figure 5 The target model in the corresponding embodiment generates the processing result of mixed speech.

[0236] This application also relates to a training device. Figure 12 This is a schematic diagram of the structure of a training device provided in an embodiment of this application. Figure 12As shown, the training device 1200 is implemented by one or more servers. The training device 1200 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 1214 (e.g., one or more processors) and memory 1232, and one or more storage media 1230 (e.g., one or more mass storage devices) for storing application programs 1242 or data 1244. The memory 1232 and storage media 1230 can be temporary or persistent storage. The program stored in the storage media 1230 may include one or more modules (not shown in the figure), each module including a series of instruction operations on the training device. Furthermore, the CPU 1214 may be configured to communicate with the storage media 1230 and execute the series of instruction operations in the storage media 1230 on the training device 1200.

[0237] The training device 1200 may also include one or more power supplies 1226, one or more wired or wireless network interfaces 1250, one or more input / output interfaces 1258; or, one or more operating systems 1241, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0238] Specifically, the training equipment can perform Figure 8 The model training method in the corresponding embodiment.

[0239] This application also relates to a computer storage medium storing a program for signal processing, which, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.

[0240] This application also relates to a computer program product that stores instructions that, when executed by a computer, cause the computer to perform steps as performed by the aforementioned execution device, or to perform steps as performed by the aforementioned training device.

[0241] The execution device, training device, or terminal device provided in this application embodiment can specifically be a chip. The chip includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip within the execution device to execute the data processing method described in the above embodiments, or to cause the chip within the training device to execute the data processing method described in the above embodiments. Optionally, the storage unit can be a storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0242] For details, please refer to Figure 13 , Figure 13 This is a schematic diagram of the chip provided in an embodiment of this application. The chip can be represented as a neural network processor (NPU) 1300. The NPU 1300 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core part of the NPU is the arithmetic circuit 1303, which is controlled by the controller 1304 to extract matrix data from the memory and perform multiplication operations.

[0243] In some implementations, the arithmetic circuit 1303 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 1303 is a two-dimensional pulsating array. The arithmetic circuit 1303 can also be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1303 is a general-purpose matrix processor.

[0244] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 1302 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 1301 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 1308.

[0245] Unified memory 1306 is used to store input and output data. Weight data is directly transferred to weight memory 1302 via Direct Memory Access Controller (DMAC) 1305. Input data is also transferred to unified memory 1306 via DMAC.

[0246] BIU stands for Bus Interface Unit, which is used for interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 1309.

[0247] The Bus Interface Unit (BIU) 1313 is used by the instruction fetch memory 1309 to fetch instructions from external memory, and also by the memory access controller 1305 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0248] The DMAC is mainly used to move input data from external memory DDR to unified memory 1306, or to weight data to weight memory 1302, or to input data to input memory 1301.

[0249] The vector computation unit 1307 includes multiple arithmetic processing units that, when needed, further process the output of the computation circuit 1303, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is mainly used for computation in non-convolutional / fully connected layers of neural networks, such as Batch Normalization, pixel-level summation, and upsampling of the predicted label plane.

[0250] In some implementations, the vector computation unit 1307 can store the processed output vector in the unified memory 1306. For example, the vector computation unit 1307 can apply a linear function, or a nonlinear function, to the output of the computation circuit 1303, such as linearly interpolating the predicted label plane extracted from the convolutional layer, or, for example, accumulating a vector of values ​​to generate activation values. In some implementations, the vector computation unit 1307 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as activation input to the computation circuit 1303, for example, for use in subsequent layers of the neural network.

[0251] The instruction fetch buffer 1309 connected to the controller 1304 is used to store the instructions used by the controller 1304;

[0252] Unified memory 1306, input memory 1301, weighted memory 1302, and instruction fetch memory 1309 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0253] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the above program.

[0254] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0255] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0256] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0257] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A speech processing method, characterized in that, The method includes: Acquire a mixed speech and a reference speech of a target object, wherein the mixed speech includes the speech of the target object and the speech of other objects besides the target object; The mixed speech and the reference speech are processed by a first model to obtain the intermediate output of the first model; The intermediate outputs of the first model and the intermediate outputs of the second model are processed by the first model to obtain the final output of the first model. The final output of the first model is used to obtain the speech of the target object. The intermediate output of the second model is obtained by performing a third process on the mixed speech and the intermediate output of the first model through the second model. The intermediate output of the second model is processed by the second model to obtain the final output of the second model. The final output of the second model is used to determine the position of the target object's speech in the mixed speech.

2. The method according to claim 1, characterized in that, The first process includes at least one of the following: encoding and processing based on a recurrent neural network, and the second process includes at least one of the following: concatenation, processing based on a recurrent neural network, mask prediction, and decoding.

3. The method according to claim 2, characterized in that, The third processing includes at least one of the following: processing based on a bidirectional long short-term memory network, and the fourth processing includes at least one of the following: splicing, processing based on a bidirectional long short-term memory network, and linear computation.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: The intermediate output of the second model is upsampled using the third model to obtain the upsampled intermediate output of the second model. The process of processing the mixed speech, the reference speech, and the intermediate output of the second model to obtain the intermediate output and the final output of the first model includes: The mixed speech, the reference speech, and the upsampled intermediate output of the second model are processed to obtain the intermediate output of the first model and the final output of the first model.

5. The method according to any one of claims 1 to 3, characterized in that, The method further includes: The intermediate output of the first model is downsampled using the third model to obtain the downsampled intermediate output of the first model. The process of processing the mixed speech and the intermediate output of the first model to obtain the intermediate output of the second model and the final output of the second model includes: The mixed speech and the downsampled intermediate output of the first model are processed to obtain the intermediate output of the second model and the final output of the second model.

6. The method according to any one of claims 1 to 3, characterized in that, The reference speech for obtaining the target object includes: Obtain information about a target object, the information including at least one of the following: an image of the target object, text about the target object, and an identifier of the target object; From a pre-set speech library, obtain the reference speech of the target object corresponding to the information.

7. The method according to any one of claims 1 to 3, characterized in that, The reference speech for obtaining the target object includes: The mixed speech is divided into multiple speech segments, and the multiple speech segments include the target speech segment; If a target speech segment corresponds to the same object, the object is identified as the target object, and the target speech segment is identified as the reference speech of the target object.

8. A model training method, characterized in that, The method includes: Acquire a mixed speech and a reference speech of a target object, wherein the mixed speech includes the speech of the target object and the speech of other objects besides the target object; The mixed speech and the reference speech are processed by the first training model to obtain the intermediate output of the first training model. The intermediate outputs of the first training model and the second training model are processed by the first training model to obtain the final output of the first training model. The final output of the first training model is used to obtain the speech of the target object. The intermediate output of the mixed speech and the intermediate output of the first training model are processed by the second training model to obtain the intermediate output of the second training model. The intermediate output of the second training model is processed by the second training model to obtain the final output of the second training model. The final output of the second training model is used to determine the position of the speech of the target object in the mixed speech. Based on the final output of the first training model and the final output of the second training model, the first training model and the second training model are trained to obtain a first model and a second model. The first model and the second model are used to process the mixed speech and the reference speech to determine the position of the speech of the target object in the mixed speech.

9. The method according to claim 8, characterized in that, The first process includes at least one of the following: encoding and processing based on a recurrent neural network, and the second process includes at least one of the following: concatenation, processing based on a recurrent neural network, mask prediction, and decoding.

10. The method according to claim 8, characterized in that, The third processing includes at least one of the following: processing based on a bidirectional long short-term memory network, and the fourth processing includes at least one of the following: splicing, processing based on a bidirectional long short-term memory network, and linear computation.

11. The method according to any one of claims 8 to 10, characterized in that, The method further includes: The intermediate output of the second training model is upsampled by the third training model to obtain the upsampled intermediate output of the second training model. The process of processing the mixed speech, the reference speech, and the intermediate output of the second model to be trained to obtain the intermediate output and the final output of the first model to be trained includes: The mixed speech, the reference speech, and the upsampled intermediate output of the second training model are processed to obtain the intermediate output of the first training model and the final output of the first training model.

12. The method according to any one of claims 8 to 10, characterized in that, The method further includes: The intermediate output of the first training model is downsampled by the third training model to obtain the downsampled intermediate output of the first training model. The process of processing the mixed speech and the intermediate output of the first model to be trained to obtain the intermediate output and the final output of the second model to be trained includes: The mixed speech and the downsampled intermediate output of the first training model are processed to obtain the intermediate output of the second training model and the final output of the second training model.

13. The method according to any one of claims 8 to 10, characterized in that, The reference speech for obtaining the target object includes: Obtain information about a target object, the information including at least one of the following: an image of the target object, text about the target object, and an identifier of the target object; From a pre-set speech library, obtain the reference speech of the target object corresponding to the information.

14. The method according to any one of claims 8 to 10, characterized in that, The reference speech for obtaining the target object includes: The mixed speech is divided into multiple speech segments, and the multiple speech segments include the target speech segment; If a target speech segment corresponds to the same object, the object is identified as the target object, and the target speech segment is identified as the reference speech of the target object.

15. A voice processing device, characterized in that, The device includes: An acquisition module is used to acquire mixed speech and reference speech of a target object, wherein the mixed speech includes the speech of the target object and the speech of other objects besides the target object; The first processing module is configured to: perform a first processing on the mixed speech and the reference speech using a first model to obtain an intermediate output of the first model; and perform a second processing on the intermediate output of the first model and the intermediate output of the second model using the first model to obtain a final output of the first model, wherein the final output of the first model is used to acquire the speech of the target object. The second processing module is configured to: perform a third processing on the intermediate output of the mixed speech and the first model through the second model to obtain the intermediate output of the second model; and perform a fourth processing on the intermediate output of the second model through the second model to obtain the final output of the second model, wherein the final output of the second model is used to determine the position of the speech of the target object in the mixed speech.

16. A model training device, characterized in that, The device includes: An acquisition module is used to acquire mixed speech and reference speech of a target object, wherein the mixed speech includes the speech of the target object and the speech of other objects besides the target object; The first processing module is configured to: perform a first processing on the mixed speech and the reference speech using a first training model to obtain an intermediate output of the first training model; and perform a second processing on the intermediate output of the first training model and the intermediate output of a second training model using the first training model to obtain a final output of the first training model, wherein the final output of the first training model is used to acquire the speech of the target object. The second processing module is used to perform a third processing on the intermediate output of the mixed speech and the first training model through the second training model to obtain the intermediate output of the second training model; and to perform a fourth processing on the intermediate output of the second training model through the second training model to obtain the final output of the second training model. The final output of the second training model is used to determine the position of the speech of the target object in the mixed speech. The training module is used to train the first training model and the second training model based on the final output of the first training model and the final output of the second training model to obtain a first model and a second model, wherein the first model and the second model are used to process the mixed speech and the reference speech to determine the position of the speech of the target object in the mixed speech.

17. A voice processing device, characterized in that, The device includes a memory and a processor; the memory stores code, and the processor is configured to execute the code, wherein when the code is executed, the speech processing device performs the method as described in any one of claims 1 to 14.

18. A computer storage medium, characterized in that, The computer storage medium stores one or more instructions that, when executed by one or more computers, cause the one or more computers to perform the method of any one of claims 1 to 14.

19. A computer program product, characterized in that, The computer program product stores instructions that, when executed by a computer, cause the computer to perform the method described in any one of claims 1 to 14.

Images