Model determination method, model application method, and related device

Abstract

Embodiments of the present application disclose a model determination method, a model application method, and a related device. An initial voice separation model comprises an initial number determination module used for analyzing the number of voice producing objects, and an initial voice separation module used for performing voice separation on the basis of the number of voice producing objects determined by the initial number determination module, and sample voice information only needs to be inputted, so that a voice separation result can be obtained by means of separation by the model. On the basis of the difference between an accurate voice separation result corresponding to the sample voice information and a model output, the accuracy of the model for analysis of the number of the voice producing objects and the accuracy of the model for the separation of the voice information can be reflected. Therefore, parameter adjustment is performed on the initial voice separation model on the basis of the difference, so that the model can learn at the same time how to accurately analyze the number of the voice producing objects and accurately separate the voice information, the obtained voice separation model can realize accurate voice separation without other information input than voice information to be separated, thereby improving the voice separation efficiency.

Description

Model determination method, model application method and related device

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on July 6, 2023, application number 202310824944.X, and application name “Model determination method, model application method and related device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of machine learning technology, and in particular to model determination and model application technology. Background Art

[0003] Speech recognition is one of the primary areas of model application. Speech recognition models can identify the subject of speech information. In some scenarios, due to the presence of multiple people speaking simultaneously, the collected speech information may contain multiple subjects' speech information. In these cases, a speech separation model is needed to separate the speech information of these multiple subjects and accurately recognize the speech information of each subject.

[0004] In related technologies, when applying a speech separation model, it is necessary to determine in advance the number of speakers included in the speech information to be separated. This is necessary to achieve relatively accurate speech separation results. Therefore, in related technologies, before inputting the speech information to be separated into the speech separation model, it is necessary to first determine the number of speakers corresponding to the speech information.

[0005] It can be seen that the speech separation process in related technologies is relatively complicated and has high requirements on the information input side, making it difficult to achieve efficient and concise speech information separation.

[0006] Summary of the Invention

[0007] In order to solve the above technical problems, the present application provides a model determination method. The model trained by this method has the ability to automatically identify the number of pronunciation objects corresponding to the speech information to be separated, and can automatically separate multiple sub-speech information corresponding to the speech information to be separated based on the number of pronunciation objects. There is no need for preliminary processing of the speech information to be separated, thereby improving the efficiency and convenience of speech separation.

[0008] The embodiments of this application disclose the following technical solutions:

[0009] In a first aspect, an embodiment of the present application discloses a model determination method, which is executed by a computer device and includes:

[0010] Acquire a sample information set, the sample information set including a plurality of sample voice information, each of the plurality of sample voice information having a corresponding plurality of sample sub-voice information, each of the sample sub-voice information corresponding one-to-one to a sample pronunciation object;

[0011] For each of the plurality of sample voice information, determining, by an initial quantity determination module in an initial voice separation model, information on the number of undetermined pronunciation objects corresponding to the sample voice information, and performing voice separation on the sample voice information by an initial voice separation module in the initial voice separation model based on the information on the number of undetermined pronunciation objects, to obtain a plurality of first voice information;

[0012] Determining the number of actual pronunciation objects corresponding to the sample voice information according to the plurality of sample sub-voice information contained in the sample voice information;

[0013] According to the difference between the information on the number of objects to be pronounced and the information on the number of actual objects to be pronounced, and according to the difference between the multiple sample sub-speech information of the sample speech information and the multiple first speech information, the model parameters corresponding to the initial speech separation model are adjusted to obtain a speech separation model. The speech separation model is used to perform speech separation on the speech information to be separated to obtain the corresponding multiple sub-speech information, and each of the sub-speech information corresponds one-to-one to the pronunciation object to be identified.

[0014] In a second aspect, an embodiment of the present application discloses a model application method, which is executed by a computer device and includes:

[0015] Acquiring speech information to be separated, wherein the speech information to be separated is composed of sub-speech information corresponding to a plurality of pronunciation objects to be recognized;

[0016] According to the speech information to be separated, the number information of pronunciation objects corresponding to the speech information to be separated is determined by the quantity determination module in the speech separation model, and according to the number information of pronunciation objects, the speech information to be separated is speech-separated by the speech separation module in the speech separation model to obtain the number of pronunciation objects, the number of sub-speech information, the number of pronunciation objects corresponds one-to-one to the pronunciation objects to be identified, the pronunciation objects number information is used to identify the number of pronunciation objects corresponding to the speech information to be separated, and the speech separation model is determined by the model determination method described in the aforementioned aspect.

[0017] In a third aspect, an embodiment of the present application discloses a model determination device, which is deployed on a computer device and includes a first acquisition unit, a first determination unit, a second determination unit, and an adjustment unit:

[0018] The first acquiring unit is configured to acquire a sample information set, the sample information set including a plurality of sample voice information, each of the plurality of sample voice information having a corresponding plurality of sample sub-voice information, each of the sample sub-voice information corresponding one-to-one to a sample pronunciation object;

[0019] The first determining unit is configured to determine, for each sample voice information in the plurality of sample voice information, the number of undetermined pronunciation objects corresponding to the sample voice information by an initial number determining module in an initial voice separation model, and perform voice separation on the sample voice information based on the undetermined pronunciation object number information by an initial voice separation module in the initial voice separation model to obtain a plurality of first voice information;

[0020] The second determining unit is configured to determine the number of actual pronunciation objects corresponding to the sample voice information based on the plurality of sample sub-voice information contained in the sample voice information;

[0021] The adjustment unit is used to adjust the model parameters corresponding to the initial speech separation model according to the difference between the information on the number of objects to be pronounced and the information on the number of actual pronunciation objects, and according to the difference between the multiple sample sub-speech information contained in the sample speech information and the multiple first speech information, so as to obtain a speech separation model. The speech separation model is used to perform speech separation on the speech information to be separated to obtain the corresponding multiple sub-speech information, and each sub-speech information corresponds one-to-one to the pronunciation object to be identified.

[0022] In a fourth aspect, an embodiment of the present application discloses a model application device, which is deployed on a computer device and includes a second acquisition unit and a third determination unit:

[0023] The second acquiring unit is configured to acquire speech information to be separated, wherein the speech information to be separated is composed of sub-speech information corresponding to a plurality of pronunciation objects to be recognized;

[0024] The third determination unit is used to determine the number information of pronunciation objects corresponding to the voice information to be separated through the number determination module in the voice separation model according to the voice information to be separated, and to perform voice separation on the voice information to be separated according to the number information of pronunciation objects through the voice separation module in the voice separation model to obtain the number of pronunciation objects, the number of pronunciation objects and the sub-voice information correspond one-to-one to the pronunciation objects, and the pronunciation object number information is used to identify the number of pronunciation objects corresponding to the voice information to be separated, and the voice separation model is determined by the model determination method described in the above aspects.

[0025] In a fifth aspect, an embodiment of the present application discloses a computer device, comprising a processor and a memory:

[0026] The memory is used to store a computer program and transmit the computer program to the processor;

[0027] The processor is configured to execute the model determination method described in any one of the first aspects, or execute the model application method described in any one of the second aspects, according to instructions in the computer program;

[0028] In a sixth aspect, an embodiment of the present application discloses a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, wherein the computer program is used to execute the model determination method described in any one of the first aspects, or execute the model application method described in any one of the second aspects;

[0029] In the seventh aspect, an embodiment of the present application discloses a computer program product including a computer program, which, when running on a computer device, enables the computer device to execute the model determination method described in any one of the first aspects, or execute the model application method described in any one of the second aspects.

[0030] It can be seen from the above technical solution that in order to enable the model to have the ability to analyze the number of pronunciation objects in the voice information by itself, the initial voice separation model in the present application can include an initial number determination module and an initial voice separation module, wherein the initial number determination module can determine the number of pending pronunciation objects corresponding to the sample voice information, and the pending pronunciation object number information is used to identify the number of pronunciation objects corresponding to the sample voice information analyzed by the initial number determination module; through the initial voice separation module, the sample voice information can be separated based on the number of pronunciation objects identified by the pending pronunciation object number information to obtain multiple first voice information. Through the multiple sample sub-voice information corresponding to the sample voice information, it can be reflected that when the sample voice information is accurately voice-separated, the number of pronunciation objects corresponding to the sample voice information and the multiple sub-voice information obtained by separation can be reflected. Therefore, the actual number of pronunciation objects corresponding to the sample voice information can be determined by multiple sample sub-voice information. The difference between the actual number of pronunciation objects and the number of pending pronunciation objects can reflect the accuracy of the initial number determination module for the analysis of the number of pronunciation objects. The difference between multiple sample sub-voice information and multiple first voice information can reflect the accuracy of the initial voice separation module when performing voice information separation. Thus, by combining the differences in these two dimensions to adjust the parameters of the initial voice separation model, on the one hand, the initial number determination module can learn how to accurately analyze the number of pronunciation objects included in the voice information, and on the other hand, the initial voice model can learn how to accurately perform voice separation on the voice information based on the number of pronunciation objects information, and obtain voice information that can form an accurate one-to-one correspondence between the pronunciation objects. It can be seen that the model determination method of the present application can generate a voice separation model that can independently perform voice object analysis and accurately perform voice separation based on the number of pronunciation objects obtained by analysis. When applied, only the voice information to be separated needs to be input to obtain the voice separation result, and there is no need to perform voice object recognition on the voice information before input. While ensuring the accuracy of voice separation, the convenience and efficiency of voice separation are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0032] FIG1 is a schematic diagram of a model determination method in a practical application scenario provided by an embodiment of the present application;

[0033] FIG2 is a flow chart of a model determination method provided in an embodiment of the present application;

[0034] FIG3 is a schematic diagram of an initial quantity determination module provided in an embodiment of the present application;

[0035] FIG4 is a schematic diagram of an initial speech separation module provided in an embodiment of the present application;

[0036] FIG5 is a schematic diagram of an initial feature extraction module provided in an embodiment of the present application;

[0037] FIG6 is a flow chart of a model application method provided in an embodiment of the present application;

[0038] FIG7 is a schematic diagram of a model application method provided in an embodiment of the present application;

[0039] FIG8 is a structural block diagram of a model determination device provided in an embodiment of the present application;

[0040] FIG9 is a structural block diagram of a model application device provided in an embodiment of the present application;

[0041] FIG10 is a structural diagram of a terminal provided in an embodiment of the present application;

[0042] FIG11 is a structural diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] The embodiments of the present application are described below with reference to the accompanying drawings.

[0044] In related technologies, speech separation models generally include two types. The first type of speech separation model generates sample speech information for a specific pronunciation object during model training, so that the speech separation model can learn the speech characteristics of the pronunciation object based on the sample speech information during training, such as the frequency characteristics, timbre characteristics, etc. of the speech, so that the speech information of the pronunciation object can be accurately identified and separated from the speech information to be separated that is mixed with the speech information of multiple pronunciation objects; the second type of speech separation model needs to be told how many speech information of pronunciation objects are in the speech information to be separated when it is applied, so that the speech separation model can perform more accurate speech separation.

[0045] The first speech separation model requires the preparation of a large number of training samples for the same pronunciation object during the initial model training process, which is relatively complex. At the same time, the speech separation model trained in this way can only accurately separate the features of the pronunciation object corresponding to the training samples, which is very limited. In addition, since different pronunciation objects may have similarities in speech features, for example, the frequency distributions of different pronunciation objects may overlap to a certain extent. If the frequency band of the filter in the speech separation model is set too loosely, the separated speech information will be mixed between different pronunciation objects, resulting in impure separated speech information. If the frequency band of the filter is set too tight, although it can prevent aliasing between speech information of different pronunciation objects to a certain extent, it will cause the separated speech information to be lost. The second speech separation model requires manual determination of the number of pronunciation objects corresponding to the speech information to be separated before inputting the speech information into the model. Therefore, a pre-calculation and labeling process is required, which makes speech separation more cumbersome and inefficient.

[0046] Based on this, in order to solve the technical problems in the relevant technology, the present application provides a model determination method, which can include an initial number determination module for analyzing the number of pronunciation objects in the initial speech separation model, and an initial speech separation module for determining the number of pronunciation objects to be determined based on the initial speech separation model to perform speech separation. It only takes input of sample speech information to obtain the speech separation result through the initial speech separation model. The difference between the accurate speech separation result corresponding to the sample speech information and the speech separation result output by the model can reflect the accuracy of the initial speech separation model in analyzing the number of pronunciation objects on the one hand, and on the other hand, it can reflect the accuracy of the speech information separated by the initial speech separation model. Therefore, adjusting the parameters of the initial speech separation model based on the difference can enable the model to learn how to accurately analyze the number of pronunciation objects and accurately separate speech information at the same time, so that the obtained speech separation model can achieve accurate speech separation without the input of other information except the speech information to be separated, thereby reducing the difficulty of speech separation and improving the efficiency of speech separation.

[0047] It is understood that the method can be applied to a computer device that is capable of model training and model application, such as a terminal device or a server. The method can be executed independently by a terminal device or a server, or it can be applied to a network scenario where a terminal device and a server communicate, and executed by the terminal device and the server in cooperation. The terminal device can be a mobile phone, tablet computer, laptop computer, desktop computer, smart TV, vehicle-mounted device, etc. The server can be understood as an application server or a web server. In actual deployment, the server can be a standalone server, a cluster server, or a cloud server.

[0048] In order to facilitate understanding of the technical solution provided by this application, a model determination method provided in an embodiment of this application will be introduced below in combination with an actual application scenario.

[0049] Refer to Figure 1, which is a schematic diagram of a model determination method in an actual application scenario provided by an embodiment of the present application. In this actual application scenario, the computer device can be a model determination server 101 with a model determination function.

[0050] First, the model determination server 101 can obtain a set of sample voice information, which includes multiple sample voice information (for example, N sample voice information, represented as sample voice information 1, sample voice information 2,..., sample voice information N), each sample voice information has corresponding multiple sample sub-voice information, and the multiple sample sub-voice information corresponding to the sample voice information is the accurate voice separation result obtained when the sample voice information is accurately voice separated, and each sample sub-voice information corresponds to a sample pronunciation object.

[0051] Taking the sample voice information 1 as an example, the model determination server 101 can input the sample voice information 1 into the initial voice separation model. The initial quantity determination module in the initial voice separation model can determine the corresponding information on the number of pending pronunciation objects based on the sample voice information 1. The information on the number of pending pronunciation objects is used to identify the number of pronunciation objects analyzed by the module. The initial voice separation module in the initial voice separation model can perform voice separation on the sample voice information 1 based on the sample voice information 1 and the information on the number of pending pronunciation objects to obtain multiple first voice information. The multiple first voice information are the multiple voice information separated by the model.

[0052] Based on the multiple sample sub-speech information corresponding to the sample speech information 1, the actual number of pronunciation objects corresponding to the sample speech information 1 can be determined, and the actual number of pronunciation objects is used to identify the actual number of pronunciation objects corresponding to the sample speech information 1. For example, as can be seen from Figure 1, the sample speech information 1 actually corresponds to 5 pronunciation objects. Therefore, based on the difference between the actual number of pronunciation objects and the pending number of pronunciation objects, the accuracy of the initial speech separation model in analyzing the number of pronunciation objects can be reflected; through the difference between the multiple first speech information and the multiple sample sub-speech information, the overall speech separation accuracy of the initial speech separation model when performing speech separation on the sample speech information 1 can be reflected. Based on this, the model determination server 101 can combine the difference in the number of pronunciation objects and the difference in speech information to adjust the parameters of the initial speech separation model, so that the initial speech separation model learns how to accurately analyze the number of pronunciation objects and separate speech information, and obtains an adjusted speech separation model. When the model is applied, only the voice information to be separated needs to be input into the voice separation model, without the need to input other information. The voice separation model can automatically achieve accurate separation of the voice information to be separated and determine the multiple sub-voice information corresponding to the pronunciation object to be identified, thereby reducing the preliminary preparation required for voice separation, reducing the difficulty of voice separation, and improving the efficiency of voice separation. For example, in actual application, the voice information to be separated can be audio information corresponding to video information including multiple pronunciation objects. Through the voice separation model, the voice information corresponding to the multiple pronunciation objects can be separated, so that the recipient of the video information can select the voice information of the pronunciation object of interest to receive.

[0053] Next, the model determination method and model application method provided in the embodiments of the present application will be introduced in conjunction with the accompanying drawings.

[0054] Referring to FIG. 2 , FIG. 2 is a flow chart of a model determination method provided in an embodiment of the present application. In this embodiment, the method can be executed by a computer device. The method includes:

[0055] S201: Obtain a sample information set.

[0056] The sample information set is used to train an initial speech separation model. The sample information set may include multiple sample speech information. The sample speech information may be any speech information mixed from speech information of multiple pronunciation objects. Each of the multiple sample speech information has corresponding multiple sample sub-speech information, and the sample sub-speech information corresponds one-to-one with the sample pronunciation object. For example, any one of the multiple sample speech information is composed of a combination of the corresponding multiple sample sub-speech information. That is, when accurate speech separation is performed on the sample speech information, the multiple sample sub-speech information constitutes the speech separation result corresponding to the sample speech information.

[0057] For example, the sample voice information may be audio information corresponding to video information including multiple voice objects (e.g., a movie or TV series including multiple characters), and the multiple sample sub-voice information may be voice information corresponding to each voice object in the video information (e.g., voice information of a character). A voice object refers to an object that can generate voice information, such as a person object, an animal object, or other objects. The sample voice object may be the voice object that generates the sample voice information.

[0058] S202: For each sample voice information in the multiple sample voice information, determine the number information of the pending pronunciation objects corresponding to the sample voice information through the initial number determination module in the initial voice separation model, and based on the number information of the pending pronunciation objects, perform voice separation on the sample voice information through the initial voice separation module in the initial voice separation model to obtain multiple first voice information.

[0059] In order to enable the speech separation model to perform speech separation based only on the input speech information, and at the same time not have the limitations of speech separation in the dimension of pronunciation objects, the computer device needs to enable the speech separation model itself to have the ability to analyze the number of pronunciation objects during the model training process. Based on this, in an embodiment of the present application, the initial speech separation model may include an initial quantity determination module and an initial speech separation module, the initial quantity determination module is used to analyze the number of pronunciation objects included in the speech information, and the initial speech separation module is used to separate the speech information based on the number of pronunciation objects obtained by analysis to obtain a separation result. Thus, based on the addition of the initial quantity determination module, the model can have the ability of autonomous quantity analysis, and there is no need to input the number of pronunciation objects into the model during speech separation.

[0060] During the model training process, taking any sample speech information as an example, the computer device can input the sample speech information into the initial speech separation model, and the initial quantity determination module in the initial speech separation model can determine the number of pending pronunciation objects corresponding to the sample speech information. Based on the pending pronunciation object number information, the initial speech separation module in the initial speech separation model can perform speech separation on the sample speech information to obtain multiple first speech information. The pending pronunciation object number information is used to identify the number of pronunciation objects included in the sample speech information analyzed by the initial quantity determination module, and the multiple first speech information are the speech separation results output by the initial speech separation model, and the number of first speech information is the number of pronunciation objects identified by the pending pronunciation object number information.

[0061] S203: Determine the number of actual pronunciation objects corresponding to the sample voice information according to the plurality of sample sub-voice information contained in the sample voice information.

[0062] Because multiple sample sub-speech information pieces represent the accurate speech separation results corresponding to a single sample speech piece, and the sample sub-speech information pieces correspond one-to-one to the sample pronunciation objects, the number of actual pronunciation objects corresponding to the sample speech piece can be determined using the multiple sample sub-speech information pieces, and thus the actual pronunciation object quantity information can be determined. This actual pronunciation object quantity information is used to identify the actual number of pronunciation objects included in the sample speech piece. For example, a computer device can directly determine the number of sample sub-speech information pieces as the actual pronunciation object quantity information.

[0063] S204: According to the difference between the information on the number of pending pronunciation objects and the information on the number of actual pronunciation objects, and according to the difference between the multiple sample sub-speech information and the multiple first speech information contained in the sample speech information, the model parameters corresponding to the initial speech separation model are adjusted to obtain the speech separation model.

[0064] Since the information on the number of pending pronunciation objects can identify the number of pronunciation objects included in the sample voice information analyzed by the initial number determination module, the information on the number of actual pronunciation objects is used to identify the actual number of pronunciation objects included in the sample voice information. Therefore, based on the difference between the information on the number of pending pronunciation objects and the information on the number of actual pronunciation objects, the accuracy of the initial number determination module in analyzing the number of pronunciation objects can be reflected.

[0065] At the same time, since the multiple first voice information are the voice separation results corresponding to the sample voice information output by the initial voice separation model, and the multiple sample sub-voice information are the accurate voice separation results corresponding to the sample voice information, the difference between the multiple first voice information and the multiple sample sub-voice information can reflect the accuracy of the initial voice separation model as a whole when performing voice separation. Therefore, the parameters of the initial voice separation model are adjusted in combination with the difference in these two dimensions. In the process of gradually narrowing the difference in these two dimensions, on the one hand, the initial quantity determination module can learn how to accurately analyze the number of pronunciation objects in the voice information, and on the other hand, the model as a whole can separate more accurate voice separation results, thereby obtaining a voice separation model that can accurately analyze the pronunciation objects included in the voice information and accurately perform voice separation based on the accurately analyzed number of pronunciation objects. The initial quantity determination module can obtain the quantity determination module in the voice separation model through parameter adjustment, and the initial voice separation module can obtain the voice separation module in the voice separation model through parameter adjustment.

[0066] The speech separation model can be used to perform speech separation on the speech information to be separated to obtain corresponding multiple sub-speech information. The speech information to be separated can be any speech information that needs to be speech separated. The sub-speech information obtained by the speech separation model corresponds one-to-one to the pronunciation object to be recognized.

[0067] It can be seen from the above technical solution that in order to enable the model to have the ability to analyze the number of pronunciation objects in the voice information by itself, the initial voice separation model in the present application can include an initial number determination module and an initial voice separation module, wherein the initial number determination module can determine the number of pending pronunciation objects corresponding to the sample voice information, and the pending pronunciation object number information is used to identify the number of pronunciation objects corresponding to the sample voice information analyzed by the initial number determination module; through the initial voice separation module, the sample voice information can be separated based on the number of pronunciation objects identified by the pending pronunciation object number information to obtain multiple first voice information. Through the multiple sample sub-voice information corresponding to the sample voice information, it can be reflected that when the sample voice information is accurately voice-separated, the number of pronunciation objects corresponding to the sample voice information and the multiple sub-voice information obtained by separation can be reflected. Therefore, the actual number of pronunciation objects corresponding to the sample voice information can be determined by multiple sample sub-voice information. The difference between the actual number of pronunciation objects and the number of pending pronunciation objects can reflect the accuracy of the initial number determination module for the analysis of the number of pronunciation objects. The difference between multiple sample sub-voice information and multiple first voice information can reflect the accuracy of the initial voice separation module when performing voice information separation. Thus, by combining the differences in these two dimensions to adjust the parameters of the initial voice separation model, on the one hand, the initial number determination module can learn how to accurately analyze the number of pronunciation objects included in the voice information, and on the other hand, the initial voice model can learn how to accurately perform voice separation on the voice information based on the number of pronunciation objects information, and obtain voice information that can form an accurate one-to-one correspondence between the pronunciation objects. It can be seen that the model determination method of the present application can generate a voice separation model that can independently perform voice object analysis and accurately perform voice separation based on the number of pronunciation objects obtained by analysis. When applied, only the voice information to be separated needs to be input to obtain the voice separation result, and there is no need to perform voice object recognition on the voice information before input. While ensuring the accuracy of voice separation, the convenience and efficiency of voice separation are improved.

[0068] Among them, the form of the pronunciation object quantity information can include multiple. In a possible implementation method, the pronunciation object quantity information can be information composed of counting identifiers. When the undetermined pronunciation object quantity information corresponding to the sample voice information is determined by the initial quantity determination module, the sample voice information can be first separated in the voice feature dimension to determine the multiple second sub-voice features corresponding to the sample voice information. The multiple second sub-voice features are the separation results of the initial quantity determination module for separating the sample voice information in the voice feature dimension. The voice feature refers to the feature of the voice information, which can identify the information characteristics of the voice information.

[0069] It is understandable that the characteristics of the speech information of different pronunciation objects are usually different. For example, the speech information of different pronunciation objects is usually different in time domain features and frequency domain features. Therefore, the speech features can characterize the pronunciation objects to a certain extent. Therefore, in the present application, the initial quantity determination module can analyze the multiple second sub-speech features separately to determine whether each second sub-speech feature can characterize a pronunciation object. The computer device can use the initial quantity determination module to determine the pending counting information corresponding to the multiple second sub-speech features according to the multiple second sub-speech features. The pending counting information is one of the counting identifiers, and the pending counting information is used to identify the probability that the corresponding second sub-speech feature corresponds to a single sample pronunciation object. That is, by analyzing each second sub-speech feature, the initial quantity determination module can determine whether each second sub-speech feature can characterize a single sample pronunciation object, thereby determining the pending counting information.

[0070] The computer device may determine the pending count information corresponding to each of the plurality of second sub-speech features as information on the number of pending pronunciation objects. The information on the number of pending pronunciation objects is used to identify the number of second sub-speech features corresponding to a single sample pronunciation object. Because the pending count information can identify the probability that the second sub-speech feature corresponds to a single sample pronunciation object, the number of second sub-speech features corresponding to the single sample pronunciation object among the plurality of second sub-speech features can be determined by combining the pending count information corresponding to each of the plurality of second sub-speech features, thereby identifying the number of sample pronunciation objects.

[0071] Similarly, when determining the number of actual pronunciation objects corresponding to the sample voice information based on the multiple sample sub-voice information contained in the sample voice information, the computer device may execute S2031-S2033 (not shown in the figure). S2031-S2033 is a possible implementation of S203:

[0072] S2031: Determine second sub-speech information corresponding to a plurality of second sub-speech features.

[0073] It can be understood that the voice features are extracted from the voice information, so the voice information can be restored based on the voice features, that is, the voice features corresponding to the second sub-voice information are the second sub-voice features, so the second sub-voice information can be restored based on the second sub-voice features.

[0074] S2032: For each second sub-voice information in the plurality of second sub-voice information, determine the maximum value of the similarities respectively corresponding to the second sub-voice information and the plurality of sample sub-voice information as the actual count information corresponding to the second sub-voice information.

[0075] As mentioned above, the pending count information is used to identify the probability that the corresponding second sub-speech feature corresponds to a single sample pronunciation object, and the sample sub-speech information is the speech information corresponding to a single sample pronunciation object. Therefore, through the similarity between the second sub-speech information and the sample sub-speech information, it can reflect the probability that the second sub-speech information is the speech information corresponding to a single sample pronunciation object, thereby reflecting the probability that the second sub-speech feature corresponding to the second sub-speech information corresponds to a single sample pronunciation object.

[0076] Based on this, the computer device can calculate the similarity between each second sub-voice information in the plurality of second sub-voice information and the plurality of sample sub-voice information. The maximum value among the similarities can indicate that the second sub-voice information has the highest similarity with the sample sub-voice information, i.e., the second sub-voice information is most likely to correspond to the sample pronunciation object corresponding to the sample sub-voice information. Therefore, the computer device can determine the maximum value among the similarities between the second sub-voice information and the plurality of sample sub-voice information as the actual count information corresponding to the second sub-voice information, thereby measuring whether the second sub-voice information corresponds to a single sample pronunciation object.

[0077] S2033: Determine the actual count information corresponding to the plurality of second sub-speech features as actual pronunciation object quantity information.

[0078] Since the actual counting information corresponding to each second sub-speech feature can reflect the probability of whether the second sub-speech feature actually corresponds to a single sample pronunciation object, the computer device can determine the actual counting information corresponding to multiple second sub-speech features as the actual pronunciation object quantity information.

[0079] When executing S204, the computer device may execute S2041-S2042 (not shown in the figure). S2041-S2042 is a possible implementation of S204:

[0080] S2041: For each second sub-speech feature among the plurality of second sub-speech features, based on a difference between pending count information corresponding to the second sub-speech feature and actual count information, adjusting a model parameter corresponding to an initial quantity determination module.

[0081] The pending count information corresponding to the second sub-speech feature can reflect the probability of corresponding to a single sample pronunciation object analyzed by the initial speech separation model. The actual pronunciation object quantity information is the probability that the second sub-speech feature actually corresponds to a single sample pronunciation object. Therefore, the difference can reflect the accuracy of the initial quantity determination module in analyzing whether the second sub-speech feature corresponds to a single sample pronunciation object. Adjusting the initial quantity determination module based on the difference can enable the module to learn how to accurately analyze whether the sub-speech feature corresponds to a single pronunciation object, and then learn how to determine the accurate pronunciation object quantity information.

[0082] S2042: Adjusting model parameters corresponding to the initial speech separation model according to differences between the plurality of sample sub-speech information and the plurality of first speech information contained in the sample speech information.

[0083] Since the speech separation result is obtained by the collaboration of multiple modules in the initial speech separation model, the computer device can adjust the initial speech separation model as a whole based on the differences in the speech separation results to obtain an accurate and effective speech separation model.

[0084] As mentioned above, the number of pronunciation objects is key information in the speech separation process. Since the speech features of the sample speech information can be separated by the initial quantity determination module to obtain multiple second sub-speech features, and the pending count information corresponding to the multiple second sub-speech features can identify the probability that the second sub-speech feature corresponds to a single sample pronunciation object, therefore, based on the pending count information corresponding to the multiple second sub-speech features, the number of second sub-speech features corresponding to a single sample pronunciation object in the multiple second sub-speech features can be determined. This number is the pending pronunciation object number information analyzed by the initial quantity determination module.

[0085] Based on this, when the initial speech separation module in the initial speech separation model performs speech separation on the sample speech information based on the information about the number of undetermined pronunciation objects and obtains multiple first speech information, the computer device can determine the number of second sub-speech features among the multiple second sub-speech features whose corresponding probability identified by the undetermined counting information is greater than a first threshold as the number of undetermined pronunciation objects. Once the number of undetermined pronunciation objects is determined, the initial speech separation module can use the number of undetermined pronunciation objects as the basis for speech separation and perform speech separation on the sample speech information based on the number of undetermined pronunciation objects to obtain the number of first speech information of the undetermined pronunciation objects.

[0086] It can be seen from the above content that the pending count information corresponding to the second sub-speech feature is one of the key information in the speech separation process, and the pending count information is determined based on the second sub-speech feature. Therefore, the analysis accuracy of the second sub-speech feature by the initial quantity determination module is also one of the important factors affecting the accuracy of speech separation.

[0087] Based on this, in a possible implementation, in order to further improve the accuracy of speech separation, the computer device may improve the analysis accuracy of the second sub-speech feature by the initial quantity determination module.

[0088] When executing S2041, the computer device may execute S20411 (not shown in the figure). S20411 is a possible implementation of S2041:

[0089] S20411: Based on the difference between the second sub-speech information corresponding to the second sub-speech feature and the target sub-speech information, and the difference between the pending count information corresponding to the second sub-speech feature and the actual count information, adjust the model parameters corresponding to the initial quantity determination module.

[0090] Among them, the target sub-voice information is the sample sub-voice information with the greatest similarity to the second sub-voice information corresponding to the second sub-voice feature among multiple sample sub-voice information, that is, the sample pronunciation object represented by the second sub-voice feature is likely to be the sample pronunciation object corresponding to the target sub-voice information. If the initial quantity determination module extracts the second sub-voice feature more accurately, the second sub-voice feature is closer to the voice feature of the sample pronunciation object, then the second sub-voice information corresponding to the second sub-voice feature should be closer to the target sub-voice information.

[0091] Based on this, on the one hand, the computer device can, based on the difference between the second sub-speech information corresponding to the second sub-speech feature and the target sub-speech information, enable the initial quantity determination module to learn how to analyze and extract the speech features of the speech information during the model training process, so that the extracted speech features can effectively represent each pronunciation object; on the other hand, based on the difference between the pending counting information corresponding to the second sub-speech feature and the actual counting information, the initial quantity determination module can learn how to accurately analyze the counting information based on the extracted speech features, thereby improving the accuracy of the determined pronunciation object quantity information in these two dimensions, thereby improving the accuracy of speech separation.

[0092] As mentioned above, the speech features of different pronunciation objects are different, and the speech features can represent the pronunciation objects to a certain extent. Therefore, in one possible implementation method, the computer device can guide the speech information separation process of the initial speech separation module based on the representation of the pronunciation object by the second sub-speech features obtained by the initial speech separation module analysis.

[0093] When determining the number of pending pronunciation objects corresponding to the sample voice information, the computer device can also determine multiple second sub-voice features corresponding to the sample voice information, and the second sub-voice features are consistent with the second sub-voice features mentioned above. The computer device can determine the number of pending pronunciation objects corresponding to the sample voice information based on the multiple second sub-voice features. The number of pending pronunciation objects is used to identify multiple standard features corresponding to a single sample pronunciation object among the multiple second sub-voice features, that is, the standard feature is the second sub-voice feature determined by the initial quantity determination module to correspond to the single sample pronunciation object. For example, the computer device can analyze the above-mentioned pending count information and determine the second sub-voice feature with a higher probability identified by the pending count information as the standard feature, wherein the current standard feature can be used to identify the pronunciation feature of the pronunciation object corresponding to the current standard feature. The pronunciation feature refers to the feature when the pronunciation object emits voice information. The current standard feature is the voice feature corresponding to the pronunciation object. The current standard feature can be any standard feature.

[0094] When the sample voice information is subjected to voice separation by the initial voice separation module in the initial voice separation model based on the information on the number of undetermined pronunciation objects to obtain multiple first voice information, the computer device can use multiple standard features as current standard features respectively, and the computer device can determine the first voice information corresponding to the current standard feature based on the current standard feature and the sample voice information, that is, the computer device can perform information separation from the sample voice information based on the current standard feature, and separate the voice information whose corresponding voice features are more consistent with the current standard feature as the first voice information corresponding to the current standard feature. Since the current standard feature can identify the pronunciation features of the corresponding pronunciation object, the first voice information corresponding to the current standard feature can be determined as the voice information corresponding to the pronunciation object corresponding to the current standard feature.

[0095] Therefore, when performing speech separation, on the one hand, there is a constraint on the number of pronunciation objects, so that the separated speech information can fit the number of pronunciation objects included in the speech information to be separated; on the other hand, there is a guidance of standard features, which can enable speech separation to obtain speech information that accurately corresponds to the pronunciation objects to be identified in the speech information to be separated, thereby improving the overall accuracy of speech separation.

[0096] When determining the first voice information corresponding to the current standard feature based on the current standard feature and the sample voice information, in a possible implementation, the computer device may make the determination based on the similarity between the voice features.

[0097] First, the computer device can determine a sample speech feature corresponding to the sample speech information. The sample speech feature is the speech feature corresponding to the entire sample speech information. Therefore, the sample speech feature contains speech features corresponding to speech information of multiple sample pronunciation objects. The computer device can extract a first sub-speech feature corresponding to the current standard feature from the sample speech feature based on the similarity between the current standard feature and each speech feature portion of the sample speech feature. The similarity between the first sub-speech feature and the current standard feature is greater than a second threshold. The second threshold is used to determine whether the speech features have a high degree of similarity. In other words, if the first sub-speech feature has a high degree of similarity with the current standard feature, the first sub-speech feature is likely to correspond to the same sample pronunciation object as the current standard feature. Thus, the computer device can determine the first speech information corresponding to the current standard feature based on the first sub-speech feature corresponding to the current standard feature. The first speech information can be determined to be the speech information corresponding to the pronunciation object represented by the current standard feature.

[0098] It can be understood that since standard features correspond one-to-one to sample pronunciation objects, different standard features usually correspond to different sample pronunciation objects. Therefore, in one possible implementation method, in order to further improve the accuracy of speech separation, the computer device can not only extract similar speech features based on the current standard features, but also eliminate the parts of the extracted speech features that have a probability of corresponding to other sample pronunciation objects based on other standard features.

[0099] The computer device can extract the first sub-speech features corresponding to the multiple standard features in the above manner. Each first sub-speech feature can, to a certain extent, reflect the speech information corresponding to the pronunciation object represented by the corresponding standard feature in the sample speech information. Therefore, when determining the first speech information corresponding to the current standard feature based on the first sub-speech feature corresponding to the current standard feature, the computer device can first use the first sub-speech features corresponding to the remaining standard features in the multiple standard features as multiple comparison sub-speech features. If there is a similar feature part between the first sub-speech feature corresponding to the current standard feature and the comparison sub-speech feature, it means that the feature part has a high probability of corresponding to other pronunciation objects, and when determining the speech information based on the first sub-speech feature, there is a high probability that the speech information corresponding to the other pronunciation object will be determined.

[0100] Based on this, the computer device can extract the first speech feature corresponding to the current standard feature from the first sub-speech feature corresponding to the current standard feature based on multiple comparison sub-speech features. The first speech feature is a first sub-speech feature whose similarity with multiple comparison sub-speech features is less than a third threshold value. The remaining standard features are standard features other than the current standard feature among the multiple standard features. The third threshold value is used to determine whether there is a small similarity between the speech features. That is, based on the third threshold value, the computer device can eliminate the speech feature portion with a high similarity to the comparison sub-speech feature in the first sub-speech feature corresponding to the current standard feature, thereby eliminating the speech feature portion with a high probability of corresponding to other sample pronunciation objects, making the association between the first speech feature and the sample pronunciation object represented by the current standard feature closer.

[0101] Furthermore, the computer device can determine the first voice information corresponding to the current standard feature based on the first voice feature corresponding to the current standard feature, so that the correspondence between the first voice information and the pronunciation object represented by the current standard feature can be more accurate, reducing the interference of voice information of other pronunciation objects on the first voice information, and obtaining a more accurate voice separation result.

[0102] Specifically, in one possible implementation, to further improve the accuracy of speech separation, after extracting the first sub-speech feature corresponding to the current standard feature using the above method, the computer device may further, through the initial speech separation module, extract a second speech feature corresponding to the current standard feature from the first sub-speech feature corresponding to the current standard feature based on the current standard feature. Similar to the method for extracting the first sub-speech feature, the computer device may extract based on the similarity between each feature portion of the first sub-speech feature and the current standard feature, to obtain a second speech feature that is closer to the current standard feature. The second speech feature is a first sub-speech feature whose similarity to the current standard feature is greater than a fourth threshold, where the fourth threshold is greater than the second threshold. This process further refines the first sub-speech feature based on the current standard feature.

[0103] When determining the first speech information corresponding to the current standard feature based on the first speech feature corresponding to the current standard feature, the computer device can combine the feature extraction based on the current standard feature with the feature extraction based on other standard features, fuse the first speech feature corresponding to the current standard feature and the second speech feature corresponding to the current standard feature, and generate a speech feature corresponding to the current standard feature, so that the speech feature has a high degree of fit for the pronunciation feature of the pronunciation object identified by the current standard feature on the one hand, and on the other hand, the probability of the speech feature containing the speech feature of the pronunciation object identified by other standard features is low. Therefore, the computer device can determine the first speech information corresponding to the current standard feature based on the speech feature corresponding to the current standard feature, further improve the degree of correlation between the first speech information and the pronunciation object represented by the current standard feature, and thereby improve the accuracy of speech separation.

[0104] As can be seen from the above, the overall speech characteristics of the sample speech information are also one of the key pieces of information used for speech separation. Therefore, the computer equipment can not only optimize the analysis of the number of pronunciation objects and speech separation, but also more accurately process the speech feature extraction of the sample speech information, thereby obtaining more accurate speech features that reflect the characteristics of the sample speech information.

[0105] In one possible implementation, the initial speech separation model may further include an initial feature extraction module, which is configured to extract sample speech features corresponding to the sample speech information. The sample speech features are configured to characterize information features corresponding to the sample speech information, such as features in the time domain or frequency domain. Therefore, in the embodiment of the present application, the sample speech features corresponding to the sample speech information may also be extracted by the initial feature extraction module.

[0106] Based on this, when executing S202, the computer device may execute S2021 (not shown in the figure), which is a possible implementation of S202:

[0107] S2021: According to the sample voice features, the initial quantity determination module in the initial voice separation model is used to determine the quantity information of the pending pronunciation objects corresponding to the sample voice information, and based on the quantity information of the pending pronunciation objects and the sample voice features, the initial voice separation module in the initial voice separation model is used to perform voice separation on the sample voice information to obtain multiple first voice information.

[0108] In the initial quantity determination module, the sample speech features can be analyzed to extract the above-mentioned multiple second sub-speech features, thereby analyzing the number of features corresponding to a single sample pronunciation object to determine the number information of the pending pronunciation objects; in the initial speech separation module, based on the number information of the pending pronunciation objects, the above-mentioned multiple first sub-speech features can be extracted from the sample speech features to determine multiple first speech information.

[0109] Referring to Figure 3, Figure 3 is a schematic diagram of an initial quantity determination module provided by an embodiment of the present application. The initial quantity determination module may include two two-dimensional convolutional networks (2D Convolutional Neural Network, referred to as 2DCNN), padding and reshaping layers (Reshape), a bidirectional long short-term memory network model (Bidirectional Long Short-Term Memory, referred to as BLSTM) and an activation function (Relu). The computer device can input the sample speech features into the first two-dimensional convolutional network in the initial quantity determination module to obtain multiple features. After padding and reshaping the features based on a preset mask matrix size, a mask matrix can be obtained through the second two-dimensional convolutional network. The mask matrix is ​​a feature matrix composed of multiple second sub-speech features. As shown in Figure 3, the length of the mask matrix can be a preset length, for example, set to N, then the maximum number of pronunciation objects determined by the initial quantity determination module is N, and N can be, for example, 128. The size of the mask matrix is ​​scalable, and by changing it in the padding and reshaping stages, the size (i.e., dimension) of the mask matrix can be controlled.

[0110] Through the bidirectional long short-term memory network model and activation function, the pending count information corresponding to each second sub-speech feature can be analyzed to obtain the probability that the second sub-speech feature corresponds to a single pronunciation object. Based on this pending count information, the 01 coding vector corresponding to the second sub-speech feature can be determined, where 1 is used to indicate that the corresponding second sub-speech feature corresponds to a single sample pronunciation object, and 0 is used to indicate that the corresponding second sub-speech feature does not correspond to a single sample pronunciation object. Therefore, the number of pending pronunciation objects can be reflected by the number of 1s in the 01 coding vector. For example, the computer device can determine the 01 coding vector of the second sub-speech feature whose probability identified by the pending count information is greater than a threshold as 1, otherwise it is determined as 0. The module can directly input the number of pending pronunciation objects and the second sub-speech feature whose corresponding 01 coding vector is 1 into the initial speech separation module for speech separation.

[0111] Referring to FIG4 , FIG4 is a schematic diagram of an initial speech separation module provided in an embodiment of the present application. The initial speech separation module is composed of multiple decoders connected in parallel, the number of which is the number of pending pronunciation objects analyzed by the initial quantity determination module, that is, the decoders are one-to-one with the pronunciation objects, and one decoder is used to separate the speech information corresponding to one pronunciation object. The input of each decoder is the sample speech feature corresponding to the sample speech information and the standard feature (also called mask vector) corresponding to a single pronunciation object in the mask matrix.

[0112] As can be seen in Figure 4, each encoder is composed of an attention calculation mechanism, a two-dimensional convolutional network, and Mel-frequency cepstrum calculation. Taking the current standard feature as an example, the current standard feature and the sample speech feature are first input into an attention calculation mechanism. The current standard feature calculates its similarity with each speech feature in the sample speech feature, and then outputs a two-dimensional feature (i.e., the first sub-speech feature) through the two-dimensional convolutional network. The application of the two-dimensional convolutional network is mainly to expand the features. Because a large number of small convolution kernels may be used for calculation when extracting sample speech features, this calculation will gradually reduce the size of the speech features. Therefore, in order to balance the feature size, two two-dimensional convolutional networks are added to the decoder to expand the feature size, so that the time length of the final output speech information can be the same as the time length of the original speech information.

[0113] As shown in Figure 4, in addition to the decoder's own decoding chain, there is also an attention mechanism calculation between different decoders. After determining the first sub-speech feature, the first sub-speech features calculated by n decoders are concatenated to form a multi-layer feature. Then, the attention mechanism calculation is performed. The calculation process is to use the first sub-speech feature output by the decoder corresponding to the current standard feature to calculate the similarity with the first sub-speech features output by the other n-1 decoders. Parts with excessive similarity to the first sub-speech features output by the other n-1 decoders are eliminated to obtain the first speech feature. Then, in the decoder corresponding to the current standard feature, the second speech feature extracted from the first sub-speech feature is fused using the second attention calculation mechanism and the two-dimensional convolutional network to obtain the speech feature corresponding to the current standard feature. After the Mel-frequency cepstrum calculation, the first speech information is obtained. This allows the decoder to perceive the speech information of the pronunciation object decoded by other decoders, thereby reducing the weight of the information dimension that the decoder focuses on by other decoders. This achieves information exchange between multiple decoders and improves the purity of the final speech separation.

[0114] It is understood that speech information is typically composed of information in two dimensions: time domain information and frequency domain information. Time domain information can, for example, reflect characteristics such as the pronunciation rhythm of the speaker, while frequency domain information can, for example, reflect characteristics such as the pronunciation pitch of the speaker. Based on this, in one possible implementation, to highlight the characteristics of the target sample speech information in both the time and frequency domains, the computer device can perform speech feature extraction based on the time and frequency domain features of the target sample speech information, respectively.

[0115] Specifically, when extracting sample speech features corresponding to the sample speech information, the computer device may extract a first feature corresponding to the sample speech information based on the time domain information corresponding to the sample speech information, and extract a second feature corresponding to the sample speech information based on the frequency domain information corresponding to the sample speech information. Thus, the first feature can reflect the time domain features of the sample speech information, and the second feature can reflect the frequency domain features of the sample speech information. The computer device can determine the sample speech features based on the first and second features, thereby enabling the sample speech features to better represent both the time domain features and the frequency domain features of the sample speech information, thereby facilitating more accurate analysis and processing of the sample speech features.

[0116] Because speech information is composed of both frequency and time domain information, extracting features based solely on a single dimension of information can significantly reduce the amount of extracted speech features. For example, extracting features based solely on time domain information can significantly reduce the amount of frequency domain information, potentially resulting in a poor overall representation of the sample speech information. Therefore, in one possible implementation, the computer device can extract both the first and second features using a combination of information from both dimensions.

[0117] In addition, it can be understood that feature extraction is the process of extracting some representative information from information. When the amount of information to be extracted is the same, the more feature extractions are performed in the feature extraction process, the more detailed the feature extraction of the voice information will be to a certain extent, and more details of the voice information can be noticed. For example, when extracting voice features by convolution, the related art usually uses a larger convolution kernel to obtain voice features based on a single feature extraction; in this application, the large convolution kernel can be split into multiple small convolution kernels, and a small convolution kernel corresponds to a single feature extraction, so that the voice features corresponding to the voice information can be extracted through multiple feature extractions. On the one hand, it expands the deeper convolution network layer, and on the other hand, the use of small convolution kernels can pay attention to more information details when performing feature extraction, because under the calculation of the entire convolution kernel, the smaller the scale, the more changes in information details can be noticed.

[0118] Therefore, in the present application, the computer device can combine information from multiple dimensions and extract the sample speech features corresponding to the sample speech information through multiple feature extractions. For example, when extracting the first feature corresponding to the sample speech information based on the time domain information corresponding to the sample speech information, the computer device can determine the first feature corresponding to the sample speech information through N feature extractions, where N is a positive integer greater than 1. The output information of the i-th feature extraction is the input information of the i+1-th feature extraction, that is, the latter feature extraction is a feature extraction of the information output by the previous feature extraction, and i is a positive integer less than N.

[0119] In order to enable the first feature to highlight the time domain characteristics of the sample voice information, the first M feature extractions in the N feature extractions are based on the time domain information corresponding to the sample voice information, and the subsequent N minus M feature extractions are based on the frequency domain information corresponding to the sample voice information, where M is a positive integer less than N. The advantage of this feature extraction method is that, since the first M feature extractions are based on the complete sample voice information, the feature extraction based on the time domain information can maximize the retention of the time domain characteristics corresponding to the sample voice information. The subsequent N minus M feature extractions are based on the frequency domain information on the information obtained from the first M feature extractions. Therefore, although feature extraction cannot be based on the complete frequency domain information, both frequency domain information and time domain information are taken into account to achieve the extraction of the first feature.

[0120] Similarly, when extracting the second feature corresponding to the sample voice information based on the frequency domain information corresponding to the sample voice information, the computer device can determine the second feature corresponding to the sample voice information through P feature extractions, where P is a positive integer greater than 1. The output information of the kth feature extraction is the input information of the k+1th feature extraction, the first Q feature extractions are based on the frequency domain information corresponding to the sample voice information, and the subsequent PQ feature extractions are based on the time domain information corresponding to the sample voice information, and both k and Q are positive integers less than N. Since the first Q feature extractions are performed on the basis of the complete sample voice information, first performing feature extraction based on the frequency domain information can retain the frequency domain features corresponding to the sample voice information to the greatest extent possible. The subsequent PQ feature extractions are based on the information obtained from the first Q feature extractions and the frequency domain information. Therefore, although feature extraction cannot be performed based on the complete time domain information, both the frequency domain information and the time domain information are taken into account to achieve the extraction of the second feature. It can be seen from this that although the first feature and the second feature both take into account the feature information of the two dimensions of time domain and frequency domain for feature extraction, to a certain extent, the first feature can more accurately reflect the time domain features of the sample voice information, and the second feature can more accurately reflect the frequency domain features of the sample voice information. While ensuring the rationality of feature extraction, it strengthens the representation effect of the sample voice features on the features of each dimension. At the same time, this application realizes fine-grained analysis of the sample voice information through multiple feature extractions, thereby being able to focus on more information details in the sample voice information and improve the accuracy of voice feature extraction.

[0121] As mentioned above, the subsequent feature extractions in multiple feature extractions are based on the output information of the previous feature extractions. In some cases, if each feature extraction is based on information of a single dimension, it will, to a certain extent, result in the inability to fully extract the information features of a certain dimension. For example, when extracting the first feature, if the first M feature extractions are based only on time domain information without considering the frequency domain information corresponding to the sample voice information, some frequency domain features corresponding to the sample voice information may be ignored. Although the last N minus M feature extractions are based on frequency domain information, if feature extraction is performed based on the output information of the Mth feature extraction, the loss of frequency domain information during the first M feature extractions may be irreversible.

[0122] Based on this, in a possible implementation, in order to reduce information loss of the sample speech information during feature extraction, the computer device may determine the sample speech features in combination with the overall information features of the sample speech information.

[0123] Based on the time and frequency domain information corresponding to the sample speech information, the computer device can perform a single feature extraction through an initial feature extraction module to obtain a direct mapping feature corresponding to the sample speech information. Because this feature extraction takes into account both time and frequency domain information and only involves a single feature extraction, while it cannot specifically characterize features in a specific time or frequency domain dimension, it does retain the information characteristics of the sample speech information to the greatest extent possible, minimizing information loss.

[0124] When determining a first feature corresponding to the sample speech information through N feature extractions, the computer device may determine a first undetermined feature corresponding to the sample speech information through the N feature extractions, where the first undetermined feature is the output of the Nth feature extraction, thereby enabling the first undetermined feature to more prominently characterize the time domain features. The computer device may combine the first undetermined feature with the direct mapping feature to obtain the first feature corresponding to the sample speech information, thereby enabling the first feature to highlight the time domain features of the sample speech information while also taking into account the overall information features of the sample speech information, thereby reducing the loss of frequency domain features during the information feature extraction process.

[0125] Similarly, when determining the second feature corresponding to the sample speech information through P feature extractions, the computer device can determine the second undetermined feature corresponding to the sample speech information through P feature extractions. The second undetermined feature is the output of the P-th feature extraction, so that the first undetermined feature can more prominently represent the frequency domain features. The computer device can combine the second undetermined feature and the direct mapping feature to obtain the second feature corresponding to the sample speech information, so that the first feature can highlight the frequency domain features of the sample speech information while taking into account the overall information features of the sample speech information, thereby reducing the loss of time domain features during the information feature extraction process.

[0126] In addition to the above-mentioned combination methods, the computer device may also adopt a method of directly combining multiple features to determine the sample voice feature corresponding to the sample voice information.

[0127] Specifically, the computer device can determine the third feature corresponding to the sample voice information through T feature extractions, wherein the wth feature extraction is used to perform feature extraction based on the time domain information and frequency domain information corresponding to the sample voice information, T is a positive integer greater than 1, and w is a positive integer not exceeding T. That is, in an embodiment of the present application, in order to obtain a more accurate overall feature extraction corresponding to the sample voice information, the third feature can be obtained by combining information of two dimensions for multiple feature extractions. For example, the computer device can extract through a small two-dimensional convolution kernel during each feature extraction, and refine the feature analysis granularity while combining the two-dimensional information. The number of feature extractions for the above-mentioned multiple feature extractions can be set based on actual needs and is not limited here.

[0128] When determining the sample speech feature based on the first feature and the second feature, the computer device may determine the sample speech feature based on the first feature, the second feature, and the third feature. The first feature and the second feature can respectively highlight the characteristics of the sample speech information in the time domain and the frequency domain, while the third feature can reflect the overall characteristics of the sample speech information in the two information dimensions. This allows the sample speech feature to take into account both single-dimensional characteristics and multi-dimensional overall characteristics, thereby improving the representation effect of the sample speech feature on the sample speech information and facilitating subsequent accurate speech separation of the sample speech information based on the sample speech feature.

[0129] See Figure 5, which is a schematic diagram of an initial feature extraction module provided in an embodiment of the present application. The module is divided into three branches, namely the left branch, the middle branch, and the right branch. In the three branches, the left and right branches are both composed of a convolutional network composed of a one-dimensional small convolution kernel and a residual link, and the middle branch is composed of a convolutional network composed of a two-dimensional small convolution kernel. The three branches then merge features to obtain the average feature (mean), and finally pass through a large pooling layer to obtain the sample speech feature, such as a coding feature map.

[0130] After the sample speech information is input, its corresponding Mel feature is calculated first, and then feature extraction is performed through three branches. The upper half of the left branch is based on feature extraction of time domain information, that is, the first three of the six feature extractions are one-dimensional convolution and one-dimensional pooling based on time domain information, and the lower half is based on feature extraction of frequency domain information, that is, the last three feature extractions are one-dimensional convolution and one-dimensional pooling in the frequency domain dimension. The architecture of the right branch is opposite to that of the left branch. The purpose of this architecture is to allow half of the network layers to focus on perceptual learning of time domain information, and the other half to focus on perceptual learning of frequency domain information. This allows the network to obtain detailed information in both the time domain and the frequency domain. Then, a residual link is added to the left and right branches respectively, that is, a large convolution kernel and a large pooling layer are used to extract the direct mapping features corresponding to the sample speech information at one time, and concatenate the features with the results of the last branch. For example, the first pending feature obtained by the left branch and the second pending feature obtained by the right branch are concatenated to obtain the first feature and the second feature, so that the sample speech features extracted by the model can be combined with the overall features of the underlying sample speech information and the detailed features obtained by multiple feature extractions at the high level.

[0131] The middle branch uses a two-dimensional convolutional network composed of two-dimensional small convolution kernels. Through a three-layer network, three feature extractions are performed based on time domain information and frequency domain information to obtain the third feature. The features output by the three branches are then averaged. Finally, after passing through a pooling layer on a channel, the final sample speech feature output can be obtained.

[0132] Based on the speech separation model determined in the above embodiments, the present application also provides a model application method, wherein the model application method can be applied to the speech separation model determined in any of the above embodiments.

[0133] Referring to FIG. 6 , FIG. 6 is a flow chart of a model application method provided in an embodiment of the present application. In this embodiment, the method may be executed by a computer device, and the method includes:

[0134] S601: Acquire voice information to be separated.

[0135] The speech information to be separated can be any speech information that needs to be separated. The speech information to be separated is composed of sub-speech information corresponding to multiple pronunciation objects to be identified. That is, through the speech information to be separated, sub-speech information corresponding to multiple pronunciation objects to be identified can be received. For example, the speech information to be separated can be the audio information corresponding to a movie, a conference recording, etc. The pronunciation objects to be identified can be pronunciation objects that need to be distinguished during the model application process, such as actors in a movie, speakers at a conference, etc.

[0136] S602: According to the voice information to be separated, the number of pronunciation objects corresponding to the voice information to be separated is determined by the number determination module in the voice separation model, and according to the number of pronunciation objects, the voice information to be separated is voice separated by the voice separation module in the voice separation model to obtain the number of pronunciation objects.

[0137] The sub-speech information of the number of pronunciation objects is the speech separation result corresponding to the speech information to be separated, wherein the sub-speech information of the number of pronunciation objects corresponds one-to-one to the pronunciation objects to be identified, and the number of pronunciation objects information is used to identify the number of pronunciation objects corresponding to the speech information to be separated. The number determination module is a module determined by the initial number determination module through the above-mentioned parameter adjustment method, and the speech separation module is a module determined by the initial speech separation module through the above-mentioned parameter adjustment method.

[0138] It should be noted that the method provided in the embodiment of the present application can be applied to various scenarios where voice separation is required, such as creative platforms, live broadcast platforms, etc. In the creative platform, the voice of the pronunciation object (i.e., the speaker) is separated for the numerous materials in the massive film and television drama material library, and then the user can specifically pick out all the lines and voice clips of one of the speakers according to his or her preferences, so that there are more ways to play in the editing. For example, for a compilation of highlights of a star, the embodiment of the present application can separate the voice of the star that is mixed with other stars, so that the pure voice of the star can be extracted.

[0139] On live streaming platforms, when multiple live streams are connected, their voices are often mixed. The live streams of the anchors are often recorded. Live streaming platforms also provide a voice separation function. Accurately recognizing mixed voices is difficult. The present embodiment of the application provides a method for separating mixed voices from recorded videos, allowing for separate voice recognition and subtitle generation. This improves the viewing experience of recorded videos on live streaming platforms.

[0140] As mentioned above, speech features can play a role in characterizing the pronunciation objects during the speech separation process. Similar to the model determination process, during the model application process, when determining the number of pronunciation objects corresponding to the speech information to be separated, the computer device can determine the number of sub-speech features of the pronunciation objects corresponding to the speech information to be separated. The number of sub-speech features of the pronunciation objects corresponds one-to-one to the pronunciation objects to be identified, and the sub-speech features are used to characterize the speech features of the corresponding pronunciation objects to be identified. This capability of the model is based on the parameter adjustment of the initial quantity determination module based on multi-dimensional differences during the above-mentioned model determination process, so that the module can learn how to determine the accurate second sub-speech feature and accurately analyze the counting information corresponding to the second sub-speech feature, thereby accurately determining the sub-speech feature corresponding to a single pronunciation object.

[0141] When the speech information to be separated is speech-separated by the speech separation module in the speech separation model based on the information on the number of pronunciation objects, and sub-speech information of the number of pronunciation objects is obtained, the speech separation module can be used to speech-separate the speech information to be separated based on the information on the number of pronunciation objects and the sub-speech features of the number of pronunciation objects to obtain sub-speech information of the number of pronunciation objects, thereby achieving the quantitative restriction of speech information based on the information on the number of pronunciation objects and the representation of the pronunciation objects to be identified by the sub-speech features, and determining the sub-speech information of the number of pronunciation objects corresponding to the speech information to be separated. Through the sub-speech features, the computer device can extract speech information that matches each sub-speech feature from the speech information to be separated, and then obtain the speech information corresponding to the pronunciation object to be identified represented by each sub-speech feature. Among them, the speech separation model in the model application method can be a speech separation model determined by any of the above-mentioned model determination methods.

[0142] It is understandable that the number of pronunciation objects included in different speech information to be separated may vary. In this application, the speech separation process for each pronunciation object is the same, that is, the model parameters required for speech separation for each pronunciation object are the same. Based on this, in one possible implementation, the computer device can use parallel processing to separate the speech information corresponding to multiple pronunciation objects.

[0143] When performing speech separation on the speech information to be separated based on the information on the number of pronunciation objects and the sub-speech features of the pronunciation objects through the speech separation module to obtain the sub-speech information of the number of pronunciation objects, the computer device can construct the number of pronunciation object separation sub-modules based on the model parameters corresponding to the speech separation module and the information on the number of pronunciation objects. The model parameters corresponding to the number of pronunciation object separation sub-modules are the same and are all model parameters determined based on the model parameters of the speech separation module. The number of pronunciation object separation sub-modules correspond one-to-one to the number of pronunciation object sub-speech features, that is, each separation sub-module is used to separate the speech information corresponding to the pronunciation object represented by the corresponding sub-speech feature from the speech information to be separated. Therefore, during the speech separation process, the speech information corresponding to multiple pronunciation objects can be separated synchronously, without having to wait for the separation of the speech information corresponding to one pronunciation object to be completed before performing the speech information separation of the next pronunciation object, thereby improving the speech separation efficiency.

[0144] The computer device can use the number of separation sub-modules of the pronunciation objects as the current sub-modules respectively, and can separate the sub-voice information corresponding to the pronunciation object to be identified from the voice information to be separated through the current sub-module according to the sub-voice features corresponding to the current sub-module and the voice features of the voice information to be separated. The sub-voice information is the voice information that is more consistent with the sub-voice features, and the pronunciation object to be identified is the pronunciation object represented by the sub-voice features.

[0145] Refer to Figure 7, which is a schematic diagram of a model application method provided by an embodiment of the present application. The speech separation model may include an initial feature extraction module, a quantity determination module, and a speech separation module. The initial feature extraction module can determine the speech features corresponding to the speech information to be separated of the input model. The quantity determination module can determine the n sub-speech features corresponding to the semantic features and the number of pronunciation objects n. The n sub-speech features correspond one-to-one to the pronunciation objects. The quantity determination module can input this information into the speech separation module. The speech separation module constructs n separation submodules based on the number of pronunciation objects n, that is, n decoders. Each decoder corresponds to a sub-speech feature. The decoder can separate the corresponding sub-speech information based on the corresponding sub-speech feature and the speech features of the speech information to be separated, and obtain n sub-speech information. The n sub-speech information corresponds one-to-one to the n pronunciation objects, which is the separation result of the speech information to be separated.

[0146] Based on the model determination method provided in the above embodiment, the present application further provides a model determination device. See FIG8 , which is a structural block diagram of a model determination device provided in an embodiment of the present application. The device 800 includes a first acquisition unit 801, a first determination unit 802, a second determination unit 803, and an adjustment unit 804:

[0147] The first acquiring unit 801 is configured to acquire a sample information set, wherein the sample information set includes a plurality of sample speech information, each of the plurality of sample speech information has a corresponding plurality of sample sub-speech information, and each of the sample sub-speech information corresponds one-to-one to a sample pronunciation object;

[0148] The first determining unit 802 is configured to determine, for each sample voice information in the plurality of sample voice information, the number of undetermined pronunciation objects corresponding to the sample voice information by an initial number determining module in an initial voice separation model, and perform voice separation on the sample voice information based on the undetermined pronunciation object number information by an initial voice separation module in the initial voice separation model to obtain a plurality of first voice information;

[0149] The second determining unit 803 is configured to determine the number of actual pronunciation objects corresponding to the sample voice information based on the multiple sample sub-voice information contained in the sample voice information;

[0150] The adjustment unit 804 is used to adjust the model parameters corresponding to the initial speech separation model according to the difference between the information on the number of objects to be pronounced and the information on the number of actual pronunciation objects, and according to the difference between the multiple sample sub-speech information of the sample speech information and the multiple first speech information, so as to obtain a speech separation model. The speech separation model is used to perform speech separation on the speech information to be separated to obtain the corresponding multiple sub-speech information, and each sub-speech information corresponds one-to-one to the pronunciation object to be identified.

[0151] In a possible implementation, the first determining unit 802 is specifically configured to:

[0152] Determining a plurality of second sub-speech features corresponding to the target sample speech information;

[0153] determining a plurality of second sub-speech features corresponding to the sample speech information;

[0154] Determining, based on the plurality of second sub-speech features, pending count information corresponding to each of the plurality of second sub-speech features, the pending count information being used to identify a probability that the corresponding second sub-speech feature corresponds to a single sample pronunciation object;

[0155] Determining the pending count information corresponding to each of the plurality of second sub-speech features as the pending pronunciation object quantity information, wherein the pending pronunciation object quantity information is used to identify the number of second sub-speech features corresponding to a single sample pronunciation object;

[0156] The second determining unit 803 is specifically configured to:

[0157] Determining second sub-speech information corresponding to each of the plurality of second sub-speech features;

[0158] For each second sub-voice information in the plurality of second sub-voice information, determining a maximum value of the similarities respectively corresponding to the second sub-voice information and the plurality of sample sub-voice information as actual count information corresponding to the second sub-voice information;

[0159] Determining actual count information corresponding to each of the plurality of second sub-speech features as the actual pronunciation object quantity information;

[0160] The adjustment unit 804 is specifically configured to:

[0161] For each second sub-speech feature in the plurality of second sub-speech features, adjusting a model parameter corresponding to the initial quantity determination module based on a difference between pending count information corresponding to the second sub-speech feature and actual count information;

[0162] According to the differences between the plurality of sample sub-speech information contained in the sample speech information and the plurality of first speech information, the model parameters corresponding to the initial speech separation model are adjusted.

[0163] In a possible implementation, the first determining unit 802 is specifically configured to:

[0164] Determining, among the plurality of second sub-speech features, the number of second sub-speech features whose corresponding pending counting information indicates a probability greater than a first threshold as the number of pending pronunciation objects;

[0165] The sample voice information is voice-separated based on the number of objects to be pronounced to obtain first voice information of the number of objects to be pronounced.

[0166] In a possible implementation, the adjusting unit 804 is specifically configured to:

[0167] Based on the difference between the second sub-voice information corresponding to the second sub-voice feature and the target sub-voice information, and the difference between the pending count information corresponding to the target second sub-voice feature and the actual count information, the model parameters corresponding to the initial quantity determination module are adjusted, and the target sub-voice information is the sample sub-voice information with the greatest similarity to the second sub-voice information corresponding to the second sub-voice feature among the multiple sample sub-voice information.

[0168] In a possible implementation, the first determining unit 802 is specifically configured to:

[0169] determining a plurality of second sub-speech features corresponding to the sample speech information;

[0170] Determining, based on the plurality of second sub-speech features, information on the number of undetermined pronunciation objects corresponding to the sample speech information, wherein the information on the number of undetermined pronunciation objects is used to identify a plurality of standard features corresponding to a single sample pronunciation object in the plurality of second sub-speech features;

[0171] The multiple standard features are respectively used as current standard features, and the first voice information corresponding to the current standard feature is determined based on the current standard feature and the sample voice information. The first voice information corresponding to the current standard feature is the voice information corresponding to the sample pronunciation object corresponding to the current standard feature.

[0172] In a possible implementation, the first determining unit 802 is specifically configured to:

[0173] Determining a sample voice feature corresponding to the sample voice information;

[0174] extracting a first sub-speech feature corresponding to the current standard feature from the sample speech feature according to the current standard feature, wherein a similarity between the first sub-speech feature and the current standard feature is greater than a second threshold;

[0175] The first speech information corresponding to the current standard feature is determined according to the first sub-speech feature corresponding to the current standard feature.

[0176] In a possible implementation, the first determining unit 802 is specifically configured to:

[0177] using first sub-speech features corresponding to the remaining standard features in the multiple standard features as multiple comparison sub-speech features, and extracting a first speech feature corresponding to the current standard feature from the first sub-speech feature corresponding to the current standard feature based on the multiple comparison sub-speech features, wherein the first speech feature is a first sub-speech feature whose similarity with the multiple comparison sub-speech features is less than a third threshold, and the remaining standard features are standard features in the multiple standard features other than the current standard feature;

[0178] The first voice information corresponding to the current standard feature is determined according to the first voice feature corresponding to the current standard feature.

[0179] In a possible implementation, the device is further configured to:

[0180] extracting, based on the current standard feature, a second speech feature corresponding to the current standard feature from a first sub-speech feature corresponding to the current standard feature, wherein the second speech feature is the first sub-speech feature having a similarity with the current standard feature greater than a fourth threshold, and the fourth threshold is greater than the second threshold;

[0181] The first determining unit 802 is specifically configured to:

[0182] fusing a first speech feature corresponding to the current standard feature and a second speech feature corresponding to the current standard feature to generate a speech feature corresponding to the current standard feature;

[0183] The first voice information corresponding to the current standard feature is determined according to the voice feature corresponding to the current standard feature.

[0184] In a possible implementation, the initial speech separation model further includes an initial feature extraction module, and the apparatus is further configured to:

[0185] Extracting sample speech features corresponding to the sample speech information by the initial feature extraction module;

[0186] The first determining unit 802 is specifically configured to:

[0187] According to the sample speech features, the initial quantity determination module in the initial speech separation model is used to determine the quantity information of the pending pronunciation objects corresponding to the sample speech information, and based on the quantity information of the pending pronunciation objects and the sample speech features, the initial speech separation module in the initial speech separation model is used to perform speech separation on the sample speech information to obtain the multiple first speech information.

[0188] In a possible implementation, extracting, by the initial feature extraction module, sample speech features corresponding to the sample speech information includes:

[0189] Extracting a first feature corresponding to the sample voice information based on time domain information corresponding to the sample voice information, and extracting a second feature corresponding to the sample voice information based on frequency domain information corresponding to the sample voice information;

[0190] The sample speech feature is determined according to the first feature and the second feature.

[0191] In a possible implementation, extracting the first feature corresponding to the sample voice information based on the time domain information corresponding to the sample voice information includes:

[0192] Determining a first feature corresponding to the sample voice information by performing N feature extractions, wherein output information of the i-th feature extraction is input information of the i+1-th feature extraction, the first M feature extractions are performed based on time domain information corresponding to the sample voice information, and the next N minus M feature extractions are performed based on frequency domain information corresponding to the sample voice information, where N is a positive integer greater than 1, and M and i are positive integers less than N;

[0193] The extracting the second feature corresponding to the sample voice information based on the frequency domain information corresponding to the sample voice information includes:

[0194] The second feature corresponding to the sample voice information is determined through P feature extractions, wherein the output information of the kth feature extraction is the input information of the k+1th feature extraction, the first Q feature extractions are based on the frequency domain information corresponding to the sample voice information, and the last PQ feature extractions are based on the time domain information corresponding to the sample voice information. P is a positive integer greater than 1, and k and Q are positive integers less than N.

[0195] In a possible implementation, the device is further configured to:

[0196] Based on the time domain information and frequency domain information corresponding to the sample speech information, the initial feature extraction module performs one feature extraction to obtain a direct mapping feature corresponding to the sample speech information;

[0197] The determining the first feature corresponding to the sample speech information by performing N feature extractions includes:

[0198] Determining a first undetermined feature corresponding to the sample speech information through the N-th feature extraction, where the first undetermined feature is an output of the N-th feature extraction;

[0199] Combining the first undetermined feature and the directly mapped feature to obtain a first feature corresponding to the sample speech information;

[0200] The determining the second feature corresponding to the sample speech information through P feature extractions includes:

[0201] Determining a second undetermined feature corresponding to the sample speech information through the P-th feature extraction, where the second undetermined feature is the output of the P-th feature extraction;

[0202] The second undetermined feature and the directly mapped feature are combined to obtain a second feature corresponding to the sample speech information.

[0203] In a possible implementation, the device is further configured to:

[0204] Determining a third feature corresponding to the sample speech information through T feature extractions, wherein the wth feature extraction is used to perform feature extraction based on time domain information and frequency domain information corresponding to the sample speech information;

[0205] The determining the sample speech feature according to the first feature and the second feature includes:

[0206] The sample speech feature is determined according to the first feature, the second feature, and the third feature, where T is a positive integer greater than 1, and w is a positive integer not greater than T.

[0207] Based on the model application method provided in the above embodiment, the present application further provides a model application device. See FIG9 , which is a structural block diagram of a model application device provided in an embodiment of the present application. The device 900 includes a second acquisition unit 901 and a third determination unit 902:

[0208] The second acquiring unit 901 is configured to acquire speech information to be separated, wherein the speech information to be separated is composed of sub-speech information corresponding to a plurality of pronunciation objects to be recognized;

[0209] The third determination unit 902 is used to determine the number information of pronunciation objects corresponding to the voice information to be separated through the number determination module in the voice separation model according to the voice information to be separated, and to perform voice separation on the voice information to be separated through the voice separation module in the voice separation model according to the number information of pronunciation objects to obtain the number of pronunciation object sub-voice information, the number of pronunciation object sub-voice information corresponding one-to-one to the pronunciation objects to be identified, the pronunciation object number information is used to identify the number of pronunciation objects corresponding to the voice information to be separated, and the voice separation model is obtained based on the model determination method provided in the aforementioned embodiment.

[0210] In a possible implementation, the third determining unit 902 is specifically configured to:

[0211] Determine the number of sub-speech features of the pronunciation object corresponding to the speech information to be separated, wherein the number of sub-speech features of the pronunciation object corresponds one-to-one to the pronunciation object to be identified, and the sub-speech features are used to characterize the speech features of the corresponding pronunciation object to be identified;

[0212] According to the information on the number of pronunciation objects and the number of sub-speech features of the pronunciation objects, the speech information to be separated is subjected to speech separation by the speech separation module to obtain the number of sub-speech information of the pronunciation objects.

[0213] In a possible implementation, the third determining unit 902 is specifically configured to:

[0214] According to the model parameters corresponding to the speech separation module and the information on the number of pronunciation objects, a number of separation submodules of the number of pronunciation objects is formed, wherein the model parameters corresponding to the number of separation submodules of the number of pronunciation objects are the same, and the number of separation submodules of the number of pronunciation objects corresponds one to one with the number of sub-speech features of the number of pronunciation objects;

[0215] The number of separation submodules of the pronunciation objects are respectively used as current submodules, and based on the sub-speech features corresponding to the current submodules and the speech features of the speech information to be separated, the sub-speech information corresponding to the pronunciation object to be identified is separated from the speech information to be separated through the current submodule.

[0216] The embodiment of the present application further provides a computer device, as shown in FIG10 , which may be a terminal device. For example, the terminal device is a mobile phone.

[0217] Figure 10 shows a block diagram of a partial structure of a mobile phone related to the terminal device provided in an embodiment of the present application. Referring to Figure 10, the mobile phone includes components such as a radio frequency (RF) circuit 710, a memory 720, an input unit 730, a display unit 740, a sensor 750, an audio circuit 760, a wireless fidelity (WiFi) module 770, a processor 780, and a power supply 790. The input unit 730 may include a touch panel 731 and other input devices 732, the display unit 740 may include a display panel 741, and the audio circuit 760 may include a speaker 761 and a microphone 762. It will be understood that the mobile phone structure shown in Figure 10 does not constitute a limitation on the mobile phone, and may include more or fewer components than shown, or combine certain components, or arrange components differently.

[0218] The memory 720 can be used to store software programs and modules. The processor 780 executes the various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 720. The memory 720 can mainly include a program storage area and a data storage area. The program storage area can store an operating system and at least one application required for a function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created based on the use of the mobile phone (such as audio data, a phone book, etc.). In addition, the memory 720 can include high-speed random access memory and non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0219] Processor 780 is the control center of the phone, connecting all parts of the phone using various interfaces and circuits. By running or executing software programs and / or modules stored in memory 720 and accessing data stored in memory 720, it performs various phone functions and processes data, thereby performing overall phone testing. Optionally, processor 780 may include one or more processing units; preferably, processor 780 may integrate an application processor and a modem processor, where the application processor primarily handles the operating system, user interface, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into processor 780.

[0220] Although not shown, the mobile phone may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0221] In this embodiment, the processor 780 included in the terminal device also has any one of the above-mentioned model determination functions or model application functions.

[0222] The embodiment of the present application also provides a server, as shown in Figure 11, which is a structural diagram of the server 800 provided in the embodiment of the present application. The server 800 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPUs) 822 (for example, one or more processors) and a memory 832, and one or more storage media 830 (for example, one or more mass storage devices) for storing application programs 842 or data 844. Among them, the memory 832 and the storage medium 830 can be temporary storage or permanent storage. The program stored in the storage medium 830 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 822 can be configured to communicate with the storage medium 830 to execute a series of instruction operations in the storage medium 830 on the server 800.

[0223] The server 800 may also include one or more power supplies 826, one or more wired or wireless network interfaces 850, one or more input and output interfaces 858, and / or one or more operating systems 841, such as Windows Server 2000. TM , Mac OS X TM , UnixTM, Linux TM , FreeBSD TM etc.

[0224] The steps executed by the server in the above embodiment may be based on the server structure shown in FIG11 .

[0225] An embodiment of the present application also provides a computer-readable storage medium for storing a computer program, which is used to execute any implementation of the model determination method or model application method described in the aforementioned embodiments.

[0226] An embodiment of the present application also provides a computer program product including a computer program, which, when executed on a computer device, enables the computer device to execute the model determination method or model application method described in any one of the above embodiments.

[0227] It is understandable that in the specific implementation of this application, when it involves object information (such as voice information) and other related data, when the above embodiments of this application are applied to specific products or technologies, it is necessary to obtain the object's permission or consent, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0228] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the above-mentioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the above-mentioned storage medium can be at least one of the following media: read-only memory (English: read-only memory, abbreviated: ROM), RAM, magnetic disk or optical disk, etc., various media that can store program codes.

[0229] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments. The device and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0230] The above is merely one specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A model determination method, the method being executed by a computer device, the method comprising: Acquire a sample information set, the sample information set comprising a plurality of sample voice information, each of the plurality of sample voice information having a corresponding plurality of sample sub-voice information, each of the sample sub-voice information corresponding one-to-one to a sample pronunciation object; For each sample voice information in the plurality of sample voice information, determining the number information of pending pronunciation objects corresponding to the sample voice information by an initial number determination module in an initial voice separation model, and performing voice separation on the sample voice information based on the pending pronunciation object number information by an initial voice separation module in the initial voice separation model to obtain a plurality of first voice information; Determining the number of actual pronunciation objects corresponding to the sample voice information according to the multiple sample sub-voice information of the sample voice information; According to the difference between the information on the number of objects to be pronounced and the information on the number of actual objects to be pronounced, and according to the difference between the multiple sample sub-speech information of the sample speech information and the multiple first speech information, the model parameters corresponding to the initial speech separation model are adjusted to obtain a speech separation model, and the speech separation model is used to perform speech separation on the speech information to be separated to obtain the corresponding multiple sub-speech information, and each of the sub-speech information corresponds one-to-one to the pronunciation object to be identified.

2. According to the method of claim 1, the step of determining the number of pending pronunciation objects corresponding to the sample speech information comprises: Determining a plurality of second sub-speech features corresponding to the sample speech information; Determine, according to the plurality of second sub-speech features, pending count information corresponding to the plurality of second sub-speech features respectively, wherein the pending count information is used to identify a probability that the corresponding second sub-speech feature corresponds to a single sample pronunciation object; Determine the pending count information corresponding to the plurality of second sub-speech features as the pending pronunciation object quantity information, wherein the pending pronunciation object quantity information is used to identify the number of second sub-speech features corresponding to a single sample pronunciation object; The determining, according to the plurality of sample sub-speech information contained in the sample speech information, the actual number of pronunciation objects corresponding to the sample speech information includes: Determine the second sub-speech information corresponding to the plurality of second sub-speech features respectively; For each second sub-voice information in the plurality of second sub-voice information, determining the maximum value of the similarities respectively corresponding to the second sub-voice information and the plurality of sample sub-voice information as the actual count information corresponding to the second sub-voice information; Determine the actual count information respectively corresponding to the plurality of second sub-speech features as the actual pronunciation object quantity information; The adjusting the model parameters corresponding to the initial speech separation model according to the difference between the information on the number of objects to be pronounced and the information on the number of objects to be pronounced, and according to the difference between the plurality of sample sub-speech information and the plurality of first speech information in the sample speech information, comprises: For each second sub-speech feature of the plurality of second sub-speech features, based on a difference between pending count information corresponding to the second sub-speech feature and actual count information, adjusting a model parameter corresponding to the initial quantity determination module; According to the difference between the multiple sample sub-speech information contained in the sample speech information and the multiple first speech information, the model parameters corresponding to the initial speech separation model are adjusted.

3. The method according to claim 2, wherein based on the information on the number of pending pronunciation objects, the initial speech separation module in the initial speech separation model performs speech separation on the sample speech information to obtain a plurality of first speech information, comprising: Determining the number of second sub-speech features whose corresponding pending counting information indicates a probability greater than a first threshold value among the plurality of second sub-speech features as the number of pending pronunciation objects; The sample voice information is voice-separated based on the number of pending pronunciation objects to obtain first voice information of the number of pending pronunciation objects.

4. The method according to claim 2, wherein adjusting the model parameters corresponding to the initial quantity determination module based on the difference between the pending count information corresponding to the second sub-speech feature and the actual count information comprises: Based on the difference between the second sub-voice information corresponding to the second sub-voice feature and the target sub-voice information, and the difference between the pending count information corresponding to the target second sub-voice feature and the actual count information, the model parameters corresponding to the initial quantity determination module are adjusted, and the target sub-voice information is the sample sub-voice information having the greatest similarity with the second sub-voice information corresponding to the second sub-voice feature among the multiple sample sub-voice information.

5. According to the method of claim 1, the step of determining the number of pending pronunciation objects corresponding to the sample speech information comprises: Determining a plurality of second sub-speech features corresponding to the sample speech information; Determining the number of pending pronunciation objects corresponding to the sample speech information according to the plurality of second sub-speech features, wherein the number of pending pronunciation objects is used to identify a plurality of standard features corresponding to a single sample pronunciation object in the plurality of second sub-speech features; The method of performing speech separation on the sample speech information based on the information on the number of the pending pronunciation objects by using an initial speech separation module in the initial speech separation model to obtain a plurality of first speech information includes: The multiple standard features are respectively used as current standard features, and the first voice information corresponding to the current standard features is determined according to the current standard features and the sample voice information. The first voice information corresponding to the current standard features is the voice information corresponding to the sample pronunciation object corresponding to the current standard features.

6. The method according to claim 5, wherein determining the first voice information corresponding to the current standard feature according to the current standard feature and the sample voice information comprises: Determining a sample voice feature corresponding to the sample voice information; Extracting a first sub-speech feature corresponding to the current standard feature from the sample speech feature according to the current standard feature, wherein the similarity between the first sub-speech feature and the current standard feature is greater than a second threshold; The first speech information corresponding to the current standard feature is determined according to the first sub-speech feature corresponding to the current standard feature.

7. The method according to claim 6, characterized in that The determining, according to the first sub-speech feature corresponding to the current standard feature, the first speech information corresponding to the current standard feature comprises: The first sub-speech features corresponding to the remaining standard features in the plurality of standard features are respectively used as a plurality of comparative sub-speech features, and the first sub-speech features corresponding to the current standard feature are extracted from the first sub-speech features corresponding to the current standard feature according to the plurality of comparative sub-speech features. a first speech feature corresponding to the current standard feature, wherein the first speech feature is a first sub-speech feature whose similarity with the plurality of comparison sub-speech features is less than a third threshold, and the remaining standard features are standard features among the plurality of standard features except the current standard feature; The first voice information corresponding to the current standard feature is determined according to the first voice feature corresponding to the current standard feature.

8. The method according to claim 7, further comprising: According to the current standard feature, extracting a second speech feature corresponding to the current standard feature from a first sub-speech feature corresponding to the current standard feature, wherein the second speech feature is a first sub-speech feature whose similarity to the current standard feature is greater than a fourth threshold, and the fourth threshold is greater than the second threshold; The determining, according to the first voice feature corresponding to the current standard feature, the first voice information corresponding to the current standard feature comprises: Fusion of a first speech feature corresponding to the current standard feature and a second speech feature corresponding to the current standard feature to generate a speech feature corresponding to the current standard feature; The first voice information corresponding to the current standard feature is determined according to the voice feature corresponding to the current standard feature.

9. The method according to claim 1, wherein the initial speech separation model further comprises an initial feature extraction module, and the method further comprises: Extracting the sample speech features corresponding to the sample speech information by the initial feature extraction module; The method of determining the number of pending pronunciation objects corresponding to the sample voice information by an initial number determination module in an initial voice separation model, and performing voice separation on the sample voice information by an initial voice separation module in the initial voice separation model based on the pending pronunciation object number information to obtain a plurality of first voice information includes: According to the sample speech features, the initial quantity determination module in the initial speech separation model determines the quantity information of the pending pronunciation objects corresponding to the sample speech information, and based on the quantity information of the pending pronunciation objects and the sample speech features, the initial speech separation module in the initial speech separation model performs speech separation on the sample speech information to obtain the multiple first speech information.

10. The method according to claim 9, wherein extracting the sample speech features corresponding to the sample speech information by the initial feature extraction module comprises: Extracting a first feature corresponding to the sample voice information based on time domain information corresponding to the sample voice information, and extracting a second feature corresponding to the sample voice information based on frequency domain information corresponding to the sample voice information; The sample speech feature is determined according to the first feature and the second feature.

11. The method according to claim 10, wherein extracting the first feature corresponding to the sample voice information based on the time domain information corresponding to the sample voice information comprises: Determine the first feature corresponding to the sample voice information by N feature extractions, wherein the output information of the i-th feature extraction is the input information of the i+1-th feature extraction, the first M feature extractions are based on the time domain information corresponding to the sample voice information, and the last N minus M feature extractions are based on the frequency domain information corresponding to the sample voice information, N is a positive integer greater than 1, and M and i are positive integers less than N; The extracting the second feature corresponding to the sample voice information based on the frequency domain information corresponding to the sample voice information includes: The second feature corresponding to the sample voice information is determined through P feature extractions, wherein the output information of the kth feature extraction is the input information of the k+1th feature extraction, the first Q feature extractions are based on the frequency domain information corresponding to the sample voice information, and the last PQ feature extractions are based on the time domain information corresponding to the sample voice information, P is a positive integer greater than 1, and k and Q are positive integers less than N.

12. The method according to claim 11, further comprising: Based on the time domain information and frequency domain information corresponding to the sample speech information, the initial feature extraction module performs one feature extraction to obtain a direct mapping feature corresponding to the sample speech information; The determining the first feature corresponding to the sample speech information by extracting features N times includes: Determine a first undetermined feature corresponding to the sample voice information by extracting the features N times, where the first undetermined feature is the output of the Nth feature extraction; Combining the first undetermined feature and the directly mapped feature to obtain a first feature corresponding to the sample speech information; The determining the second feature corresponding to the sample speech information through P feature extractions includes: Determine a second undetermined feature corresponding to the sample voice information through the P-th feature extraction, where the second undetermined feature is the output of the P-th feature extraction; The second undetermined feature and the directly mapped feature are combined to obtain a second feature corresponding to the sample speech information.

13. The method according to claim 10, further comprising: Determine a third feature corresponding to the sample voice information by performing T feature extractions, wherein the wth feature extraction is used to perform feature extraction based on time domain information and frequency domain information corresponding to the sample voice information; The determining the sample speech feature according to the first feature and the second feature includes: The sample speech feature is determined according to the first feature, the second feature and the third feature, where T is a positive integer greater than 1, and w is a positive integer not greater than T.

14. A model application method, the method being executed by a computer device, the method comprising: Acquire speech information to be separated, wherein the speech information to be separated is composed of sub-speech information corresponding to a plurality of pronunciation objects to be recognized; According to the speech information to be separated, the number information of pronunciation objects corresponding to the speech information to be separated is determined by a number determination module in a speech separation model, and according to the number information of pronunciation objects, the speech information to be separated is speech-separated by a speech separation module in the speech separation model to obtain the number of pronunciation objects, sub-speech information, the number of pronunciation objects, sub-speech information and the pronunciation objects to be identified one-to-one, the pronunciation objects, and the number information are used to identify the number of pronunciation objects corresponding to the speech information to be separated, and the speech separation model is determined by the model determination method described in any one of claims 1 to 13.

15. The method according to claim 14, wherein the step of determining the number of pronunciation objects corresponding to the speech information to be separated comprises: Determine the number of sub-speech features of the pronunciation object corresponding to the speech information to be separated, the number of sub-speech features of the pronunciation object corresponds to the pronunciation object to be identified one by one, and the sub-speech features are used to characterize the speech features of the corresponding pronunciation object to be identified; The method of performing voice separation on the voice information to be separated by a voice separation module in the voice separation model according to the information on the number of pronunciation objects to obtain voice information on the number of pronunciation objects includes: According to the information on the number of pronunciation objects and the number of sub-speech features of the pronunciation objects, the speech information to be separated is speech-separated by the speech separation module to obtain the number of sub-speech information of the pronunciation objects.

16. The method according to claim 15, wherein the voice separation module performs voice separation on the voice information to be separated according to the information on the number of pronunciation objects and the number of sub-voice features of the pronunciation objects to obtain the number of sub-voice information of the pronunciation objects, comprising: According to the model parameters corresponding to the speech separation module and the information on the number of pronunciation objects, forming the number of separation submodules of the pronunciation objects, the model parameters corresponding to the number of separation submodules of the pronunciation objects are the same, and the number of separation submodules of the pronunciation objects corresponds one to one with the number of sub-speech features of the pronunciation objects; The number of separation submodules of the pronunciation object is respectively used as the current submodule, and the sub-speech information corresponding to the pronunciation object to be identified is separated from the speech information to be separated through the current submodule according to the sub-speech features corresponding to the current submodule and the speech features of the speech information to be separated.

17. A model determination device, the device being deployed on a computer device, the device comprising a first acquisition unit, a first determination unit, a second determination unit and an adjustment unit: The first acquisition unit is used to acquire a sample information set, the sample information set includes a plurality of sample voice information, each of the plurality of sample voice information has a corresponding plurality of sample sub-voice information, and each of the sample sub-voice information has a one-to-one correspondence with a sample pronunciation object; The first determination unit is used to determine, for each sample voice information in the multiple sample voice information, the number information of pending pronunciation objects corresponding to the sample voice information through the initial number determination module in the initial voice separation model, and perform voice separation on the sample voice information based on the pending pronunciation object number information through the initial voice separation module in the initial voice separation model to obtain multiple first voice information; The second determining unit is used to determine the actual pronunciation object quantity information corresponding to the sample voice information according to the multiple sample sub-voice information of the sample voice information; The adjustment unit is used to adjust the model parameters corresponding to the initial speech separation model according to the difference between the information on the number of objects to be pronounced and the information on the number of actual objects to be pronounced, and according to the difference between multiple sample sub-speech information contained in the sample speech information and the multiple first speech information, so as to obtain a speech separation model. The speech separation model is used to perform speech separation on the speech information to be separated to obtain corresponding multiple sub-speech information, and each sub-speech information corresponds one-to-one to a pronunciation object to be identified.

18. A model application device, the device being deployed on a computer device, the device comprising a second acquisition unit and a third determination unit: The second acquisition unit is used to acquire the voice information to be separated, wherein the voice information to be separated is composed of sub-voice information corresponding to a plurality of pronunciation objects to be recognized; The third determination unit is used to determine the number information of pronunciation objects corresponding to the voice information to be separated through the number determination module in the voice separation model according to the voice information to be separated, and to perform voice separation on the voice information to be separated through the voice separation module in the voice separation model according to the number information of pronunciation objects to obtain the number of pronunciation object sub-voice information, wherein the number of pronunciation object sub-voice information corresponds one-to-one to the pronunciation objects, and the pronunciation object number information is used to identify the number of pronunciation objects corresponding to the voice information to be separated, and the voice separation model is determined by the model determination method described in any one of claims 1 to 13.

19. A computer device, comprising a processor and a memory: The memory is used to store a computer program and transmit the computer program to the processor; The processor is used to execute the model determination method described in any one of claims 1 to 13, or execute the model application method described in any one of claims 14 to 16 according to the instructions in the computer program.

20. A computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, wherein the computer program is used to execute the model determination method described in any one of claims 1 to 13, or execute the model application method described in any one of claims 14 to 16.

21. A computer program product, comprising a computer program, which, when executed on a computer, enables the computer to execute the model determination method described in any one of claims 1 to 13, or the model application method described in any one of claims 14 to 16.

Images